Test results not published

[akurtakov]

Starting https://download.eclipse.org/eclipse/downloads/drops4/I20250307-1800/ test results are not published as the newest jvm on project storage is Java 17 but SDK used during build needs Java 21.
Running full SDK on project storage is against webmaster recommendations thus asking for Java 21 doesn't feel the right thing to do.

[akurtakov]

Example failing job: https://ci.eclipse.org/releng/job/Releng/job/ep-collectResults/6032/console
Contents of relevant dirs:

10:58:11 + ssh genie.releng@projects-storage.eclipse.org ls -al /opt/public/common/java/
10:58:12 total 24
10:58:12 drwxrwsr-x   4 root common  4096 Jan 17  2020 .
10:58:12 drwxrwsr-x 153 root common 12288 Feb 20  2024 ..
10:58:12 drwxrwsr-x  12 root common  4096 Mar 20  2023 openjdk
10:58:12 drwxrwsr-x  12 root common  4096 Apr 30  2021 oracle

10:59:53 + ssh genie.releng@projects-storage.eclipse.org ls -al /opt/public/common/java/oracle
10:59:54 total 48
10:59:54 drwxrwsr-x 12 root     common 4096 Apr 30  2021 .
10:59:54 drwxrwsr-x  4 root     common 4096 Jan 17  2020 ..
10:59:54 drwxrwsr-x  8 root     common 4096 Mar 22  2018 jdk-10-ga_x64
10:59:54 drwxrwsr-x  8 root     common 4096 Jun 22  2018 jdk-10.0.1_x64
10:59:54 lrwxrwxrwx  1 root     common   14 Jun 22  2018 jdk-10_x64-latest -> jdk-10.0.1_x64
10:59:54 drwxrwsr-x  8 root     root   4096 Oct  2  2018 jdk-11_x64
10:59:54 lrwxrwxrwx  1 root     root     10 Oct  2  2018 jdk-11_x64-latest -> jdk-11_x64
10:59:54 lrwxrwxrwx  1    63303 common   17 Apr 30  2021 jdk-8_x64-latest -> jdk1.8.0_202_x64/
10:59:54 drwxrwsr-x  8 root     common 4096 Jan  8  2018 jdk-9-ga_x64
10:59:54 drwxrwsr-x  8 root     common 4096 Dec 14  2017 jdk-9.0.1
10:59:54 lrwxrwxrwx  1 root     common   13 Oct 13  2017 jdk-9_x64-latest -> jdk-9-ga_x64/
10:59:54 drwxr-xr-x  8 uucp        143 4096 Dec 19  2017 jdk1.8.0_162
10:59:54 drwxr-xr-x  8 uucp        143 4096 Dec 20  2017 jdk1.8.0_162_x64
10:59:54 drwxr-sr-x  7 fgurr    common 4096 Oct  6  2018 jdk1.8.0_192_x64
10:59:54 drwxr-sr-x  7 mbarbero common 4096 Dec 15  2018 jdk1.8.0_202
10:59:54 drwxr-sr-x  7 mbarbero common 4096 Dec 15  2018 jdk1.8.0_202_x64

10:56:54 + ssh genie.releng@projects-storage.eclipse.org ls -al /opt/public/common/java/openjdk
10:56:55 total 48
10:56:55 drwxrwsr-x 12 root common 4096 Mar 20  2023 .
10:56:55 drwxrwsr-x  4 root common 4096 Jan 17  2020 ..
10:56:55 drwxrwxr-x 10 root common 4096 Jan 12  2021 jdk-11_11.0.10+9
10:56:55 drwxrwxr-x 10 root common 4096 Apr  6  2020 jdk-11_11.0.7+10
10:56:55 lrwxrwxrwx  1 root common   16 Jan 21  2021 jdk-11_x64-latest -> jdk-11_11.0.10+9
10:56:55 drwxrwsr-x  8 root common 4096 Mar 27  2019 jdk-12
10:56:55 lrwxrwxrwx  1 root common    6 Mar 27  2019 jdk-12_x64-latest -> jdk-12
10:56:55 drwxrwsr-x  8 root common 4096 Sep  3  2019 jdk-13
10:56:55 lrwxrwxrwx  1 root common    6 May 29  2019 jdk-13_x64-latest -> jdk-13
10:56:55 drwxrwsr-x  8 root common 4096 Sep 23  2019 jdk-14_ea15
10:56:55 drwxrwsr-x  8 root common 4096 Jan  7  2020 jdk-14_ea30
10:56:55 drwxrwsr-x  8 root common 4096 Mar 16  2020 jdk-14_rc
10:56:55 lrwxrwxrwx  1 root common    9 Mar 16  2020 jdk-14_x64-latest -> jdk-14_rc
10:56:55 drwxrwxr-x  9 root common 4096 Jan 21  2021 jdk-15.0.2+7
10:56:55 lrwxrwxrwx  1 root common   12 Feb  5  2021 jdk-15_x64-latest -> jdk-15.0.2+7
10:56:55 drwxrwsr-x  8 root common 4096 Feb  5  2021 jdk-16_rc_build35
10:56:55 lrwxrwxrwx  1 root common   17 Feb  5  2021 jdk-16_x64-latest -> jdk-16_rc_build35
10:56:55 drwxr-sr-x  8 root common 4096 Mar 20  2023 jdk-17.0.2
10:56:55 lrwxrwxrwx  1 root common   10 Mar 20  2023 jdk-17_x64-latest -> jdk-17.0.2

[akurtakov]

Until this one is fixed test results can be seen at https://ci.eclipse.org/releng/job/AutomatedTests/ ep436I-* jobs

[akurtakov]

It will require some significant changes to the process but it will not execute eclipse on project storage and will move things under releng control is:

• Prepare eclipse to be used in job workspace
• Download files to be modified to the job workspace
• Run local eclipse to modify downloaded files
• scp modified files to project storage

As I don't know how much time and when I can spend on this one - help is more than welcome.

[iloveeclipse]

Running full SDK on project storage is against webmaster recommendations thus asking for Java 21 doesn't feel the right thing to do.

I can't understand that. Why can't we ask to install Java 21? Why 17 is "OK" to install but not 21?

[akurtakov]

IMO it was not "OK" even with Java 17 , I did it a year ago (https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/2835) but considering the deprecation of PHP on downloads by end of this year (2025 #2715 ) I see continuing to band-aid current approach for few more months not worth the time and better be (partially) revamped whenever and issue arise.
Current state of webpages being all broken (even https://eclipse.org/eclipse gives 404 now) proves to me that waiting just gives worse results.
FWIW, I have never said "we can't ask" so if there are people with time to spend on band-aiding things for next few months they can do so.

[laeubi]

Can someone explain the current issue / process and why we need Java "somewhere else"?

Why can't we for example use a docker-container with php+java installed and let it emit static HTML pages we upload to "somewhere"? Maybe we can even have a github pages repo and upload stuff via git, Tycho is doing it for its documentation:

https://github.com/eclipse-tycho/eclipse-tycho.github.io

[akurtakov]

Can someone explain the current issue / process and why we need Java "somewhere else"?

That's the current approach (spread in too many "unknown" places) to execute eclipse apps (e.g. https://github.com/eclipse-platform/eclipse.platform.releng.buildtools/ ) on downloads directly.

Why can't we for example use a docker-container with php+java installed and let it emit static HTML pages we upload to "somewhere"?

That is my goal eventually. I even believe that php could not be needed at all . Main issue is that these download pages are being done that way before even I joined the project and there is no one left to say what happens where for sure thus reluctance to change it as it would certainly means certain things not being uncovered (and fixed if they are even useful nowadays) for extended period. What is for sure is that it would require quite some time to get it understood and changed.

Maybe we can even have a github pages repo and upload stuff via git, Tycho is doing it for its documentation:

https://github.com/eclipse-tycho/eclipse-tycho.github.io

[iloveeclipse]

Can someone explain the current issue / process and why we need Java "somewhere else"?

I've found a brief explanation of the current test result publishing process, see https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/wiki/How-to-Do-Miscellaneous-Releng-Tasks#how-to-change-to-web-pages-for-builds-or-test-results

Here is the summary from what I've understood so far:

The problem in #2906 with "collectResults" job shows failures: https://ci.eclipse.org/releng/job/Releng/job/ep-collectResults is that it depends on running SDK application and SDK application requires Java 21 now, which is not installed on the machine running the job.

If I understood this right, the collectResults job is triggered once tests are done.

The job seem to use this script: https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/2c0c9ef1bb4f7bce2ca3b2f386465411884587af/scripts/collectResultsLocal.sh

The script depends on execution of TestResultsGenerator.java code via ant task running in the SDK, that is supposed to create html page used by this php file:

eclipse.platform.releng.aggregator/eclipse.platform.releng.tychoeclipsebuilder/eclipse/publishingFiles/staticDropFiles/testResults.php

Lines 119 to 129 in 2c0c9ef

	// testResultsTables.html is generated by a custom ant task in
	// build tools (see TestResultsGenerator.java). It consist of
	// one to three tables: test results, missing files, files missing from
	// testManifest.xml. The later two are are rarely produced, since usually nothing
	// is missing.
	$rowResultsFile="testResultsTables.html";
	if (file_exists($rowResultsFile)) {
	include $rowResultsFile;
	} else {
	include "testResultsTablesPending.html";
	}

The ant task that should be executed is this one: https://github.com/eclipse-platform/eclipse.platform.releng.buildtools/blob/412edbfad978d91deccec3f56c3eda444e318784/bundles/org.eclipse.build.tools/src/org/eclipse/releng/generators/TestResultsGenerator.java

Additionally to the current problem with not being able to run SDK on the collectResults job because of missing Java 21 we will face the problem with the php shutdown at Eclipse.org by the end of 2025.

I see the proposal here from @laeubi and last one on this thread from @HannesWell are very similar.

So we ideally should address two issues ("no Java 21" now + "php shutdown" later), but not necessarily at same time.

In any way, I believe not being able to easily access test results is a big problem and has to be fixed ASAP.

The short term tasks could be

1. Run TestResultsGenerator.java at the end of the each test execution, where we do have latest Java we need anyway and generate test results html file that can be fetched by collectResultsLocal.sh. Update collectResultsLocal.sh to fetch the generated data instead of generating that on the fly.
2. Or ask infra team to install Java 21 on the machine that runs collectResultsLocal.sh. That's the easiest one of course :-)

The long term tasks could be

1. Change collectResultsLocal.sh to do something that would produce test results in a format suitable for "static" html page, replacing TestResultsGenerator.java execution via SDK application. May be this data can be already generated after test execution in the test execution job itself, where we do have latest Java we need anyway, and just fetched by collectResultsLocal.sh script.
2. Convert testResults.php to a static html file.
3. Add some javascript based code to that new static testResults.html file that would use generated test result data to produce test results tables.

The question is: can @MohananRahul , @laeubi, @HannesWell or someone else (@howlger ?) to work on this?

[laeubi]

Before invest much time here maybe we should first decide what is actually required here, e.g. Jenkins can already produce test result summary like this:

https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/master/916/testReport/

would it be maybe enough to have such reports produced in the individual jobs? Jenkins should also be able to aggregate different job results if required to collect them from dependent jobs.

[akurtakov]

So https://download.eclipse.org/eclipse/downloads/drops4/I20250309-1800/testResults.php shows results but also shows "[Missing testManifest.xml Files]" for which I have no clue.
At least with #2911 we are in control of JVM used so test results are shown and script is ~20 lines shorter.

[subyssurendran666]

1. Change collectResultsLocal.sh to do something that would produce test results in a format suitable for "static" html page, replacing TestResultsGenerator.java execution via SDK application. May be this data can be already generated after test execution in the test execution job itself, where we do have latest Java we need anyway, and just fetched by collectResultsLocal.sh script.
2. Convert testResults.php to a static html file.
3. Add some javascript based code to that new static testResults.html file that would use generated test result data to produce test results tables.

Can we add PHP-CLI to the build pipeline, and once testResults.php is generated, convert it to HTML using php testResults.php > testResults.html , then delete testResults.php? Would this approach minimise time and require less rework?

[akurtakov]

@subyssurendran666 This question could be answered only when someone spends the time to try it out. Volunteers are welcome.

[akurtakov]

From what I can tell neither collectResultsLocal.sh is used at all not I can see any php generated in it.