Add Clang 15 C++17 build with error tolerance

[lucaam]

This PR adds Clang 15 to the build matrix C++17 support and set it as non-voting (continue-on-error: true) and use the -k option to cmake continue going even if error occurs.

[joka921]

This can be the first step towards such a CI check.
I think for this to be very useful, we need also a postprocessor that reads the error output of the compilation, and lists the failing and non-failing .cpp fiiles that can and can't be compiled.
This can then be combined with an explicit whitelist (Files that we have already fixed) to really check, that we don't regress, and that we correctly track our progress.
But I assume, that this is your initial step, and as you get the output of the CI pipeline you can used that to develop the further tools.

Let me know if I can be of assistance.

[joka921]

As long as this compilation doesn't work in fact, I would prefer having an additional
variable to be used here, such that this reads if (COMPILER_VERSION LESS 16 && !COMPILER_VERSION_CHECK_DEACTIVATED

[lucaam]

Sure, way better solution, gonna push it soon

[joka921]

I have just started the pipeline, this would be an initial step to track the progress.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.59%. Comparing base (db6825c) to head (041b309).
Report is 81 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1764      +/-   ##
==========================================
+ Coverage   89.98%   90.59%   +0.60%     
==========================================
  Files         395      414      +19     
  Lines       37693    40100    +2407     
  Branches     4241     4536     +295     
==========================================
+ Hits        33919    36329    +2410     
+ Misses       2478     2419      -59     
- Partials     1296     1352      +56     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[lucaam]

This can be the first step towards such a CI check. I think for this to be very useful, we need also a postprocessor that reads the error output of the compilation, and lists the failing and non-failing .cpp fiiles that can and can't be compiled. This can then be combined with an explicit whitelist (Files that we have already fixed) to really check, that we don't regress, and that we correctly track our progress. But I assume, that this is your initial step, and as you get the output of the CI pipeline you can used that to develop the further tools.

Let me know if I can be of assistance.

I think we could think about it in the future. Would you be open to a Python-based solution for this (we could include in a Github Workflow)?

[lucaam]

Hello @joka921, any chance to have a look?

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[lucaam]

Hello @joka921, could you please take a look and tell me if something else is needed?

[joka921]

Hi @lucaam
Sorry for the delay,
I have just started the checks again, so we can have a look at what the actual output looks like.
I have some questions

1. Was your choice to support a clang based compiler deliberate? I think our target platform is GCC, so something like GCC 10, or even the actual target 8.3 would seem more appropriate for me.

2. I hope that I did not block your further work, I hope that you don't have some workflow that depends on this being merged.

3. If you want to discuss anything general about the further extensions of these CI checks to make them more useful by reporting the failed and succesful builds, you can contact me via email (you should have access to the address)

Best regards
Johannes

[joka921]

Hi @lucaam ,
I previously had three comments/issues:

1. Use GCC instead of clang, as it is our target platform
2. Don't fail the CI if the tests fail. (The ctest of everything must be "do not fail on error" and the checks after that (E2E and Printing benchmark examples) must be disabled.
3. Add a (hard) check for a whitelisted set of tests.

In your latest commit I see point 3 addressed, but 1 and 2 are still missing.
Best regards

[lucaam]

Hi @lucaam , I previously had three comments/issues:

1. Use GCC instead of clang, as it is our target platform

2. Don't fail the CI if the tests fail. (The ctest of everything must be "do not fail on error" and the checks after that (E2E and Printing benchmark examples) must be disabled.

3. Add a (hard) check for a whitelisted set of tests.

In your latest commit I see point 3 addressed, but 1 and 2 are still missing. Best regards

Hi, I will continue working on that to add all the requested changes and let you know once is done! Thanks!

[lucaam]

Hello @joka921, could you please have a look?

[joka921]

Hi @lucaam
I have just started the checks, so we can see how they behave.
I will have a look at this later today.

[joka921]

As we already have very many compilers, maybe GCC-8 sufficies, and you can drop the clang-15 check for now.
Otherwise this looks fine if it works as expected (checks are still running, as the CI is very busy at the moment).

[joka921]

Do we still need Clang15?
The CI runs long enough as is, so if it works we can maybe only add the Gcc8 check for now?

[joka921]

Hi @lucaam
Thanks for integrating the Gcc-8 compiler.
However currently this doesn't do anything useful because we explicitly pass the -fcoroutines flag, which gcc-8 doesn't have.
So for now let's keep the clang-15 build and don't activate the gcc-8 build. Please leave the code to run it in, and just deactivate the given configuration, s.t. it doesn't waste our CI time.
Sorry for the confusion, but all the work you did was required and now it's working.

TLDR:
keep but deactivate the GCC-8 build for now, then we can merge this.
Best regards

[lucaam]

Hi @joka921, thanks for the feedback, I just pushed a commit to avoid -fcoroutines flag for gcc8 but I don't understand what do you mean with "keep but deactivate the GCC-8". Do you mean remove it from include in the matrix but keep the commits already pushed?

[joka921]

Hi @lucaam
Yes, that's what I meant. But if the removed flag also works and we can compile the whitelisted test with GCC-8 (let's wait for the check), then we can do it the other way round and remove Clang-15 and keep GCC-8.

While analyzing the last build output I found out that the build passes if the whitelisted tests are not run, because they couldn't be compiled, but I am not sure if we can do something about that (let me know if you have any idea).

I'll be away for the weekend soon, but hopefully we can finally merge this with exactly one additional CI check at the beginning of next week.

[sparql-conformance]

Conformance check passed ✅

No test result changes.

Details: https://qlever.cs.uni-freiburg.de/sparql-conformance-ui?cur=041b309c4264bc6e9f888f434e5f79d9daa80ad9&prev=f022cde36c5c7a5de4c5f9122831848c907518ce

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[joka921]

Hi @lucaam
GCC with the removed -fcoroutines flag now works as expected, so you can remove the clang-15 build for now
(just comment it out in the matrix, s.t. we know how to reincluded should we ever want to.
Then we can merge this and discuss further steps.

Best regards
Johannes

[lucaam]

Hi @lucaam GCC with the removed -fcoroutines flag now works as expected, so you can remove the clang-15 build for now (just comment it out in the matrix, s.t. we know how to reincluded should we ever want to. Then we can merge this and discuss further steps.

Best regards Johannes

Hi @joka921, I would keep both if you think this is possible, the main idea was having clang15 enabled. Let me know.