Please add TruffleRuby to your CI

[postmodern]

Recently simplecov-0.22.0 which uses Coverage.running?. Unfortunately, truffleruby does not support Coverage.running?. This broke the CI of many of my gems which I both use simplecov and test against truffleruby (stable). I feel like this could have been avoided by adding truffleruby to simplecov's own CI, which would have caught the issue early, and allowed time for the TruffleRuby team to add Coverage.running?.

[eregon]

I can make a PR for it after #1043 is merged.

[PragTob]

Morning folks,

as the reason issue #1 on truffle ruby exists I can hardly say no.

However, does TruffleRuby support basic coverage yet? (@eregon ?) I haven't kept up well through the last years, but I remember that last time it didn't or so I think/thought.

In that case, not running coverage on TruffleRuby on some side might be pregerable.

[eregon]

Yes TruffleRuby supports Line coverage since many years and it passes the specs except for 1 failure: #1043 (comment)

[eregon]

In that case, not running coverage on TruffleRuby on some side might be preferable.

That's an interesting question. For most gems I would expect it's enough to run coverage on CRuby, and redundant to run coverage on JRuby/TruffleRuby. Although that's not true for gems which have some code depending on RUBY_ENGINE for instance.

Coverage has some overhead for TruffleRuby IIRC so it might be best to disable it in gems CI for CI speed.
We should measure again though, it might not be significant enough to change anything.
And it's not clear what's a clean way to "disable by default" or so (option in TruffleRuby? option in SimpleCov? etc).

For now what's needed is #1043.

[eregon]

Coverage has some overhead for TruffleRuby IIRC so it might be best to disable it in gems CI for CI speed.

In the case of the syntax_tree test suite it adds (both measuring the coverage and generating the report) about 5 seconds (ruby-syntax-tree/syntax_tree@ed6e206) to a test suite taking (without it) ~32 seconds (so ~15% overhead in that case). That's quick measurements with other apps open, etc though and a fair amount of variance:
Without: 32.071 29.791 30.989 vs With: 36.462 36.325 36.821. The times from minitest don't seem to change much (all around 26s), so it seems mostly the extra time for generating the coverage report.
On CRuby 3.1.3 it makes a 1 second difference, so probably we could optimize the report generation better on TruffleRuby.

[eregon]

This is no longer a problem with https://github.com/oracle/truffleruby/releases/tag/vm-23.0.0-preview1.
We should still add TruffleRuby to SimpleCov CI, which is #1053
But that really needs someone to merge/review PRs here, this repo hasn't seen a single commit in 2023 yet.

[eregon]

New PR to add truffleruby in CI: #1079

[PragTob]

That's an interesting question. For most gems I would expect it's enough to run coverage on CRuby, and redundant to run coverage on JRuby/TruffleRuby. Although that's not true for gems which have some code depending on RUBY_ENGINE for instance.

Yeah, I think most coverage reporting services also struggle with it. It can be an option to multi run it but the use case for this is by far the minority (imo).

As for how to run/not run it - I think it should be dedicated by an environment variable and only then do you run (or not) run the CI. This can easily be configured in a build matrix/build steps then and so for me that's on the user and neither TruffleRuby nor SimpleCov need to do anything.

[PragTob]

Thanks to @nirvdrum for pinging me and sorry for all the silence!