Feedback about Test Analytics from Codecov ❄ 📊

[rohan-at-sentry]

Thanks for dropping by! 👋

We've recently released a whole new feature around Test analytics and flaky test detection .

We'd love to hear feedback on

How your setup experience was.
How easy/useful the PR comment is
How actionable and accurate our flake reporting is
How useful our Tests Dashboards are in finding and fixing problematic tests in your test suite

This issue is intended to share and collect feedback about the tool. If you have support needs or questions, please let us know!

[Bibo-Joshi]

Thanks for the new thread :)
Repeating my comments from the old thread

How easy/useful the PR comment is

TBH I don't much like automated PR comments. I'd prefer to have that in a job summary. For the coverage reports, one can already disable the comments.

[fmigneault]

Bringing back my comments that also got lost in the massive thread:

#304 (comment)
#304 (comment)

In summary, I like the feature overall. It was easy to set up.

However, I would prefer for the posted PR comment including flaky test details to preserve the original test results summary as well. Both coverage results and flaky test information are relevant simultaneously. Currently, if a flaky test does happen to result in a failure, the coverage of all other tests are overridden in the comment, which forces me to go through the CodeCov UI and dig back the information. Some tests might be temporarily accepted although flaky/failing, while being interested about covering other aspects of the code.

At this time, I have to disable the flaky test analysis because it hinders our development of other features while some known flaky tests are happening, and this feature masks away coverage results in PR comments.

[rohan-at-sentry]

@fmigneault - thanks for raising, this is feedback that we've heard before as well. I wanted to ask though, if there is a test that fails, it's possible (depending on the test runner) that coverage information is possibly incorrect.

Is that a risk you anticipate?

[fmigneault]

I've tried 2 different pytest plugins, https://github.com/pytest-dev/pytest-rerunfailures and https://github.com/str0zzapreti/pytest-retry, each of which had different outcomes.

The first one was indeed erroneously inserting some <failure> entries in the XML report, even when retries flipped a failing flaky test to a successful run. This indeed misdirected CodeCov in thinking that the tests were always failing, which resulted in invalid reporting.

With the second plugin, the <failure> are not inserted when flaky tests went from failing to success following retries, as expected. However, I happened to have other tests that resulted in real and replicable failures, which were annotated correctly as such.

Therefore, I intend to move forward with the second plugin that gives the expected result, but I expect CodeCov to provide both the coverage summary report and the flaky test analysis, so I can address either situation.

Here is the comment that CodeCov posts on the corresponding PR modifying the utility: crim-ca/weaver#790 (comment)
If you look at the "edits" of that comment, you will notice that CodeCov modifies its contents back-and-forth between "Codecov Reports" and the "Test Analytics", which are inline with the status-per-commit obtained in relations to the file changes (https://github.com/crim-ca/weaver/pull/790/files). My expectation would be that both are kept in such case, rather than "Test Analytics" overriding the "Codecov Reports" contents.

[cosmith]

Hello, our main use case is finding and fixing flaky tests. We switched from Datadog CI because it was too expensive and we didn't need most of their features.

It's working ok for us with Codecov, but we need a view with all branches to be able to prioritize which tests to fix. We have dozens of open PRs where CI is running, and looking only at our main branch for flakyness doesn't give us much data to work with (there are many more CI runs on feature branches than on the main branch).

Happy to discuss it in more details if needed.

[rohan-at-sentry]

Hi @cosmith - this is something we're actively working on, you can follow along here (and also leave feedback :D ) codecov/engineering-team#2542

We're working through some performance improvements currently and should be able to ship 2542 along with that.

[rChaoz]

Hi! It would be great if there was an option on the regular Action to enable test analytics, since most people are already using that. There is no need to add another action just to upload a single file. Thanks!

[tarungunampalli1]

I would like to dynamically pull data about test flakiness in our main branch to skip/ignore failures from tests with high failure rates. The test results api is supposed to return this data, but it lacks some critical features that would be useful for this case, like filtering by flag, filtering to be above a certain failure rate, and also the return value from my testing doesn't even show the failure rate, even though that's listed as a required output in the docs.

Also, the docs say that we can use the test results API to use custom reports, but it just links to the two test results get requests, which doesn't explain how to upload custom reports.

I'm also curious if we are able to use an access token to call the graphql endpoints that are called when viewing dashboards. Those seems to return all the data I need, with options for sorting and filtering by flag, so that would work relatively well for my use case.