Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dart CI package tests are run with the noisy compact reporter for package:test #54929

Open
whesse opened this issue Feb 15, 2024 · 0 comments
Open
Labels
analyzer-test area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. area-test Cross-cutting test issues (use area- labels for specific failures; not used for package:test).

Comments

@whesse
Copy link
Contributor

whesse commented Feb 15, 2024

The Dart CI and CQ run tests for the packages in the Dart SDK source on builders like

https://ci.chromium.org/ui/p/dart/builders/luci.dart.ci.sandbox/pkg-linux-release

These builders run package tests written using package:test framework the same way they run any
Dart test on the runtime VM, by running the program using the Dart VM, with no special arguments because it uses the package:test framework.

The default reporter for package:test is --reporter compact which prints ANSI console control codes in stdout to try
and make a console keep printing progress on the same line without scrolling. In our test logs, they appear as strings like �[32m+10�[0m�[31m -1�[0m on every line.

It would be better to use the expanded, github, or json reporters, to produce better logs on stdout for users to see using our CI infrastructure. This could be done by passing the --reporter flag in the commands created by package:test_runner, or by setting the option in a dart_test.yaml file at the root of each package.

The json report can also be sent to a file, in addition to another report format printed on stdout.
There has been discussion of a failures-only reporting format as well on dart-lang/test#829

If we reported a structured format, then our test results systems could even read that format and track individual tests in a test file separately in our CI databases, but that would be an extensive feature addition.

So we should decide on the format to report on stdout, and change package:test_runner or else add dart_test.yaml files to our individual packages, to add the correct --reporter option when running these tests.

@whesse whesse added area-test Cross-cutting test issues (use area- labels for specific failures; not used for package:test). area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. analyzer-test labels Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
analyzer-test area-infrastructure Use area-infrastructure for SDK infrastructure issues, like continuous integration bot changes. area-test Cross-cutting test issues (use area- labels for specific failures; not used for package:test).
Projects
None yet
Development

No branches or pull requests

1 participant