Add support for --compiler flag

pkgs/test/CHANGELOG.md

@@ -1,3 +1,21 @@
+## 1.24.0-dev
+
+* Support the `--compiler` flag, which can be used to configure which compiler
+  to use.
+  * To specify a compiler by platform, the argument supports platform selectors
+    through this syntax `[<platform>:]<compiler>`. For example the command line
+    argument `--compiler vm:source` would run all vm tests from source instead
+    of compiling to kernel first.
+  * If no given compiler is compatible for a platform, it will use its default
+    compiler instead.
+* Support compiler identifiers in platform selectors.
+* List the supported compilers for each platform in the usage text.
+* Update all reporters to print the compiler along with the platform name
+  when configured to print the platform. Extend the logic for printing platofrm
+  information to do so if any compilers are explicitly configured.
+* Deprecate `--use-data-isolate-strategy`. It is now an alias for `-c vm:source`
+  which is roughly equivalent. If this is breaking for you please file an issue.
+
 ## 1.23.1
 
 * Fix running paths by absolute path (with drive letter) on windows.

pkgs/test/README.md

@@ -13,6 +13,7 @@
   * [Collecting Code Coverage](#collecting-code-coverage)
   * [Restricting Tests to Certain Platforms](#restricting-tests-to-certain-platforms)
   * [Platform Selectors](#platform-selectors)
+  * [Compiler Selectors](#compiler-selectors)
   * [Running Tests on Node.js](#running-tests-on-nodejs)
 * [Asynchronous Tests](#asynchronous-tests)
   * [Stream Matchers](#stream-matchers)
@@ -187,6 +188,12 @@ reported on the command line just like for VM tests. In fact, you can even run
 tests on both platforms with a single command: `dart test -p "chrome,vm"
 path/to/test.dart`.
 
+By default each platform has a default compiler, but some of them support
+more than one compiler. You can choose which compiler to use by passing
+`dart test -c source`, which would run all VM tests from source instead of
+compiling them to kernel. This also supports targetting a specific platform
+using normal platform selectors, like this `dart test -c vm:source`.
+
 ### Test Path Queries
 
 Some query parameters are supported on test paths, which allow you to filter the
@@ -384,6 +391,14 @@ only supports boolean operations. The following identifiers are defined:
 * `posix`: Whether the test is running on a POSIX operating system. This is
   equivalent to `!windows`.
 
+* `dart2js`: Whether the test has been compiled with Dart2Js.
+
+* `dart2wasm`: Whether the test has been compiled with Dart2Wasm.
+
+* `kernel`: Whether the test has been compiled to kernel.
+
+* `source`: Whether the test has been run with no compiler (from source).
+
 For example, if you wanted to run a test on every browser but Chrome, you would
 write `@TestOn('browser && !chrome')`.
 

pkgs/test/doc/configuration.md

@@ -121,14 +121,14 @@ verbose_trace: true
 
 ### `chain_stack_traces`
 
-This boolean field controls whether or not stack traces are chained. 
+This boolean field controls whether or not stack traces are chained.
 Disabling [`stack trace chaining`][stack trace chaining] will improve
-performance for heavily asynchronous code at the cost of debuggability. 
+performance for heavily asynchronous code at the cost of debuggability.
 
 [stack trace chaining]: https://github.com/dart-lang/stack_trace/blob/master/README.md#stack-chains
 
 ```yaml
-chain_stack_traces: false 
+chain_stack_traces: false
 ```
 
 ### `js_trace`
@@ -163,7 +163,7 @@ This field is not supported in the
 
 ### `retry`
 
-This int field controls how many times a test is retried upon failure. 
+This int field controls how many times a test is retried upon failure.
 
 ```yaml
 tags:
@@ -417,6 +417,19 @@ platforms:
 - firefox
 ```
 
+### `compilers`
+
+This field indicates which compilers tests should be compiled with by default.
+It allows the same compiler selectors that can be passed to `--compiler`. If
+a given platform has no supported compiler configured, it will use its default.
+
+```yaml
+compilers: [source]
+
+compilers:
+- source
+```
+
 ### `concurrency`
 
 This field indicates the default number of test suites to run in parallel. More
@@ -511,8 +524,8 @@ package are folded.
 ```yaml
 fold_stack_frames:
   except:
-  - test 
-  - stream_channel 
+  - test
+  - stream_channel
 ```
 
 Sample stack trace, note the absence of `package:test`

pkgs/test/lib/src/runner/browser/browser_manager.dart

@@ -9,7 +9,7 @@ import 'package:async/async.dart';
 import 'package:pool/pool.dart';
 import 'package:stream_channel/stream_channel.dart';
 // ignore: deprecated_member_use
-import 'package:test_api/backend.dart' show Runtime, StackTraceMapper;
+import 'package:test_api/backend.dart' show Compiler, Runtime, StackTraceMapper;
 import 'package:test_core/src/runner/application_exception.dart'; // ignore: implementation_imports
 import 'package:test_core/src/runner/configuration.dart'; // ignore: implementation_imports
 import 'package:test_core/src/runner/environment.dart'; // ignore: implementation_imports
@@ -219,12 +219,13 @@ class BrowserManager {
   /// If [mapper] is passed, it's used to map stack traces for errors coming
   /// from this test suite.
   Future<RunnerSuite> load(String path, Uri url, SuiteConfiguration suiteConfig,
-      Map<String, Object?> message,
+      Map<String, Object?> message, Compiler compiler,
       {StackTraceMapper? mapper}) async {
     url = url.replace(
         fragment: Uri.encodeFull(jsonEncode({
       'metadata': suiteConfig.metadata.serialize(),
-      'browser': _runtime.identifier
+      'browser': _runtime.identifier,
+      'compiler': compiler.serialize(),
     })));
 
     var suiteID = _suiteID++;
@@ -256,7 +257,7 @@ class BrowserManager {
       try {
         controller = deserializeSuite(
             path,
-            currentPlatform(_runtime),
+            currentPlatform(_runtime, compiler),
             suiteConfig,
             await _environment,
             suiteChannel.cast(),

pkgs/test/lib/src/runner/browser/platform.dart

@@ -17,7 +17,7 @@ import 'package:shelf_static/shelf_static.dart';
 import 'package:shelf_web_socket/shelf_web_socket.dart';
 // ignore: deprecated_member_use
 import 'package:test_api/backend.dart'
-    show Runtime, StackTraceMapper, SuitePlatform;
+    show Compiler, Runtime, StackTraceMapper, SuitePlatform;
 import 'package:test_core/src/runner/configuration.dart'; // ignore: implementation_imports
 import 'package:test_core/src/runner/dart2js_compiler_pool.dart'; // ignore: implementation_imports
 import 'package:test_core/src/runner/load_exception.dart'; // ignore: implementation_imports
@@ -215,6 +215,14 @@ class BrowserPlatform extends PlatformPlugin
       throw ArgumentError('$browser is not a browser.');
     }
 
+    var compiler = platform.compiler;
+    // Technically, ddc is supported through --precompiled. But we don't expect
+    // any specific compiler setting in that case.
+    if (compiler != Compiler.dart2js) {
+      throw StateError('Unsupported compiler for $browser platform '
+          '$compiler');
+    }
+
     var htmlPathFromTestPath = '${p.withoutExtension(path)}.html';
     if (File(htmlPathFromTestPath).existsSync()) {
       if (_config.customHtmlTemplatePath != null &&
@@ -271,7 +279,8 @@ class BrowserPlatform extends PlatformPlugin
     var browserManager = await _browserManagerFor(browser);
     if (_closed || browserManager == null) return null;
 
-    var suite = await browserManager.load(path, suiteUrl, suiteConfig, message,
+    var suite = await browserManager.load(
+        path, suiteUrl, suiteConfig, message, platform.compiler,
         mapper: _mappers[path]);
     if (_closed) return null;
     return suite;

pkgs/test/lib/src/runner/node/platform.dart

@@ -13,7 +13,7 @@ import 'package:path/path.dart' as p;
 import 'package:stream_channel/stream_channel.dart';
 // ignore: deprecated_member_use
 import 'package:test_api/backend.dart'
-    show Runtime, StackTraceMapper, SuitePlatform;
+    show Compiler, Runtime, StackTraceMapper, SuitePlatform;
 import 'package:test_core/src/runner/application_exception.dart'; // ignore: implementation_imports
 import 'package:test_core/src/runner/configuration.dart'; // ignore: implementation_imports
 import 'package:test_core/src/runner/dart2js_compiler_pool.dart'; // ignore: implementation_imports
@@ -82,7 +82,11 @@ class NodePlatform extends PlatformPlugin
   @override
   Future<RunnerSuite> load(String path, SuitePlatform platform,
       SuiteConfiguration suiteConfig, Map<String, Object?> message) async {
-    var pair = await _loadChannel(path, platform.runtime, suiteConfig);
+    if (platform.compiler != Compiler.dart2js) {
+      throw StateError(
+          'Unsupported compiler for the Node platform ${platform.compiler}.');
+    }
+    var pair = await _loadChannel(path, platform, suiteConfig);
     var controller = deserializeSuite(
         path, platform, suiteConfig, PluginEnvironment(), pair.first, message);
 
@@ -96,12 +100,14 @@ class NodePlatform extends PlatformPlugin
   /// Returns that channel along with a [StackTraceMapper] representing the
   /// source map for the compiled suite.
   Future<Pair<StreamChannel<Object?>, StackTraceMapper?>> _loadChannel(
-      String path, Runtime runtime, SuiteConfiguration suiteConfig) async {
+      String path,
+      SuitePlatform platform,
+      SuiteConfiguration suiteConfig) async {
     final servers = await _loopback();
 
     try {
-      var pair =
-          await _spawnProcess(path, runtime, suiteConfig, servers.first.port);
+      var pair = await _spawnProcess(
+          path, platform.runtime, suiteConfig, servers.first.port);
       var process = pair.first;
 
       // Forward Node's standard IO to the print handler so it's associated with

pkgs/test/lib/src/runner/wasm/platform.dart

@@ -15,7 +15,7 @@ import 'package:shelf_packages_handler/shelf_packages_handler.dart';
 import 'package:shelf_static/shelf_static.dart';
 import 'package:shelf_web_socket/shelf_web_socket.dart';
 // ignore: deprecated_member_use
-import 'package:test_api/backend.dart' show Runtime, SuitePlatform;
+import 'package:test_api/backend.dart' show Compiler, Runtime, SuitePlatform;
 import 'package:test_core/src/runner/configuration.dart'; // ignore: implementation_imports
 import 'package:test_core/src/runner/package_version.dart'; // ignore: implementation_imports
 import 'package:test_core/src/runner/platform.dart'; // ignore: implementation_imports
@@ -201,11 +201,16 @@ class BrowserWasmPlatform extends PlatformPlugin
   @override
   Future<RunnerSuite?> load(String path, SuitePlatform platform,
       SuiteConfiguration suiteConfig, Map<String, Object?> message) async {
+    if (platform.compiler != Compiler.dart2wasm) {
+      throw StateError(
+          'Unsupported compiler for experimental-chrome-wasm platform '
+          '${platform.compiler}');
+    }
+
     if (suiteConfig.precompiledPath != null) {
       throw UnsupportedError(
           'The wasm platform doesn\'t support precompiled suites');
     }
-
     var browser = platform.runtime;
     assert(suiteConfig.runtimes.contains(browser.identifier));
 
@@ -216,6 +221,7 @@ class BrowserWasmPlatform extends PlatformPlugin
     // TODO: Support custom html?
 
     Uri suiteUrl;
+
     await _compileSuite(path, suiteConfig);
 
     if (_closed) return null;
@@ -227,7 +233,8 @@ class BrowserWasmPlatform extends PlatformPlugin
     var browserManager = await _browserManagerFor(browser);
     if (_closed || browserManager == null) return null;
 
-    var suite = await browserManager.load(path, suiteUrl, suiteConfig, message);
+    var suite = await browserManager.load(
+        path, suiteUrl, suiteConfig, message, platform.compiler);
     if (_closed) return null;
     return suite;
   }

pkgs/test/pubspec.yaml

@@ -1,5 +1,5 @@
 name: test
-version: 1.23.1
+version: 1.24.0-dev
 description: >-
   A full featured library for writing and running Dart tests across platforms.
 repository: https://github.com/dart-lang/test/tree/master/pkgs/test
@@ -32,8 +32,8 @@ dependencies:
   webkit_inspection_protocol: ^1.0.0
   yaml: ^3.0.0
   # Use an exact version until the test_api and test_core package are stable.
-  test_api: 0.4.18
-  test_core: 0.4.24
+  test_api: 0.5.0
+  test_core: 0.5.0
 
 dev_dependencies:
   fake_async: ^1.0.0

pkgs/test/test/runner/browser/compact_reporter_test.dart

@@ -28,7 +28,7 @@ void main() {
         ['-p', 'chrome', '-p', 'vm', '-j', '1', 'test.dart'],
         reporter: 'compact');
 
-    expect(test.stdout, containsInOrder(['[Chrome]', '[VM]']));
+    expect(test.stdout, containsInOrder(['[Chrome, Dart2Js]', '[VM, Kernel]']));
     await test.shouldExit(0);
   }, tags: 'chrome');
 }

pkgs/test/test/runner/browser/expanded_reporter_test.dart

@@ -28,8 +28,8 @@ void main() {
       'test.dart'
     ]);
 
-    expect(test.stdoutStream(), emitsThrough(contains('[VM]')));
-    expect(test.stdout, emitsThrough(contains('[Chrome]')));
+    expect(test.stdoutStream(), emitsThrough(contains('[VM, Kernel]')));
+    expect(test.stdout, emitsThrough(contains('[Chrome, Dart2Js]')));
     await test.shouldExit(0);
   }, tags: ['chrome']);
 }

pkgs/test/test/runner/browser/loader_test.dart

@@ -67,6 +67,7 @@ void main() {
     test('returns a suite with the file path and platform', () {
       expect(suite.path, equals(p.join(d.sandbox, 'a_test.dart')));
       expect(suite.platform.runtime, equals(Runtime.chrome));
+      expect(suite.platform.compiler, equals(Runtime.chrome.defaultCompiler));
     });
 
     test('returns tests with the correct names', () {
@@ -142,8 +143,10 @@ Future main() {
         .cast<RunnerSuite>()
         .toList();
     expect(suites[0].platform.runtime, equals(Runtime.vm));
+    expect(suites[0].platform.compiler, equals(Runtime.vm.defaultCompiler));
     expect(suites[0].path, equals(path));
     expect(suites[1].platform.runtime, equals(Runtime.chrome));
+    expect(suites[1].platform.compiler, equals(Runtime.chrome.defaultCompiler));
     expect(suites[1].path, equals(path));
 
     for (var suite in suites) {