Build hook spec

pkgs/native_assets_builder/doc/protocol/v1.md

@@ -0,0 +1,231 @@
+Build Hook Specification
+========================
+
+Version 1: As of 2024-01-24 implemented behind `--enable-experiment=native-assets`
+
+### Concepts
+
+#### Asset
+
+An asset is a file together with metadata that tells the Dart and Flutter SDK how to bundle such asset in an application bundle.
+
+Currently, all such assets are dynamic libraries.
+
+An asset must have an `assetId`. Dart code that uses an asset, references the asset using the `assetId`.
+
+An asset must have a target os and target architecture. When accessing the asset at runtime, the asset for the current os and architecture is accessed.
+
+An asset must have a link mode. Currently, the only supported link mode is dynamic. (Static linking of native code will be added later.)
+
+An asset must specify a path type for dynamic linking. 
+
+* `absolute` paths are absolute paths in the output dir (see below).
+* `system` paths are expected to resolve on the target machine PATH. In this case the asset is not a file but only metadata.
+* `process` "paths" have no path, symbols are resolved in the current process. In this case the asset is not a file but only metadata.
+* `executable` "paths" have no path, symbols are resolved in the current executable. In this case the asset is not a file but only metadata.
+
+#### AssetId
+An asset must have an `assetId`. Dart code that uses an asset, references the asset using the `assetId`.
+
+A package must prefix all assetIds it defines with: `package:<package>/`, this ensures assets don't conflict between packages.
+
+Conventionally, an asset referenced from `lib/src/foo.dart` in `package:foo` has `assetId = 'package:foo/src/foo.dart'`.
+
+#### Target os and architecture
+
+A target specifies the operating system and architecture of the targeted platform.
+
+### `build.dart`
+To bundle native assets with a package, that package must define a `build.dart`
+script which accepts a `--config` option as follows:
+
+```console
+$ dart build.dart --config <build_config.yaml>
+```
+
+The `--config` option specifies location of the `build_config.yaml` file.
+
+The `build.dart` file MUST:
+ * Read the `build_config.yaml` file,
+ * Build assets using configuration from `build_config.yaml`.
+   * If `dry_run: true` in `build_config.yaml`, then this may be skipped.
+ * Write assets into `out_dir` from `build_config.yaml`.
+   * If `dry_run: true` in `build_config.yaml`, then this may be skipped.
+   * Filename are unrelated to `assetId`.
+   * Arbitrary file names are fine, `build_output.yaml` will map `assetId` to files. 
+   * MUST avoid file name `build_output.yaml`.
+ * Write `build_output.yaml` into `out_dir` (from `build_config.yaml`).
+   * This maps `assetId` to assets previous written in `out_dir`.
+   * There may be multiple assets for a given `assetId` depending on
+     characteristics like `target`, `link_mode`, etc.
+   * If `dry_run: true` in `build_config.yaml`, the list of assets that would be
+     generated must this be enumerated. Mapping `assetId`s to non-existing files
+     is expected.
+
+
+### `build_config.yaml`
+The Dart and Flutter SDK invoke `build.dart` of all packages in the transitive
+dependencies and pass a `build_config.yaml` as the `--config` option.
+
+```yaml
+# Build in dry-run mode.
+#
+# Running in dry-run mode `<out_dir>/build_output.yaml` must be written, but
+# the files it references need not exist.
+dry_run: true | false
+
+# Build Mode.
+#
+# A hint `build.dart` can use to determined which optimizations to enable and
+# whether or not to include debug symbols in the format relevant for the asset.
+#
+# Not provided on dry runs.
+build_mode: release | debug
+
+# Metadata as output from build.dart from direct dependencies.
+#
+# Not provided on dry runs.
+dependency_metadata:
+  # package name of direct dependency.
+  some_package_name:
+    # key value pairs.
+    some_key: some_value
+
+# Preferred link mode
+link_mode_preference: dynamic | static | prefer-dynamic | prefer-static
+
+# Path to output directory where assets should be placed.
+#
+# This is also where `build_output.yaml` should be written.
+#
+# Remark: Avoid using the name "build_output.yaml" for an asset file, this is
+#         forbidden.
+out_dir: /absolute/path/to/out_dir/
+
+# Name of the package that contains the `build.dart`
+#
+# Remark: This is entirely redundant since this is a config file specified to
+# `build.dart`, and the author of `build.dart` probably knows the name of the
+# package they are writing.
+package_name: my_package_with_native_assets
+
+# Path to root folder for the package that contains `build.dart`.
+#
+# This is useful if `build.dart` wishes to find source code files embedded in
+# its own package and compile them to an asset.
+#
+# Note that this will be most likely a path in the pub cache when the package
+# with a `build.dart` is a dependency of another package.
+package_root: /absolute/path/to/my_package_with_native_assets
+
+# Target architecture
+#
+# Combined with `target_os` this specifies the "target" for which assets
+# should be built.
+#
+# Not provided on dry runs.
+target_architecture: x64 | ia32 | arm | arm64 | riscv32 | riscv64
+
+# Target operating system
+#
+# Combined with `target_architecture` this specifies the "target" for which
+# assets should be built.
+target_os: android | ios | linux | macos | windows
+
+# Schema version of this file.
+version: 1.0.0
+```
+
+### `build_output.yaml`
+Mapping from `assetId`s to files created by `build.dart`.
+
+```yaml
+# The list of assets.
+#
+# In dry runs, must contain assets for each architecture for the requested os.
+assets:
+  - id: 'package:my_package_with_native_assets/src/foo.dart'
+    link_mode: dynamic | static | prefer-dynamic | prefer-static
+    path:
+      path_type: absolute | system | process | executable
+      # Only provided for path_type absolute and system.
+      #
+      # If path_type absolute: The absolute path to the file name.
+      #
+      # If path_type system: The path of the dynamic library as available on 
+      # the target machine's PATH.
+      uri: /absolute/path/to/outdir/arbitrary_filename.whatever
+    target: linux_x64
+  ...
+
+# The files used by this build.
+#
+# If any of the files in [dependencies] are modified after [timestamp], the
+# build will be re-run.
+#
+# Not output on dry runs.
+dependencies:
+  - /absolute/path/to/my_package_with_native_assets/build.dart
+  ...
+
+# The time the build this output is for started.
+#
+# Must be before any of the files in dependencies are read.
+timestamp: 2024-01-02 17:05:35.000
+
+# Metadata usable for build.dart of packages directly depending on this package.
+#
+# Not output in dry runs.
+metadata:
+  # Key value pairs.
+  some_key: some_value
+
+# Schema version of this file.
+version: 1.0.0
+```
+
+
+### `.dart_tool/native_assets.yaml`
+
+(This is not part of the build.dart protocol.)
+
+This file is internal to the Dart SDK and Flutter SDK.
+It produced by `native_asset_builder` (as embedded in the SDKs).
+For each `target` it maps from `assetId` to a path type with an optional path.
+
+These paths are used to resolve `@Native() external` functions in Dart code.
+
+The path types in this file are as follows:
+
+* `absolute` paths are absolute paths on the system where the Dart executable is running. This path type is used when running in JIT mode on a developers' host machine, or in an Android app where the root path is the Android bundle.
+* `relative` paths are relative to the kernel or aot snapshot. This path type is used for shipping native assets in a directory together with a kernelsnapshot, aotsnapshot, or standalone executable (dartaotruntime+aotsnapshot).
+* `system` paths are expected to resolve on the target machine PATH.
+* `process` "paths" have no path, symbols are resolved in the current process.
+* `executable` "paths" have no path, symbols are resolved in the current executable.
+
+```yaml
+# Schema version of this file.
+format-version: [1, 0, 0]
+
+# A mapping from target and assetId to an asset.
+native-assets:
+  # A <target>.
+  linux_x64:
+    # An <assetId>.
+    'package:my_package_with_native_assets/src/foo.dart':
+      # A list of path_type and optionally a path.
+      - absolute | relative | system | process | executable
+      - <url> # Only provided for absolute, relative, and system path types.
+  ...
+```
+
+This file is used by the embedded SDK to resolve `assetId`s when running in
+development mode or build a deployable application (or standalone executable).
+
+## Wishes to improve for v2
+
+* Multiple asset types. Rename `Asset` to `NativeCodeAsset` and introduce `DataAsset`s.
+* The path_types really only make sense for `link_mode: dynamic`.
+* Fix the mismatch between `target` for output but `os` and `architecture` for config.
+  * Should the architecture field of native code assets be optional and be ommitted in dry runs?
+* Introduce a `link.dart` protocol with a link input and link output.