- Where: zoom.us (see Registration below)
- When: June 13 2024, 17:00-18:00 UTC
- Contact:
- Name: Pat Hickey and Bailey Hayes
- Email: pat@moreproductive.org and bailey@cosmonic.com
The meeting is open to CG members only. You can join the CG here.
If this is your first time attending, please fill out the registration form to receive an invite. Please make sure you have joined the CG as above, and that your name appears on the membership page, before registering.
The meeting will be on a zoom.us video conference.
- Opening, welcome and roll call
- Please help add your name to the meeting notes.
- Please help take notes.
- Thanks!
- Announcements
- Submit a PR to add your announcement here
- Proposals and discussions
- Bailey Hayes
- Colin Murphy
- Pat Hickey
- Alex Crichton
- Dave Bakker
- David Bryant
- Fedor SMirnov
- Jeff Charles
- Kate Goldenring
- Maximilian Seidler
- Merlijn Sebrechts
- mosakka
- Nick Fitzgerald
- Piotr Sikora
- Marcin Kolny
- Robin Brown
- Sean Isom
- Sehyo Chang
- Dan Gohman
- Taylor Thomas
- Yong He
- Johnnie Birch
- Zalim Bashorov
- Stephen Berard
- Trevor Elliott
- Yosh Wuyts
- Joel Dice
- Mike Warres
- Hung-Ying Tai
Yosh Wuyts Two weeks ago presented about since and unstable gates. We are putting that into practice and we realized we want to tweak how proposal changes make it through the phase process.
Yosh In my last presentation I said that “extensions to existing specifications need to independently advance through the phase process” and “starting with phase 0”. We all liked the first part, not so much the second part, because it adds a bunch of incremental votes to advance something pretty small. So we are asking, can we advance from phase 0 to phase 3 in a single vote. That seems like a lot of process. So probably amendments to existing proposals that are at phase 3 already meet the phase 2 criteria by definition of being just an amendment. The phase 2 criteria say: portability criteria is documented. Because we are modifying an existing proposal, the criteria is already documented (using wasi-clocks timezone addition as example). Overview document also exists in the repo. And the wit exists. So, adding an extension behind an unstable gate already meets that - its in the wit, its just not part of a release. Final phase 2 criteria is that all dependencies are also phase 2.
Yosh When we think about process, I am trying to balance the agency for the proposal champion to develop their proposal, while also having the subgroup ensure that proposals meet our standards and goals. We do that second part by having a vote at releases. So the way I’m interpreting our process is that: making changes to existing proposals is done at the discretion of the proposal champions. Stabilizing changes to existing proposals must always go through a subgroup vote. The biggest guard rail is that changes to existing proposals cannot be backwards-incompatible outside of major revisions.
Yosh So here's the process by which i suggest we approach amending proposals: 1. File a change as @unstable to a proposal. 2. Champions submit change as a new phase 2 proposal for wasi, that we track as its own item. 3. Champion lands the change 4. so that it can be used behind the gate by implementors and experimenters. 5. Champion decides the feature is ready for the next release, calls for the subgroup to vote.
Yosh Next time I’ll be back here for the actual release process for 0.2.1
Pat Hickey: I am in total agreement with this proposal and think this will involve the WASI SG at the appropriate time. Thank you Yosh for talking about rules and process.
Taylor Thomas: I am also in favor of this. I think this strikes the right balance as one of the champions for a lot of the wasi-cloud repos. Especially now when we are looking to ramp adoption of a number of interfaces within the community.
Yosh: I would like us to present regular changes even if these aren’t for a vote. Encourage champions to share regularly.
Pat: This meeting is a great place to have discussions. In addition that can happen in async communications on the GitHub PRs where these additions are proposed. We want to maximize the number of ways people can give feedback and share their work.
Dan: This mirrors the CG process. Shares the flexibility and the same way the CG votes.
Yosh I will follow a PR on the WASI repo that reflects that timezones addition is phase 2, and a separate PR clarifying that amendments are tracked and follow the process I outlined here.
https://docs.google.com/presentation/d/13FbrlS1mPcG2nhypU6zLCDfw2Ghdmt-Q2xCcfeOLQ0s
Luke Wagner This is a slice of the presentation I gave at the WASM CG meeting last week, many of you may have seen it. This is for extra visibility and I gave more detail at the end.
Luke Motivation is adding support to new wasip2 functionality to existing core wasm engines. Adding core imports is easier than implementing the whole component model. Supporting incremental development of those existing core wasm engines, and some use cases only want one core module.
Luke Idea is to add a core module “build target” to the component model preview 2. So heres an example of taking a wit file, parsing and resolving it and it turns into a component type in the component binary. To target that with a component I compile C into some object files, and package it into a component using wasm-component-ld. That component can run in any component runtime that supports that world.
Luke Additionally we can take the component type definition and derive, mechanically, the module type signature (according to the canonical abi) that corresponds to that component type. That will have all the same imports. You can take the same object files above and the regular old wasm-ld and it produces a core module that is of that same signature, and any core-only runtime should be able to run it if it provides those imports.
Luke for wasi-sdk we are suggesting that if it uses --target=wasm32-wasip2-component it takes the wasm-component-ld path, and --target=wasm32-wasip2-module takes the wasm-ld path. Over in go, GOOS=wasip2 for both, -buildmode=module|component.
Luke All this core module is, is the input to wasm-tools component new that trivially turns that module into a component. So really what we are doing is documenting what the input to wasm-tools component new is, which we hadnt gotten to doing before.
Luke Why would you use components if these modules exist? Well, components have a bigger feature set - shared everything dynamic linking (e.g. cpython), shared nothing cross-language reuse, tweak the ABI to use different string encoding, or wasm-gc, or a custom realloc. So we arent bringing the full power of components to modules, just the ability to use canonical abi imports and exports, which seems to be what a lot of people want right now.
Marcin Kolny Is this just a plan for preview 2 or is this going to happen for every release?
Luke This is part of the component model spec so it goes for all releases of WASI that use CM.
Maximillian Seidler How do we treat strings?
Luke Default to utf-8. If you need more options you should use full components.
Merlijn Are these modules - can runtimes that support full components how do they treat these modules?
Luke It makes sense for those runtimes to just automatically accept those core modules and upgrade them to components internally, but thats an implementation decision
Bailey This is what two implementations I’m aware of - wasmcloud and spin - already do - they accept a core module and then transform it to a component, and execute the component.
Alex Just to clarify the behavior of wasm-tools component new, that tool requires there to be type information in some custom sections, that type information has to come from somewhere so the upgrade tool will need to have its own wit/component type information to map the names to component types if the module is stripped and doesnt provide its own. Also you can play with all these internals in the existing implementations. There are plenty of tweaks we can still make to the exact form of the core module in the spec because some of the decisions we made implementing wasm-tools component new were arbitrary and we never asked for input on them because it wasnt submitted to a spec.
Colin: So does this mean this removes the need for the adapter?
Pat: The reason the adapter exists, before we standardized p2, rust, go, etc. These all use p1in the standard libraries and so we needed to transform those apis from p1 to p2 rather than build our own custom toolchain as part of p2 development, and p2 was unstable so that was changing constantly. Now that we are past that standardization phase, those core libraries can adopt p2 directly. They need to do that to support this target. The adapter is very complex and is the ladder we want to throw away behind us. Randy is doing this for GOOS in go. wasm-ld will produce component.
Colin: I wasn’t sure about the triple?
Pat: The triple will have wasip2 in it and not use the wasip1 API’s. That doesn’t work yet in some places like in the wasi-sdk/libc.
Marcin: This is great, thanks a lot. I think that basically enables us to do what's described in: WebAssembly/WASI#595
Merlijn: What about from the perspective of a host component.
Luke: Like for the host implementation side?
Merlijn: To get a basic bare bones code, we needed to ensure that we were using the abi correctly from within wasmtime. Now looking at this, it would be possible to use components from within WAMR. How do we write the host side of that interface within WAMR? Will this give us some way to get the right bindings that allows us to get the right interfaces.
Luke: Wit will be the input for a host bindgen that each engine will implement. I don’t know how hard or easy it is for that to exist in e.g. WAMR, it should be possible.
Alex: this alternative lowering will increase the likelihood that there are divergent behaviors between implementations. For example resources have some subtle invariants that are hard to get right. So this makes it more important, to me, to have a test suite that covers the nuanced behavior of these general component model wit to module signature things, its a very complicated problem that we have tried to work on in the past so we know its hard.
Luke More implementations means more opportunities to get it subtly wrong so yes more important to have a test suite that makes it easy to get all the implementations conforming.
Pat: There is a wasmtime crate for wasmtime host bindgen. The way this works for WAMR, it’s up to the WAMR developers. That’s the tool that creates the host bindings that is specific to the host.
Robin (in chat): It seems like it'd be nice to be able to take a fairly plain module (no semantic custom sections) and pass the name/path for a wit world and have that work.
Joel Dice (in chat): You can do that indirectly today by taking a bare module and telling wasm-tools to add a custom section to it using WIT as the source.
wasm-tools component embed
Robin (in chat): It'd be nice if wasm-tools component new could do that for you some producers could avoid an extra step and make their tool only aware of implementing the module type of the world they care about.
Merlijn: What tool does the WAMR runtime use to do this bindings generation?
Pat: The whole difficulty of it is, “how does your runtime do stuff?”. Take JCO for example, it uses the standard JS API for WebAssembly and builds a component runtime out of that. Wasmtime has it’s own implementation. So WAMR is still responsible for making an implementation of this wit to core wasm bindings. This is making the work for core runtimes simpler if they don’t need all of the power of the component model, and just want to use canonical abi interfaces.
Nick (in chat): every wasm engine has its own APIs
Luke: Student implementation where there could be some shared tooling, a hole that could be filled by a new project.
Merlijn: One more question about strings. Since this seems to be the thing used for WAMR. Maybe not all platforms will be using UTF-8 strings natively. Is there a way to describe a non-canonical ABI way to describe a string.
Luke: With a single fixed build target, you’d be able to do a copy into linear memory. There is also the externref approach that JS is taking. If you create an abstraction in your language, you can do this by creating an opaque handle. This is what GC languages are doing. Right now we are focused on the linear memory languages work well. We can do better using reference types especially in the GC context in the future.
Alex I wouldn't want to go the route of having the -component or -module in the target triple. As a user the only thing they care about is whether they are targeting -wasip2.
Luke: Yes that lines up with what the go folks want too. What would be the flag?
Alex: I think it should be some additional option - I think a linker flag that defaults to invoke wasm-component-ld, and then if they specify it will just invoke wasm-ld instead.
Luke: Yes, that’s why there's a unicode snowman on the slide. We can tweak that together and collaborate on the proposal.
Joel Dice (in chat): Any thoughts on supporting the upcoming async lift/lower ABI (and corresponding built-in functions) at the module level?
Luke: When we shift to CM Preview 3 we will have to answer those questions for all the new things that are added in p3. Maybe there are a couple ways to name core functions and you get to pick what signature to use based on the name you choose.
Luke: What about avoiding cabi_realloc? I want to address that problem orthogonally. Theres a PR CM/#304 to add fixed-length lists that allows a more efficient canonical abi, so you dont have to pass the length as an argument. So what if we extend the signature of read to pass in a parameter length: u64, and the return type is now list<u8; ..length>. This looks a bit like a dependent type but its enforced dynamically, so its an extra contract captured in the type. This is already an invariant that read documents, but we’d be specifying it mechanically here, and then the canonical ABI says because the caller specified the length then they are passing in a buffer that is already guaranteed to be of that length, and the callee no longer needs cabi_realloc because they already have the buffer.
Luke: So these ideas subsumes the “wasit2” idea that we had thrown around in some github discussions, its now been split into two orthogonal parts: define a core module target, and provide a way to avoid cabi_realloc.
Robin: What if the function does need realloc?
Luke: then the runtime calls the realloc function. We are not forcing anyone to use this, some complex types really do need realloc to be passed across the canonical abi and if the host runtime doesn't want to support using realloc then it just can't support worlds that use types like that. The trick is that in core wasm, you have mutable function tables.
Luke: I am writing this up in BuildTargets.md of the component model repo.
Bailey: We are publishing the wasi 0.2 release on Github as packages now! That makes it easier for many tools to fetch the wit/component type definitions of wasi, which previously there was manual work for.
Dan: For wasi proposals that depend on other wasi proposals, can we use this to manage dependencies
Bailey: We need a tool that completes that - most of the work is done in wasm-package-tools https://github.com/bytecodealliance/wasm-pkg-tools, the cli is named wackage [editor: spelling, link to this]. Taylor, James, and others have been working on it. The tool isnt 100% complete yet but most of this really just started working for a demo last week. So, please folks who are interest start trying to use it and tell us whats working.
Dave Bakker: What is a package on github?
Bailey: Github supports GHCR, which is an OCI registry. What we are really publishing are OCI artifacts. We opted for this because we are already publishing tarball releases on Github, so this is just a way Github makes it easier to do this mechanically and works with more tools. We are talking to Github people about improving the UI around this - right now the instructions say to docker pull it, which actually doesnt work for these artifacts, we are asking them for tweaks that tell the user how to consume these wasm artifacts.