Add schema migration system to Defra

[AndrewSisley]

Part of #1002

Using Lens.

Copy-pasted from discord thread dev-db.LensVM on read(https://discord.com/channels/427944769851752448/1102730749007904798):

• Lens on local read, and on update (from anywhere, local or P2P), transform outcome to be stored (cached) in datastore
• Can tolerate discarding P2P updates/creates for unknown schema within release cycle, but must not discard stuff by the time release is cut. By that point unknown schema version updates need to be held in a hidden (from normal queries) branch
• Current datastore schema version tracked per document initially, long term we may wish to offer this per field, but not right now

From discord discussion RE unique secondary indexes:

• The loss of datastore uniqueness can be tolerated in the short term.
• Long term, this will likely use the same mechanics as the secondary index backfill stuff, as the problem also exists when defining new unique indexes on existing fields. It could be that if a Lens migration is for an object/field with a unique index it is eagerly evaluated (probably in the background).
• Something worth noting, that I had forgotten is that non-unique secondary indexes are being implemented first, it could be that this will not even be a problem within 0.6, depending on when unique secondary indexes are implemented

From discord discussion RE wasm module persistence:

We could just rely on the user/admins preserving wherever they exist at time of migration registration (1). But we could also copy them somewhere defra specific (2), or store them as blobs within a datastore (3).

• (1) is fine short term, and is likely the easiest, it is not a blocker for release
• (2) is preferred short term if low cost
• going from (1) to (2) should be fairly cheap, so if (1) is implemented first and everyone hates it, it shouldnt block merge and moving to (2) can likely be done immediately after within the release if required
• (3) is Andy's and John's preferred long term solution, but:
• (2) and (3) actually share the same public interfaces, and so the differences between the two can really be seen as implementation details

Personal/wip thoughts:

• wasmer.NewModule(store, content) is slow (40-50ms), and should not be done in user/query time. This will mean they need to be cached somewhere.
• Lens module instances are not threadsafe, and will probably need some worker-like channel stuff to ensure that db actions on different threads do not concurrently access the same lens module.
• A Lens module instance should not be shared between multiple defra requests/actions/contexts at the same time. This is to protect against the leaking of state between contexts. This is particularly important the proposed Lens enumerable stuff, but is otherwise just a very good thing to have anyway.
• The client package func(s) to manage migrations should not be coupled to mutating schema, although it should be possible to rely on the same transaction to commit/discard the changes together. I.e. it should be possible to modify the registered migrations at any point in time (e.g. for security updates to migration modules).
• The migration system should still work following a restart. This means that the modules need to be persisted locally and the config (re)loaded on db start. The Lens engine itself can be referenced directly, only the wasm modules need to be dynamic.
• The Lens related code can be nice and isolated/contained, within a defra lens sub-package.
• If PR ready before Lens is open-sourced, we can vendor the dependency in the short term: https://stackoverflow.com/questions/68544611/what-is-the-purpose-of-go-mod-vendor-command
• The Defra load-Lens code should avoid doing anything too Defra specific, and should instead be used to find deficiencies in the Lens config/interfaces.
• The update code (including via planner) uses docFetcher, so by handling the Lens magic in the fetcher-space our lens update requirements should be met auto-magically. It still needs tests though.

Tasks

Preview Give feedback

We're having trouble rendering your tasklist, it's most likely has one of the following formatting errors:

• indented tasks (i.e. nested tasklist; support for this is coming!)
• an empty task (i.e. - [ ] on a line by itself)
• any empty new lines (before/after the tasklist or in between tasks)
• duplicate issue/pr links
• a draft task exceeds 512 characters

Please check the tasklists documentation for more information. 🔗 Feedback Discussion

Thank you for participating in the Private Beta ❤️

No tasks being tracked yet.

[AndrewSisley]

I've added #1082 as a prerequisite, as we should ensure that migrations are not lost on database restart, and the linked ticket can provide a mechanism for testing this.

[AndrewSisley]

WIP branch is here: https://github.com/AndrewSisley/defradb/pull/new/1448-lens-in-defra

I may be pulling bits out into smaller PRs before the main body of work is put into PR.