universal-lock: Lock with all extras, trim down on installation #3700
Comments
I think there is a hitch here right? Because some packages will have "dev" extras, and we specifically don't want to bring those into the lock file right? Basically that a package shouldn't inherit the dev dependencies of its dependencies. How do we deal with that? |
|
Development dependencies are separate from extras, technically. They can be placed in an extra, but generally they are separate and not a part of the standard metadata (i.e. it's project management tool specific). |
|
There's a bigger problem here though... extras can conflict with each other. Are we going to solve with them all turned on and ban conflicts? Are we going to lock all combinations? |
|
If we want to support conflicting extras, the user needs to specify which those are and we apply the same split-resolve-merge as for conflicting disjoint markers. The ensures that extras are generally unified and conflict free and avoids combinatorial explosion. |
|
Do we know how they're going to express the conflict? Can you just do it with |
|
We need additional |
|
I can take this if we want to start by "including all extras, then trimming at install time". |
|
You may want to take inspiration from pixi, which handles this via features (equivalent to a group of extra), that are included in environment(s). |
Add workspace support when using `-r <path>/pyproject.toml` or `-e <path>` in the pip interface. It is limited to all-editable static-metadata workspaces, and tests only include a single main workspace, ignoring path dependencies in another workspace. This can be considered the MVP for workspace support: You can create a workspace, you can install from it, but some options and conveniences are still missing. I'll file follow-up tickets (support in lockfiles, support path deps in other workspace, #3625) There is also support in `uv run`, but we need #3700 first to properly support using different current projects in the bluejay interface, currently the resolution and therefore the lockfile depends on the current project. I'd do this change first (it's big enough already), then #3700, and then add workspace support properly to bluejay. Fixes #3404
## Summary This PR adds extras to the lockfile, and enables users to selectively sync extras in `uv sync` and `uv run`. The end result here was fairly simple, though it required a few refactors to get here. The basic idea is that `DistributionId` now includes `extra: Option<ExtraName>`, so we effectively treat extras as separate packages. Generating the lockfile, and generating the resolution from the lockfile, fall out of this naturally with no special-casing or additional changes. The main downside here is that it bloats the lockfile significantly. Specifically: - We include _all_ distribution URLs and hashes for _every_ extra variant. - We include all dependencies for the extra variant, even though that are dependencies of the base package. We could normalize this representation by changing each distribution have an `optional-dependencies` hash map that keys on extras, but we actually don't have the information we need to create that right now (specifically, we can't differentiate between dependencies that _require_ the extra and dependencies on the base package). Closes #3700.
The lockfile needs to be built on a resolution with all extras of the root project and workspace requirements, but when installing we only want the extras and workspace packages the user requested. The need to trim the resolution, probably by a DAG traversal. This includes mapping unnamed requirements to their named counterpart in the resolution map.
The text was updated successfully, but these errors were encountered: