Annotate sources of requirements

[palfrey]

Summary

Fixes #1343. This is kinda a first draft at the moment, but does at least mostly work locally (barring some bits of the test suite that seem to not work for me in general).

Test Plan

Mostly running the existing tests and checking the revised output is sane

Outstanding issues

Most of these come down to "AFAIK, the existing tools don't support these patterns, but uv does" and so I'm not sure there's an existing good answer here! Most of the answers so far are "whatever was easiest to build"

• Is "-r pyproject.toml" correct? Should it show something else or get skipped entirely No it wasn't. Fixed in 3044fa8
• If the requirements file is stdin, that just gets skipped. Should it be recorded?
• Overrides get shown as "--override<override.txt>". Correct?
• Some of the tests (e.g. dependency_excludes_non_contiguous_range_of_compatible_versions) make assumptions about the order of package versions being outputted, which this PR breaks. I'm not sure if the text is fairly arbitrary and can be replaced or whether the behaviour needs fixing? - fixed by removing the custom pubgrub PartialEq/Hash
• Are all the TrackedFromStr et al changes needed, or is there an easier way? I don't think so, I think it's necessary to track these sort of things fairly comprehensively to make this feature work, and this sort of invasive change feels necessary, but happy to be proved wrong there :)
• If you have a requirement coming in from two or more different requirements files only one turns up. I've got a closed-source example for this (can go into more detail if needed), mostly consisting of a complicated set of common deps creating a larger set. It's a rarer case, but worth considering. 042432b
• Doesn't add annotations for setup.py yet
  • This is pretty hard, as the correct location to insert the path is crates/pypi-types/src/metadata.rs's parse_pkg_info, which as it's based off a source distribution has entirely thrown away such matters as "where did this package requirement get built from". Could add "built package name" as a dep, but that's a little odd.

[palfrey]

I am not entirely sure this is the right package, but AFAIK, it's the only crate that everything that needs it can depend on....

[charliermarsh]

I want to experiment with a slightly different approach here. Instead of a trait, could we just always use None for the source, and then add a method to set the source after parsing? Since the parsing itself doesn't depend on the source at all, right?

[charliermarsh]

(I can give it a try.)

[palfrey]

Sure. If you can make it work, all good!

[ThiefMaster]

I just tested it with the requirements files of https://github.com/indico/indico/, and for requirements.{in,txt} it works great - no changes beyond the comments on top! :)

For requirements.dev.{in,txt}, however, I noticed one bug: Instead of -c requirements.txt it uses -r requirements.txt in the source comments. That aside, it seems to do everything fine there as well.

[zanieb]

Exciting! I'm not a good review for this one, but I wonder if it could be used to address #1854 next?

[charliermarsh]

Could we instead omit this here, and re-connect the requirements to their paths when we build the ResolutionGraph?

[charliermarsh]

I think that could also enable correct annotation for (e.g.) -c requirements.txt, since then you could identify requirements that came from constraints.

[palfrey]

I started writing "here's why this doesn't work" and then found out in the course of writing it, that yes, it can be done! A tool I really want but can't seem to find with a bit of googling is some sort of code visualisation thing for Rust that'd let me visually see all of this v.s. the "lets cram it all into my head" approach I've been using, which clearly isn't the best option.

Here's my notes:

The only code that actually notices -r/-c requirements is parse_entry in the requirements-txt crate, which then only gets used by parse/parse_inner in RequirementsTxt. This results in path data in RequirementEntry and source values in the Requirement's that are the constraints in the RequirementsTxt struct.

That in turn gets called by RequirementsSpecification from_source/from_sources, which similarly builds requirements/constraints with path/source data.

pip_compile in the uv crate then calls all that, feeds it into the pubgrub bits for resolution, and then feeds the results into DisplayResolutionGraph for display.

So, yeah. I could take the data from RequirementsSpecification, fish out a package -> sources hashmap and skip adding it to Pubgrub entirely... let me have a look at that.

[palfrey]

Fixed! Also resolved some test failures :)

[palfrey]

I just tested it with the requirements files of https://github.com/indico/indico/, and for requirements.{in,txt} it works great - no changes beyond the comments on top! :)

For requirements.dev.{in,txt}, however, I noticed one bug: Instead of -c requirements.txt it uses -r requirements.txt in the source comments. That aside, it seems to do everything fine there as well.

9655e91 should fix that.

[palfrey]

@charliermarsh I'm seeing some test failures on Windows, and they appear to be related to path redactions in insta. There's some nice work for doing that cross-platform in crates/uv/tests/common/mod.rs, but I'm needing it for things like crates/requirements-txt/src/lib.rs and that would be a circular dependency. Any thoughts on how to approach sorting those out? I don't have an easy way to do Windows work sadly, so this might be a bit fiddly!

[ThiefMaster]

9655e91 should fix that.

Indeed, the output is exactly what I expect now!

[adrian@claptrap:~/dev/indico/src:master *$]> git diff
diff --git a/docs/requirements.txt b/docs/requirements.txt
index af095861bf..e16ac62be7 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,9 +1,5 @@
-#
-# This file is autogenerated by pip-compile with Python 3.12
-# by the following command:
-#
-#    pip-compile --strip-extras docs/requirements.in
-#
+# This file was autogenerated by uv via the following command:
+#    uv pip compile -o docs/requirements.txt docs/requirements.in
 alabaster==0.7.16
     # via sphinx
 babel==2.14.0
diff --git a/requirements.dev.txt b/requirements.dev.txt
index 6146f0535c..3fdd46a5bd 100644
--- a/requirements.dev.txt
+++ b/requirements.dev.txt
@@ -1,9 +1,5 @@
-#
-# This file is autogenerated by pip-compile with Python 3.12
-# by the following command:
-#
-#    pip-compile --strip-extras requirements.dev.in
-#
+# This file was autogenerated by uv via the following command:
+#    uv pip compile -o requirements.dev.txt requirements.dev.in
 aiosmtpd==1.4.5
     # via pytest-localserver
 alabaster==0.7.16
@@ -125,6 +121,8 @@ parso==0.8.4
     #   pyquotes
 pep8-naming==0.13.3
     # via -r requirements.dev.in
+pip==24.0
+    # via pip-tools
 pip-tools==7.4.1
     # via -r requirements.dev.in
 plantweb==1.2.1
@@ -190,6 +188,10 @@ ruff==0.4.1
     # via -r requirements.dev.in
 schemainspect==3.1.1663587362
     # via migra
+setuptools==69.5.1
+    # via
+    #   flake8-quotes
+    #   pip-tools
 six==1.16.0
     # via
     #   -c requirements.txt
@@ -267,7 +269,3 @@ werkzeug==3.0.2
     #   pytest-localserver
 wheel==0.43.0
     # via pip-tools
-
-# The following packages are considered to be unsafe in a requirements file:
-# pip
-# setuptools
diff --git a/requirements.txt b/requirements.txt
index 32521e6a04..a8ec374820 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,9 +1,5 @@
-#
-# This file is autogenerated by pip-compile with Python 3.12
-# by the following command:
-#
-#    pip-compile --strip-extras
-#
+# This file was autogenerated by uv via the following command:
+#    uv pip compile -o requirements.txt requirements.in
 alembic==1.13.1
     # via
     #   -r requirements.in

[jardarbo]

@palfrey Very nice of you to look at this!

Testing your version with the following pyproject.toml:

$ cat pyproject.toml
[project]
name = "uv_test"
dynamic = ["version"]
dependencies = ["click"]

Running uv pip compile with your version yields the following:

$ uv pip compile pyproject.toml
Resolved 1 package in 4ms
# This file was autogenerated by uv via the following command:
#    uv pip compile pyproject.toml
click==8.1.7
    # via -r pyproject.toml

Running pip-compile (https://github.com/jazzband/pip-tools) yields:

$ pip-compile pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
#    pip-compile pyproject.toml
#
click==8.1.7
    # via uv_test (pyproject.toml)

The difference is # via -r pyproject.toml vs. # via uv_test (pyproject.toml).

I don't know if you think we should match the output of pip-compile, but it makes it easier to mix the usage of uv pip compile and pip-compile.

Running this on Ubuntu on Windows 11 WSL.

[ThiefMaster]

+1 on matching the pip-compile output, it makes transitioning from pip-tools to uv much more straightforward (as you immediately see that nothing changed), and -r pyproject.toml seems weird anyway (unless pip install -r pyproject.toml is valid which I'd find surprising)

[palfrey]

The difference is # via -r pyproject.toml vs. # via uv_test (pyproject.toml).

I don't know if you think we should match the output of pip-compile, but it makes it easier to mix the usage of uv pip compile and pip-compile.

I didn't investigate this path much, but had seen -r pyproject.toml and wasn't sure offhand how this was meant to be handled. Now I know, 3044fa8 fixes this.

[jardarbo]

Now I know, 3044fa8 fixes this.

Wow, this was quick @palfrey, thanks! This does indeed make output same as pip-compile.

$ uv pip compile pyproject.toml
Resolved 1 package in 4ms
# This file was autogenerated by uv via the following command:
#    uv pip compile pyproject.toml
click==8.1.7
    # via uv-test (pyproject.toml)

[charliermarsh]

I have a Windows machine so I can debug on that.

[palfrey]

BTW, it doesn't seem to add sources for editables e.g. the poetry_editable = { path = "../poetry_editable", editable = true } in tool_uv_sources just comes out as -e ../poetry_editable with no source. This might well be like the setup.py scenario I listed in the issues in the description, and may well be effectively unsolvable (or at least worth pushing further down the line post this PRs merge)

Nope :) Got a local fix, pushing shortly...

[palfrey]

eb3ea98 fixes editables. I needed to change the sources key to String to cope with both package names and editables urls, which I'm a little uncertain about.

Just going to fix that clippy issue, then get some sleep 💤. Thanks for the review and work here!

[palfrey]

🥳

[zanieb]

Thanks for the contribution!