22.1: netrc file is only parsed when challenged with HTTP/401

[ginoledesma]

Description

The change in NETRC handling per #10979 may have introduced an unexpected side-effect.

https://github.com/pypa/pip/compare/22.0.4...22.1b1?diff=split

In the current release, netrc is only parsed when the PyPI repo returns an HTTP/401 error due to a change in the method signature's default value from True to False:

    def _get_new_credentials(
        self,
        original_url: str,
        *,
        allow_netrc: bool = False,
        allow_keyring: bool = False,
    ) -> AuthInfo:

And further:

    def handle_401(self, resp: Response, **kwargs: Any) -> Response:

        # Query the keyring for credentials:
        username, password = self._get_new_credentials(
            resp.url,
            allow_netrc=False,
            allow_keyring=True,
        )

However, some PyPI repositories (such as those by JFrog Artifactory) can return HTTP/404 when auth isn't presented, and HTTP/200 otherwise, like so:

GET /api/pypi/pypi-private/simple/my-awesome-lib/ HTTP/1.1
Host: artifacts.example.com
User-Agent: pip/22.1 {"ci":null,"cpu":"x86_64","distro":{"name":"macOS","version":"12.4"},"implementation":{"name":"CPython","version":"3.10.3"},"installer":{"name":"pip","version":"22.1"},"openssl_version":"OpenSSL 1.1.1n  15 Mar 2022","python":"3.10.3","setuptools_version":"58.1.0","system":{"name":"Darwin","release":"21.5.0"}}
Accept-Encoding: gzip, deflate
Accept: text/html
Connection: keep-alive
Cache-Control: max-age=0

HTTP/1.1 404 
Server: ArtifactoryHttpServer/47a3fd1d6c202c53658865bf00439b03f16f48e4
Date: Thu, 12 May 2022 04:56:57 GMT
Content-Type: application/json
Transfer-Encoding: chunked
X-JFrog-Version: Artifactory/7.37.14 73714900
X-Artifactory-Id: 090ed27bcfd2435b77f88876e97cddc25c8604f6
Content-Encoding: gzip
X-Proxy-Cache-Status: HIT
X-Proxy-Cache-Status: EXPIRED
X-B3-TraceId: 6e3a84b1422a9e79
Strict-Transport-Security: max-age=31536000
Connection: keep-alive

Whereas in 22.0.4:

GET /api/pypi/pypi-private/simple/my-awesome-lib/ HTTP/1.1
Host: artifacts.example.com
User-Agent: pip/22.0.4 {"ci":null,"cpu":"x86_64","distro":{"name":"macOS","version":"12.4"},"implementation":{"name":"CPython","version":"3.10.3"},"installer":{"name":"pip","version":"22.0.4"},"openssl_version":"OpenSSL 1.1.1n  15 Mar 2022","python":"3.10.3","setuptools_version":"58.1.0","system":{"name":"Darwin","release":"21.5.0"}}
Accept-Encoding: gzip, deflate
Accept: text/html
Connection: keep-alive
Cache-Control: max-age=0
Authorization: Basic ****REDACTED****

HTTP/1.1 200 
Server: ArtifactoryHttpServer/47a3fd1d6c202c53658865bf00439b03f16f48e4
Date: Thu, 12 May 2022 05:13:08 GMT
Content-Type: text/html
Transfer-Encoding: chunked
X-JFrog-Version: Artifactory/7.37.14 73714900
X-Artifactory-Id: 090ed27bcfd2435b77f88876e97cddc25c8604f6
Cache-Control: max-age=300
Content-Encoding: gzip
X-Proxy-Cache-Status: EXPIRED
X-Proxy-Cache-Status: MISS
X-B3-TraceId: 2dbc36045a554171
Strict-Transport-Security: max-age=31536000
Connection: keep-alive

Expected behavior

Credentials from netrc are used at the first request

pip version

22.1

Python version

3.10.3

OS

macOS 12.4

How to Reproduce

1. pip install --upgrade pip==22.1
2. Create a ~/.netrc file
3. pip install some-package --index-url=https://artifacts.example.com/simple

Assumptions:

• PyPI server returns different results depending on when auth is first presented or not

Output

~/ $ pip3 install --debug --verbose my-awesome-lib --index-url=https://artifacts.example.com/api/pypi/pypi-private/simple --no-cache-dir
Using pip 22.1 from /Users/gledesma/.pyenv/versions/3.10.3/lib/python3.10/site-packages/pip (python 3.10)
Looking in indexes: https://artifacts.example.com/api/pypi/pypi-private/simple
ERROR: Could not find a version that satisfies the requirement my-awesome-lib (from versions: none)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[q0w]

Hmm, I think #10998 was totally incorrect, because auth has already prioritized url credentials over .netrc, here

pip/src/pip/_internal/network/auth.py

Lines 118 to 126 in 0a982f6

	url, netloc, url_user_password = split_auth_netloc_from_url(
	original_url,
	)
	# Start with the credentials embedded in the url
	username, password = url_user_password
	if username is not None and password is not None:
	logger.debug("Found credentials in url for %s", netloc)
	return url_user_password

Seems, the problem in #10979 was with Google Artifacts Registry #10904 (comment)

[potiuk]

Yeah. I think we need to revive #10904 - but I am waiting for comments from @pradyunsg and other maintainers whether my proposal is good there.

It's not really a "problem" of Google Artifact Registry, it's the incompatibility between the way how pypi "/simple" repository was defined in https://peps.python.org/pep-0503/ and it's basic incompatibiity (at least without interpretation) with RFC7617 regarding basic authentication and repositories that can host mutliple projects. Basically PEP 503 missed the fact that authentication will have to be RFC7617 compliant for cases where single domain hosts multiple pip repositories and current implementation will need to be fixed to account for that.

The maintainers were busy with 22.1 so i respectfully was not pushing for a quick merge - it's better to do it "well" rather than "quick" that's why I converted it into draft while waiting so that it won't get merged accidentally.

But I think the proposal I made with treating "/simple" as special case might work in vast majority of cases. It's not perfect and unless we can somehow figure out the "base" repository URL at the moment we resolve auth, it might work.

[pradyunsg]

Let's revert #10998 then -- we can figure out the proper fix for it in a follow up. :)

[pradyunsg]

Can someone file a PR for this?

[raybellwaves]

sorry not entirely related to this issue but thought it was worth noting for others seeing this issue.

One thing I did to get around this was to put the auth in index-url in my pip.conf instead of using a .netrc file

index-url = https://${USER}:${PW}@${REGISTRY}

where REGISTRY may look something like ${COMPANY}.jfrog.io/artifactory/api/pypi/pypi/simple

[eric-spitler]

sorry not entirely related to this issue but thought it was worth noting for others seeing this issue.

One thing I did to get around this was to put the auth in index-url in my pip.conf instead of using a .netrc file

index-url = https://${USER}:${PW}@${REGISTRY}

where REGISTRY may look something like ${COMPANY}.jfrog.io/artifactory/api/pypi/pypi/simple

The problem with putting credentials in the pip.conf is if you use a tool like pip-compile it will put comments into the requirements.txt indicating the command used, and therefore the URLs will show your credentials. Using .netrc means the credentials are handled silently and securely (well, as securely as a clear-text permission-controlled file can be).

[raybellwaves]

The problem with putting credentials in the pip.conf is if you use a tool like pip-compile it will put comments into the requirements.txt indicating the command used, and therefore the URLs will show your credentials. Using .netrc means the credentials are handled silently and securely (well, as securely as a clear-text permission-controlled file can be).

Thanks for sharing. I'll revert back to using a .netrc once fixed