[Bug?]: yarn npm audit fails with 400 error

[SimenB]

Self-service

• I'd be willing to implement a fix

Describe the bug

Running yarn npm audit -AR fails in the Jest repo with a 400 error from the npm registry.

To reproduce

$ git clone git@github.com:facebook/jest.git
$ cd jest
$ yarn
$ yarn npm audit -AR

Environment

System:
    OS: macOS 12.1
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
  Binaries:
    Node: 16.14.0 - /private/var/folders/gj/0mygpdfn6598xh34njlyrqzc0000gn/T/xfs-9e7a75e5/node
    Yarn: 3.2.0-rc.15.git.20220211.hash-32c522a7c - /private/var/folders/gj/0mygpdfn6598xh34njlyrqzc0000gn/T/xfs-9e7a75e5/yarn
    npm: 8.4.1 - ~/.nvm/versions/node/v16.14.0/bin/npm
  npmPackages:
    jest: workspace:* => 28.0.0-alpha.0

Additional context

I've added a log statement after the network call, and the reported error from npm is

{
  statusCode: 400,
  error: 'Bad Request',
  message: 'Invalid package tree, run  npm install  to rebuild your package-lock.json'
}

[arcanis]

Hm - I can reproduce it on this very repository as well. I'm sure it used to work, so perhaps a backend change broke something 🙁

[SimenB]

@arcanis I tried a yarn workspaces foreach exec yarn npm audit -R and I get a failure in just jest-website, the others exits successfully. So it seems to be some specific combination rather than the entire -A being broken.

[revyh]

I have the same issue with the command yarn workspace workspace-name npm audit --recursive.

  System:
    OS: Linux 5.13 Debian GNU/Linux 10 (buster) 10 (buster)
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  Binaries:
    Node: 16.14.2 - /tmp/xfs-04e866f4/node
    Yarn: 3.2.0 - /tmp/xfs-04e866f4/yarn
    npm: 8.5.0 - /usr/bin/npm
  npmPackages:
    jest: ^27.4.5 => 27.4.7 

Don't know what other debug information I can provide

[revyh]

Looks like I managed to make a repro. It's offline because yarn version in the sherlock playground is 4.0.0rc2 and I failed to change it to required 3.2.0.

Reproduction requires only package.json as follows

{
  "packageManager": "yarn@3.2.0",
  "devDependencies": {
    "@vue/cli-service": "~5.0.4"
  }
}

and command
docker run -it --rm --name my-running-script -v "$PWD":/usr/src/app -w /usr/src/app node:lts corepack enable && yarn install && yarn npm audit --recursive

[RebeccaStevens]

I believe I'm getting this error too. Here's my output:

❯ yarn npm audit -AR
➤ YN0035: Bad Request
➤ YN0035:   Response Code: 400 (Bad Request)
➤ YN0035:   Request Method: POST
➤ YN0035:   Request URL: https://registry.yarnpkg.com/-/npm/v1/security/audits/quick

➤ Errors happened when preparing the environment required to run this command.
➤ This might be caused by packages being missing from the lockfile, in which case running "yarn install" might help.

[Drew-Garratt]

I've recently run into the same error

Interestingly for me, the error doesn't occur when running --environment production

➤ YN0035: Bad Request
➤ YN0035:   Response Code: 400 (Bad Request)
➤ YN0035:   Request Method: POST
➤ YN0035:   Request URL: https://registry.yarnpkg.com/-/npm/v1/security/audits/quick

➤ Errors happened when preparing the environment required to run this command.
➤ This might be caused by packages being missing from the lockfile, in which case running "yarn install" might help.

[noisekit]

For me the issue happened when one of the deep deps declared dependency in this weird format

'ethers-v4': 'npm:ethers@4'

As soon as I upgraded to newer version that no longer has this line - issue stopped happening.

The offender was this lib:

npm info @eth-optimism/core-utils@0.0.1-alpha.30 dependencies

Before that I did some debugging inside yarn code itself and the actual error happens on npm side when generated on the fly "package lock" from yarn gets submitted for audit. Npm essentially replies with error that some packages are missing from the lock file (I don't remember exact error though as I fixed this problem a few weeks ago).

[Lexouden]

I managed to find another situation where this occurs.

@docusaurus/core has a dependency listed like:

"react-loadable": "npm:@docusaurus/react-loadable@5.5.2"

Between yarn@3.1.1 and yarn@3.2.x something changed that broke the "package lock" sent to npm when a dependency version is resolved with a npm: protocol, so I don't think there was something that changed on the backend.

I made it work by downgrading Yarn back to 3.1.1 but this is ofc not ideal. So I hope a better solution can be found, as I can imagine more people will run into this at some point.

[Glandos]

It should be great if yarn was able to output the payload sent to the registry server.
Right now, the only way to inspect it seems to be changing npmRegistryServer to an HTTP one, and listen with tcpdump/wireshark for the incoming payload.

[burk3rt]

I've looked in the payload by logging the request in the source code, but yes a native way would be great @Glandos.

The dependency I have issues with is "@vue/cli-service": "~5.0.1" with yarn@3.1.0 and yarn@4.0.0-rc.26.dev, just like @revyh in this comment

Does anybody have any clue by now?

[jj811208]

now npm audit use /-/npm/v1/security/advisories/bulk and only fall back to /-/npm/v1/security/audits/quick if it fails. (code)

npm-audit.md

As of version 7, npm uses the much faster Bulk Advisory endpoint to optimize the speed of calculating audit results.

I cloned the npm repository and tested the same scenario, and if npm doesn't use /-/npm/v1/security/advisories/bulk, they also receive 400 errors

In my opinion, the solution to this issue is:

1. yarn npm audit lacks maintenance, it still always uses /-/npm/v1/security/audits/quick, but it should have the same behavior as npm.
2. should there be an error in this case /-/npm/v1/security/audits/quick 🤔? if it can be fixed, then there will be no error here.

[sargunv]

I've published a minimal repro of the bug here: https://github.com/sargunv/audit-bug-repro/. The repo has three branches. The yarn-v3 branch repros this issue, while the yarn-v1 and npm branches show it can be audited successfully on those package managers.

The minimal library I published to trigger the bug is here: https://www.npmjs.com/package/@sargunv/testlib-c

[sargunv]

The specific trigger is a construction like this in the dependency tree: https://github.com/sargunv/audit-bug-repro/blob/72cd19f2eb33a29505e4c8c653e4e7b551fc7138/yarn.lock#L19

"@sargunv/testlib-c@npm:^0.1.0":
  version: 0.1.0
  resolution: "@sargunv/testlib-c@npm:0.1.0"
  dependencies:
    "@sargunv/fake-testlib-a": "npm:@sargunv/testlib-a@^0.1.0". # THIS LINE
  checksum: 026c9a8be2f5ddee32a4bfba97d093ac0ba947851c74755240bc87536af579f95ab2c2b9e17fbcc9cde840d7842e7502ed904599a7fabd6324185e3eee889683
  languageName: node
  linkType: hard

[sargunv]

I did a bit of digging with an HTTPS proxy.

Yarn v3

Here's a sample request body for the /quick endpoint that Yarn uses, generated from my above linked repro project.

POST https://registry.yarnpkg.com/-/npm/v1/security/audits/quick

{
  "requires": {
    "@sargunv/testlib-c": "^0.1.0"
  },
  "dependencies": {
    "audit-bug-repro": {
      "version": "0.0.0-use.local",
      "integrity": "0069b1507df7fda1a72cbcea230138bbf288eb1b1217be39c92e38aa99e645d8c2590978a7fb82eaf176ad2861b62ce5e1766063d605534146ccb96874f44f73",
      "requires": {
        "@sargunv/testlib-c": "^0.1.0"
      },
      "dev": false
    },
    "@sargunv/testlib-c": {
      "version": "0.1.0",
      "integrity": "6cb68cb06e6edf019ff5a5cd314ca8a3d00ec0853a3e9f19af0d108c169f1c85b364641654cc452dd53cc50b0745a3fedd78b8899f4cf02a5ea7f97a25e34400",
      "requires": {
        "@sargunv/fake-testlib-a": "@sargunv/testlib-a@^0.1.0"
      },
      "dev": false
    },
    "@sargunv/testlib-a": {
      "version": "0.1.0",
      "integrity": "7732f389a7070c3eae9d0b14ef941a3d7d36568d5b83ea58d823d70a070e80ff8ff6c8354a4391762da3ffc243167e117f2a247ded5cb0d38fca24dfca5bc2f7",
      "requires": {},
      "dev": false
    }
  }
}

My best guess is that the audit endpoint rejects it because the dependency tree doesn't look complete; there appears to be a dependency @sargunv/fake-testlib-a that's not present. Looking at this format, it's unclear how multiple versions of the same package in one tree would be represented, and I can't find an NPM API spec anywhere.

The error from NPM's /quick endpoint for the above request is as follows:

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Invalid package tree, run  npm install  to rebuild your package-lock.json"
}

Additionally, I can confirm that this issue affects any released version of Yarn Berry since yarn npm audit was added (tested as far back as v2.4.0 and as far forward as v4.0.0-rc.34).

NPM

Now, here's what the request looks like when using the /bulk endpoint (with the npm cli, on the same project):

POST https://registry.npmjs.org/-/npm/v1/security/advisories/bulk

{
  "@sargunv/testlib-a": [
    "0.1.0"
  ],
  "@sargunv/testlib-c": [
    "0.1.0"
  ]
}

The "fake" dependency name is no longer relevant; the request simply includes a hash of package names, each with a list of versions included in the tree. This request succeeds.

Yarn v1

Finally, this is the request Yarn v1 sends. It uses neither the /quick nor the /bulk endpoint, but a different /audits endpoint.

POST https://registry.yarnpkg.com/-/npm/v1/security/audits

{
  "name": "audit-bug-repro",
  "install": [],
  "remove": [],
  "metadata": {},
  "requires": {
    "@sargunv/testlib-c": "^0.1.0"
  },
  "dependencies": {
    "@sargunv/testlib-c": {
      "version": "0.1.0",
      "integrity": "sha512-yYzZOaFg/OKwogF4IXxrB/fKctJOTE3MhhbyaR7LU+C0uAlXfzvlLP2Ie6WYU1IXEWLFfOXDy8lmEmRyQk1ZsQ==",
      "requires": {
        "@sargunv/fake-testlib-a": "npm:@sargunv/testlib-a@^0.1.0"
      },
      "dependencies": {},
      "dev": false
    },
    "@sargunv/fake-testlib-a": {
      "version": "0.1.0",
      "integrity": "sha512-Fn64vahG+47n9xScHD9vViPF+twOhSg9g7sH9hTsJ4DwRHJZgIgeYf82MRoX6nURgzpksRoE4n3hxcd3o8XFxg==",
      "requires": {},
      "dependencies": {},
      "dev": false
    }
  },
  "dev": false
}

In this case, Yarn v1 forms the request using the "fake" name of the dependency key, not the "real" name of the dependency version/resolution. This request also succeeds. Sure enough, if I manually tweak the original request to do the same, NPM no longer says "bad request"!:

[nocive]

Still seeing this with 3.6.3.
Even after nuking the node_modules folder and the yarn.lock file, the problem persists.

$ yarn npm audit --recursive --json --all --environment development
➤ YN0035: Bad Request
➤ YN0035:   Response Code: 400 (Bad Request)
➤ YN0035:   Request Method: POST
➤ YN0035:   Request URL: https://registry.yarnpkg.com/-/npm/v1/security/audits/quick

➤ Errors happened when preparing the environment required to run this command.
➤ This might be caused by packages being missing from the lockfile, in which case running "yarn install" might help.

$ yarn --version
3.6.3

In my case, the problem must be deep in my dependency tree and not in direct dependencies, because if I don't --recursive it works.
Also worth noting that there's nothing special about my setup, no private registries, no proxy.

[arcanis]

As mentioned in the PR that closed this issue, since it's a major change to the implementation, it won't be released in 3.x. You can already use the RCs, though, and the stable isn't very far.

[nocive]

@arcanis thanks for the info.

Unfortunately v4 (4.0.0-rc.50) doesn't work properly with the offline package cache in node 18.x.

I get tons of Cache entry required but missing for PKG errors despite a valid .yarn/cache/.
Under node 20.x seems to run fine though.

node 18.x

$ node --version
v18.17.1

$ yarn install --immutable --immutable-cache
➤ YN0000: · Yarn 4.0.0-rc.50
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Post-resolution validation
➤ YN0060: │ typescript is listed by your project with version 5.2.2, which doesn't satisfy what msw requests (^4.4.0).
➤ YN0002: │ subscription-preferences-ui@workspace:. doesn't provide webpack (p1e1b9), requested by html-webpack-plugin
➤ YN0002: │ subscription-preferences-ui@workspace:. doesn't provide webpack (p8dcb6), requested by ts-loader
➤ YN0086: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0056: │ @aashutoshrathi/word-wrap@npm:1.2.6: Cache entry required but missing for @aashutoshrathi/word-wrap@npm:1.2.6

(...)
(...one error message for each resolved dependency...)

node 20.x

$ node --version
v20.5.1

$ yarn install --immutable --immutable-cache
➤ YN0000: · Yarn 4.0.0-rc.50
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Post-resolution validation
➤ YN0060: │ typescript is listed by your project with version 5.2.2, which doesn't satisfy what msw requests (^4.4.0).
➤ YN0002: │ subscription-preferences-ui@workspace:. doesn't provide webpack (p1e1b9), requested by html-webpack-plugin
➤ YN0002: │ subscription-preferences-ui@workspace:. doesn't provide webpack (p8dcb6), requested by ts-loader
➤ YN0086: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 0s 458ms
➤ YN0000: ┌ Link step
➤ YN0007: │ msw@npm:1.2.5 [9c327] must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:3.32.1 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js-pure@npm:3.32.1 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 2s 812ms
➤ YN0000: · Done with warnings in 3s 698ms

[arcanis]

The global cache is now opt-in rather than opt-out in 4.x - the migration to keep the old behaviour (enableGlobalCache: false) should have been done automatically, but perhaps it failed for some reason?

The Node version shouldn't change anything for this check - perhaps you ran one of the commands without the immutable flags and populated your cache, and thought it was caused by the Node version change? 🤔

[nocive]

Seems to be specific to alpine/musl and not related to the node version. Does that make more sense?
Happens on node 18.x and 20.x and likely other versions too.

$ cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.17.5
PRETTY_NAME="Alpine Linux v3.17"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

When running yarn install --immutable --immutable-cache outside of my alpine based containerized environment it doesn't occur anymore.

[nocive]

Running yarn install once in my containerized environment actually brought the missing dependencies to the cache, while running it on my host did not 🤷‍♂️

Thanks for the support @arcanis

EDIT: Seems to have been caused by the missing enableGlobalCache: false in configuration, which was not automatically done during migration.