[Bug] yarn npm audit doesn't report all vulnerabilities

[xfournet]

• I'd be willing to implement a fix

Describe the bug

yarn npm audit -R (yarn2) doesn't report some vulnerabilities which are detected with npm audit and yarn audit (yarn1)
In my case when a dependencies is present with 2 different version, only the newer one seems to be taken into consideration, which hide the vulnerabilities of the older one

To Reproduce

Reproduction

await packageJsonAndInstall({
dependencies: {
  [`react`]: `16.14.0`,
},
devDependencies: {
  [`@storybook/react`]: `6.1.18`,
  [`react-scripts`]: `4.0.3`,
},
});

const output = await yarn(`npm`, `audit`, `-R`);
expect(output).not.toContain(`No audit suggestions`);

package.json

{
  "name": "foo",
  "version": "1.0.0",
  "private": true,
  "dependencies": {
    "react": "16.14.0"
  },
  "devDependencies": {
    "@storybook/react": "6.1.18",
    "react-scripts": "4.0.3"
  }
}

With yarn2, no vulnerability is found

yarn install
[...]
yarn npm audit -R
➤ YN0001: No audit suggestions

With npm, a vulnerability related to immer is reported

npm install --force
[... report 4 vulnerabilities]
npm audit
# npm audit report

immer  <8.0.1
Severity: high
Prototype Pollution - https://npmjs.com/advisories/1603
fix available via `npm audit fix --force`
Will install @storybook/react@4.0.2, which is a breaking change
node_modules/immer
  react-dev-utils  6.0.6-next.9b4009d7 - 11.0.2
  Depends on vulnerable versions of immer
  node_modules/react-dev-utils
    @storybook/core  4.0.3 - 6.2.0-alpha.2
    Depends on vulnerable versions of react-dev-utils
    node_modules/@storybook/core
    @storybook/react  4.0.3 - 6.2.0-alpha.2
    Depends on vulnerable versions of react-dev-utils
    node_modules/@storybook/react

4 high severity vulnerabilities

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

With yarn1, the same vulnerability to immer is reported

yarn install
[...]
❯ yarn audit
yarn audit v1.22.10
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ high          │ Prototype Pollution                                          │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ immer                                                        │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >=8.0.1                                                      │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ @storybook/react                                             │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ @storybook/react > @storybook/core > react-dev-utils > immer │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://www.npmjs.com/advisories/1603                        │
└───────────────┴──────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ high          │ Prototype Pollution                                          │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ immer                                                        │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >=8.0.1                                                      │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ @storybook/react                                             │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ @storybook/react > react-dev-utils > immer                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://www.npmjs.com/advisories/1603                        │
└───────────────┴──────────────────────────────────────────────────────────────┘
2 vulnerabilities found - Packages audited: 2087
Severity: 2 High
Done in 1.69s.

Even with yarn2, immer dependency 1.10.0 is present in the dependency tree and should have been reported. Note that in that case immer is present in two version, 1.10.0 which is vulnerable and 8.0.1 which is not vulnerable/

yarn why -R immer
└─ foo@workspace:.
   ├─ @storybook/react@npm:6.1.18 [6847a] (via npm:6.1.18 [6847a])
   │  ├─ react-dev-utils@npm:10.2.1 (via npm:^10.0.0)
   │  │  └─ immer@npm:1.10.0 (via npm:1.10.0)
   │  └─ @storybook/core@npm:6.1.18 [1e78a] (via npm:6.1.18 [1e78a])
   │     └─ react-dev-utils@npm:10.2.1 (via npm:^10.0.0)
   └─ react-scripts@npm:4.0.3 [6847a] (via npm:4.0.3 [6847a])
      └─ react-dev-utils@npm:11.0.3 (via npm:^11.0.3)
         └─ immer@npm:8.0.1 (via npm:8.0.1)

Removing react-scripts dependency from package.json and now the immer vulnerability is reported with yarn2 too (still as with npm and yarn1):

yarn npm audit -R
└─ immer: 1.10.0
   ├─ Issue: Prototype Pollution
   ├─ URL: https://npmjs.com/advisories/1603
   ├─ Severity: high
   ├─ Vulnerable Versions: <8.0.1
   ├─ Patched Versions: >=8.0.1
   ├─ Via: @storybook/react
   └─ Recommendation: Upgrade to version 8.0.1 or later

So it seems that when a dependency is present with 2 different versions, only the newer one is used for the audit command which is a flaw in the vulnerability detection.

yarn npm audit feature has been added in #1892

Environment if relevant (please complete the following information):

• OS: Windows 10
• Node version: 15.9.0
• Yarn version: 2.4.0

[yarnbot]

This issue reproduces on master:

Error: expect(received).not.toContain(expected) // indexOf

Expected substring: not "No audit suggestions"
Received string:        "➤ YN0001: No audit suggestions
"
    at module.exports (evalmachine.<anonymous>:13:20)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:13
    at async executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:16)
    at async Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:12)
    at async ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-1.0.38-d4f5e2dbf3-63f998598d.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:38)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-a57989414f.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)

[xfournet]

Quick & dirty plugin i'm using in meantime (since i'm using yarn workspaces i can't fallback to npm)

const { once } = require('events');
const fs = require('fs');
const readline = require('readline');

module.exports = {
  name: `plugin-npmaudit`,
  factory: (require) => {
    const { Command } = require(`clipanion`);
    const { Configuration, Project, httpUtils, structUtils } = require(`@yarnpkg/core`);

    class NpmAuditCommand extends Command {
      async execute() {
        const configuration = await Configuration.find(this.context.cwd, this.context.plugins);
        const { project, workspace } = await Project.find(configuration, this.context.cwd);

        if (!workspace) {
          throw new WorkspaceRequiredError(project.cwd, this.context.cwd);
        }

        await project.restoreInstallState();

        const request = {};
        let dependenciesCount = 0;
        const putResolution = (name, version) => {
          const versions = request[name] ?? [];
          if (versions.length === 0) {
            request[name] = versions;
          }
          versions.push(version);
          dependenciesCount++;
        };

        for (const pkg of project.storedPackages.values()) {
          const { reference, version } = pkg;
          if (reference.startsWith('npm:')) {
            putResolution(structUtils.stringifyIdent(pkg), version);
          }
        }

        const vulnerableDependencies = await httpUtils.post(
          'https://registry.npmjs.org/-/npm/v1/security/advisories/bulk',
          request,
          {
            configuration,
            jsonResponse: true
          });

        const vulnerableDependenciesCount = Object.keys(vulnerableDependencies).length;
        const report = {
          vulnerableDependencies,
          vulnerableDependenciesCount,
          dependenciesCount,
        };

        this.context.stdout.write(JSON.stringify(report, undefined, 2));

        return vulnerableDependenciesCount === 0 ? 0 : 1;
      }
    }

    NpmAuditCommand.addPath('npmaudit');

    return {
      commands: [
        NpmAuditCommand,
      ],
    };
  }
};

With the package.json used in the repro case and then use this command:

❯ yarn npmaudit
{
  "vulnerableDependencies": {
    "immer": [
      {
        "id": 1603,
        "url": "https://npmjs.com/advisories/1603",
        "title": "Prototype Pollution",
        "severity": "high",
        "vulnerable_versions": "<8.0.1"
      }
    ]
  },
  "vulnerableDependenciesCount": 1,
  "dependenciesCount": 1772
}

immer vulnerability is reported as expected

[Ariane-B]

Yarn reports no vulnerabilities (or rather, "YN0001: No audit suggestions") on our whole tree of dependencies, no matter if we pass it various options, while NPM reports 329 vulnerabilities (3 low, 324 moderate, 2 high). It's enough to prevent us from migrating from npm for the time being.

Looking forward to yarn npm audit working at least as well as npm audit and we can enjoy all the other Yarn goodies.

[C0Nd3Mnd]

Same here:

yarn npm audit
➤ YN0001: No audit suggestions

Adding the --recursive flag actually results in an HTTP error:

yarn npm audit --recursive
➤ YN0001: HTTPError: Response code 400 (Bad Request)
➤ Errors happened when preparing the environment required to run this command.
➤ This might be caused by packages being missing from the lockfile, in which case running "yarn install" might help.

(the suggested yarn install doesn't help)

[Shinigami92]

Same here:

yarn npm audit
➤ YN0001: No audit suggestions

Adding the --recursive flag actually results in an HTTP error:

yarn npm audit --recursive
➤ YN0001: HTTPError: Response code 400 (Bad Request)
➤ Errors happened when preparing the environment required to run this command.
➤ This might be caused by packages being missing from the lockfile, in which case running "yarn install" might help.

(the suggested yarn install doesn't help)

I can reproduce this in a project in our company

[illiakovalenko]

Same here:

yarn npm audit
➤ YN0001: No audit suggestions

Adding the --recursive flag actually results in an HTTP error:

yarn npm audit --recursive
➤ YN0001: HTTPError: Response code 400 (Bad Request)
➤ Errors happened when preparing the environment required to run this command.
➤ This might be caused by packages being missing from the lockfile, in which case running "yarn install" might help.

(the suggested yarn install doesn't help)

Same for me. Will it be fixed? Impossible to audit vulnerabilities

[wezter96]

This issue still occurs.