[BUG] Infinite loop installing a package in a workspace

[targos]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Running npm install in my repository without a package-lock.json file enters an infinite loop inside npm for a dependency in a workspace.

Looping part of the logs with `--loglevel silly`

npm timing idealTree:node_modules/@nivo/core Completed in 6ms
npm timing idealTree:node_modules/@nivo/recompose Completed in 1ms
npm sill placeDep ROOT @nivo/annotations@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/axes@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/colors@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/legends@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/scales@0.70.0 OK for: @nivo/scatterplot@0.70.1 want: 0.70.0
npm sill placeDep ROOT @nivo/voronoi@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep node_modules/@nivo/scatterplot react-motion@0.5.2 OK for: @nivo/scatterplot@0.70.1 want: ^0.5.2
npm sill placeDep ROOT react@16.14.0 REPLACE for: react-motion@0.5.2 want: ^0.14.9 || ^15.3.0 || ^16.0.0
npm sill timing Tried to end timer that doesn't exist: idealTree:#root
npm sill placeDep ROOT @nivo/scatterplot@0.70.1 OK for: repro-ws@0.0.0 want: ^0.70.1
npm sill placeDep ROOT @nivo/core@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/tooltip@0.70.1 OK for: @nivo/core@0.70.1 want: 0.70.1
npm sill placeDep ROOT prop-types@15.7.2 OK for: @nivo/core@0.70.1 want: >= 15.5.10 < 16.0.0
npm timing idealTree:ws Completed in 6ms
npm sill placeDep ROOT @nivo/recompose@0.70.0 OK for: @nivo/core@0.70.1 want: 0.70.0
npm sill placeDep ROOT @react-spring/web@9.1.2 OK for: @nivo/core@0.70.1 want: 9.1.2
npm sill placeDep ROOT react-dom@17.0.2 OK for: @react-spring/web@9.1.2 want: >=16.8
npm sill placeDep ROOT react@17.0.2 REPLACE for: react-dom@17.0.2 want: 17.0.2
npm timing idealTree:node_modules/@nivo/core Completed in 6ms
npm timing idealTree:node_modules/@nivo/recompose Completed in 0ms
npm sill placeDep ROOT @nivo/annotations@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/axes@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/colors@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/legends@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/scales@0.70.0 OK for: @nivo/scatterplot@0.70.1 want: 0.70.0
npm sill placeDep ROOT @nivo/voronoi@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep node_modules/@nivo/scatterplot react-motion@0.5.2 OK for: @nivo/scatterplot@0.70.1 want: ^0.5.2
npm sill placeDep ROOT react@16.14.0 REPLACE for: react-motion@0.5.2 want: ^0.14.9 || ^15.3.0 || ^16.0.0
npm sill timing Tried to end timer that doesn't exist: idealTree:#root
npm sill placeDep ROOT @nivo/scatterplot@0.70.1 OK for: repro-ws@0.0.0 want: ^0.70.1
npm sill placeDep ROOT @nivo/core@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/tooltip@0.70.1 OK for: @nivo/core@0.70.1 want: 0.70.1
npm sill placeDep ROOT prop-types@15.7.2 OK for: @nivo/core@0.70.1 want: >= 15.5.10 < 16.0.0
npm timing idealTree:ws Completed in 7ms
npm sill placeDep ROOT @nivo/recompose@0.70.0 OK for: @nivo/core@0.70.1 want: 0.70.0
npm sill placeDep ROOT @react-spring/web@9.1.2 OK for: @nivo/core@0.70.1 want: 9.1.2
npm sill placeDep ROOT react-dom@17.0.2 OK for: @react-spring/web@9.1.2 want: >=16.8
npm sill placeDep ROOT react@17.0.2 REPLACE for: react-dom@17.0.2 want: 17.0.2
npm timing idealTree:node_modules/@nivo/core Completed in 6ms
npm timing idealTree:node_modules/@nivo/recompose Completed in 0ms
npm sill placeDep ROOT @nivo/annotations@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/axes@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/colors@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/legends@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/scales@0.70.0 OK for: @nivo/scatterplot@0.70.1 want: 0.70.0
npm sill placeDep ROOT @nivo/voronoi@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep node_modules/@nivo/scatterplot react-motion@0.5.2 OK for: @nivo/scatterplot@0.70.1 want: ^0.5.2
npm sill placeDep ROOT react@16.14.0 REPLACE for: react-motion@0.5.2 want: ^0.14.9 || ^15.3.0 || ^16.0.0
npm sill timing Tried to end timer that doesn't exist: idealTree:#root
npm sill placeDep ROOT @nivo/scatterplot@0.70.1 OK for: repro-ws@0.0.0 want: ^0.70.1
npm sill placeDep ROOT @nivo/core@0.70.1 OK for: @nivo/scatterplot@0.70.1 want: 0.70.1
npm sill placeDep ROOT @nivo/tooltip@0.70.1 OK for: @nivo/core@0.70.1 want: 0.70.1
npm sill placeDep ROOT prop-types@15.7.2 OK for: @nivo/core@0.70.1 want: >= 15.5.10 < 16.0.0
npm timing idealTree:ws Completed in 25ms
npm sill placeDep ROOT @nivo/recompose@0.70.0 OK for: @nivo/core@0.70.1 want: 0.70.0
npm sill placeDep ROOT @react-spring/web@9.1.2 OK for: @nivo/core@0.70.1 want: 9.1.2
npm sill placeDep ROOT react-dom@17.0.2 OK for: @react-spring/web@9.1.2 want: >=16.8
npm sill placeDep ROOT react@17.0.2 REPLACE for: react-dom@17.0.2 want: 17.0.2
npm timing idealTree:node_modules/@nivo/core Completed in 17ms
npm timing idealTree:node_modules/@nivo/recompose Completed in 0ms

Expected Behavior

It should finish to install without errors. The behavior is correct if the dependency is not in a workspace.

Steps To Reproduce

1. git clone https://github.com/targos/npm-repro-infinite-loop.git
2. cd npm-repro-infinite-loop
3. npm install

Environment

• OS: Windows 10, CentOS 8
• Node: 16.3.0
• npm: 7.16.0

[targos]

I can still reproduce with npm 7.17.0.

[targos]

I can still reproduce with npm 7.18.1

[targos]

I can still reproduce with npm 7.19.0

[Newbie012]

Any idea what's the cause of that? having the same issue on 7.19.0 as well

[adiun]

Our team is hitting this issue too with workspaces. We have a checked-in package-lock.json that we're relying on but if we delete the package-lock.json and try to do an npm install over the workspace it gets into an infinite loop

[targos]

I can still reproduce with npm 7.19.1

[ejazahm3d]

Having the same issue.

[adiun]

Not sure if this helps anyone but I was able to resolve my issue by (a) making sure all package versions matched across all workspaces (b) bisecting my dependencies (deleting half of the dependencies, rerunning npm install, repeat, until success).

I found through this process that the offending package was graphql-hooks. When I removed just this package, npm install succeeded!

So then I deleted the package-lock file, removed graphql-hooks, and ran an npm install to generate the lock file from scratch. Then added graphql-hooks back and npm install again. Now all packages work with successive npm install.

I still think this a bad bug with npm 7/workspaces but I hope this helps someone.

[targos]

@ruyadorno I'm sorry for pinging you directly, but after more than a month, I lost hope that someone from the npm team would see this issue.

[ruyadorno]

no worries @targos! I'm sorry about the delay 😊

I'm not sure this is a workspaces issue and I'm more inclined to believe this is a peer dependencies failure, if I try running npm install inside that @nivo/scatterplot package I'm able to see the proper ERESOLVE error:

$ pacote extract @nivo/scatterplot ./nivo-scatterplot
{
  resolved: 'https://registry.npmjs.org/@nivo/scatterplot/-/scatterplot-0.73.0.tgz',
  integrity: 'sha512-kYkGf0Cya3s6HEH3aTv5rA0pTNYI/77mBjKvg++YsJSPpz0IqKtWiJTiHcUiQjDW4dZfgvxHyEdPO8s8OyFnEA==',
  from: '@nivo/scatterplot@'
}

$ cd ./nivo-scatterplot/


$ npm install
npm ERR! code ERESOLVE
npm ERR! ERESOLVE unable to resolve dependency tree
npm ERR!
npm ERR! Found: react@17.0.2
npm ERR! node_modules/react
npm ERR!   peer react@">= 16.14.0 < 18.0.0" from the root project
npm ERR!   peer react@">= 16.14.0 < 18.0.0" from @nivo/annotations@0.73.0
npm ERR!   node_modules/@nivo/annotations
npm ERR!     @nivo/annotations@"0.73.0" from the root project
npm ERR!   11 more (@nivo/core, @nivo/axes, @nivo/colors, @nivo/legends, ...)
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer react@"^0.14.9 || ^15.3.0 || ^16.0.0" from react-motion@0.5.2
npm ERR! node_modules/react-motion
npm ERR!   react-motion@"^0.5.2" from the root project
npm ERR!   react-motion@"^0.5.2" from @nivo/colors@0.73.0
npm ERR!   node_modules/@nivo/colors
npm ERR!     @nivo/colors@"0.73.0" from the root project
npm ERR!     1 more (@nivo/annotations)
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
npm ERR!
npm ERR! See $HOME/.npm/eresolve-report.txt for a full report.

npm ERR! A complete log of this run can be found in:
npm ERR!     $HOME/.npm/_logs/2021-07-14T21_26_44_693Z-debug.log

This def needs some more debugging and digging to find the root cause. I know that @isaacs is in the middle of a larger peer dependencies refactor at the moment so it might be better to wait for that to be done before starting to tackle this bug.

[isaacs]

I have good news, bad news, and a workaround.

the good news

The good news is that I have a fix for this already in the npm/arborist branch I'm working on where I'm refactoring the buildIdealTree method. So this does not trigger an infinite loop or fatal crash once that lands. It's almost done, I'm just finishing getting test coverage back up to 100%, and combing through the nitpicky issues that always come from that. But the bulk of it is done, and working already.

the bad news

The bad news is that, even with this fix, we end up creating a not-strictly-valid tree, because of this:

  peer react@"17.0.2" from react-dom@17.0.2
  node_modules/react-dom
    peer react-dom@"^16.8.0  || ^17.0.0" from @react-spring/web@9.2.0
    node_modules/@react-spring/web
      @react-spring/web@"9.2.0" from @nivo/annotations@0.72.0
      node_modules/@nivo/annotations
        @nivo/annotations@"0.72.0" from @nivo/scatterplot@0.72.0
        node_modules/@nivo/scatterplot
          @nivo/scatterplot@"^0.72.0" from repro-ws@0.0.0
          ws
            repro-ws@0.0.0
            node_modules/repro-ws
              workspace ws from the root project

@react-spring/web depends on react-dom@16 || 17. Ok, so it picks react-dom@17.0.2 as the latest and greatest resolution, and react@17.0.2 along with it.

Then, it encounters a module that needs react@16. But, since the react/react-dom peer set cannot be nested (since everything along the chain has a peer dep on it), and the thing depending on react@16 doesn't include any version of react-dom to replace 17.0.2, it's an ERESOLVE. It correctly determines that the collision is coming from something deep in the package tree, so reduces this to a warning by default. (Unless you have --strict-peer-dependencies enabled, in which case it crashes. Still better than infinitely looping! But not great.)

In general dependency-graph terms, you have something like this:

k -> PEER(y@1 || y@2), PEER(z@1 || z@2)
z@1 -> PEER(y@1)
z@2 -> PEER(y@2)
v -> PEER(y@1)
root -> lib
lib -> (k, v) PEER(y)

When resolving the tree, we try to place k first, taking the latest and greatest versions allowed, and get here:

root
+-- lib
+-- k (could nest k, but doesn't help us)
+-- y@2 (cannot nest under lib, because lib has a peer dep on some version of y)
+-- z@2

Then we try to place v (and its peer dependency on y@1), and find that we can't. We can't nest the (k, y, z) peer set, because of lib's peer dependency on y, and we also can't nest the (v, y) peer set under lib for the same reason.

The solution (obvious to meatbrains, because we're pretty great at NP-hard problems) is to roll back y@1 and z@1. However, in general terms, this means that we have to walk the dependency graph back indefinitely to find the point of divergence. If it was not 1 path into the conflict, but rather dozens or hundreds, that could be quite a lot of CPU spinning, with no guarantee of success. Pubgrub is the gold standard here, and it's where Arborist is heading, but it's still some work to get there.

the workaround

The workaround is that if you add "react-dom": "16" to your root package.json file, it'll work fine in both the current and future versions of npm, avoid the infinite loop and produce a strictly valid tree.

[raut-rahul]

Surprisingly I was able to successfully install packages using npm version 7.19.1. But as soon as I upgraded npm version to 7.24.0 it again stuck in infinite loop.

[lukekarrys]

This specific instance of npm install getting stuck in an infinite loop was fixed by 97cb5ec in https://github.com/npm/cli/releases/tag/v7.20.3. I was able to confirm with the original https://github.com/targos/npm-repro-infinite-loop.git repo, so I'm going to close this as fixed.

@raut-rahul If you're having a similar problem, you can open a new issue. It's possible this is still happening in other cases (like #3787) but it's easier to debug in a new issue with a reproducible test case.