Poetry install and update resolve versions differently when requirements aren't pinned

[iserko]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian running in WSL2
• Poetry version: 1.1.0b2
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/iserko/ae0b5240bdb6af0a63264f834a3b4bee

Issue

Using the pyproject.toml I can run poetry install (log) and get the latest dbt installed (at the time that's 0.17.2), but then when I run poetry update (log) it downgrades dbt and updates requests.

The workaround for this is to pin dbt to dbt = "^0.17.2", however I was surprised to see poetry just downgrade.

[dimbleby]

In general I don't think there's any guarantee that locking will give the same answer if you do it twice.

If the latest versions of two packages - dbt and requests - are not compatible, then at most one of them can 'win', and which one it is will depend on the exact path that poetry takes through the search.

The promise that poetry makes is to find a solution where the packages are compatible with one another: not that it always finds the same solution.

If you care more about dbt being at the latest than you do about requests being at the latest, then telling poetry so (as you have done) is the right thing to do.

So while I agree that this behaviour is slightly surprising, I don't think it's a bug.

[TBBle]

I would say it's a bug because the path that poetry takes through a search shouldn't be different for the same inputs. Either whatever is randomising that order needs to stop being random, or (better) the solution-evaluation rule needs to be extended in what is I guess a tie-break case.

I'd also suggest that trivially, in a tie-break case, if one of the options is "What's already locked", then it should choose that one, using hysteresis to avoid flip-flopping. That would take care of all the existing bug reports about this behaviour, since none are "It's choosing wrong when this", they're all "It's choosing differently from what it chose 20 seconds ago". Then whatever randomness is being done now would stop actually being a problem except when one deletes the lockfile and relocks, in which case a different solution is less-surprising.

That might actually be a simple change, since the flip-flopping around poetry update suggests that it knows it's in a tie-break, but is choosing the option that isn't the currently-locked result.

You could also define more rules for the tie-break cases, but I suspect no one will really care that much which one it's choosing, as long as it's consistent.

[dimbleby]

I would say it's a bug ...

It still isn't, though ;-)

FWIW I can't reproduce this today even with old versions of poetry, presumably because the universe of packages that poetry has to choose from has changed in the mean time.

If anyone cares about this enough to dig into it: one possibility that comes to mind is the code in _choose_package_version which decides which package to try and resolve next. It does that by looking for the package with fewest remaining versions available, afaik in case of a tie there's no particular reason that one or the other should win. Perhaps you could construct a repro or unit test with that clue...

But IMO this remains at worst "slightly surprising" and not worth bothering about.

[JacobHayes]

There's a reproducible example in #6470, as of today at least.

[radoering]

From #6470:

I would expect the code paths to be equivalent and poetry update to basically be a fancy poetry lock && poetry install, though I acknowledge that's probably naive.

For those who are interested:

poetry lock ignores an existing lock file and creates a new lock file from scratch.
poetry update takes a whitelist of packages to update. That's why it's more like a poetry lock --no-update except for the whitelisted packages. It considers an existing lock file and tries not to update packages that are not on the whitelist.
If you don't pass any names to poetry update all locked packages are on the whitelist, but the lockfile is still taken into consideration which can result in a different resolution order.

I drafted #6477 to make poetry update more determistic and similar to poetry lock.

[dimbleby]

duplicate #3779 can also be closed

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.