Brew-installed Poetry can't use pyenv-managed Python versions

[maxcountryman]

• ran brew update and can still reproduce the problem?
• ran brew doctor, fixed all issues and can still reproduce the problem?
• ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)

Hi there, I've been mucking around with setting up a consistent Python environment for my team. We use Poetry to manage our Python project dependencies. While a nice improvement over historical alternatives, it's hardly been a silver bullet so it's required some extra work to get things working smoothly across local environments. Specifically, in order to keep the project on a locked version of Python, we've paired Poetry with pyenv.

This has generally worked, but recently we've switch from installing Poetry via the standard cURL method, to using Homebrew. Why brew? Well once again it's hard to keep everyone's environments aligned and specifically the brew bundle facility allows us to ship a Brewfile with our projects thus keeping everyone on a consistent set of tools. Problem solved, right?

Well, not exactly. In theory we should be able to use the Brewfile to provide Poetry, pyenv, and other system-level dependencies. While it does in fact provide those things, it seems that by moving to installing Poetry with brew we're no longer able to point Poetry at Python versions managed by pyenv.

What happened (include command output)

For example, let's say I want to use Python 3.6 and I install that with pyenv and then set my local project to 3.6.10:

pyenv install 3.6.10
$ pyenv local 3.6.10 
$ python --version
Python 3.6.10

Now, I've already configured my pyproject.toml to require this specific version of Python and I'll also instruct Poetry to set up the virtualenv to use it:

$ poetry env use python
Recreating virtualenv {project} in /Users/max/Documents/projects/{project}/.venv
Using virtualenv: /Users/max/Documents/projects/{project}/.venv

Unfortunately this didn't work as intended:

$ poetry run python --version
Python 3.7.7

What you expected to happen

I'd hope that we could still manage Python versions with pyenv after installing Poetry with brew. It seems we aren't able to (see above). Cursory investigation seems to reveal that the Poetry bin installed by brew is hardcoding Python:

$ cat (which poetry)
#!/usr/local/opt/python/bin/python3
import sys

sys.path.insert(0, "/usr/local/Cellar/poetry/1.0.5_1/libexec/lib/python3.7/site-packages")
sys.path.insert(0, "/usr/local/Cellar/poetry/1.0.5_1/libexec/vendor/lib/python3.7/site-packages")

if __name__ == "__main__":
    from poetry.console import main
    main()

This doesn't necessarily seem like the intended behavior so I figured it probably warrants an issue.

Output of brew config and brew doctor commands

$ brew config
HOMEBREW_VERSION: 2.2.13
ORIGIN: https://github.com/Homebrew/brew
HEAD: 3d9cf83fec45a75af61551f53d25383abe009d31
Last commit: 34 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 7c0e823b76a2500c93d1d1dd394ce856963c7a75
Core tap last commit: 3 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_EDITOR: nvim
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 8
CPU: octa-core 64-bit kabylake
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
Clang: 11.0 build 1103
Git: 2.26.0 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
Java: 9.0.1, 1.8.0_152
macOS: 10.15.4-x86_64
CLT: 11.4.0.0.1.1583798712
Xcode: 11.4

$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and which additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew-provided
script of the same name. We found the following "config" scripts:
  /Users/max/.pyenv/shims/python3.6m-config
  /Users/max/.pyenv/shims/python-config
  /Users/max/.pyenv/shims/python3-config
  /Users/max/.pyenv/shims/python3.6-config

[MikeMcQuaid]

Are you asking for a Homebrew-installed application to be able to use a Python of your choice rather than Homebrew's Python?

[maxcountryman]

Are you asking for a Homebrew-installed application to be able to use a Python of your choice rather than Homebrew's Python?

Correct.

This is seemingly a pretty fundamental feature because Poetry manages the underlying virtualenv your project's dependencies are installed into. Without being able to control the precise version of Python that Poetry uses (or minimally, installs into the virtualenv) it's very difficult to use Poetry. (The exception of course is if the version of Python installed by Homebrew exactly matches what you'd like your project to run on).

One way around this could be use to Homebrew to install Python. This would be fine for us I think, but it seems there isn't much control over which version of Python Homebrew installs (I see python and python@3.8 but it isn't clear to me how I could install e.g. 3.6 or a more specific point release--I may very well just be ignorant of how to do this, so please correct me if you know of a better way).

From my perspective, the most natural tool for Python version management in the contemporary toolchain is pyenv so it does seem like it could be nice if Homebrew-installed Poetry could be made capable of utilizing that.

[MikeMcQuaid]

Sorry, it's expected behaviour that Homebrew-installed tools use the Homebrew-installed dependencies when run. It sounds like you may be best to install Poetry outside of Homebrew. Thanks!

[maxcountryman]

Sorry, it's expected behaviour that Homebrew-installed tools use the Homebrew-installed dependencies when run. It sounds like you may be best to install Poetry outside of Homebrew. Thanks!

Hm but this fundamentally limits Poetry.

Wouldn’t it be reasonable to expect that brew-installed pyenv could interact with brew-installed Poetry? That’s simply not the case today...

[MikeMcQuaid]

Hm but this fundamentally limits Poetry.

Perhaps but you can install it with pip in any Python or virtualenv of your choice.

Wouldn’t it be reasonable to expect that brew-installed pyenv could interact with brew-installed Poetry? That’s simply not the case today...

Whether it's reasonable or not: it's not what we currently do or plan to change, sorry.

[maxcountryman]

Perhaps but you can install it with pip in any Python or virtualenv of your choice.

I don't think this is a workable solution for our use-case.

We might be able to workaround this by using brew to install a specific version of Python, but then this won't actually work when we have projects with conflicting Python versions (we currently are in this situation).

Whether it's reasonable or not: it's not what we currently do or plan to change, sorry.

What's the point of installing Poetry in this way then?

I'd love to better understand this rationale. This takes a tool which relies heavily on the specific Python version you're running, locks that down in a way that isn't configurable (or seemingly barely so), and then doesn't provide any way to interact with the standard toolset (e.g. pyenv) for managing Python versions.

Why not provide an option to install Poetry configured to use pyenv as part of the build environment?

[Bo98]

The problem comes with Poetry's dependencies. You can already override the Python used to execute Poetry (as with any shebang), but you would have to be using the exact same Python as Poetry was configured to use (3.7). Python 3.8 in your pyenv would likely not work well with Homebrew Poetry. Homebrew will change Poetry to use 3.8 in the future without notice.

[MikeMcQuaid]

What's the point of installing Poetry in this way then?

The thousands of other users who are using it in the last few months would be able to tell you.

@Bo98 has provided more explanation: I think we've hit the limit of productive discussion: this is the Homebrew issue tracker and this is not a bug.