Poetry on fresh install thinks virtual env is in "/lsiopy", despite "poetry config virtualenvs.in-project true" that should force .venv in project folder

[ManiMatter]

Description

I am desperately trying to get poetry to run, and have create a installation script (setup.sh, below) that is supposed to install everything on a clean slate.

However, despite the setting to create the virtual environment in the project folder ("Test/.venv"), for some reason the virtual env keeps pointing to a folder called "/lsiopy", for which poetry complains that the Virtual Env is not valid (output below).

I've tried cleaning caches, reinstalling everything a hundred times, but the result is the same.
I think I am staring at something very obvious - any help is appreciated.

Thank you.

#!/bin/bash

## To run:
# chmod +x setup.sh
# ./setup.sh

# Install necessary dependencies
sudo apt-get update 
sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa # Contains newer historic versions of python

# Update package lists
sudo apt-get update 
sudo apt-get install -y \
    python3.10 \
    python3-pip \
    git \
    pre-commit
sudo apt auto-remove -y

# Create a python alias pointing to python3.10 
sudo rm /usr/bin/python
sudo ln -s $(which python3.10) /usr/bin/python

# # # Installing Poetry
rm -rf /config/.cache/pypoetry
rm -rf ~/.config/pypoetry
curl -sSL https://install.python-poetry.org | python3 - --uninstall 
curl -sSL https://install.python-poetry.org | python3.10

# # # Ensure Poetry creates virtual environments in the project directory
poetry config virtualenvs.in-project true

# # # Show the virtual env
poetry env info

Virtualenv
Python: 3.10.16
Implementation: CPython
Path: /lsiopy
Executable: python
> Valid: False

Base
Platform: linux
OS: posix
Python: 3.10.16
Path: /usr
Executable: /usr/bin/python3.10

Workarounds

none

Poetry Installation Method

curl -sSL https://install.python-poetry.org | python3.10

Operating System

Ubuntu 24.04.1 LTS

Poetry Version

1.8.5

Poetry Configuration

cache-dir = "/config/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /config/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

Bash Session

Example pyproject.toml

TOML

Poetry Runtime Logs

I am adding the poetry install logs, which show at the very beginning the output that the venv /lsiopy is broken (which is expected, since /lsiopy should not be used in the first place, but the .venv inside the cloudbackup folder)

Bash Session

[dimbleby]

Using virtualenv: /docker/own_coding/cloudbackup/.venv

wasn't that what you wanted all along? what's the problem?

[ManiMatter]

Using virtualenv: /docker/own_coding/cloudbackup/.venv

wasn't that what you wanted all along? what's the problem?

Thanks for the fast reply.

Yes, in theory that is what I want.
But after installing to docker/own_coding/cloudbackup/.venv, poetry will still complain on poetry env info that the path is /lsiopy, and I am back to square one… poetry install eill then again attempt to recreate the venv, and afterwards the path is again /lsiopy…..

[dimbleby]

well it's your computer, I expect you have a broken virtual environment in /lsiopy. Fix or delete it?

[ManiMatter]

I tried that before - the folder is long gone. I even tried to manually recreate it, to see if then poetry install works, same result.

The following log shows how the folder is /lsiopy; poetry install finds that this directory is not working and installs to .venv (as desired), but the log at the end (poetry show) lists all packages red (e.g. not installed).

Step 1: Removal of broken /lsiopy
abc@eefc547deed6:/docker/own_coding/cloudbackup$ sudo rm -rf /lsiopy

Step 2: Check if /lsiopy still exists
abc@eefc547deed6:/docker/own_coding/cloudbackup$ test -d /lsiopy && echo "Directory exists" || echo "Directory does not exist"

Directory does not exist

Step 3: Make sure the venvs are put into the project folder
abc@eefc547deed6:/docker/own_coding/cloudbackup$ poetry config virtualenvs.in-project true

Step 4: Checking teh .env path -> should be the project folder but it isn't
abc@eefc547deed6:/docker/own_coding/cloudbackup$ poetry env info --path

/lsiopy

Step 5: Trying an install nonetheless. It finds /lsiopy to be broken and correctly installs to .venv
abc@eefc547deed6:/docker/own_coding/cloudbackup$ poetry install

The virtual environment found in /lsiopy seems to be broken.
Recreating virtualenv cloudbackup in /docker/own_coding/cloudbackup/.venv
Installing dependencies from lock file

Package operations: 19 installs, 0 updates, 0 removals

• Installing typing-extensions (4.12.2)
• Installing astroid (3.3.8)
• Installing etc. etc.

Step 6: Verifying the venv path. Latest with above installation it should be .venv, but it is still /lsiopy
abc@eefc547deed6:/docker/own_coding/cloudbackup$ poetry env info --path

/lsiopy

Step 7: Checking if poetry shell at least has the packages. It argues lsiopy is not valid, and recreates .venv, thereby killing the installs from above
abc@eefc547deed6:/docker/own_coding/cloudbackup$ poetry shell

The virtual environment found in /lsiopy seems to be broken.
Recreating virtualenv cloudbackup in /docker/own_coding/cloudbackup/.venv
Spawning shell within /docker/own_coding/cloudbackup/.venv

Step 8: Checking the packages - they are all not installed.
(cloudbackup-py3.10) abc@eefc547deed6:/docker/own_coding/cloudbackup$ poetry show --no-ansi

astroid (!) 3.3.8 An abstract syntax tree for Python with inference support.
autoflake (!) 2.3.1 Removes unused imports and unused variables
black (!) 24.10.0 The uncompromising code formatter.
etc. etc.

[dimbleby]

poetry isn't going to be pulling "lsiopy" out of thin air, there is something on your computer pointing at it or using it or whatever.

looks like you're on docker, suggest sharing a minimal dockerfile that reproduces this. There's probably a pretty good chance that in the process of trying to construct that repro you will understand what is going on.

[ManiMatter]

I am using the image: lscr.io/linuxserver/code-server:latest

I see in the code of above image that they use the /lsiopy as virtual python environment.

When I use their image, at the beginning there doesn't seem to be python installed (hence I install it with the script I shared further above). Looks to me that they use python in their build process, but then don't include it to keep the package lean.

What to me is unclear is why poetry looks for venvs (and finds /lsiopy by doing so) in the first place. Since I turned on the below setting, I would have expected it to simply consider the project folder, and nothing else.
poetry config virtualenvs.in-project true

Any ideas?

Update:
I see that the lscr.io image comes with a ENV variable called 'VIRTUAL_ENV': '/lsiopy'.
Looking at the poetry code, I see that you use the same env variable as well.

My take would be that poetry gives priority to the VIRTUAL_ENV variable when chosing which venv to use, and thus in my case - given the VIRTUAL_ENV is present - poetry ignores the .venv present in the project folder, eventhough the setting poetry config virtualenvs.in-project true.

Is my understanding correct? If yes, is that desired behavior (or potentially a bug)?

Looking at the changelog sentence below, my reading would have been that the VIRTUAL_ENV variable should have no effect in my case, since it is not an activated environment (actually, the /lsiopy is an empty folder).

poetry/CHANGELOG.md

Line 1681 in d9f1914

- Fixed venv detection so that it only uses `VIRTUAL_ENV` to detect activated virtualenvs.

If the current implementation is as it should be, is there a way to force poetry to use the .venv in the project folder, eventhough a VIRTUAL_ENV variable is present? (Update 2: Isn't that exactly what this PR is about? #9902)

[dimbleby]

If there is a bad VIRTUAL_ENV variable then I would say all bets are off. In that case you will likely want to unset it, and raise a bug against the source of that container.

[ManiMatter]

If there is a bad VIRTUAL_ENV variable then I would say all bets are off. In that case you will likely want to unset it, and raise a bug against the source of that container.

Already done :)
linuxserver/docker-code-server#189

I am still surprised though that the setting poetry config virtualenvs.in-project true does not override the VIRTUAL_ENV. Hope the PR above gives the possibility to enforce that.

[ManiMatter]

If there is a bad VIRTUAL_ENV variable then I would say all bets are off. In that case you will likely want to unset it, and raise a bug against the source of that container.

After pondering more about it, I think we might actually have a bug here in poetry. Let me try to expain.

Firstly, the changelog above says…

Fixed venv detection so that it only uses VIRTUAL_ENV to detect activated virtualenvs.

… but that doesn‘t seem to be the case. While in my container said VIRTUAL_ENV is present, the virtual env is not activated. Nonetheless, poetry tries to start in that dir, contradicting the changelog.

Of course the changelog entry dates back sometime, and the statement is not accurate anymore.

Second reason why I believe there is a bug, is because poetry acts inconsistently.

It sees that the VIRTUAL_ENV is set, and then immediately argues that venv is invalid (and does not seem to activate, and hence fails to install).

Then it tries to local project folder (due to virtualenvs.in-project true) and here recreates it (despite being a perfectly valid venv) and then activates it.

So the inconsistency is that the VIRTUAL_ENV it seeems not to try to activate/recreate, while the own it does.

I am not that bothered about the second discrepancy; but if the expected behavior -as the changelog suggests- is to only consider active VIRTUAL_ENV then I think this is a bug since in my case the env is not active and thus VIRTUAL_ENV should be entirely disregarded

[dimbleby]

the virtual env is not activated

the virtual env is activated, that is what it means to have a VIRTUAL_ENV environment variable.

[ManiMatter]

I am trying to make sense of this statement:

We create that venv folder and set the env var in the baseimage because pip no longer uses system folder to install things into.

All of our images that utilize python activate that venv.

You can either override that env var, or choose to use that path for the venv. For the latter, you would have to activate the venv at that path (the mod may have to do that as well).

I am not activating that venv. Sure, they set the venv variable to the /lsiopy dir, but according to the last sentence, unless I activate it myself, it should not be active.

Doesn‘t this contradict with your statement that the env variable itself indicates that the venv is active?

Is there a way to verify if the venv is actually active (eg to verify that the env variable is a reliable indicator)? My reading from above would be that they create the env variable, but its not a representation of whether the venv is active or not

[dimbleby]

the only thing that activating a virtual environment does is to set that variable and tweak the path. A virtual environment is active exactly when VIRTUAL_ENV says so: the meaning of that variable is "this is the active virtual environment".

[ManiMatter]

Thank you- i‘m following up with the code server team. Appreciate your help, expertise and patience with this issue (and me 🙂)

[ManiMatter]

They dont recommend installing software in their containers, which I was doing.

rather, use their mods.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.