Install script does not alway use the right venv path (IDFGH-8471)

[fossum]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

release/v4.3-9e863ec

Operating System used.

Linux

How did you build your project?

Other (please specify in More Information)

If you are using Windows, please specify command line type.

No response

What is the expected behavior?

I wanted to get the tools given the documentation.
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/linux-macos-setup.html#step-3-set-up-the-tools

What is the actual behavior?

Instead it crashes after installing the venv. The path it's looking for doesn't exist, but python does exist in another folder.

Error:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ericfoss/.espressif/python_env/idf4.3_py3.10_env/bin/python'

File Listings:
➜ esp-idf git:(release/v4.3) ✗ ls '/home/ericfoss/.espressif/python_env/idf4.3_py3.10_env'
lib local pyvenv.cfg
➜ esp-idf git:(release/v5.0) ls '/home/ericfoss/.espressif/python_env/idf4.3_py3.10_env/local/bin'
activate activate.fish activate.ps1 deactivate.nu pip3 pip3.10 python3 wheel wheel-3.10
activate.csh activate.nu activate_this.py pip pip-3.10 python python3.10 wheel3 wheel3.10

Steps to reproduce.

1. On a fresh Ubuntu system, do steps 1 and 2 of the linked documentation.
2. Do the first part of the linked set up the tools documentation.

Build or installation Logs.

➜  esp cd ~/esp/esp-idf
./install.sh esp32
Detecting the Python interpreter
Checking "python" ...
/home/ericfoss/esp/esp-idf/tools/detect_python.sh: line 16: python: command not found
Checking "python3" ...
Python 3.10.4
"python3" has been detected
Installing ESP-IDF tools
WARNING: File /home/ericfoss/.espressif/idf-env.json was not found. 
Creating /home/ericfoss/.espressif/idf-env.json
Selected targets are: esp32
Installing tools: xtensa-esp32-elf, esp32ulp-elf, openocd-esp32
Skipping xtensa-esp32-elf@esp-2021r2-8.4.0 (already installed)
Skipping esp32ulp-elf@2.28.51-esp-20191205 (already installed)
Skipping openocd-esp32@v0.10.0-esp32-20211111 (already installed)
Installing Python environment and packages
Creating a new Python environment in /home/ericfoss/.espressif/python_env/idf4.3_py3.10_env
created virtual environment CPython3.10.4.final.0-64 in 1773ms
  creator CPython3Posix(dest=/home/ericfoss/.espressif/python_env/idf4.3_py3.10_env, clear=False, no_vcs_ignore=False, global=False)
  seeder PipInvoke(download=False, pip=bundle, setuptools=bundle, wheel=bundle)
    added seed packages: 
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Installing Python packages from /home/ericfoss/esp/esp-idf/requirements.txt
Traceback (most recent call last):
  File "/home/ericfoss/esp/esp-idf/tools/idf_tools.py", line 1822, in <module>
    main(sys.argv[1:])
  File "/home/ericfoss/esp/esp-idf/tools/idf_tools.py", line 1818, in main
    action_func(args)
  File "/home/ericfoss/esp/esp-idf/tools/idf_tools.py", line 1539, in action_install_python_env
    subprocess.check_call(run_args, stdout=sys.stdout, stderr=sys.stderr, env=env_copy)
  File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 345, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/ericfoss/.espressif/python_env/idf4.3_py3.10_env/bin/python'

More Information.

Seems to be fixed in v5 but v4.3 is the cloned branch, so it seems like this should be fixed. v4.4 might need to be tested as well.

[igrr]

Hi @fossum,
just to help us understand the cause of this issue better, could you please tell what is the output of

python3 -m sysconfig | grep installation

idf_tools.py has an assumption that virtual environments' binaries are in <venv_path>/bin/, however this seems to be not the case based on your log.

This might be related to this recently fixed issue in virtualenv: pypa/virtualenv#2350 (comment).

[fossum]

Sure I can do that, but it'll be Monday when I'm back at work.

…

On Thu, Oct 6, 2022, 14:02 Ivan Grokhotkov ***@***.***> wrote: Hi @fossum <https://github.com/fossum>, just to help us understand the cause of this issue better, could you please tell what is the output of python3 -m sysconfig | grep installation idf_tools.py has an assumption that virtual environments' binaries are in <venv_path>/bin/, however this seems to be not the case based on your log. This might be related to this recently fixed issue in virtualenv: pypa/virtualenv#2350 (comment) <pypa/virtualenv#2350 (comment)>. — Reply to this email directly, view it on GitHub <#9931 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADNQRV5VH7JGVIFJFTB443WB446TANCNFSM6AAAAAAQ66N3AI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fossum]

Some more output if it's helpful.

➜  temp virtualenv .
created virtual environment CPython3.10.4.final.0-64 in 125ms
  creator CPython3Posix(dest=/home/ericfoss/temp, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ericfoss/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.3.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator                                                                                   /0.4s
➜  temp ls       
lib  local  pyvenv.cfg                                                                                                                                                                          /0.0s
➜  temp ls local 
bin  lib                                                                                                                                                                                        /0.0s
➜  temp ls local/bin 
activate  activate.csh  activate.fish  activate.nu  activate.ps1  activate_this.py  deactivate.nu  pip  pip3  pip-3.10  pip3.10  python  python3  python3.10  wheel  wheel3  wheel-3.10  wheel3 /0.0s
➜  temp 

[fossum]

You may be on the same path already, but I found this SO question. Which says there will be a solution, but I'm on the latest virtualenv (virtualenv==20.16.5) as of now.

[dobairoland]

I think that probably detecting a the right path would be preferable over limiting the version of virtualenv or setuptools.

@fossum Could you please try to manually change the code here and replace

subdir = 'bin'

with

subdir = 'local/bin'

Is this enough for you to fix the issue? Will everything work as expected (build, flash, not just install)?

[fossum]

Yes, that is all I did to fix the issue.

…

On Tue, Oct 11, 2022, 07:19 Roland Dobai ***@***.***> wrote: I think that probably detecting a the right path would be preferable over limiting the version of virtualenv or setuptools. @fossum <https://github.com/fossum> Could you please try to manually change the code here <https://github.com/espressif/esp-idf/blob/release/v4.3/tools/idf_tools.py#L1018> and replace subdir = 'bin' with subdir = 'local/bin' Is this enough for you fix the issue? Will everything work as expected (build, flash, not just install)? — Reply to this email directly, view it on GitHub <#9931 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADNQRQIB3RRTV7SPZ7ESHDWCVZPJANCNFSM6AAAAAAQ66N3AI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dobairoland]

@fossum Thank you for your report and help tracking this down.

A workaround has been merged to the release branches (v4.4 and earlier).