Environment variables in Pipfile are not being expanded when executing pipenv lock

[ptzianos]

We want to use a private git repo as a dependency for a project. We added the following line in our Pipfile:
"private" = {git = "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.private.com/path/to/repo.git", editable = true, ref = "v0.0.4"}
When running pipenv install everything works fine but when running pipenv lock it fails because the environment variable is not being expanded.

I think this is a side effect of this PR: #2141

$ python -m pipenv.help output

Pipenv version: '2018.05.18'

Pipenv location: '/home/pavlos/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 2.7.15: /usr/bin/python

• 2.7.15: /usr/bin/python2

• 3.6.5: /usr/bin/python3

PEP 508 Information:

`
{'implementation_name': 'cpython',
'implementation_version': '3.6.5',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '4.15.0-22-generic',
'platform_system': 'Linux',
'platform_version': '#24-Ubuntu SMP Wed May 16 12:15:17 UTC 2018',
'python_full_version': '3.6.5',
'python_version': '3.6',
'sys_platform': 'linux'}

System environment variables:

  - `GS_LIB`
  - `KDE_FULL_SESSION`
  - `CI_JOB_TOKEN`
  - `LS_COLORS`
  - `LC_MEASUREMENT`
  - `LESSCLOSE`
  - `LC_PAPER`
  - `LC_MONETARY`
  - `LANG`
  - `PROFILEHOME`
  - `DISPLAY`
  - `OLDPWD`
  - `SHELL_SESSION_ID`
  - `COLORTERM`
  - `PYENV_VIRTUALENV_INIT`
  - `XDG_VTNR`
  - `PAM_KWALLET5_LOGIN`
  - `SSH_AUTH_SOCK`
  - `MANDATORY_PATH`
  - `LC_NAME`
  - `XDG_SESSION_ID`
  - `USER`
  - `DESKTOP_SESSION`
  - `DEFAULTS_PATH`
  - `PWD`
  - `HOME`
  - `SSH_AGENT_PID`
  - `QT_ACCESSIBILITY`
  - `XDG_SESSION_TYPE`
  - `XDG_DATA_DIRS`
  - `KONSOLE_DBUS_SESSION`
  - `XDG_SESSION_DESKTOP`
  - `LC_ADDRESS`
  - `LC_NUMERIC`
  - `KONSOLE_DBUS_WINDOW`
  - `TERM`
  - `SHELL`
  - `KONSOLE_DBUS_SERVICE`
  - `XDG_SESSION_CLASS`
  - `XDG_SEAT_PATH`
  - `XCURSOR_THEME`
  - `XDG_CURRENT_DESKTOP`
  - `GPG_AGENT_INFO`
  - `KONSOLE_PROFILE_NAME`
  - `PYENV_SHELL`
  - `XDG_SEAT`
  - `SHLVL`
  - `COLORFGBG`
  - `LANGUAGE`
  - `WINDOWID`
  - `LC_TELEPHONE`
  - `LOGNAME`
  - `DBUS_SESSION_BUS_ADDRESS`
  - `XDG_RUNTIME_DIR`
  - `PAM_KWALLET_LOGIN`
  - `XAUTHORITY`
  - `XDG_SESSION_PATH`
  - `QT_AUTO_SCREEN_SCALE_FACTOR`
  - `XDG_CONFIG_DIRS`
  - `PATH`
  - `LC_IDENTIFICATION`
  - `KDE_SESSION_UID`
  - `KDE_SESSION_VERSION`
  - `SESSION_MANAGER`
  - `LESSOPEN`
  - `LC_TIME`
  - `_`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_PYTHON_PATH`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/home/pavlos/.local/bin:/home/pavlos/.pyenv/plugins/pyenv-virtualenv/shims:/home/pavlos/.pyenv/shims:/home/pavlos/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/pavlos/.cargo/bin:/home/pavlos/.pyenv/bin`
  - `SHELL`: `/bin/bash`
  - `LANG`: `en_US.UTF-8`
  - `PWD`: `/home/pavlos/project`


---------------------------

Contents of `Pipfile` ('/home/pavlos/project/Pipfile'):

```toml
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
"private" = {git = "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.private.com/path/to/private.git", editable = true, ref = "v0.0.4"}

[dev-packages]
"flake8" = "*"

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/home/pavlos/project/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "0cb99a6489baaf74e36062af015490390b4a4f954a71939ba898a0fc666c7e40"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "private": {
            "editable": true,
            "git": "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.private.com/path/to/private.git",
            "ref": "v0.0.4"
        },
    },
    "develop": {
        "flake8": {
            "hashes": [
                "sha256:7253265f7abd8b313e3892944044a365e3f4ac3fcdcfb4298f55ee9ddf188ba0",
                "sha256:c7841163e2b576d435799169b78703ad6ac1bbb0f199994fc05f700b2a90ea37"
            ],
            "index": "pypi",
            "version": "==3.5.0"
        },
    }
}

---

Expected result

That pipenv lock works the same way as install, meaning that it will expand the environment variables and produce the Pipfile.lock.

Actual result

Locking [dev-packages] dependencies…
Locking [packages] dependencies…
ne 202, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/home/pavlos/.local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 256, in get_legacy_dependencies
    result = reqset._prepare_file(self.finder, ireq, ignore_requires_python=True)
  File "/home/pavlos/.local/lib/python3.6/site-packages/pipenv/patched/notpip/req/req_set.py", line 526, in _prepare_file
    req_to_install.update_editable(not self.is_download)
  File "/home/pavlos/.local/lib/python3.6/site-packages/pipenv/vendor/pip9/req/req_install.py", line 586, in update_editable
    vcs_backend.obtain(self.source_dir)
  File "/home/pavlos/.local/lib/python3.6/site-packages/pipenv/vendor/pip9/vcs/git.py", line 142, in obtain
    self.run_command(['clone', '-q', url, dest])
  File "/home/pavlos/.local/lib/python3.6/site-packages/pipenv/vendor/pip9/vcs/__init__.py", line 325, in run_command
    spinner)
  File "/home/pavlos/.local/lib/python3.6/site-packages/pipenv/vendor/pip9/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip9.exceptions.InstallationError: Command "git clone -q https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.private.com/path/to/private.git /tmp/tmp3ffjjgu1source/private" failed with error code 128 in None

Steps to replicate

Add a private repo with an environment variable in the url for the access token and then run pipenv install and pipenv lock. The first command will run just fine while the second will fail with a message like the one above.

[uranusjr]

Same as #2317. Please help contribute to the documentation so people don’t step on this!

[astrojuanlu]

The alternative is to use a properly configured .netrc: https://docs.gitlab.com/ce/user/project/new_ci_build_permissions_model.html#dependent-repositories