Work around Pipenv bug when using --system (#1842)

After moving Pipenv to its own virtual environment in #1840 (to fix
leaking Pipenv's dependencies into the app environment), apps that
explicitly depend on a package that happen to also be a Pipenv
dependency were seeing `ModuleNotFoundError` failures.

For example:

```
ModuleNotFoundError: No module named 'certifi'
```

Or:

```
ModuleNotFoundError: No module named 'packaging'
```

It appears that Pipenv's `--system` feature has a bug whereby it sees
its own dependencies (installed in its own environment) as already
installed (even though they aren't actually installed in the app
environment), so incorrectly skips installing them.

In general, it seems Pipenv's support for `--system` is pretty buggy
(for example, it doesn't work with `pipenv run` either):
https://github.com/pypa/pipenv/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22--system%20flag%22

The classic Python buildpack doesn't currently use virtual environments
(since it installs a dedicated Python install for the app, that's
already separate from system Python), and switching to them would
be a significant change that may cause other types of breakage
(for example, to apps that hardcode the path to Python).

As such for now, we need to continue to use `pipenv install --system`
(as the buildpack has done for some time), but I have found the bug
can be worked around if we set `VIRTUAL_ENV` to the Python install
location, to trick Pipenv into thinking it's operating in a virtual
environment.

The Pipenv tests have been updated to confirm that an explicit
`certifi` install now works.

GUS-W-19118553.

Author: edmorley

CHANGELOG.md

@@ -2,6 +2,7 @@
 
 ## [Unreleased]
 
+- Work around a Pipenv bug when using `--system`, that causes packages to not be installed correctly if they are also a dependency of Pipenv (such as `certifi` ). ([#1842](https://github.com/heroku/heroku-buildpack-python/pull/1842))
 
 ## [v292] - 2025-07-23
 

lib/pipenv.sh

@@ -93,17 +93,29 @@ function pipenv::install_pipenv() {
 	export PATH="${pipenv_bin_dir}:${PATH}"
 	# Force Pipenv to manage the system Python site-packages instead of using venvs.
 	export PIPENV_SYSTEM="1"
+	# Hide Pipenv's notice about finding/using an existing virtual environment.
+	export PIPENV_VERBOSITY="-1"
+	# Work around a Pipenv bug when using `--system`, whereby it doesn't correctly install
+	# dependencies that happen to also be a dependency of Pipenv (such as `certifi`).
+	# In general Pipenv's support for its `--system` mode seems very buggy. Longer term we
+	# should explore moving to venvs, however, that will need to be coordinated across all
+	# package managers and also change paths for Python which could break other use cases.
+	export VIRTUAL_ENV="${python_home}"
 
 	# Set the same env vars in the environment used by later buildpacks.
 	cat >>"${export_file}" <<-EOF
 		export PATH="${pipenv_bin_dir}:\${PATH}"
 		export PIPENV_SYSTEM="1"
+		export PIPENV_VERBOSITY="-1"
+		export VIRTUAL_ENV="${python_home}"
 	EOF
 
 	# And the environment used at app run-time.
 	cat >>"${profile_d_file}" <<-EOF
 		export PATH="${pipenv_bin_dir}:\${PATH}"
 		export PIPENV_SYSTEM="1"
+		export PIPENV_VERBOSITY="-1"
+		export VIRTUAL_ENV="${python_home}"
 	EOF
 }
 

spec/fixtures/pipenv_basic/Pipfile

@@ -4,6 +4,7 @@ verify_ssl = true
 name = "pypi"
 
 [packages]
+certifi = "*"
 typing-extensions = "*"
 
 [dev-packages]

spec/fixtures/pipenv_basic/Pipfile.lock

@@ -104,18 +104,22 @@
                  LIBRARY_PATH=/app/.heroku/python/lib
                  PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/
                  PIPENV_SYSTEM=1
+                 PIPENV_VERBOSITY=-1
                  PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config
                  PYTHONUNBUFFERED=1
+                 VIRTUAL_ENV=/app/.heroku/python
           -----> Inline app detected
           LANG=en_US.UTF-8
           LD_LIBRARY_PATH=/app/.heroku/python/lib
           LIBRARY_PATH=/app/.heroku/python/lib
           PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/
           PIPENV_SYSTEM=1
+          PIPENV_VERBOSITY=-1
           PYTHONHASHSEED=random
           PYTHONHOME=/app/.heroku/python
           PYTHONPATH=/app
           PYTHONUNBUFFERED=true
+          VIRTUAL_ENV=/app/.heroku/python
           -----> No test-setup command provided. Skipping.
           -----> Running test command `./bin/print-env-vars.sh && pytest --version`...
           CI=true
@@ -127,10 +131,12 @@
           LIBRARY_PATH=/app/.heroku/python/lib
           PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/
           PIPENV_SYSTEM=1
+          PIPENV_VERBOSITY=-1
           PYTHONHASHSEED=random
           PYTHONHOME=/app/.heroku/python
           PYTHONPATH=/app
           PYTHONUNBUFFERED=true
+          VIRTUAL_ENV=/app/.heroku/python
           WEB_CONCURRENCY=5
           pytest .+
           -----> test command `./bin/print-env-vars.sh && pytest --version` completed successfully

spec/hatchet/ci_spec.rb

@@ -22,10 +22,12 @@
           remote: LIBRARY_PATH=/app/.heroku/python/lib
           remote: PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
           remote: PIPENV_SYSTEM=1
+          remote: PIPENV_VERBOSITY=-1
           remote: PYTHONHASHSEED=random
           remote: PYTHONHOME=/app/.heroku/python
           remote: PYTHONPATH=/app
           remote: PYTHONUNBUFFERED=true
+          remote: VIRTUAL_ENV=/app/.heroku/python
           remote: 
           remote: \\['',
           remote:  '/app',
@@ -37,6 +39,7 @@
           remote: pipenv, version #{PIPENV_VERSION}
           remote: Package           Version
           remote: ----------------- -+
+          remote: certifi           2025.7.14
           remote: typing_extensions 4.12.2
           remote: 
           remote: <module 'typing_extensions' from '/app/.heroku/python/lib/python3.13/site-packages/typing_extensions.py'>