Skip to content

Commit f6bbc99

Browse files
edmorleyedmorley
authored
Update Pipenv to 2025.0.4 and improve installation/caching (#1840)
* Updates Pipenv from 2024.0.1 to 2025.0.4. Changelog: https://github.com/pypa/pipenv/blob/main/CHANGELOG.md#202504-2025-07-07 * Switches to installing Pipenv into its own virtual environment, so that it and its dependencies don't leak into the app environment. * Stops installing pip (in addition to Pipenv) when the chosen package manager is Pipenv, to match the behaviour when using Poetry or uv. * Forces the build cache to be cleared if the contents of `Pipfile.lock` has changed since the last build. This is required to work around `pipenv {install,sync}` by design not uninstalling packages if they are removed from the lockfile (?!). We also can't use `pipenv clean` since it doesn't work with `--system` / `PIPENV_SYSTEM`. Apps that have been inadvertently depending on Pipenv's own dependencies implicitly, will need to add those dependencies explicitly to their `Pipfile` and regenerate `Pipfile.lock`. Similarly, any apps that for some reason use pip later in the build in addition to Pipenv, will need to either add pip as an explicit dependency in their `Pipfile`, or else for adhoc use-cases (such as in one-off dynos), can temporarily install pip using `python -m ensurepip --default-pip`. Supersedes #1828. GUS-W-17482289. GUS-W-17482412. GUS-W-19116903.
1 parent e5206cb commit f6bbc99

File tree

9 files changed

+164
-78
lines changed

9 files changed

+164
-78
lines changed

‎CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
## [Unreleased]
44

5+
- Updated Pipenv from 2024.0.1 to 2025.0.4. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840))
6+
- Fixed the way Pipenv is installed, so that it and its dependencies are installed into a separate virtual environment rather than same environment as the app. If your app inadvertently depended on Pipenv's internal dependencies, you will need to add those dependencies explicitly to your `Pipfile`. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840))
7+
- Stopped installing pip when Pipenv is the chosen package manager. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840))
8+
- The build cache is now cleared when using Pipenv if the contents of `Pipfile.lock` has changed since the last build. This is required to work around Pipenv not uninstalling packages when they are removed from the lockfile. ([#1840](https://github.com/heroku/heroku-buildpack-python/pull/1840))
59
- The build now errors when using Pipenv without its lockfile (`Pipfile.lock`). This replaces the warning displayed since November 2024. ([#1833](https://github.com/heroku/heroku-buildpack-python/pull/1833))
610

711
## [v291] - 2025-07-10

‎bin/compile

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -172,9 +172,7 @@ case "${package_manager}" in
172172
pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}"
173173
;;
174174
pipenv)
175-
# TODO: Stop installing pip when using Pipenv.
176-
pip::install_pip_setuptools_wheel "${python_home}" "${python_major_version}"
177-
pipenv::install_pipenv
175+
pipenv::install_pipenv "${python_home}" "${python_major_version}" "${EXPORT_PATH}" "${PROFILE_PATH}"
178176
;;
179177
poetry)
180178
poetry::install_poetry "${python_home}" "${python_major_version}" "${CACHE_DIR}" "${EXPORT_PATH}"

‎lib/cache.sh

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ function cache::restore() {
8787
# later removed, pip will not uninstall the package. This check can be removed if we
8888
# ever switch to only caching pip's HTTP/wheel cache rather than site-packages.
8989
# TODO: Remove the `-f` check once the setup.py fallback feature is removed.
90+
# TODO: Switch this to using sha256sum like the Pipenv implementation.
9091
if [[ -f "${build_dir}/requirements.txt" ]] && ! cmp --silent "${cache_dir}/.heroku/requirements.txt" "${build_dir}/requirements.txt"; then
9192
cache_invalidation_reasons+=("The contents of requirements.txt changed")
9293
fi
@@ -102,9 +103,14 @@ function cache::restore() {
102103
elif [[ "${cached_pipenv_version}" != "${PIPENV_VERSION:?}" ]]; then
103104
cache_invalidation_reasons+=("The Pipenv version has changed from ${cached_pipenv_version} to ${PIPENV_VERSION}")
104105
fi
105-
# TODO: Remove this next time the Pipenv version is bumped (since it will trigger cache invalidation of its own)
106-
if [[ -d "${cache_dir}/.heroku/src" ]]; then
107-
cache_invalidation_reasons+=("The editable VCS repository location has changed (and Pipenv doesn't handle this correctly)")
106+
# `pipenv {install,sync}` by design don't actually uninstall packages on their own (!!):
107+
# and we can't use `pipenv clean` since it isn't compatible with `--system`.
108+
# https://github.com/pypa/pipenv/issues/3365
109+
# We have to explicitly check for the presence of the Pipfile.lock.sha256 file,
110+
# since we only started writing it to the build cache as of buildpack v292+.
111+
local pipfile_lock_checksum_file="${cache_dir}/.heroku/python/Pipfile.lock.sha256"
112+
if [[ -f "${pipfile_lock_checksum_file}" ]] && ! sha256sum --check --strict --status "${pipfile_lock_checksum_file}"; then
113+
cache_invalidation_reasons+=("The contents of Pipfile.lock changed")
108114
fi
109115
;;
110116
poetry)
@@ -194,7 +200,12 @@ function cache::save() {
194200
# TODO: Simplify this once multiple package manager files being found is turned into an
195201
# error and the setup.py fallback feature is removed.
196202
if [[ "${package_manager}" == "pip" && -f "${build_dir}/requirements.txt" ]]; then
203+
# TODO: Switch this to using sha256sum like the Pipenv implementation.
197204
cp "${build_dir}/requirements.txt" "${cache_dir}/.heroku/"
205+
elif [[ "${package_manager}" == "pipenv" ]]; then
206+
# This must use a relative path for the lockfile, since the output file will contain
207+
# the path specified, and the build directory path changes every build.
208+
sha256sum Pipfile.lock >"${cache_dir}/.heroku/python/Pipfile.lock.sha256"
198209
fi
199210

200211
meta_time "cache_save_duration" "${cache_save_start_time}"

‎lib/pipenv.sh

Lines changed: 103 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -6,44 +6,105 @@ set -euo pipefail
66

77
PIPENV_VERSION=$(utils::get_requirement_version 'pipenv')
88

9-
# TODO: Either enable or remove these.
10-
# export CLINT_FORCE_COLOR=1
11-
# export PIPENV_FORCE_COLOR=1
12-
139
function pipenv::install_pipenv() {
14-
meta_set "pipenv_version" "${PIPENV_VERSION}"
15-
16-
output::step "Installing Pipenv ${PIPENV_VERSION}"
17-
18-
# TODO: Install Pipenv into a venv so it isn't leaked into the app environment.
19-
# TODO: Skip installing Pipenv if its version hasn't changed (once it's installed into a venv).
20-
# TODO: Explore viability of making Pipenv only be available during the build, to reduce slug size.
10+
local python_home="${1}"
11+
local python_major_version="${2}"
12+
local export_file="${3}"
13+
local profile_d_file="${4}"
14+
15+
# Ideally we would only make Pipenv available during the build to reduce slug size, however,
16+
# the buildpack has historically not done that and so some apps are relying on it at run-time
17+
# (for example via `pipenv run` commands in their Procfile). As such, we have to store it in
18+
# the build directory, but must do so via the symlinked `/app/.heroku/python` path so the
19+
# venv doesn't break when the build directory is relocated to /app at run-time.
20+
local pipenv_root="${python_home}/pipenv"
21+
22+
# We nest the venv and then symlink the `pipenv` script to prevent the rest of `venv/bin/`
23+
# (such as entrypoint scripts from Pipenv's dependencies, or the venv's activation scripts)
24+
# from being added to PATH and exposed to the app.
25+
local pipenv_bin_dir="${pipenv_root}/bin"
26+
local pipenv_venv_dir="${pipenv_root}/venv"
2127

22-
# shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled.
23-
if ! {
24-
pip \
25-
install \
26-
--disable-pip-version-check \
27-
--no-cache-dir \
28-
--no-input \
29-
--quiet \
30-
"pipenv==${PIPENV_VERSION}" \
31-
|& output::indent
32-
}; then
33-
output::error <<-EOF
34-
Error: Unable to install Pipenv.
35-
36-
In some cases, this happens due to a temporary issue with
37-
the network connection or Python Package Index (PyPI).
38-
39-
Try building again to see if the error resolves itself.
28+
meta_set "pipenv_version" "${PIPENV_VERSION}"
4029

41-
If that doesn't help, check the status of PyPI here:
42-
https://status.python.org
43-
EOF
44-
meta_set "failure_reason" "install-package-manager::pipenv"
45-
exit 1
30+
# The earlier buildpack cache invalidation step will have already handled the case where the
31+
# Pipenv version has changed, so here we only need to check that a Pipenv install exists.
32+
# Note: We don't need to use the `-e` trick of `install_poetry()` since we're installing into
33+
# a constant path, rather than the cache directory (which can change location).
34+
if [[ -f "${pipenv_bin_dir}/pipenv" ]]; then
35+
output::step "Using cached Pipenv ${PIPENV_VERSION}"
36+
else
37+
output::step "Installing Pipenv ${PIPENV_VERSION}"
38+
39+
mkdir -p "${pipenv_root}"
40+
41+
# We use the pip wheel bundled within Python's standard library to install Pipenv,
42+
# since Pipenv vendors its own pip, so doesn't need an install in the venv.
43+
# shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled.
44+
if ! python -m venv --without-pip "${pipenv_venv_dir}" |& output::indent; then
45+
output::error <<-EOF
46+
Internal Error: Unable to create virtual environment for Pipenv.
47+
48+
The 'python -m venv' command to create a virtual environment did
49+
not exit successfully.
50+
51+
See the log output above for more information.
52+
EOF
53+
meta_set "failure_reason" "create-venv::pipenv"
54+
exit 1
55+
fi
56+
57+
local bundled_pip_module_path
58+
bundled_pip_module_path="$(utils::bundled_pip_module_path "${python_home}" "${python_major_version}")"
59+
60+
# We must call the venv Python directly here, rather than relying on pip's `--python`
61+
# option, since `--python` was only added in pip v22.3, so isn't supported by the older
62+
# pip versions bundled with Python 3.9/3.10.
63+
# shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled.
64+
if ! {
65+
"${pipenv_venv_dir}/bin/python" "${bundled_pip_module_path}" \
66+
install \
67+
--disable-pip-version-check \
68+
--no-cache-dir \
69+
--no-input \
70+
--quiet \
71+
"pipenv==${PIPENV_VERSION}" \
72+
|& output::indent
73+
}; then
74+
output::error <<-EOF
75+
Error: Unable to install Pipenv.
76+
77+
In some cases, this happens due to a temporary issue with
78+
the network connection or Python Package Index (PyPI).
79+
80+
Try building again to see if the error resolves itself.
81+
82+
If that doesn't help, check the status of PyPI here:
83+
https://status.python.org
84+
EOF
85+
meta_set "failure_reason" "install-package-manager::pipenv"
86+
exit 1
87+
fi
88+
89+
mkdir -p "${pipenv_bin_dir}"
90+
ln --symbolic --no-target-directory "${pipenv_venv_dir}/bin/pipenv" "${pipenv_bin_dir}/pipenv"
4691
fi
92+
93+
export PATH="${pipenv_bin_dir}:${PATH}"
94+
# Force Pipenv to manage the system Python site-packages instead of using venvs.
95+
export PIPENV_SYSTEM="1"
96+
97+
# Set the same env vars in the environment used by later buildpacks.
98+
cat >>"${export_file}" <<-EOF
99+
export PATH="${pipenv_bin_dir}:\${PATH}"
100+
export PIPENV_SYSTEM="1"
101+
EOF
102+
103+
# And the environment used at app run-time.
104+
cat >>"${profile_d_file}" <<-EOF
105+
export PATH="${pipenv_bin_dir}:\${PATH}"
106+
export PIPENV_SYSTEM="1"
107+
EOF
47108
}
48109

49110
# Previous versions of the buildpack used to cache the checksum of the lockfile to allow
@@ -62,7 +123,8 @@ function pipenv::install_dependencies() {
62123
export PIP_EXTRA_INDEX_URL
63124
fi
64125

65-
# Note: We can't use `pipenv sync` since it doesn't validate that the lockfile is up to date.
126+
# Note: We can't use `pipenv sync` since it doesn't support `--deploy` and so won't abort
127+
# if the lockfile is out of date.
66128
local pipenv_install_command=(
67129
pipenv
68130
install
@@ -77,11 +139,14 @@ function pipenv::install_dependencies() {
77139
# We only display the most relevant command args here, to improve the signal to noise ratio.
78140
output::step "Installing dependencies using '${pipenv_install_command[*]}'"
79141

142+
# TODO: Expose app config vars to the install command as part of doing so for all package managers.
143+
# `PIPENV_NOSPIN`: Disable progress spinners.
144+
# `PIP_SRC`: Override the editable VCS repo location from its default of inside the build directory
145+
# (Pipenv uses pip internally, and doesn't offer its own config option for this).
80146
# shellcheck disable=SC2310 # This function is invoked in an 'if' condition so set -e will be disabled.
81147
if ! {
82-
"${pipenv_install_command[@]}" \
83-
--extra-pip-args='--src=/app/.heroku/python/src' \
84-
--system \
148+
PIPENV_NOSPIN="1" PIP_SRC="/app/.heroku/python/src" \
149+
"${pipenv_install_command[@]}" \
85150
|& tee "${WARNINGS_LOG:?}" \
86151
|& output::indent
87152
}; then

‎requirements/pipenv.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
pipenv==2024.0.1
1+
pipenv==2025.0.4

‎spec/fixtures/pipenv_basic/bin/compile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,8 @@ echo
1717

1818
pipenv --version
1919
# We have to resort to using pip to list installed packages, since `pipenv graph` doesn't support `--system`.
20-
pip list --disable-pip-version-check
20+
python -m ensurepip --default-pip >/dev/null
21+
pip list --disable-pip-version-check --exclude pip
2122
echo
2223

2324
python -c 'import typing_extensions; print(typing_extensions)'

‎spec/hatchet/ci_spec.rb

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,6 @@
8787
-----> Python app detected
8888
-----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version
8989
-----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION}
90-
-----> Installing pip #{PIP_VERSION}
9190
-----> Installing Pipenv #{PIPENV_VERSION}
9291
-----> Installing dependencies using 'pipenv install --deploy --dev'
9392
Installing dependencies from Pipfile.lock \\(.+\\)...
@@ -103,14 +102,16 @@
103102
LC_ALL=C.UTF-8
104103
LD_LIBRARY_PATH=/app/.heroku/python/lib
105104
LIBRARY_PATH=/app/.heroku/python/lib
106-
PATH=/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/
105+
PATH=/app/.heroku/python/pipenv/bin:/app/.heroku/python/bin::/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/
106+
PIPENV_SYSTEM=1
107107
PKG_CONFIG_PATH=/app/.heroku/python/lib/pkg-config
108108
PYTHONUNBUFFERED=1
109109
-----> Inline app detected
110110
LANG=en_US.UTF-8
111111
LD_LIBRARY_PATH=/app/.heroku/python/lib
112112
LIBRARY_PATH=/app/.heroku/python/lib
113-
PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/
113+
PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/
114+
PIPENV_SYSTEM=1
114115
PYTHONHASHSEED=random
115116
PYTHONHOME=/app/.heroku/python
116117
PYTHONPATH=/app
@@ -124,7 +125,8 @@
124125
LANG=en_US.UTF-8
125126
LD_LIBRARY_PATH=/app/.heroku/python/lib
126127
LIBRARY_PATH=/app/.heroku/python/lib
127-
PATH=/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/
128+
PATH=/app/.heroku/python/bin:/app/.heroku/python/pipenv/bin:/usr/local/bin:/usr/bin:/bin:/app/.sprettur/bin/:/app/.sprettur/bin/
129+
PIPENV_SYSTEM=1
128130
PYTHONHASHSEED=random
129131
PYTHONHOME=/app/.heroku/python
130132
PYTHONPATH=/app
@@ -140,8 +142,7 @@
140142
-----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version
141143
-----> Restoring cache
142144
-----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION}
143-
-----> Installing pip #{PIP_VERSION}
144-
-----> Installing Pipenv #{PIPENV_VERSION}
145+
-----> Using cached Pipenv #{PIPENV_VERSION}
145146
-----> Installing dependencies using 'pipenv install --deploy --dev'
146147
Installing dependencies from Pipfile.lock \\(.+\\)...
147148
Installing dependencies from Pipfile.lock \\(.+\\)...

‎spec/hatchet/package_manager_spec.rb

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,6 @@
115115
remote:
116116
remote: -----> Using Python #{DEFAULT_PYTHON_MAJOR_VERSION} specified in .python-version
117117
remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION}
118-
remote: -----> Installing pip #{PIP_VERSION}
119118
remote: -----> Installing Pipenv #{PIPENV_VERSION}
120119
remote: -----> Installing dependencies using 'pipenv install --deploy'
121120
remote: Installing dependencies from Pipfile.lock \\(.+\\)...

0 commit comments

Comments
 (0)