From 42499ff4dd7101e9e10c4bba7a3b8d39ca120a65 Mon Sep 17 00:00:00 2001 From: Ed Morley <501702+edmorley@users.noreply.github.com> Date: Wed, 9 Oct 2024 08:40:40 +0000 Subject: [PATCH] Refactor Python version handling (#1658) The previous implementation of Python version detection/resolution revolved heavily around the `runtime.txt` file, even though the version could originate from other sources. For example, a fake `runtime.txt` would be written out into the build directory containing the desired Python version, even if the version originated from `Pipfile.lock`. This meant all later Python version handling in the buildpack would see a `runtime.txt` file, along with versions specified in the syntax of that file (e.g. `python-N.N.N` strings), even though that wasn't the format in which the user had specified the version. Now, the buildpack explicitly tracks the requested version and its origin (rather than using the `runtime.txt` file as an API), along with the resolved Python major and full versions (which makes later Python version conditionals less fragile). In addition, the Python version specifiers are validated upfront at the point of parsing the relevant data source, so that clearer error messages can be shown. Lastly, the Python version resolution (the mapping of major Python versions to the latest patch release) has been decoupled from the Pipenv version implementation and made more robust, so it can also be used by the upcoming `.python-version` file support. GUS-W-16821309. GUS-W-7924371. GUS-W-8104668. --- CHANGELOG.md | 4 + bin/compile | 64 ++-- bin/default_pythons | 19 -- bin/steps/pipenv-python-version | 66 ---- bin/steps/python | 124 +++----- lib/python_version.sh | 286 ++++++++++++++++++ .../Pipfile | 2 +- .../pipenv_lockfile_invalid_json/Pipfile.lock | 1 + spec/fixtures/pipenv_python_3.6/Pipfile.lock | 29 -- .../pipenv_python_full_version/Pipfile | 5 +- .../pipenv_python_full_version/Pipfile.lock | 11 +- .../Pipfile | 2 +- .../Pipfile.lock | 10 +- .../requirements.txt | 0 spec/fixtures/python_2.7/runtime.txt | 1 + .../fixtures/python_3.10_outdated/runtime.txt | 1 - .../fixtures/python_3.11_outdated/runtime.txt | 1 - .../fixtures/python_3.12_outdated/runtime.txt | 1 - spec/fixtures/python_3.6/requirements.txt | 1 - spec/fixtures/python_3.6/runtime.txt | 1 - spec/fixtures/python_3.7/requirements.txt | 1 - .../requirements.txt | 0 .../runtime.txt | 1 + .../requirements.txt | 0 .../runtime.txt | 1 + .../requirements.txt | 0 .../runtime.txt | 0 .../requirements.txt | 0 .../runtime.txt | 0 spec/hatchet/ci_spec.rb | 16 +- spec/hatchet/pip_spec.rb | 20 +- spec/hatchet/pipenv_spec.rb | 200 ++++++------ spec/hatchet/python_update_warning_spec.rb | 100 +++--- spec/hatchet/python_version_spec.rb | 170 ++++++++--- spec/hatchet/stack_spec.rb | 10 +- spec/spec_helper.rb | 3 +- 36 files changed, 680 insertions(+), 471 deletions(-) delete mode 100755 bin/default_pythons delete mode 100755 bin/steps/pipenv-python-version create mode 100644 lib/python_version.sh rename spec/fixtures/{pipenv_python_3.6 => pipenv_lockfile_invalid_json}/Pipfile (84%) create mode 100644 spec/fixtures/pipenv_lockfile_invalid_json/Pipfile.lock delete mode 100644 spec/fixtures/pipenv_python_3.6/Pipfile.lock rename spec/fixtures/{python_3.10_outdated => python_2.7}/requirements.txt (100%) create mode 100644 spec/fixtures/python_2.7/runtime.txt delete mode 100644 spec/fixtures/python_3.10_outdated/runtime.txt delete mode 100644 spec/fixtures/python_3.11_outdated/runtime.txt delete mode 100644 spec/fixtures/python_3.12_outdated/runtime.txt delete mode 100644 spec/fixtures/python_3.6/requirements.txt delete mode 100644 spec/fixtures/python_3.6/runtime.txt rename spec/fixtures/{python_3.11_outdated => python_version_non_existent_major}/requirements.txt (100%) create mode 100644 spec/fixtures/python_version_non_existent_major/runtime.txt rename spec/fixtures/{python_3.12_outdated => python_version_non_existent_patch}/requirements.txt (100%) create mode 100644 spec/fixtures/python_version_non_existent_patch/runtime.txt rename spec/fixtures/{python_3.8_outdated => python_version_outdated}/requirements.txt (100%) rename spec/fixtures/{python_3.9_outdated => python_version_outdated}/runtime.txt (100%) rename spec/fixtures/{python_3.9_outdated => python_version_outdated_and_deprecated}/requirements.txt (100%) rename spec/fixtures/{python_3.8_outdated => python_version_outdated_and_deprecated}/runtime.txt (100%) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9345e2bc7..d0fd0fd39 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## [Unreleased] +- Improved build log output about the detected Python version. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) +- Improved error messages shown when the requested Python version is not a valid version string or is for an unknown/non-existent major Python version. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) +- Improved error messages shown when `Pipfile.lock` is not valid JSON. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) +- Fixed invalid Python versions being silently ignored when they were specified via the `python_version` field in `Pipfile.lock`. ([#1658](https://github.com/heroku/heroku-buildpack-python/pull/1658)) - Added support for Python 3.9 on Heroku-24. ([#1656](https://github.com/heroku/heroku-buildpack-python/pull/1656)) - Added buildpack metrics for use of outdated Python patch versions and occurrences of internal errors. ([#1657](https://github.com/heroku/heroku-buildpack-python/pull/1657)) - Improved the robustness of buildpack error handling by enabling `inherit_errexit`. ([#1655](https://github.com/heroku/heroku-buildpack-python/pull/1655)) diff --git a/bin/compile b/bin/compile index 4c9c0a5f0..12c74f3b6 100755 --- a/bin/compile +++ b/bin/compile @@ -24,6 +24,7 @@ source "${BUILDPACK_DIR}/lib/output.sh" source "${BUILDPACK_DIR}/lib/package_manager.sh" source "${BUILDPACK_DIR}/lib/pip.sh" source "${BUILDPACK_DIR}/lib/pipenv.sh" +source "${BUILDPACK_DIR}/lib/python_version.sh" source "${BUILDPACK_DIR}/lib/utils.sh" compile_start_time=$(nowms) @@ -46,9 +47,6 @@ export BUILD_DIR CACHE_DIR ENV_DIR S3_BASE_URL="${BUILDPACK_S3_BASE_URL:-"https://heroku-buildpack-python.s3.us-east-1.amazonaws.com"}" # This has to be exported since it's used by the geo-libs step which is run in a subshell. -# Default Python Versions -source "${BUILDPACK_DIR}/bin/default_pythons" - # Common Problem Warnings: # This section creates a temporary file in which to stick the output of `pip install`. # The `warnings` subscript then greps through this for common problems and guides @@ -123,10 +121,14 @@ fi # Runs a `bin/pre_compile` script if found in the app source, allowing build customisation. source "${BUILDPACK_DIR}/bin/steps/hooks/pre_compile" -# Sticky runtimes. If there was a previous build, and it used a given version of Python, -# continue to use that version of Python in perpetuity. +# TODO: Clear the cache if this isn't a valid version, as part of the cache refactor. +# (Currently the version is instead validated in `read_requested_python_version()`) if [[ -f "$CACHE_DIR/.heroku/python-version" ]]; then - CACHED_PYTHON_VERSION=$(cat "$CACHE_DIR/.heroku/python-version") + cached_python_version="$(cat "${CACHE_DIR}/.heroku/python-version")" + # `python-X.Y.Z` -> `X.Y` + cached_python_version="${cached_python_version#python-}" +else + cached_python_version= fi # We didn't always record the stack version. @@ -140,29 +142,37 @@ fi package_manager="$(package_manager::determine_package_manager "${BUILD_DIR}")" meta_set "package_manager" "${package_manager}" -# Pipenv Python version support. -# Detect the version of Python requested from a Pipfile (e.g. python_version or python_full_version). -# Convert it to a runtime.txt file. -source "${BUILDPACK_DIR}/bin/steps/pipenv-python-version" - -if [[ -f runtime.txt ]]; then - # PYTHON_VERSION_SOURCE may have already been set by the pipenv-python-version step. - # TODO: Refactor this and stop pipenv-python-version using runtime.txt as an API. - PYTHON_VERSION_SOURCE=${PYTHON_VERSION_SOURCE:-"runtime.txt"} - puts-step "Using Python version specified in ${PYTHON_VERSION_SOURCE}" - meta_set "python_version_reason" "specified" -elif [[ -n "${CACHED_PYTHON_VERSION:-}" ]]; then - puts-step "No Python version was specified. Using the same version as the last build: ${CACHED_PYTHON_VERSION}" - echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" - meta_set "python_version_reason" "cached" - echo "${CACHED_PYTHON_VERSION}" >runtime.txt -else - puts-step "No Python version was specified. Using the buildpack default: ${DEFAULT_PYTHON_VERSION}" - echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" - meta_set "python_version_reason" "default" - echo "${DEFAULT_PYTHON_VERSION}" >runtime.txt +# TODO: Move this warning to lib/package_manager.sh once `output::warning()` exists +# (puts-warn outputs to stdout, which would break `determine_package_manager()` as is). +# TODO: Adjust this warning to mention support for missing Pipfile.lock will be removed soon. +if [[ "${package_manager}" == "pipenv" && ! -f "${BUILD_DIR}/Pipfile.lock" ]]; then + puts-warn "No 'Pipfile.lock' found! We recommend you commit this into your repository." fi +# We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function +# without having to hardcode globals. See: https://stackoverflow.com/a/38997681 +python_version::read_requested_python_version "${BUILD_DIR}" "${package_manager}" "${cached_python_version}" requested_python_version python_version_origin +meta_set "python_version_reason" "${python_version_origin}" + +case "${python_version_origin}" in + default) + puts-step "No Python version was specified. Using the buildpack default: Python ${requested_python_version}" + echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" + ;; + cached) + puts-step "No Python version was specified. Using the same version as the last build: Python ${requested_python_version}" + echo " To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes" + ;; + *) + puts-step "Using Python ${requested_python_version} specified in ${python_version_origin}" + ;; +esac + +python_full_version="$(python_version::resolve_python_version "${requested_python_version}" "${python_version_origin}")" +python_major_version="${python_full_version%.*}" +meta_set "python_version" "${python_full_version}" +meta_set "python_version_major" "${python_major_version}" + # The directory for the .profile.d scripts. mkdir -p "$(dirname "$PROFILE_PATH")" # The directory for editable VCS dependencies. diff --git a/bin/default_pythons b/bin/default_pythons deleted file mode 100755 index 51e9cc247..000000000 --- a/bin/default_pythons +++ /dev/null @@ -1,19 +0,0 @@ -#!/usr/bin/env bash - -# Disable unused env var warning, since shellcheck doesn't take into account -# that this file is sourced. We don't want to use export since it exposes -# the env vars to subprocesses. -# shellcheck disable=2034 - -LATEST_312="python-3.12.7" -LATEST_311="python-3.11.10" -LATEST_310="python-3.10.15" -LATEST_39="python-3.9.20" -LATEST_38="python-3.8.20" -# TODO: Remove these EOL versions once pipenv-python-version is refactored. -LATEST_37="python-3.7.17" -LATEST_36="python-3.6.15" -LATEST_35="python-3.5.10" -LATEST_34="python-3.4.10" -LATEST_27="python-2.7.18" -DEFAULT_PYTHON_VERSION="${LATEST_312}" diff --git a/bin/steps/pipenv-python-version b/bin/steps/pipenv-python-version deleted file mode 100755 index de2aad798..000000000 --- a/bin/steps/pipenv-python-version +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env bash -# shellcheck disable=SC2154 # TODO: Env var is referenced but not assigned. -# shellcheck disable=SC2250 # TODO: Use braces around variable references even when not strictly required. - -# TODO: Move this to lib/ as part of the refactoring for .python-version support. - -# Detect Python-version with Pipenv. - -if [[ "${package_manager}" == "pipenv" ]]; then - - if [[ ! -f $BUILD_DIR/runtime.txt ]]; then - if [[ ! -f $BUILD_DIR/Pipfile.lock ]]; then - puts-warn "No 'Pipfile.lock' found! We recommend you commit this into your repository." - fi - if [[ -f $BUILD_DIR/Pipfile.lock ]]; then - # Ignore unused env var warning since this is used by bin/compile. - # shellcheck disable=2034 - PYTHON_VERSION_SOURCE='Pipfile.lock' - set +e - PYTHON=$(jq -r '._meta.requires.python_full_version' "$BUILD_DIR/Pipfile.lock") - if [[ "$PYTHON" != "null" ]]; then - echo "python-$PYTHON" >"$BUILD_DIR/runtime.txt" - fi - set -e - - if [[ "$PYTHON" == "null" ]]; then - PYTHON=$(jq -r '._meta.requires.python_version' "$BUILD_DIR/Pipfile.lock") - case "${PYTHON}" in - 2.7) - echo "${LATEST_27}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.4) - echo "${LATEST_34}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.5) - echo "${LATEST_35}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.6) - echo "${LATEST_36}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.7) - echo "${LATEST_37}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.8) - echo "${LATEST_38}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.9) - echo "${LATEST_39}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.10) - echo "${LATEST_310}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.11) - echo "${LATEST_311}" >"${BUILD_DIR}/runtime.txt" - ;; - 3.12) - echo "${LATEST_312}" >"${BUILD_DIR}/runtime.txt" - ;; - # TODO: Make this case an error - *) ;; - esac - fi - - fi - fi -fi diff --git a/bin/steps/python b/bin/steps/python index a2bdb0f12..725846d15 100755 --- a/bin/steps/python +++ b/bin/steps/python @@ -4,54 +4,21 @@ set -euo pipefail -PYTHON_VERSION=$(cat runtime.txt) -# Remove leading and trailing whitespace. Note: This implementation relies upon -# `extglob` being set, which is the case thanks to `bin/utils` being run earlier. -PYTHON_VERSION="${PYTHON_VERSION##+([[:space:]])}" -PYTHON_VERSION="${PYTHON_VERSION%%+([[:space:]])}" - -function eol_python_version_error() { - local major_version="${1}" - local eol_date="${2}" - display_error <<-EOF - Error: Python ${major_version} is no longer supported. - - Python ${major_version} reached upstream end-of-life on ${eol_date}, and is - therefore no longer receiving security updates: - https://devguide.python.org/versions/#supported-versions - - As such, it is no longer supported by this buildpack. - - Please upgrade to a newer Python version. - - For a list of the supported Python versions, see: - https://devcenter.heroku.com/articles/python-support#supported-runtimes - EOF - meta_set "failure_reason" "python-version-eol" - exit 1 -} - -# We check for EOL prior to checking if the archive exists on S3, to ensure the more specific EOL error -# message is still shown for newer stacks where the EOL Python versions might not have been built. -case "${PYTHON_VERSION}" in - python-3.7.+([0-9])) - eol_python_version_error "3.7" "June 27th, 2023" - ;; - python-3.6.+([0-9])) - eol_python_version_error "3.6" "December 23rd, 2021" - ;; - *) ;; -esac - # The Python runtime archive filename is of form: 'python-X.Y.Z-ubuntu-22.04-amd64.tar.zst' # The Ubuntu version is calculated from `STACK` since it's faster than calling `lsb_release`. UBUNTU_VERSION="${STACK/heroku-/}.04" ARCH=$(dpkg --print-architecture) -PYTHON_URL="${S3_BASE_URL}/${PYTHON_VERSION}-ubuntu-${UBUNTU_VERSION}-${ARCH}.tar.zst" - +PYTHON_URL="${S3_BASE_URL}/python-${python_full_version}-ubuntu-${UBUNTU_VERSION}-${ARCH}.tar.zst" + +# The Python version validation earlier will have filtered out most unsupported versions. +# However, the version might still not be found if either: +# 1. It's a Python major version we've deprecated and so is only available on older stacks (i.e: Python 3.8). +# 2. If an exact Python version was requested and the patch version doesn't exist (e.g. 3.12.999). +# 3. The user has pinned to an older buildpack version and the S3 bucket location or layout has changed since. +# TODO: Update this message to be more specific once Python 3.8 support is dropped. if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}"; then display_error <<-EOF - Error: Requested runtime '${PYTHON_VERSION}' is not available for this stack (${STACK}). + Error: Python ${python_full_version} is not available for this stack (${STACK}). For a list of the supported Python versions, see: https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -60,20 +27,17 @@ if ! curl --output /dev/null --silent --head --fail --retry 3 --retry-connrefuse exit 1 fi -# TODO: Refactor Python version usage to use the non-prefixed form everywhere. -python_version_without_prefix="${PYTHON_VERSION#python-}" -meta_set "python_version" "${python_version_without_prefix}" -meta_set "python_version_major" "${python_version_without_prefix%.*}" - function warn_if_patch_update_available() { - local requested_version="${1}" - local latest_patch_version="${2}" + local requested_full_version="${1}" + local requested_major_version="${2}" + local latest_patch_version + latest_patch_version="$(python_version::resolve_python_version "${requested_major_version}" "${python_version_origin}")" # Extract the patch version component of the version strings (ie: the '5' in '3.10.5'). - local requested_patch_number="${requested_version##*.}" + local requested_patch_number="${requested_full_version##*.}" local latest_patch_number="${latest_patch_version##*.}" if ((requested_patch_number < latest_patch_number)); then puts-warn - puts-warn "A Python security update is available! Upgrade as soon as possible to: ${latest_patch_version}" + puts-warn "A Python security update is available! Upgrade as soon as possible to: Python ${latest_patch_version}" puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" puts-warn meta_set "python_version_outdated" "true" @@ -84,45 +48,31 @@ function warn_if_patch_update_available() { # We wait until now to display outdated Python version warnings, since we only want to show them # if there weren't any errors with the version to avoid adding noise to the error messages. -case "${PYTHON_VERSION}" in - python-3.12.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_312}" - ;; - python-3.11.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_311}" - ;; - python-3.10.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_310}" - ;; - python-3.9.*) - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_39}" - ;; - python-3.8.*) - puts-warn - puts-warn "Python 3.8 will reach its upstream end-of-life in October 2024, at which" - puts-warn "point it will no longer receive security updates:" - puts-warn "https://devguide.python.org/versions/#supported-versions" - puts-warn - puts-warn "Support for Python 3.8 will be removed from this buildpack on December 4th, 2024." - puts-warn - puts-warn "Upgrade to a newer Python version as soon as possible to keep your app secure." - puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" - puts-warn - warn_if_patch_update_available "${PYTHON_VERSION}" "${LATEST_38}" - ;; - # TODO: Make this case an error, since it should be unreachable. - *) ;; -esac +# TODO: Move this into lib/ as part of the warnings refactor. +if [[ "${python_major_version}" == "3.8" ]]; then + puts-warn + puts-warn "Python 3.8 will reach its upstream end-of-life in October 2024, at which" + puts-warn "point it will no longer receive security updates:" + puts-warn "https://devguide.python.org/versions/#supported-versions" + puts-warn + puts-warn "Support for Python 3.8 will be removed from this buildpack on December 4th, 2024." + puts-warn + puts-warn "Upgrade to a newer Python version as soon as possible to keep your app secure." + puts-warn "See: https://devcenter.heroku.com/articles/python-runtimes" + puts-warn +fi + +warn_if_patch_update_available "${python_full_version}" "${python_major_version}" if [[ "$STACK" != "$CACHED_PYTHON_STACK" ]]; then puts-step "Stack has changed from $CACHED_PYTHON_STACK to $STACK, clearing cache" rm -rf .heroku/python-stack .heroku/python-version .heroku/python .heroku/vendor .heroku/python .heroku/python-sqlite3-version fi +# TODO: Clean this up as part of the cache refactor. if [[ -f .heroku/python-version ]]; then - # shellcheck disable=SC2312 # TODO: Invoke this command separately to avoid masking its return value. - if [[ ! "$(cat .heroku/python-version)" == "$PYTHON_VERSION" ]]; then - puts-step "Python version has changed from $(cat .heroku/python-version) to ${PYTHON_VERSION}, clearing cache" + if [[ "${cached_python_version}" != "${python_full_version}" ]]; then + puts-step "Python version has changed from ${cached_python_version} to ${python_full_version}, clearing cache" rm -rf .heroku/python else SKIP_INSTALL=1 @@ -153,9 +103,9 @@ if [[ -f "${BUILD_DIR}/requirements.txt" ]]; then fi if [[ "${SKIP_INSTALL:-0}" == "1" ]]; then - puts-step "Using cached install of ${PYTHON_VERSION}" + puts-step "Using cached install of Python ${python_full_version}" else - puts-step "Installing ${PYTHON_VERSION}" + puts-step "Installing Python ${python_full_version}" # Prepare destination directory. mkdir -p .heroku/python @@ -163,13 +113,13 @@ else if ! curl --silent --show-error --fail --retry 3 --retry-connrefused --connect-timeout 10 "${PYTHON_URL}" | tar --zstd --extract --directory .heroku/python; then # The Python version was confirmed to exist previously, so any failure here is due to # a networking issue or archive/buildpack bug rather than the runtime not existing. - display_error "Error: Failed to download/install ${PYTHON_VERSION}." + display_error "Error: Failed to download/install Python ${python_full_version}." meta_set "failure_reason" "python-download" exit 1 fi # Record for future reference. - echo "$PYTHON_VERSION" >.heroku/python-version + echo "python-${python_full_version}" >.heroku/python-version echo "$STACK" >.heroku/python-stack hash -r diff --git a/lib/python_version.sh b/lib/python_version.sh new file mode 100644 index 000000000..8b88e6e37 --- /dev/null +++ b/lib/python_version.sh @@ -0,0 +1,286 @@ +#!/usr/bin/env bash + +# This is technically redundant, since all consumers of this lib will have enabled these, +# however, it helps Shellcheck realise the options under which these functions will run. +set -euo pipefail + +LATEST_PYTHON_3_8="3.8.20" +LATEST_PYTHON_3_9="3.9.20" +LATEST_PYTHON_3_10="3.10.15" +LATEST_PYTHON_3_11="3.11.10" +LATEST_PYTHON_3_12="3.12.7" + +DEFAULT_PYTHON_FULL_VERSION="${LATEST_PYTHON_3_12}" +DEFAULT_PYTHON_MAJOR_VERSION="${DEFAULT_PYTHON_FULL_VERSION%.*}" + +# Integer with no redundant leading zeros. +INT_REGEX="(0|[1-9][0-9]*)" +# Versions of form N.N or N.N.N. +PYTHON_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}(\.${INT_REGEX})?" +# Versions of form N.N.N only. +PYTHON_FULL_VERSION_REGEX="${INT_REGEX}\.${INT_REGEX}\.${INT_REGEX}" + +# Determine what Python version has been requested for the project. +# +# Returns a version request of form N.N or N.N.N, with basic validation checks that the version +# matches those forms. EOL version checks will be performed later, when this version request is +# resolved to an exact Python version. +# +# If an app specifies the Python version via multiple means, then the order of precedence is: +# 1. runtime.txt +# 2. Pipfile.lock (`python_full_version` field) +# 3. Pipfile.lock (`python_version` field) +# +# If a version wasn't specified by the app, then new apps/those with an empty cache will use +# a buildpack default version for the first build, and then subsequent cached builds will use +# the same Python full version in perpetuity (aka sticky versions). Sticky versioning leads to +# confusing UX so is something we want to deprecate/sunset in the future (and have already done +# so in the Python CNB). +# TODO: Change the sticky versioning implementation so it's only sticky to the major version +# rather than the full version, so apps that don't specify a Python version at least get +# security patch updates. +function python_version::read_requested_python_version() { + local build_dir="${1}" + local package_manager="${2}" + local cached_python_version="${3}" + # We use the Bash 4.3+ `nameref` feature to pass back multiple values from this function + # without having to hardcode globals. See: https://stackoverflow.com/a/38997681 + declare -n version="${4}" + declare -n origin="${5}" + local contents + + local runtime_txt_path="${build_dir}/runtime.txt" + if [[ -f "${runtime_txt_path}" ]]; then + contents="$(cat "${runtime_txt_path}")" + version="$(python_version::parse_runtime_txt "${contents}")" + origin="runtime.txt" + return 0 + fi + + if [[ "${package_manager}" == "pipenv" ]]; then + version="$(python_version::read_pipenv_python_version "${build_dir}")" + # The Python version fields in a Pipfile.lock are optional. + if [[ -n "${version}" ]]; then + origin="Pipfile.lock" + return 0 + fi + fi + + # Protect against invalid versions somehow having been written into the cache. + # TODO: Move this validation into the cache handling as part of the cache refactor? + if [[ "${cached_python_version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then + version="${cached_python_version}" + origin="cached" + else + version="${DEFAULT_PYTHON_MAJOR_VERSION}" + # shellcheck disable=SC2034 # This isn't unused, Shellcheck doesn't follow namerefs. + origin="default" + fi +} + +# Parse the contents of a runtime.txt file and return the Python version substring (e.g. `3.12.0`). +function python_version::parse_runtime_txt() { + local contents="${1}" + + # The file must contain a string of form `python-N.N.N` (leading/trailing whitespace is permitted). + if [[ "${contents}" =~ ^[[:space:]]*python-(${PYTHON_FULL_VERSION_REGEX})[[:space:]]*$ ]]; then + local version="${BASH_REMATCH[1]}" + echo "${version}" + else + display_error <<-EOF + Error: Invalid Python version in runtime.txt. + + The Python version specified in 'runtime.txt' is not in + the correct format. + + The following file contents were found: + ${contents} + + However, the version string must begin with a 'python-' prefix, + followed by the version specified as '..'. + Comments are not supported. + + For example, to request Python ${DEFAULT_PYTHON_FULL_VERSION}, use: + python-${DEFAULT_PYTHON_FULL_VERSION} + + Please update 'runtime.txt' to use a valid version string, or + else remove the file to instead use the default version + (currently Python ${DEFAULT_PYTHON_FULL_VERSION}). + EOF + meta_set "failure_reason" "python-version-invalid" + return 1 + fi +} + +# Read the Python version from a Pipfile.lock, which can exist in one of two optional fields, +# `python_full_version` (as N.N.N) and `python_version` (as N.N). If both fields are +# defined, we will use the value set in `python_full_version`. See: +# https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python +function python_version::read_pipenv_python_version() { + local build_dir="${1}" + local pipfile_lock_path="${build_dir}/Pipfile.lock" + local version + + # We currently permit using Pipenv without a `Pipfile.lock`, however, in the future we will + # require a lockfile, at which point this conditional can be removed. + if [[ ! -f "${pipfile_lock_path}" ]]; then + return 0 + fi + + if ! version=$(jq --raw-output '._meta.requires.python_full_version // ._meta.requires.python_version' "${pipfile_lock_path}" 2>&1); then + display_error <<-EOF + Error: Cannot parse Pipfile.lock. + + A Pipfile.lock file was found, however, it could not be parsed: + ${version} + + This is likely due to it not being valid JSON. + + Run 'pipenv lock' to regenerate/fix the lockfile. + EOF + meta_set "failure_reason" "pipfile-lock-invalid" + return 1 + fi + + # Neither of the optional fields were found. + if [[ "${version}" == "null" ]]; then + return 0 + fi + + # We don't use separate version validation rules for both fields (e.g. ensuring a patch version + # only exists for `python_full_version`) since Pipenv doesn't distinguish between them either: + # https://github.com/pypa/pipenv/blob/v2024.1.0/pipenv/project.py#L392-L398 + if [[ "${version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then + echo "${version}" + else + display_error <<-EOF + Error: Invalid Python version in Pipfile / Pipfile.lock. + + The Python version specified in Pipfile / Pipfile.lock by the + 'python_version' or 'python_full_version' field is not valid. + + The following version was found: + ${version} + + However, the version must be specified as either: + 1. '.' (recommended, for automatic security updates) + 2. '..' (to pin to an exact Python version) + + Please update your 'Pipfile' to use a valid Python version and + then run 'pipenv lock' to regenerate the lockfile. + + For more information, see: + https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python + EOF + meta_set "failure_reason" "python-version-invalid" + return 1 + fi +} + +# Resolve a requested Python version (which can be of form N.N or N.N.N) to a specific +# Python version of form N.N.N. Rejects Python major versions that are not supported. +function python_version::resolve_python_version() { + local requested_python_version="${1}" + local python_version_origin="${2}" + + if [[ ! "${requested_python_version}" =~ ^${PYTHON_VERSION_REGEX}$ ]]; then + # The Python version was previously validated, so this should never occur. + utils::abort_internal_error "Invalid Python version: ${requested_python_version}" + fi + + local major="${BASH_REMATCH[1]}" + local minor="${BASH_REMATCH[2]}" + + if ((major < 3 || (major == 3 && minor < 8))); then + if [[ "${python_version_origin}" == "cached" ]]; then + display_error <<-EOF + Error: The cached Python version has reached end-of-life. + + Your app does not specify a Python version, and so normally + would use the version cached from the last build (${requested_python_version}). + + However, Python ${major}.${minor} has reached its upstream end-of-life, + and is therefore no longer receiving security updates: + https://devguide.python.org/versions/#supported-versions + + As such, it is no longer supported by this buildpack. + + Please upgrade to a newer Python version by creating a + 'runtime.txt' file that contains a Python version like: + python-${DEFAULT_PYTHON_FULL_VERSION} + + For a list of the supported Python versions, see: + https://devcenter.heroku.com/articles/python-support#supported-runtimes + EOF + else + display_error <<-EOF + Error: The requested Python version has reached end-of-life. + + Python ${major}.${minor} has reached its upstream end-of-life, and is + therefore no longer receiving security updates: + https://devguide.python.org/versions/#supported-versions + + As such, it is no longer supported by this buildpack. + + Please upgrade to a newer Python version by updating the + version configured via the '${python_version_origin}' file. + + For a list of the supported Python versions, see: + https://devcenter.heroku.com/articles/python-support#supported-runtimes + EOF + fi + meta_set "failure_reason" "python-version-eol" + return 1 + fi + + if (((major == 3 && minor > 12) || major >= 4)); then + if [[ "${python_version_origin}" == "cached" ]]; then + display_error <<-EOF + Error: The cached Python version is not recognised. + + Your app does not specify a Python version, and so normally + would use the version cached from the last build (${requested_python_version}). + + However, Python ${major}.${minor} is not recognised by this version + of the buildpack. + + This can occur if you have downgraded the version of the + buildpack to an older version. + + Please switch back to a newer version of this buildpack. + EOF + else + display_error <<-EOF + Error: The requested Python version is not recognised. + + The requested Python version ${major}.${minor} is not recognised. + + Check that this Python version has been officially released, + and that the Python buildpack has added support for it: + https://devguide.python.org/versions/#supported-versions + https://devcenter.heroku.com/articles/python-support#supported-runtimes + + If it has, make sure that you are using the latest version + of this buildpack: + https://devcenter.heroku.com/articles/python-support#checking-the-python-buildpack-version + + Otherwise, switch to a supported version (such as Python ${DEFAULT_PYTHON_MAJOR_VERSION}) + by updating the version configured via the '${python_version_origin}' file. + EOF + fi + meta_set "failure_reason" "python-version-unknown" + return 1 + fi + + # If an exact Python version was requested, there's nothing to resolve. + # Otherwise map major version specifiers to the latest patch release. + case "${requested_python_version}" in + *.*.*) echo "${requested_python_version}" ;; + 3.8) echo "${LATEST_PYTHON_3_8}" ;; + 3.9) echo "${LATEST_PYTHON_3_9}" ;; + 3.10) echo "${LATEST_PYTHON_3_10}" ;; + 3.11) echo "${LATEST_PYTHON_3_11}" ;; + 3.12) echo "${LATEST_PYTHON_3_12}" ;; + *) utils::abort_internal_error "Unhandled Python major version: ${requested_python_version}" ;; + esac +} diff --git a/spec/fixtures/pipenv_python_3.6/Pipfile b/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile similarity index 84% rename from spec/fixtures/pipenv_python_3.6/Pipfile rename to spec/fixtures/pipenv_lockfile_invalid_json/Pipfile index 62a39bb91..9bf60112c 100644 --- a/spec/fixtures/pipenv_python_3.6/Pipfile +++ b/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile @@ -9,4 +9,4 @@ urllib3 = "*" [dev-packages] [requires] -python_version = "3.6" +python_version = "3.12" diff --git a/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile.lock b/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile.lock new file mode 100644 index 000000000..0e5491864 --- /dev/null +++ b/spec/fixtures/pipenv_lockfile_invalid_json/Pipfile.lock @@ -0,0 +1 @@ +INVALID JSON diff --git a/spec/fixtures/pipenv_python_3.6/Pipfile.lock b/spec/fixtures/pipenv_python_3.6/Pipfile.lock deleted file mode 100644 index 738941ddd..000000000 --- a/spec/fixtures/pipenv_python_3.6/Pipfile.lock +++ /dev/null @@ -1,29 +0,0 @@ -{ - "_meta": { - "hash": { - "sha256": "fe045e48581ffc6cb50af922ec04120b504f96ff6d6917fc535c8a9b76b8b71b" - }, - "pipfile-spec": 6, - "requires": { - "python_version": "3.6" - }, - "sources": [ - { - "name": "pypi", - "url": "https://pypi.org/simple", - "verify_ssl": true - } - ] - }, - "default": { - "urllib3": { - "hashes": [ - "sha256:076907bf8fd355cde77728471316625a4d2f7e713c125f51953bb5b3eecf4f72", - "sha256:75edcdc2f7d85b137124a6c3c9fc3933cdeaa12ecb9a6a959f22797a0feca7e1" - ], - "index": "pypi", - "version": "==1.26.14" - } - }, - "develop": {} -} diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile b/spec/fixtures/pipenv_python_full_version/Pipfile index 88c5c59bc..aa0852315 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile +++ b/spec/fixtures/pipenv_python_full_version/Pipfile @@ -9,4 +9,7 @@ urllib3 = "*" [dev-packages] [requires] -python_full_version = "3.12.2" +# Uses the oldest Python version supported by all stacks, to validate Pipenv works with it. +python_full_version = "3.9.0" +# Tests that `python_full_version` takes precedence. +python_version = "3.12" diff --git a/spec/fixtures/pipenv_python_full_version/Pipfile.lock b/spec/fixtures/pipenv_python_full_version/Pipfile.lock index 2346c8a33..888c9982e 100644 --- a/spec/fixtures/pipenv_python_full_version/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version/Pipfile.lock @@ -1,11 +1,12 @@ { "_meta": { "hash": { - "sha256": "5f4af5700c245f61e22d09479627ca936d458a6aa693452337170f369c642f40" + "sha256": "068e8668ba7ae121efca9caf66d4796267713afe33c711e01a0852fe5e9a1848" }, "pipfile-spec": 6, "requires": { - "python_full_version": "3.12.2" + "python_full_version": "3.9.0", + "python_version": "3.12" }, "sources": [ { @@ -18,12 +19,12 @@ "default": { "urllib3": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", + "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==2.2.3" } }, "develop": {} diff --git a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile index 683918cbd..1782422d3 100644 --- a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile +++ b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile @@ -9,4 +9,4 @@ urllib3 = "*" [dev-packages] [requires] -python_full_version = "X.Y.Z" +python_full_version = "3.9.*" diff --git a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock index 3b2d178ab..107e6f348 100644 --- a/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock +++ b/spec/fixtures/pipenv_python_full_version_invalid/Pipfile.lock @@ -1,11 +1,11 @@ { "_meta": { "hash": { - "sha256": "7efd722f93856d46f698123972037594c6b8621e54b5a2be8ce71f5707a13c33" + "sha256": "c629159445d228792003fe74526738c9155072a74a29f2d4920fc8aab878db0a" }, "pipfile-spec": 6, "requires": { - "python_full_version": "X.Y.Z" + "python_full_version": "3.9.*" }, "sources": [ { @@ -18,12 +18,12 @@ "default": { "urllib3": { "hashes": [ - "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d", - "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19" + "sha256:ca899ca043dcb1bafa3e262d73aa25c465bfb49e0bd9dd5d59f1d0acba2f8fac", + "sha256:e7d814a81dad81e6caf2ec9fdedb284ecc9c73076b62654547cc64ccdcae26e9" ], "index": "pypi", "markers": "python_version >= '3.8'", - "version": "==2.2.1" + "version": "==2.2.3" } }, "develop": {} diff --git a/spec/fixtures/python_3.10_outdated/requirements.txt b/spec/fixtures/python_2.7/requirements.txt similarity index 100% rename from spec/fixtures/python_3.10_outdated/requirements.txt rename to spec/fixtures/python_2.7/requirements.txt diff --git a/spec/fixtures/python_2.7/runtime.txt b/spec/fixtures/python_2.7/runtime.txt new file mode 100644 index 000000000..586b67310 --- /dev/null +++ b/spec/fixtures/python_2.7/runtime.txt @@ -0,0 +1 @@ +python-2.7.18 diff --git a/spec/fixtures/python_3.10_outdated/runtime.txt b/spec/fixtures/python_3.10_outdated/runtime.txt deleted file mode 100644 index fadb07024..000000000 --- a/spec/fixtures/python_3.10_outdated/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.10.0 diff --git a/spec/fixtures/python_3.11_outdated/runtime.txt b/spec/fixtures/python_3.11_outdated/runtime.txt deleted file mode 100644 index 335156c09..000000000 --- a/spec/fixtures/python_3.11_outdated/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.11.0 diff --git a/spec/fixtures/python_3.12_outdated/runtime.txt b/spec/fixtures/python_3.12_outdated/runtime.txt deleted file mode 100644 index 44f8fbe3a..000000000 --- a/spec/fixtures/python_3.12_outdated/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.12.0 diff --git a/spec/fixtures/python_3.6/requirements.txt b/spec/fixtures/python_3.6/requirements.txt deleted file mode 100644 index a42590beb..000000000 --- a/spec/fixtures/python_3.6/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -urllib3 diff --git a/spec/fixtures/python_3.6/runtime.txt b/spec/fixtures/python_3.6/runtime.txt deleted file mode 100644 index 6aaad49f7..000000000 --- a/spec/fixtures/python_3.6/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -python-3.6.15 diff --git a/spec/fixtures/python_3.7/requirements.txt b/spec/fixtures/python_3.7/requirements.txt index a42590beb..e69de29bb 100644 --- a/spec/fixtures/python_3.7/requirements.txt +++ b/spec/fixtures/python_3.7/requirements.txt @@ -1 +0,0 @@ -urllib3 diff --git a/spec/fixtures/python_3.11_outdated/requirements.txt b/spec/fixtures/python_version_non_existent_major/requirements.txt similarity index 100% rename from spec/fixtures/python_3.11_outdated/requirements.txt rename to spec/fixtures/python_version_non_existent_major/requirements.txt diff --git a/spec/fixtures/python_version_non_existent_major/runtime.txt b/spec/fixtures/python_version_non_existent_major/runtime.txt new file mode 100644 index 000000000..b6885d9e1 --- /dev/null +++ b/spec/fixtures/python_version_non_existent_major/runtime.txt @@ -0,0 +1 @@ +python-3.999.0 diff --git a/spec/fixtures/python_3.12_outdated/requirements.txt b/spec/fixtures/python_version_non_existent_patch/requirements.txt similarity index 100% rename from spec/fixtures/python_3.12_outdated/requirements.txt rename to spec/fixtures/python_version_non_existent_patch/requirements.txt diff --git a/spec/fixtures/python_version_non_existent_patch/runtime.txt b/spec/fixtures/python_version_non_existent_patch/runtime.txt new file mode 100644 index 000000000..f9907fe4a --- /dev/null +++ b/spec/fixtures/python_version_non_existent_patch/runtime.txt @@ -0,0 +1 @@ +python-3.12.999 diff --git a/spec/fixtures/python_3.8_outdated/requirements.txt b/spec/fixtures/python_version_outdated/requirements.txt similarity index 100% rename from spec/fixtures/python_3.8_outdated/requirements.txt rename to spec/fixtures/python_version_outdated/requirements.txt diff --git a/spec/fixtures/python_3.9_outdated/runtime.txt b/spec/fixtures/python_version_outdated/runtime.txt similarity index 100% rename from spec/fixtures/python_3.9_outdated/runtime.txt rename to spec/fixtures/python_version_outdated/runtime.txt diff --git a/spec/fixtures/python_3.9_outdated/requirements.txt b/spec/fixtures/python_version_outdated_and_deprecated/requirements.txt similarity index 100% rename from spec/fixtures/python_3.9_outdated/requirements.txt rename to spec/fixtures/python_version_outdated_and_deprecated/requirements.txt diff --git a/spec/fixtures/python_3.8_outdated/runtime.txt b/spec/fixtures/python_version_outdated_and_deprecated/runtime.txt similarity index 100% rename from spec/fixtures/python_3.8_outdated/runtime.txt rename to spec/fixtures/python_version_outdated_and_deprecated/runtime.txt diff --git a/spec/hatchet/ci_spec.rb b/spec/hatchet/ci_spec.rb index 4a29a6911..444b65e55 100644 --- a/spec/hatchet/ci_spec.rb +++ b/spec/hatchet/ci_spec.rb @@ -12,9 +12,9 @@ app.run_ci do |test_run| expect(test_run.output).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) -----> Python app detected - -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - -----> Installing python-#{DEFAULT_PYTHON_VERSION} + -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing SQLite3 -----> Installing requirements with pip @@ -70,10 +70,10 @@ expect(test_run.output).to include(<<~OUTPUT) -----> Python app detected - -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes -----> No change in requirements detected, installing from cache - -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing SQLite3 -----> Installing requirements with pip @@ -92,9 +92,9 @@ app.run_ci do |test_run| expect(test_run.output).to match(Regexp.new(<<~REGEX)) -----> Python app detected - -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - -----> Installing python-#{DEFAULT_PYTHON_VERSION} + -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} -----> Installing SQLite3 @@ -148,9 +148,9 @@ expect(test_run.output).to match(Regexp.new(<<~REGEX)) -----> Python app detected - -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} -----> Installing Pipenv #{PIPENV_VERSION} -----> Installing SQLite3 diff --git a/spec/hatchet/pip_spec.rb b/spec/hatchet/pip_spec.rb index 34f37fae6..178be593d 100644 --- a/spec/hatchet/pip_spec.rb +++ b/spec/hatchet/pip_spec.rb @@ -22,9 +22,9 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -63,10 +63,10 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> No change in requirements detected, installing from cache - remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -86,10 +86,10 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Requirements file has been changed, clearing cached dependencies - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -118,9 +118,9 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -214,9 +214,9 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip diff --git a/spec/hatchet/pipenv_spec.rb b/spec/hatchet/pipenv_spec.rb index c90092dbf..942bdb808 100644 --- a/spec/hatchet/pipenv_spec.rb +++ b/spec/hatchet/pipenv_spec.rb @@ -2,13 +2,13 @@ require_relative '../spec_helper' -RSpec.shared_examples 'builds using Pipenv with the requested Python version' do |python_version| - it "builds with Python #{python_version}" do +RSpec.shared_examples 'builds using Pipenv with the requested Python version' do |requested_version, resolved_version| + it "builds with Python #{requested_version}" do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock - remote: -----> Installing python-#{python_version} + remote: -----> Using Python #{requested_version} specified in Pipfile.lock + remote: -----> Installing Python #{resolved_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -19,24 +19,6 @@ end end -RSpec.shared_examples 'aborts the build with a runtime not available message (Pipenv)' do |requested_version| - it 'aborts the build with a runtime not available message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock - remote: - remote: ! Error: Requested runtime 'python-#{requested_version}' is not available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end -end - RSpec.describe 'Pipenv support' do context 'without a Pipfile.lock' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_no_lockfile') } @@ -46,9 +28,9 @@ expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected remote: ! No 'Pipfile.lock' found! We recommend you commit this into your repository. - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -76,9 +58,9 @@ # TODO: We should not be leaking the Pipenv installation into the app environment. expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -120,52 +102,25 @@ end end - context 'with a Pipfile.lock containing python_version 3.6' do - let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.6', allow_failure: true) } - - it 'aborts the build with an EOL message' do - app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) - remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock - remote: - remote: ! Error: Python 3.6 is no longer supported. - remote: ! - remote: ! Python 3.6 reached upstream end-of-life on December 23rd, 2021, and is - remote: ! therefore no longer receiving security updates: - remote: ! https://devguide.python.org/versions/#supported-versions - remote: ! - remote: ! As such, it is no longer supported by this buildpack. - remote: ! - remote: ! Please upgrade to a newer Python version. - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end - end - - context 'with a Pipfile.lock containing python_version 3.7' do + context 'with a Pipfile.lock containing an EOL python_version' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.7', allow_failure: true) } it 'aborts the build with an EOL message' do app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~OUTPUT)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock + remote: -----> Using Python 3.7 specified in Pipfile.lock remote: - remote: ! Error: Python 3.7 is no longer supported. + remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.7 reached upstream end-of-life on June 27th, 2023, and is + remote: ! Python 3.7 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, it is no longer supported by this buildpack. remote: ! - remote: ! Please upgrade to a newer Python version. + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the 'Pipfile.lock' file. remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -185,7 +140,7 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock + remote: -----> Using Python 3.8 specified in Pipfile.lock remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -196,7 +151,7 @@ remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-#{LATEST_PYTHON_3_8} + remote: -----> Installing Python #{LATEST_PYTHON_3_8} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -211,35 +166,53 @@ let(:allow_failure) { true } # We only support Python 3.8 on Heroku-20 and older. - include_examples 'aborts the build with a runtime not available message (Pipenv)', LATEST_PYTHON_3_8 + it 'aborts the build with a version not available message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.8 specified in Pipfile.lock + remote: + remote: ! Error: Python #{LATEST_PYTHON_3_8} is not available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end end end context 'with a Pipfile.lock containing python_version 3.9' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.9') } - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_9 + include_examples 'builds using Pipenv with the requested Python version', '3.9', LATEST_PYTHON_3_9 end context 'with a Pipfile.lock containing python_version 3.10' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.10') } - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_10 + include_examples 'builds using Pipenv with the requested Python version', '3.10', LATEST_PYTHON_3_10 end context 'with a Pipfile.lock containing python_version 3.11' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.11') } - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_11 + include_examples 'builds using Pipenv with the requested Python version', '3.11', LATEST_PYTHON_3_11 end context 'with a Pipfile.lock containing python_version 3.12' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_3.12') } - include_examples 'builds using Pipenv with the requested Python version', LATEST_PYTHON_3_12 + include_examples 'builds using Pipenv with the requested Python version', '3.12', LATEST_PYTHON_3_12 end - context 'with a Pipfile.lock containing python_full_version 3.12.2' do + # As well as testing `python_full_version`, this also tests: + # 1. That `python_full_version` takes precedence over `python_version`. + # 2. That Pipenv works on the oldest Python version supported by all stacks. + # 3. That the security update available message works for Pipenv too. + context 'with a Pipfile.lock containing python_full_version 3.9.0' do let(:allow_failure) { false } let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_full_version', allow_failure:) } @@ -247,12 +220,12 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock + remote: -----> Using Python 3.9.0 specified in Pipfile.lock remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_12} + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-3.12.2 + remote: -----> Installing Python 3.9.0 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -263,20 +236,55 @@ end end - context 'with a Pipfile.lock containing an invalid python_version', - skip: 'unknown python_version values are currently ignored (W-8104668)' do + context 'with a Pipfile.lock containing invalid JSON' do + let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_lockfile_invalid_json', allow_failure: true) } + + it 'fails the build' do + app.deploy do |app| + # The exact JQ error message varies between JQ versions, and thus across stacks. + expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + remote: -----> Python app detected + remote: + remote: ! Error: Cannot parse Pipfile.lock. + remote: ! + remote: ! A Pipfile.lock file was found, however, it could not be parsed: + remote: ! (jq: )?parse error: Invalid numeric literal at line 1, column 8 + remote: ! + remote: ! This is likely due to it not being valid JSON. + remote: ! + remote: ! Run 'pipenv lock' to regenerate/fix the lockfile. + remote: + remote: ! Push rejected, failed to compile Python app. + REGEX + end + end + end + + context 'with a Pipfile.lock containing an invalid python_version' do let(:app) { Hatchet::Runner.new('spec/fixtures/pipenv_python_version_invalid', allow_failure: true) } it 'fails the build' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock remote: - remote: ! Error: Requested runtime '^3.12' is not available for this stack (#{app.stack}). + remote: ! Error: Invalid Python version in Pipfile / Pipfile.lock. remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! The Python version specified in Pipfile / Pipfile.lock by the + remote: ! 'python_version' or 'python_full_version' field is not valid. + remote: ! + remote: ! The following version was found: + remote: ! ^3.12 + remote: ! + remote: ! However, the version must be specified as either: + remote: ! 1. '.' (recommended, for automatic security updates) + remote: ! 2. '..' (to pin to an exact Python version) + remote: ! + remote: ! Please update your 'Pipfile' to use a valid Python version and + remote: ! then run 'pipenv lock' to regenerate the lockfile. + remote: ! + remote: ! For more information, see: + remote: ! https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -291,12 +299,24 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock remote: - remote: ! Error: Requested runtime 'python-X.Y.Z' is not available for this stack (#{app.stack}). + remote: ! Error: Invalid Python version in Pipfile / Pipfile.lock. remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! The Python version specified in Pipfile / Pipfile.lock by the + remote: ! 'python_version' or 'python_full_version' field is not valid. + remote: ! + remote: ! The following version was found: + remote: ! 3.9.* + remote: ! + remote: ! However, the version must be specified as either: + remote: ! 1. '.' (recommended, for automatic security updates) + remote: ! 2. '..' (to pin to an exact Python version) + remote: ! + remote: ! Please update your 'Pipfile' to use a valid Python version and + remote: ! then run 'pipenv lock' to regenerate the lockfile. + remote: ! + remote: ! For more information, see: + remote: ! https://pipenv.pypa.io/en/latest/specifiers.html#specifying-versions-of-python remote: remote: ! Push rejected, failed to compile Python app. OUTPUT @@ -311,8 +331,8 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: -----> Installing python-#{LATEST_PYTHON_3_12} + remote: -----> Using Python #{LATEST_PYTHON_3_12} specified in runtime.txt + remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -332,9 +352,9 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -359,9 +379,9 @@ app.push! expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Using cached install of python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Using cached install of Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -380,8 +400,8 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) remote: -----> Python app detected - remote: -----> Using Python version specified in Pipfile.lock - remote: -----> Installing python-#{LATEST_PYTHON_3_12} + remote: -----> Using Python 3.12 specified in Pipfile.lock + remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 @@ -399,9 +419,9 @@ app.deploy do |app| expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX, Regexp::MULTILINE)) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing Pipenv #{PIPENV_VERSION} remote: -----> Installing SQLite3 diff --git a/spec/hatchet/python_update_warning_spec.rb b/spec/hatchet/python_update_warning_spec.rb index 284544c91..787f24a18 100644 --- a/spec/hatchet/python_update_warning_spec.rb +++ b/spec/hatchet/python_update_warning_spec.rb @@ -2,54 +2,20 @@ require_relative '../spec_helper' -RSpec.shared_examples 'warns there is a Python update available' do |requested_version, latest_version| - it 'warns there is a Python update available' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{latest_version} - remote: ! See: https://devcenter.heroku.com/articles/python-runtimes - remote: ! - remote: -----> Installing python-#{requested_version} - OUTPUT - end - end -end - -RSpec.shared_examples 'aborts the build without showing an update warning' do |requested_version| - it 'aborts the build without showing an update warning' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: - remote: ! Error: Requested runtime 'python-#{requested_version}' is not available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT - end - end -end - # NOTE: We use the oldest patch releases (ie the '.0' releases) since we also want to test against # the oldest Python versions available to users. This is particularly important given that older # patch releases will bundle older pip, and the buildpack uses that pip during bootstrapping. RSpec.describe 'Python update warnings' do - context 'with a runtime.txt containing python-3.8.0' do + context 'with a runtime.txt containing an outdated patch version that is also a deprecated major version' do let(:allow_failure) { false } - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.8_outdated', allow_failure:) } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated_and_deprecated', allow_failure:) } context 'when using Heroku-20', stacks: %w[heroku-20] do it 'warns about both the deprecated major version and the patch update' do app.deploy do |app| - expect(clean_output(app.output)).to match(Regexp.new(<<~REGEX)) + expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt + remote: -----> Using Python 3.8.0 specified in runtime.txt remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -61,11 +27,11 @@ remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_8} + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_8} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-3.8.0 - REGEX + remote: -----> Installing Python 3.8.0 + OUTPUT end end end @@ -74,31 +40,39 @@ let(:allow_failure) { true } # We only support Python 3.8 on Heroku-20 and older. - include_examples 'aborts the build without showing an update warning', '3.8.0' + it 'aborts the build without showing an update warning' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.8.0 specified in runtime.txt + remote: + remote: ! Error: Python 3.8.0 is not available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end end end - context 'with a runtime.txt containing python-3.9.0' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9_outdated') } - - include_examples 'warns there is a Python update available', '3.9.0', LATEST_PYTHON_3_9 - end - - context 'with a runtime.txt containing python-3.10.0' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.10_outdated') } - - include_examples 'warns there is a Python update available', '3.10.0', LATEST_PYTHON_3_10 - end - - context 'with a runtime.txt containing python-3.11.0' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11_outdated') } - - include_examples 'warns there is a Python update available', '3.11.0', LATEST_PYTHON_3_11 - end - - context 'with a runtime.txt containing python-3.12.0' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.12_outdated') } + context 'with a runtime.txt containing an outdated patch version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_outdated') } - include_examples 'warns there is a Python update available', '3.12.0', LATEST_PYTHON_3_12 + it 'warns there is a Python update available' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.9.0 specified in runtime.txt + remote: ! + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_9} + remote: ! See: https://devcenter.heroku.com/articles/python-runtimes + remote: ! + remote: -----> Installing Python 3.9.0 + OUTPUT + end + end end end diff --git a/spec/hatchet/python_version_spec.rb b/spec/hatchet/python_version_spec.rb index 44038a080..e0d01049d 100644 --- a/spec/hatchet/python_version_spec.rb +++ b/spec/hatchet/python_version_spec.rb @@ -2,37 +2,19 @@ require_relative '../spec_helper' -RSpec.shared_examples 'builds with the requested Python version' do |python_version| - it "builds with Python #{python_version}" do +RSpec.shared_examples 'builds with the requested Python version' do |requested_version| + it "builds with Python #{requested_version}" do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: -----> Installing python-#{python_version} + remote: -----> Using Python #{requested_version} specified in runtime.txt + remote: -----> Installing Python #{requested_version} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip remote: Collecting urllib3 (from -r requirements.txt (line 1)) OUTPUT - expect(app.run('python -V')).to include("Python #{python_version}") - end - end -end - -RSpec.shared_examples 'aborts the build with a runtime not available message' do |requested_runtime| - it 'aborts the build with a runtime not available message' do - app.deploy do |app| - expect(clean_output(app.output)).to include(<<~OUTPUT) - remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: - remote: ! Error: Requested runtime '#{requested_runtime}' is not available for this stack (#{app.stack}). - remote: ! - remote: ! For a list of the supported Python versions, see: - remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes - remote: - remote: ! Push rejected, failed to compile Python app. - OUTPUT + expect(app.run('python -V')).to include("Python #{requested_version}") end end end @@ -47,9 +29,9 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the buildpack default: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the buildpack default: Python #{DEFAULT_PYTHON_MAJOR_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} OUTPUT end end @@ -68,14 +50,14 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-3.12.3 + remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_12} + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! remote: -----> No change in requirements detected, installing from cache - remote: -----> Using cached install of python-3.12.3 + remote: -----> Using cached install of Python 3.12.3 OUTPUT expect(app.run('python -V')).to include('Python 3.12.3') end @@ -83,24 +65,25 @@ end end - context 'when runtime.txt contains python-3.6.15' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.6', allow_failure: true) } + context 'when runtime.txt contains python-2.7.18' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_2.7', allow_failure: true) } it 'aborts the build with an EOL message' do app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt + remote: -----> Using Python 2.7.18 specified in runtime.txt remote: - remote: ! Error: Python 3.6 is no longer supported. + remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.6 reached upstream end-of-life on December 23rd, 2021, and is + remote: ! Python 2.7 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, it is no longer supported by this buildpack. remote: ! - remote: ! Please upgrade to a newer Python version. + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the 'runtime.txt' file. remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -118,17 +101,18 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt + remote: -----> Using Python 3.7.17 specified in runtime.txt remote: - remote: ! Error: Python 3.7 is no longer supported. + remote: ! Error: The requested Python version has reached end-of-life. remote: ! - remote: ! Python 3.7 reached upstream end-of-life on June 27th, 2023, and is + remote: ! Python 3.7 has reached its upstream end-of-life, and is remote: ! therefore no longer receiving security updates: remote: ! https://devguide.python.org/versions/#supported-versions remote: ! remote: ! As such, it is no longer supported by this buildpack. remote: ! - remote: ! Please upgrade to a newer Python version. + remote: ! Please upgrade to a newer Python version by updating the + remote: ! version configured via the 'runtime.txt' file. remote: ! remote: ! For a list of the supported Python versions, see: remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes @@ -148,7 +132,7 @@ app.deploy do |app| expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt + remote: -----> Using Python #{LATEST_PYTHON_3_8} specified in runtime.txt remote: ! remote: ! Python 3.8 will reach its upstream end-of-life in October 2024, at which remote: ! point it will no longer receive security updates: @@ -159,7 +143,7 @@ remote: ! Upgrade to a newer Python version as soon as possible to keep your app secure. remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: -----> Installing python-#{LATEST_PYTHON_3_8} + remote: -----> Installing Python #{LATEST_PYTHON_3_8} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -174,7 +158,21 @@ let(:allow_failure) { true } # We only support Python 3.8 on Heroku-20 and older. - include_examples 'aborts the build with a runtime not available message', "python-#{LATEST_PYTHON_3_8}" + it 'aborts the build with a version not available message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python #{LATEST_PYTHON_3_8} specified in runtime.txt + remote: + remote: ! Error: Python #{LATEST_PYTHON_3_8} is not available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end end end @@ -202,10 +200,88 @@ include_examples 'builds with the requested Python version', LATEST_PYTHON_3_12 end - context 'when runtime.txt contains an invalid python version string' do + context 'when runtime.txt contains an invalid Python version string' do let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_invalid', allow_failure: true) } - include_examples 'aborts the build with a runtime not available message', 'python-3.12.0invalid' + it 'aborts the build with an invalid runtime.txt message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: + remote: ! Error: Invalid Python version in runtime.txt. + remote: ! + remote: ! The Python version specified in 'runtime.txt' is not in + remote: ! the correct format. + remote: ! + remote: ! The following file contents were found: + remote: ! python-3.12.0invalid + remote: ! + remote: ! However, the version string must begin with a 'python-' prefix, + remote: ! followed by the version specified as '..'. + remote: ! Comments are not supported. + remote: ! + remote: ! For example, to request Python 3.12.7, use: + remote: ! python-3.12.7 + remote: ! + remote: ! Please update 'runtime.txt' to use a valid version string, or + remote: ! else remove the file to instead use the default version + remote: ! (currently Python 3.12.7). + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when runtime.txt contains an non-existent Python major version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_major', allow_failure: true) } + + it 'aborts the build with an invalid runtime.txt message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.999.0 specified in runtime.txt + remote: + remote: ! Error: The requested Python version is not recognised. + remote: ! + remote: ! The requested Python version 3.999 is not recognised. + remote: ! + remote: ! Check that this Python version has been officially released, + remote: ! and that the Python buildpack has added support for it: + remote: ! https://devguide.python.org/versions/#supported-versions + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: ! + remote: ! If it has, make sure that you are using the latest version + remote: ! of this buildpack: + remote: ! https://devcenter.heroku.com/articles/python-support#checking-the-python-buildpack-version + remote: ! + remote: ! Otherwise, switch to a supported version (such as Python 3.12) + remote: ! by updating the version configured via the 'runtime.txt' file. + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end + end + + context 'when runtime.txt contains a non-existent Python patch version' do + let(:app) { Hatchet::Runner.new('spec/fixtures/python_version_non_existent_patch', allow_failure: true) } + + it 'aborts the build with a version not available message' do + app.deploy do |app| + expect(clean_output(app.output)).to include(<<~OUTPUT) + remote: -----> Python app detected + remote: -----> Using Python 3.12.999 specified in runtime.txt + remote: + remote: ! Error: Python 3.12.999 is not available for this stack (#{app.stack}). + remote: ! + remote: ! For a list of the supported Python versions, see: + remote: ! https://devcenter.heroku.com/articles/python-support#supported-runtimes + remote: + remote: ! Push rejected, failed to compile Python app. + OUTPUT + end + end end context 'when runtime.txt contains stray whitespace' do @@ -215,7 +291,7 @@ end context 'when the requested Python version has changed since the last build' do - let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.11') } + let(:app) { Hatchet::Runner.new('spec/fixtures/python_3.9') } it 'builds with the new Python version after removing the old install' do app.deploy do |app| @@ -225,10 +301,10 @@ # TODO: The output shouldn't say "installing from cache", since it's not. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> Using Python version specified in runtime.txt - remote: -----> Python version has changed from python-#{LATEST_PYTHON_3_11} to python-#{LATEST_PYTHON_3_12}, clearing cache + remote: -----> Using Python #{LATEST_PYTHON_3_12} specified in runtime.txt + remote: -----> Python version has changed from #{LATEST_PYTHON_3_9} to #{LATEST_PYTHON_3_12}, clearing cache remote: -----> No change in requirements detected, installing from cache - remote: -----> Installing python-#{LATEST_PYTHON_3_12} + remote: -----> Installing Python #{LATEST_PYTHON_3_12} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip diff --git a/spec/hatchet/stack_spec.rb b/spec/hatchet/stack_spec.rb index 35f05dfbf..0135e0628 100644 --- a/spec/hatchet/stack_spec.rb +++ b/spec/hatchet/stack_spec.rb @@ -22,15 +22,15 @@ # TODO: The requirements output shouldn't say "installing from cache", since it's not. expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-3.12.3 + remote: -----> No Python version was specified. Using the same version as the last build: Python 3.12.3 remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: ! - remote: ! A Python security update is available! Upgrade as soon as possible to: python-#{LATEST_PYTHON_3_12} + remote: ! A Python security update is available! Upgrade as soon as possible to: Python #{LATEST_PYTHON_3_12} remote: ! See: https://devcenter.heroku.com/articles/python-runtimes remote: ! remote: -----> Stack has changed from heroku-22 to heroku-24, clearing cache remote: -----> No change in requirements detected, installing from cache - remote: -----> Installing python-3.12.3 + remote: -----> Installing Python 3.12.3 remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip @@ -51,11 +51,11 @@ app.push! expect(clean_output(app.output)).to include(<<~OUTPUT) remote: -----> Python app detected - remote: -----> No Python version was specified. Using the same version as the last build: python-#{DEFAULT_PYTHON_VERSION} + remote: -----> No Python version was specified. Using the same version as the last build: Python #{DEFAULT_PYTHON_FULL_VERSION} remote: To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes remote: -----> Stack has changed from heroku-24 to heroku-22, clearing cache remote: -----> No change in requirements detected, installing from cache - remote: -----> Installing python-#{DEFAULT_PYTHON_VERSION} + remote: -----> Installing Python #{DEFAULT_PYTHON_FULL_VERSION} remote: -----> Installing pip #{PIP_VERSION}, setuptools #{SETUPTOOLS_VERSION} and wheel #{WHEEL_VERSION} remote: -----> Installing SQLite3 remote: -----> Installing requirements with pip diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index cfbb95da1..3c0632a64 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -14,7 +14,8 @@ LATEST_PYTHON_3_10 = '3.10.15' LATEST_PYTHON_3_11 = '3.11.10' LATEST_PYTHON_3_12 = '3.12.7' -DEFAULT_PYTHON_VERSION = LATEST_PYTHON_3_12 +DEFAULT_PYTHON_FULL_VERSION = LATEST_PYTHON_3_12 +DEFAULT_PYTHON_MAJOR_VERSION = DEFAULT_PYTHON_FULL_VERSION.gsub(/\.\d+$/, '') # The requirement versions are effectively buildpack constants, however, we want # Dependabot to be able to update them, which requires that they be in requirements