Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: remove Python 3.7 support #5191

Merged
merged 18 commits into from
Jul 30, 2024
Merged

feat: remove Python 3.7 support #5191

merged 18 commits into from
Jul 30, 2024

Conversation

rwgk
Copy link
Collaborator

@rwgk rwgk commented Jun 24, 2024

Description

Remove Python 3.7 support entirely. (Official end-of-life: 2023-06-27)

NOTE: We're losing testing Clang versions 3, 7, 9, 10, and GCC version 7, but we're deliberately deferring reviewing the C++ code to remove related #ifdefs.

See also (below): #5191 (comment)


For easy future reference, these were the git grep commands used to pin-point lines that may need to be changed:

#! /bin/bash
set -x
git grep 0x0307
git grep 0x0308
git grep PY_MINOR_VERSION
git grep PYPY_VERSION
git grep -I '[^0-9]37[^0-9]'
git grep -I '[^0-9]38[^0-9]'
git grep -I -E '3\.7'
git grep -I -E '3\.8'
git grep -I -E '\(3, 7'
git grep -I -E '\(3, 8'
git grep -I -E '3[^A-Za-z0-9.]+7'
git grep -I -E '3[^A-Za-z0-9.]+8'

Suggested changelog entry:

Support for Python 3.7 was removed. (Official end-of-life: 2023-06-27)

@rwgk
Copy link
Collaborator Author

rwgk commented Jun 24, 2024

Overview of all compiler versions used/lost(-)/gained(+) before this PR, and with this PR @ commit 2f24de5:

 AppleClang 15.0.0.15000040
-Clang 3.6.2    16 Jul 2015
-Clang 3.7.1     5 Jan 2016
-Clang 3.9.1    23 Dec 2016
 Clang 5.0.2
-Clang 7.1.0    10 May 2019
-Clang 9.0.1    20 Dec 2019
-Clang 10.0.1    6 Aug 2020
 Clang 11.1.0
 Clang 12.0.1
 Clang 13.0.1
 Clang 14.0.6
 Clang 15.0.7
 Clang 16.0.6
 Clang 18.1.6 with GNU-like command-line
 Clang 18.1.7
 Clang 19.0.0
-GNU 7.5.0      November 14, 2019
-GNU 8.3.0      February 22, 2019
 GNU 8.5.0
 GNU 9.4.0
+GNU 10.2.1
 GNU 10.5.0
 GNU 11.4.0
 GNU 11.4.1
 GNU 12.4.0
 GNU 13.2.0
 GNU 13.2.1
 GNU 13.3.0
 Intel 2021.10.0.20230609
 MSVC 19.29.30154.0
 MSVC 19.40.33811.0
 NVHPC 23.5.0

@rwgk
Copy link
Collaborator Author

rwgk commented Jun 24, 2024

Tracking simple findings:

Ralf W. Grosse-Kunstleve added 10 commits June 24, 2024 15:22
This reverts commit 518515a.
Revert "Try running in ubuntu:18.04 container under ubuntu-latest (to get GCC 7)"

This reverts commit b203a29.
Removes two changes made under pybind#3702
```
...
-- The CXX compiler identification is Clang 15.0.7
...
- Found Python: /usr/bin/python3.9 (found suitable version "3.9.2", minimum required is "3.7") found components: Interpreter Development.Module Development.Embed
...
/__w/pybind11/pybind11/include/pybind11/gil.h:150:13: error: 'auto key' can be declared as 'auto *key' [readability-qualified-auto,-warnings-as-errors]
            auto key = internals.tstate;
            ^~~~~
            auto *
/__w/pybind11/pybind11/include/pybind11/gil.h:174:13: error: 'auto key' can be declared as 'auto *key' [readability-qualified-auto,-warnings-as-errors]
            auto key = detail::get_internals().tstate;
            ^~~~~
            auto *
```
@henryiii
Copy link
Collaborator

I think this next release might be a good time to remove C++11 support and increase our minimum supported compilers.

@rwgk
Copy link
Collaborator Author

rwgk commented Jun 26, 2024

I think this next release might be a good time to remove C++11 support and increase our minimum supported compilers.

Awesome. That's what I was thinking, too. I'll work on this PR more after the 2.13 release has baked in a bit.

@henryiii
Copy link
Collaborator

I think we can also bump CMake support. Personally, I've been going 3.15+ everywhere as that is what was required to build scikit-build-core. Though another option is 3.12+ (FindPython). I think we has someone request 3.10 or 3.11 before, might be worth checking.

If we did go 3.15+, we maybe could start to phase out classic FindPythonInterp support.

@rwgk
Copy link
Collaborator Author

rwgk commented Jun 26, 2024

If we did go 3.15+, we maybe could start to phase out classic FindPythonInterp support.

I just looked up: CMake 3.15.0 publication date was 17 July 2019

Does that look correct? — That's already 5 years ago (almost at least), and before Python 3.8 was released (Oct 2019). — I think it's much more likely to be helpful than harmful to make CMake 3.15 the minimum, and shed cruft.

@henryiii
Copy link
Collaborator

Here's the support table for CMake, with darker red for older than 3.15:

cmake versions

@rwgk
Copy link
Collaborator Author

rwgk commented Jun 27, 2024

My thinking:

  • I want to wait about one week, JIC we need more 2.13 patch releases.
  • Then I'll make a fresh pass through this PR, looking for 3.7 and 3.8 version numbers of all sorts once more.
  • Then merge this PR, documenting in the PR description that we're losing testing with Clang 3.6 through 10, and GCC 7 as a consequence of dropping Python 3.7 support, but we're deliberately deferring cleaning up ifdefs for another release.
  • Then have a follow-on PR to bump the cmake minimum to 3.15 (independently from dropping Python 3.7) and clean out cmake cruft.

@henryiii
Copy link
Collaborator

We’ve already made a 2.13 branch, master is 2.14. No rush, but waiting isn’t required. Does save ci time though if the branches are functionally identical.

No need to defer cleanup, we have plenty of time for this to sit in master. We should specify and test minimum versions where possible. I’m hoping we get a MSVC bump. :)

But that can also be a follow-up.

@rwgk
Copy link
Collaborator Author

rwgk commented Jun 28, 2024

but waiting isn’t required.

it more prioritizing than waiting

this is up high on my list though; not a lot more to do

Ralf W. Grosse-Kunstleve added 2 commits July 27, 2024 19:48
…s below:

```
git_grep_37_38.sh |& sort | uniq -c
```

With git_grep_37_38.sh:

```
set -x
git grep 0x0307
git grep 0x0308
git grep PY_MINOR_VERSION
git grep PYPY_VERSION
git grep -I -E '3\.7'
git grep -I -E '3\.8'
git grep -I -E '\(3, 7'
git grep -I -E '\(3, 8'
git grep -I -E '3[^A-Za-z0-9.]+7'
git grep -I -E '3[^A-Za-z0-9.]+8'
```

Output:

```
      1 .appveyor.yml:    $env:CMAKE_INCLUDE_PATH = "eigen-3.3.7;$env:CMAKE_INCLUDE_PATH"
      1 .appveyor.yml:    7z x eigen-3.3.7.zip -y > $null
      1 .appveyor.yml:    Start-FileDownload 'https://gitlab.com/libeigen/eigen/-/archive/3.3.7/eigen-3.3.7.zip'
      1 CMakeLists.txt:    # Bug in macOS CMake < 3.7 is unable to download catch
      1 CMakeLists.txt:  elseif(WINDOWS AND CMAKE_VERSION VERSION_LESS 3.8)
      1 CMakeLists.txt:  if(OSX AND CMAKE_VERSION VERSION_LESS 3.7)
      1 CMakeLists.txt:    message(WARNING "CMAKE 3.7+ needed on macOS to download catch, and newer HIGHLY recommended")
      1 CMakeLists.txt:    message(WARNING "CMAKE 3.8+ tested on Windows, previous versions untested")
      1 CMakeLists.txt:    # Only tested with 3.8+ in CI.
      1 docs/advanced/functions.rst:Python 3.8 introduced a new positional-only argument syntax, using ``/`` in the
      1 docs/changelog.rst:* Adapt pybind11 to a C API convention change in Python 3.8. `pybind#1950
      1 docs/changelog.rst:* Allow thread termination to be avoided during shutdown for CPython 3.7+ via
      1 docs/changelog.rst:  considered as conversion, consistent with Python 3.8+.
      1 docs/changelog.rst:  CPython 3.8 and 3.9 debug builds.
      1 docs/changelog.rst:* Enum now has an ``__index__`` method on Python <3.8 too.
      1 docs/changelog.rst:  on Python 3.8. `pybind#1780 <https://github.com/pybind/pybind11/pull/1780>`_.
      1 docs/changelog.rst:* PyPy 3.10 support was added, PyPy 3.7 support was dropped.
      2 docs/changelog.rst:* Support PyPy 7.3.7 and the PyPy3.8 beta. Test python-3.11 on PRs with the
      1 docs/changelog.rst:* Use ``macos-13`` (Intel) for CI jobs for now (will drop Python 3.7 soon).
      1 docs/changelog.rst:* Use new Python 3.7 Thread Specific Storage (TSS) implementation if available.
      1 docs/compiling.rst:    cmake -DPYBIND11_PYTHON_VERSION=3.8 ..
      1 docs/compiling.rst:    find_package(Python 3.8 COMPONENTS Interpreter Development REQUIRED)
      1 docs/limitations.rst:- PyPy3 7.3.1 and 7.3.2 have issues with several tests on 32-bit Windows.
      1 docs/requirements.txt:idna==3.7 \
      1 + git grep 0x0307
      1 + git grep 0x0308
      1 + git grep -I -E '\(3, 7'
      1 + git grep -I -E '3\.7'
      1 + git grep -I -E '\(3, 8'
      1 + git grep -I -E '3\.8'
      1 + git grep -I -E '3[^A-Za-z0-9.]+7'
      1 + git grep -I -E '3[^A-Za-z0-9.]+8'
      1 + git grep PY_MINOR_VERSION
      1 + git grep PYPY_VERSION
      2 .github/workflows/ci.yml:        - '3.8'
      1 .github/workflows/ci.yml:        - 3.8
      1 .github/workflows/ci.yml:    - name: Add Python 3.8
      1 .github/workflows/ci.yml:        - 'pypy-3.8'
      2 .github/workflows/ci.yml:            python: '3.8'
      1 .github/workflows/ci.yml:          - python: '3.8'
      1 .github/workflows/ci.yml:          - python: 3.8
      1 .github/workflows/ci.yml:            python: 'pypy-3.8'
      1 .github/workflows/configure.yml:          cmake: "3.8"
      1 .github/workflows/configure.yml:    name: 🐍 3.8 • CMake ${{ matrix.cmake }} • ${{ matrix.runs-on }}
      1 .github/workflows/configure.yml:    - name: Setup Python 3.8
      1 .github/workflows/configure.yml:        python-version: 3.8
      1 .github/workflows/pip.yml:    name: 🐍 3.8 • 📦 & 📦 tests • ubuntu-latest
      1 .github/workflows/pip.yml:    name: 🐍 3.8 • 📦 tests • windows-latest
      2 .github/workflows/pip.yml:    - name: Setup 🐍 3.8
      2 .github/workflows/pip.yml:        python-version: 3.8
      2 include/pybind11/cast.h:#if !defined(PYPY_VERSION)
      2 include/pybind11/cast.h:#if defined(PYPY_VERSION)
      2 include/pybind11/cast.h:            // PyPy: 7.3.7's 3.8 does not implement PyLong_*'s __index__ calls.
      5 include/pybind11/detail/class.h:#if !defined(PYPY_VERSION)
      1 include/pybind11/detail/class.h:#if defined(PYPY_VERSION)
      1 include/pybind11/detail/class.h:    // This was not needed before Python 3.8 (Python issue 35810)
      1 include/pybind11/detail/common.h:    && !defined(PYPY_VERSION) && !defined(PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF)
      2 include/pybind11/detail/common.h:#    error "PYTHON < 3.8 IS UNSUPPORTED. pybind11 v2.13 was the last to support Python 3.7."
      1 include/pybind11/detail/common.h:#if defined(PYPY_VERSION) && !defined(PYBIND11_SIMPLE_GIL_MANAGEMENT)
      1 include/pybind11/detail/common.h:#if PY_VERSION_HEX < 0x03080000
      1 include/pybind11/detail/common.h:            = PYBIND11_TOSTRING(PY_MAJOR_VERSION) "." PYBIND11_TOSTRING(PY_MINOR_VERSION);        \
      1 include/pybind11/detail/internals.h:        // called. PYBIND11_TLS_FREE is PyThread_tss_free on python 3.7+. On older python, it does
      1 include/pybind11/detail/internals.h:#if PYBIND11_INTERNALS_VERSION <= 4 || defined(PYPY_VERSION)
      1 include/pybind11/detail/internals.h:// The old Python Thread Local Storage (TLS) API is deprecated in Python 3.7 in favor of the new
      1 include/pybind11/detail/type_caster_base.h:#if defined(PYPY_VERSION)
      1 include/pybind11/embed.h:#    define PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX (0x03080000)
      1 include/pybind11/embed.h:#if defined(PYPY_VERSION)
      1 include/pybind11/eval.h:    // globals if not yet present.  Python 3.8 made PyRun_String behave
      2 include/pybind11/eval.h:#if defined(PYPY_VERSION)
      2 include/pybind11/eval.h:    // was missing from PyPy3.8 7.3.7.
      2 include/pybind11/gil.h:    /// allowed during shutdown. Check _Py_IsFinalizing() on Python 3.7+, and
      1 include/pybind11/pybind11.h:#if !defined(PYPY_VERSION)
      4 include/pybind11/pybind11.h:#if !defined(PYPY_VERSION) && PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION == 9
      1 include/pybind11/pytypes.h:#endif //! defined(PYPY_VERSION)
      2 include/pybind11/pytypes.h:#if !defined(PYPY_VERSION)
      1 include/pybind11/pytypes.h:#    if defined(PYPY_VERSION_NUM) && PYPY_VERSION_NUM < 0x07030a00
      1 include/pybind11/pytypes.h:#ifdef PYPY_VERSION
      1 include/pybind11/stl/filesystem.h:#    if !defined(PYPY_VERSION)
      2 pybind11/__init__.py:if sys.version_info < (3, 8):
      2 pybind11/__init__.py:    msg = "pybind11 does not support Python < 3.8. v2.13 was the last release supporting Python 3.7."
      1 pyproject.toml:master.py-version = "3.8"
      1 pyproject.toml:python_version = "3.8"
      1 README.rst:lines of code and depend on Python (3.8+, or PyPy) and the C++
      2 README.rst:- Python 3.8+, and PyPy3 7.3 are supported with an implementation-agnostic
      1 setup.cfg:    Programming Language :: Python :: 3.8
      1 setup.cfg:python_requires = >=3.8
      1 setup.py:# TODO: use literals & overload (typing extensions or Python 3.8)
      1 tests/CMakeLists.txt:if(NOT CMAKE_VERSION VERSION_LESS 3.8)
      2 tests/constructor_stats.h:#if defined(PYPY_VERSION)
      1 tests/env.py:    doesn't work on CPython 3.8.0 with pytest==3.3.2 on Ubuntu 18.04 (pybind#2922).
      1 tests/requirements.txt:build~=1.0; python_version>="3.8"
      1 tests/requirements.txt:numpy~=1.21.5; platform_python_implementation!="PyPy" and python_version>="3.8" and python_version<"3.10"
      1 tests/requirements.txt:numpy~=1.23.0; python_version=="3.8" and platform_python_implementation=="PyPy"
      1 tests/test_buffers.py:    env.PYPY, reason="PyPy 7.3.7 doesn't clear this anymore", strict=False
      1 tests/test_builtin_casters.py:    # Before Python 3.8, `PyLong_AsLong` does not pick up on `obj.__index__`,
      2 tests/test_builtin_casters.py:    if (3, 8) <= sys.version_info < (3, 10) and env.CPYTHON:
      4 tests/test_builtin_casters.py:    # TODO: PyPy 3.8 does not behave like CPython 3.8 here yet (7.3.7)
      1 tests/test_callbacks.py:    assert m.test_callback3(z.double) == "func(43) = 86"
      2 tests/test_call_policies.cpp:#if !defined(PYPY_VERSION)
      1 tests/test_chrono.py:    diff = m.test_chrono_float_diff(43.789012, 1.123456)
      1 tests/test_constants_and_functions.py:    assert m.f3(86) == 89
      1 tests/test_eigen_matrix.py:    a_copy3[8, 1] = 11
      1 tests/test_eigen_matrix.py:    assert np.all(cornersc == np.array([[1.0, 3], [7, 9]]))
      1 tests/test_eigen_matrix.py:    assert np.all(cornersr == np.array([[1.0, 3], [7, 9]]))
      1 tests/test_eigen_matrix.py:        mymat = chol(np.array([[1.0, 2, 4], [2, 13, 23], [4, 23, 77]]))
      1 tests/test_exceptions.py:    if hasattr(pytest, unraisable):  # Python >= 3.8 and pytest >= 6
      2 tests/test_exceptions.py:@pytest.mark.xfail(env.PYPY, reason="Failure on PyPy 3.8 (7.3.7)", strict=False)
      1 tests/test_factory_constructors.py:    assert [i.alive() for i in cstats] == [13, 7]
      1 tests/test_kwargs_and_defaults.cpp:#ifdef PYPY_VERSION
      1 tests/test_local_bindings.py:    assert i1.get3() == 8
      1 tests/test_methods_and_attributes.cpp:#if !defined(PYPY_VERSION)
      1 tests/test_numpy_array.py:    a = np.arange(3 * 7 * 2) + 1
      1 tests/test_numpy_array.py:    assert str(excinfo.value) == "cannot reshape array of size 42 into shape (3,7,1)"
      2 tests/test_numpy_array.py:    assert x.shape == (3, 7, 2)
      2 tests/test_numpy_array.py:        m.reshape_tuple(a, (3, 7, 1))
      2 tests/test_numpy_array.py:    x = m.reshape_tuple(a, (3, 7, 2))
      1 tests/test_numpy_vectorize.py:    assert np.isclose(m.vectorized_func3(np.array(3 + 7j)), [6 + 14j])
      1 tests/test_pickling.cpp:#if !defined(PYPY_VERSION)
      1 tests/test_pytypes.cpp:#if (defined(__APPLE__) && defined(__clang__)) || defined(PYPY_VERSION)
      1 tests/test_smart_ptr.cpp:    m.def("make_myobject3_1", []() { return new MyObject3(8); });
      1 tests/test_smart_ptr.py:    assert cstats.values() == ["MyObject3[9]", "MyObject3[8]", "MyObject3[9]"]
      1 tests/test_stl_binders.py:    assert v_int2 == m.VectorInt([0, 99, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 88])
      1 tests/test_stl_binders.py:    assert v_int2 == m.VectorInt([0, 99, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 88, 4])
      1 tests/test_type_caster_pyobject_ptr.cpp:#if !defined(PYPY_VERSION) // It is not worth the trouble doing something special for PyPy.
      1 tools/FindPythonLibsNew.cmake:  set(PythonLibsNew_FIND_VERSION "3.8")
      1 tools/JoinPaths.cmake:# https://docs.python.org/3.7/library/os.path.html#os.path.join
      1 tools/pybind11NewTools.cmake:    Python 3.8 REQUIRED COMPONENTS ${_pybind11_interp_component} ${_pybind11_dev_component}
      1 tools/pybind11NewTools.cmake:# Python debug libraries expose slightly different objects before 3.8
      1 tools/pybind11Tools.cmake:    "3.12;3.11;3.10;3.9;3.8"
      1 tools/pybind11Tools.cmake:    if(NOT DEFINED PYPY_VERSION)
      1 tools/pybind11Tools.cmake:    message(STATUS "PYPY ${PYPY_VERSION} (Py ${PYTHON_VERSION})")
      1 tools/pybind11Tools.cmake:# Python debug libraries expose slightly different objects before 3.8
      1 tools/pybind11Tools.cmake:      set(PYPY_VERSION
```
@rwgk rwgk marked this pull request as ready for review July 28, 2024 17:45
@rwgk rwgk requested a review from henryiii as a code owner July 28, 2024 17:45
@@ -30,7 +30,7 @@ ignore_missing_imports = true


[tool.pylint]
master.py-version = "3.7"
master.py-version = "3.8"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ruff has a Python version too, right? otherwise UP036 would complain above

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it does: target-version = "py37"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will go away if we move to scikit-build-core after 0.10 comes out. ;)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this @Skylion007!

Fixed with commit 49580cf

That made me realize we need to add something like this to the list of useful git grep commands, for the next time when we want to drop a Python version:

git grep -I '[^0-9]37[^0-9]'
git grep -I '[^0-9]38[^0-9]'

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will also go away if define the minimum python version in pyproject.toml standard location instead of setup.cfg :)

@rwgk
Copy link
Collaborator Author

rwgk commented Jul 30, 2024

Thanks @Skylion007 !

@rwgk rwgk merged commit 7233072 into pybind:master Jul 30, 2024
78 checks passed
@rwgk rwgk deleted the rm_py37 branch July 30, 2024 16:18
@github-actions github-actions bot added the needs changelog Possibly needs a changelog entry label Jul 30, 2024
@henryiii henryiii removed the needs changelog Possibly needs a changelog entry label Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants