Add buildpack for pyproject.toml to configure container image

[rgaiacs]

Related to #1427

This is not yet ready for review!

The code is based on the Pipfile buildpack. We probably want to refactor some portions to avoid code duplication.

[rgaiacs]

577cb1b implements a minimal working version. It was tested with https://github.com/rgaiacs/binder-examples-pyproject.

Required changes before merge

• To parse the pyproject.toml, we require tomllib that is part of Python 3.11 standard library. This means that we will have to change the minimal version of Python required by repo2docker to Python 3.11 or later.

Changes that can be done in another pull request

• Document the use of pyproject.toml
• Migrate https://github.com/rgaiacs/binder-examples-pyproject to https://github.com/binder-examples/pyproject

[minrk]

Can we do:

Suggested change

	"""PATH="${KERNEL_PYTHON_PREFIX}/bin:$PATH" \\
	pip install --no-cache-dir --editable .
	""",
	"${KERNEL_PYTHON_PREFIX}/bin/python3 -m pip install --no-cache-dir --editable .",

to be even more explicit than relying on $PATH?

[rgaiacs]

All tests are passing after support to Python < 3.11 was dropped.

[manics]

We probably want to refactor some portions to avoid code duplication.

I've got a PR that refactors the handling of runtime.txt
#1428

[rgaiacs]

I rebased this pull request and I made minor changes to reduce the code duplication. It need to be reviewed again.

[rgaiacs]

Can we merge this? Or should we talk about it in jupyterhub/team-compass#814?

[minrk]

To parse the pyproject.toml, we require tomllib that is part of Python 3.11 standard library.

We can depend on tomli for Python < 3.11, which is the package that became tomllib in 3.11, so we should be able to:

try:
    import tomllib
except ImportError:
    import tomli as tomllib

without losing any supported environments.

I think reverting the baseline version change and depending on tomli, and fixing detection of pyproject.toml contents, I think this should be all set.

[minrk]

As discussed here, we should not consider the presence of pyproject.toml sufficient. We should return False if pyproject.toml does not contain both project and build-system.requires.

[rgaiacs]

I addressed the comments from @minrk

• use tomli to continue supporting Python < 3.11
• check for project and build-system.requires in the pyproject.toml

I also added

• single test to check if pyproject.toml were used
• list pyproject.toml in the documentation

I suggest to add more tests in another pull request.

[manics]

Suggested change

	To install your reprository like a Python package, you may include a
	`pyproject.toml` file. `repo2docker`install `pyproject.toml` files by running
	To install your repository like a Python package, you may include a
	`pyproject.toml` file. `repo2docker`installs `pyproject.toml` files by running

[manics]

Suggested change

	Support to pyproject.toml was added to pip v10.0,
	Support for pyproject.toml was added to pip v10.0,

[minrk]

This implementation looks great to me. My only comment is that I don't think we necessarily need a new buildpack to do this - the assemble steps, etc. should be identical to when setup.py is present. The only effect it should have is detecting the Python version, so it seems to me that it would be simpler to keep everything consistent if this logic is added to the python buildpack, rather than a whole new buildpack that should do all the same things.

But we can also do that as a simplification later. What do you think?

[minrk]

missing space:

Suggested change

	`pyproject.toml` file. `repo2docker`installs `pyproject.toml` files by running
	`pyproject.toml` file. `repo2docker` installs `pyproject.toml` files by running

[minrk]

this condition is always true at this point, we can avoid checking it again:

Suggested change

	return os.path.exists(pyproject_file)
	return True

[rgaiacs]

the assemble steps, etc. should be identical to when setup.py is present.

I don't like code duplication but my line of thoughts was "when we decided to drop support to setup.py, we can remove the setup.py buildpack instead of review all the code with setup.py and pyproject.toml.

Tomorrow, I will refactor the code so that there are only one Python buildpack.

[minrk]

we can remove the setup.py buildpack

We don't have a setup.py buildpack. setup.py is one small part of the python buildpack. All of the steps we take if setup.py is detected are identical to the steps taken if pyproject.toml is detected (we never invoke setup.py directly, only pip install .).

So the real change here ought to be changing the "should we run pip install ." in the Python buildpack from "setup.py exists" to a more general "is it a python package?" which checks:

• setup.py exists, OR
• pyproject exists and contains project info

Then the only other change is detecting Python version from project info, I think, at a lower priority than runtime.txt, binder dir, etc.

The rest of python buildpack should be unmodified, right?

[rgaiacs]

b755b7a merges PyprojectBuildPack into PythonBuildPack.

@minrk how much safe net we need for the users? The code only checks for the project field in detect(). If a user has pyproject.toml without the project field and setup.py in the repository, I believe that repo2docker will fail.

[minrk]

I'm not sure I quite follow - The detect you have checks for the presence build-system and project, which is what we should do. This same condition should be applied to whether to invoke pip install.

We could isolate this into a single _is_python_package method which checks setup.py present OR pyproject.toml contains build-system, and use that where we need this information, which it looks like is in two places.

This is because pyproject.toml can be used to configure tools (ruff, black, pytest, etc.) for repos that are not packages. So it's not so much a safety net as it is an accuracy of the check: pyproject.toml alone is insufficient information to decide if pip install should be called, you have to check its contents to see if it's a package or another kind of project directory.

[minrk]

I think there are a few cases not handled here:

1. pyproject_minimum_version_info contains strings, not numbers, so it will compare incorrectly sometimes, e.g. '10' < '9'
2. I think py_version_info will be empty when runtime.txt is not specified (i.e. usually), so this conflict check should only be done if runtime.txt is given

Version checking should usually be done via a version-comparison tool like packaging rather than parsing strings ourselves. The version specifier can be parsed by packaging.version.specifier.SpecifierSet, e.g.:

if "requires-python" in pyproject["project"]:
    python_specifier = SpecifierSet(pyproject["project"]["requires-python"])

    if runtime_specified:
        if not python_specifier.contains(py_version):
            # error, conflict
    else:
        # use constraint to pick version, default first
        for python_version in [major_pythons["3"], "3.10", "3.11", "3.12", "3.13", "3.14"]:
            if python_specifier.contains(python_version):
                self._python_version = python_version
                break
        else:
            # error, can't find acceptable version

[rgaiacs]

Thanks for the feedback. I'm finishing my day and I'm on holiday the rest of the week. I can pick this on Monday.