Skip to content

Commit 8bb5b6e

Browse files
lucascolleyhugovkFFY00
authored
gh-142466: Pixi package definitions for downstream use (#142469)
* WIP: ENH: Pixi package definitions for downstream development [skip ci] * linux-64 support * tidy gitignore * respond to review - switch cases on `PYTHON_VARIANT` - remove `minor_version` by using `python3` - remove runtime-only asan options * README updates * use `.md` to preview rendering * Apply suggestions from code review Co-authored-by: Hugo van Kemenade <[email protected]> * Apply suggestion from @FFY00 Co-authored-by: Filipe Laíns <[email protected]> * Apply suggestion from @FFY00 Co-authored-by: Filipe Laíns <[email protected]> * Apply suggestion from @FFY00 Co-authored-by: Filipe Laíns <[email protected]> * Apply suggestion from @lucascolley --------- Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Filipe Laíns <[email protected]>
1 parent a7501f0 commit 8bb5b6e

File tree

8 files changed

+216
-0
lines changed

8 files changed

+216
-0
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ gmon.out
4545
.pytest_cache/
4646
.ruff_cache/
4747
.DS_Store
48+
.pixi/
4849

4950
*.exe
5051

‎Tools/README‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,8 @@ patchcheck Tools for checking and applying patches to the Python source cod
4343

4444
peg_generator PEG-based parser generator (pegen) used for new parser.
4545

46+
pixi-packages Pixi package definitions for downstream from-source builds.
47+
4648
scripts A number of useful single-file programs, e.g. run_tests.py
4749
which runs the Python test suite.
4850

‎Tools/pixi-packages/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# CPython Pixi packages
2+
3+
This directory contains definitions for [Pixi packages](https://pixi.sh/latest/reference/pixi_manifest/#the-package-section)
4+
which can be built from the CPython source code.
5+
6+
Downstream developers can make use of these packages by adding them as Git dependencies in a
7+
[Pixi workspace](https://pixi.sh/latest/first_workspace/), like:
8+
9+
```toml
10+
[dependencies]
11+
python = { git = "https://github.com/python/cpython", subdirectory = "Tools/pixi-packages/asan" }
12+
```
13+
14+
This is particularly useful when developers need to build CPython from source
15+
(for example, for an ASan-instrumented build), as it does not require any manual
16+
clone or build steps. Instead, Pixi will automatically handle both the build
17+
and installation of the package.
18+
19+
Each package definition is contained in a subdirectory, but they share the build script
20+
`build.sh` in this directory. Currently defined package variants:
21+
22+
- `default`
23+
- `asan`: ASan-instrumented build with `PYTHON_ASAN=1`
24+
25+
## Maintenance
26+
27+
- Keep the `version` fields in each `recipe.yaml` up to date with the Python version
28+
- Keep the dependency requirements up to date in each `recipe.yaml`
29+
- Update `build.sh` for any breaking changes in the `configure` and `make` workflow
30+
31+
## Opportunities for future improvement
32+
33+
- More package variants (such as TSan, UBSan)
34+
- Support for Windows
35+
- Using a single `pixi.toml` and `recipe.yaml` for all package variants is blocked on https://github.com/prefix-dev/pixi/issues/4599
36+
- A workaround can be removed from the build script once https://github.com/prefix-dev/rattler-build/issues/2012 is resolved

‎Tools/pixi-packages/asan/pixi.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
[workspace]
2+
channels = ["https://prefix.dev/conda-forge"]
3+
platforms = ["osx-arm64", "linux-64"]
4+
preview = ["pixi-build"]
5+
6+
[package.build.backend]
7+
name = "pixi-build-rattler-build"
8+
version = "*"

‎Tools/pixi-packages/asan/recipe.yaml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
context:
2+
# Keep up to date
3+
version: "3.15"
4+
5+
package:
6+
name: python
7+
version: ${{ version }}
8+
9+
source:
10+
- path: ../../..
11+
12+
build:
13+
files:
14+
exclude:
15+
- "*.o"
16+
script:
17+
file: ../build.sh
18+
env:
19+
PYTHON_VARIANT: "asan"
20+
21+
# derived from https://github.com/conda-forge/python-feedstock/blob/main/recipe/meta.yaml
22+
requirements:
23+
build:
24+
- ${{ compiler('c') }}
25+
- ${{ compiler('cxx') }}
26+
- make
27+
- pkg-config
28+
# configure script looks for llvm-ar for lto
29+
- if: osx
30+
then:
31+
- llvm-tools
32+
- if: linux
33+
then:
34+
- ld_impl_${{ target_platform }}
35+
- binutils_impl_${{ target_platform }}
36+
- clang-19
37+
- llvm-tools-19
38+
39+
host:
40+
- bzip2
41+
- sqlite
42+
- liblzma-devel
43+
- zlib
44+
- zstd
45+
- openssl
46+
- readline
47+
- tk
48+
# These two are just to get the headers needed for tk.h, but is unused
49+
- xorg-libx11
50+
- xorg-xorgproto
51+
- ncurses
52+
- libffi
53+
- if: linux
54+
then:
55+
- ld_impl_${{ target_platform }}
56+
- libuuid
57+
- libmpdec-devel
58+
- expat
59+
60+
about:
61+
homepage: https://www.python.org/
62+
license: Python-2.0
63+
license_file: LICENSE

‎Tools/pixi-packages/build.sh‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
#!/bin/bash
2+
3+
if [[ "${PYTHON_VARIANT}" == "asan" ]]; then
4+
echo "BUILD TYPE: ASAN"
5+
BUILD_DIR="../build_asan"
6+
CONFIGURE_EXTRA="--with-address-sanitizer"
7+
export PYTHON_ASAN="1"
8+
export ASAN_OPTIONS="strict_init_order=true"
9+
else
10+
echo "BUILD TYPE: DEFAULT"
11+
BUILD_DIR="../build"
12+
CONFIGURE_EXTRA=""
13+
fi
14+
15+
mkdir -p "${BUILD_DIR}"
16+
cd "${BUILD_DIR}"
17+
18+
if [[ -f configure-done ]]; then
19+
echo "Skipping configure step, already done."
20+
else
21+
"${SRC_DIR}/configure" \
22+
--prefix="${PREFIX}" \
23+
--oldincludedir="${BUILD_PREFIX}/${HOST}/sysroot/usr/include" \
24+
--enable-shared \
25+
--srcdir="${SRC_DIR}" \
26+
${CONFIGURE_EXTRA}
27+
fi
28+
29+
touch configure-done
30+
31+
make -j"${CPU_COUNT}" install
32+
ln -sf "${PREFIX}/bin/python3" "${PREFIX}/bin/python"
33+
34+
# https://github.com/prefix-dev/rattler-build/issues/2012
35+
if [[ ${OSTYPE} == "darwin"* ]]; then
36+
cp "${BUILD_PREFIX}/lib/clang/21/lib/darwin/libclang_rt.asan_osx_dynamic.dylib" "${PREFIX}/lib/libclang_rt.asan_osx_dynamic.dylib"
37+
fi

‎Tools/pixi-packages/default/pixi.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
[workspace]
2+
channels = ["https://prefix.dev/conda-forge"]
3+
platforms = ["osx-arm64", "linux-64"]
4+
preview = ["pixi-build"]
5+
6+
[package.build.backend]
7+
name = "pixi-build-rattler-build"
8+
version = "*"

‎Tools/pixi-packages/default/recipe.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
context:
2+
# Keep up to date
3+
version: "3.15"
4+
5+
package:
6+
name: python
7+
version: ${{ version }}
8+
9+
source:
10+
- path: ../../..
11+
12+
build:
13+
files:
14+
exclude:
15+
- "*.o"
16+
script:
17+
file: ../build.sh
18+
19+
# derived from https://github.com/conda-forge/python-feedstock/blob/main/recipe/meta.yaml
20+
requirements:
21+
build:
22+
- ${{ compiler('c') }}
23+
- ${{ compiler('cxx') }}
24+
- make
25+
- pkg-config
26+
# configure script looks for llvm-ar for lto
27+
- if: osx
28+
then:
29+
- llvm-tools
30+
- if: linux
31+
then:
32+
- ld_impl_${{ target_platform }}
33+
- binutils_impl_${{ target_platform }}
34+
- clang-19
35+
- llvm-tools-19
36+
37+
host:
38+
- bzip2
39+
- sqlite
40+
- liblzma-devel
41+
- zlib
42+
- zstd
43+
- openssl
44+
- readline
45+
- tk
46+
# These two are just to get the headers needed for tk.h, but is unused
47+
- xorg-libx11
48+
- xorg-xorgproto
49+
- ncurses
50+
- libffi
51+
- if: linux
52+
then:
53+
- ld_impl_${{ target_platform }}
54+
- libuuid
55+
- libmpdec-devel
56+
- expat
57+
58+
about:
59+
homepage: https://www.python.org/
60+
license: Python-2.0
61+
license_file: LICENSE

0 commit comments

Comments
 (0)