Skip to content

Latest commit

 

History

History
251 lines (186 loc) · 8.04 KB

PYTHON.org

File metadata and controls

251 lines (186 loc) · 8.04 KB
Raw

Python and GNU Guix

Table of Contents

Introduction

We use Python extensively and despite its many ways of installing software and modules we have settled on using GNU Guix for development and deployment. GNU Guix comes with over 1,300 Python modules! You should look at Guix when you want clear isolation of dependencies and a reproducible environment for deploying development, testing, staging and production, with or without containers. Adding a pip package to Guix is a breeze. Also installing Python modules for testing locally is straightforward.

Install Python

After installing GNU Guix we install Python in our own profile named ~/opt/python-dev which we’ll load with the rapidjson module for testing

guix package -A python # list all Python packages
mkdir -p ~/opt
guix package -i python python-rapidjson -p ~/opt/python-dev

Now we can test by adding the profile

GUIX_PROFILE="/home/wrk/opt/python-dev"
. "$GUIX_PROFILE/etc/profile"
python3
Python 3.8.2
[GCC 7.5.0] on linux
>>> import rapidjson
>>> data = {'foo': 100, 'bar': 'baz'}
>>> rapidjson.dumps(data)
'{"bar":"baz","foo":100}'

So installing a Python package through Guix is easy.

Walking the profile we see it contains a very recent version of Python with the rapidjson module and associated dependencies. The profile acts effectively as a symlinked virtualenv. But better, because it can be shared with other generations of your setup and even other users on the system and it is binary reproducible across systems for deployment.

See also guix environment for a true way of managing packages (without virtualenv) and running Python containers.

Adding Python modules in user land with pip

So, when a Python module is part of the deployment stack you best make it part of GNU Guix. But sometimes you just want to try something. We can use pip! But when you try

pip3 install rapidjson

it will complain with “Permission denied” because it is trying to install in the (immutable) Guix store. Override with

pip3 install --user rapidjson

To figure out where Python installed the module

>>> import sys
>>> print(sys.path)
  ['',
   '/gnu/store/8ixk0ws42lz6jvrrjl12pgrrzxg2j80m-profile/lib/python3.8/site-packages',
   '/gnu/store/8ixk0ws42lz6jvrrjl12pgrrzxg2j80m-profile/lib/python38.zip',
   '/gnu/store/8ixk0ws42lz6jvrrjl12pgrrzxg2j80m-profile/lib/python3.8',
   '/gnu/store/8ixk0ws42lz6jvrrjl12pgrrzxg2j80m-profile/lib/python3.8/lib-dynload',
   '/home/wrk/.local/lib/python3.8/site-packages']

and, indeed, it contains rapidjson:

ls /home/wrk/.local/lib/python3.8/site-packages
rapidjson-1.0.0.dist-info

Isolated module installation

You can also use your own installation path. Pip and other Python installation tools install into a lib dir based on the prefix (normally /usr, and override just that). So pass in a prefix:

pip3 install --install-option="--prefix=$PREFIX_PATH" package_name

To fully guarantee isolation, set the PREFIX_PATH to something that contains the HASH value of the installed python. This can be fetched:

PYTHONBIN=$(readlink -f `which python3`)
PYTHONHASH=$(basename $(dirname $(dirname $PYTHONBIN)))
echo $PYTHONHASH
  kmfg2lq2aqgmdrr16hk7nc878aafpqhn-python-3.8.2
export PREFIX_PATH=$HOME/.python_guix/$PYTHONHASH

Now use this to install a module and make sure PATH and PYTHONPATH are updated accordingly

pip3 install --install-option="--prefix=$PREFIX_PATH" package_name
export PATH=$PREFIX_PATH/bin:$PATH
export PYTHONPATH=$PREFIX_PATH/lib/python2.8/site-packages:$PYTHONPATH

Using setup you can do something similar with

python3 setup.py install --prefix=$PREFIX_PATH

As such, you no longer need virtualenv with GNU Guix - as isolation is part of its job (more below).

With virtualenv

Still, GNU Guix supports python virtualenv. Install

guix package -i python-virtualenv

or

guix package -i python2-virtualenv

Run

virtualenv newdir
cd newdir
source bin/activate
~/newdir/bin/pip3 list

will give access to pip etc.

GNU Guix paths

The basic idea of GNU Guix is simple. A HASH value (SHA256) is calculated over the inputs to a build. This includes the source code of Python, and the switches used over configure and make. The software is installed under the HASH, for example I have Python 2.7.6 and 2.7.5 on my system sitting under

 /gnu/store/fy9arp9cn4zxzl69vsqj30p2j31w62al-python-2.7.6:
 bin  include  lib  share

 /gnu/store/yb9z2y7ndzra9r3x7l3020zjpds43yyc-python-2.7.5:
 bin  include  lib  share

and, for example, Python3 under

 /gnu/store/f01fv1v2q2bdqxsrhabryjk3rz866i3h-python-3.3.5::
 bin  lib  share

They are cleanly separated. Now if I were to change the configure for 2.1.3, for example a build without openssl, it would simply become another HASH and therefore directory.

It gets even better, the HASH value is also calculated over the dependencies. So, if you are running two different glibc’s on your system (each under its own HASH directory), or openssl’s, the python interpreter gets build against one of each and calculates a unique HASH. So you can theoretically have four concurrent Python 2.1.3 installations, compiled against any combination of two glibc’s and two openssl’s. The point, again, is that you have full control over the dependency graph!

To make a Python visible to a user, GNU Guix uses symlinks. Installing a particular Python will symlink a so-called profile in ~/.guix-profile/bin. To run Python, simply run it as

 ~/.guix-profile/bin/python --version
 Python 2.7.6

The libraries that come with Python are also symlinked via ~/.guix-profile/lib/python/. The numbering does not matter too much since it points to an immutable (read-only) directory in

 ~/.guix-profile/lib/python2.7 -> /gnu/store/fy9arp9cn4zxzl69vsqj30p2j31w62al-python-2.7.6/lib/python2.7

This means that you can access Python libraries shipped with a particular Python version, but that you can not write new files into that directory! The Python installation is carved in stone.

PYTHONPATH

To make sure Python finds the default guix Python modules that are symlinked it needs to be told where to find them

export PYTHONPATH="$HOME/.guix-profile/lib/python2.7/site-packages"

This statement is part of the file in your profile $PROFILE/etc/profile that can be loaded with

source $PROFILE/etc/profile

Typically that is enough.

Creating a GNU Guix Python package

Once you have a module that has to be embedded in deployment, the best thing is to make it part of GNU Guix.

Fortunately the Guix pypi import function made it easy by generating package definitions such as for the Python ‘prov’ package:

guix import pypi prov

renders a cut-and-paste JSON style package definition

(package
  (name "python-prov")
  (version "1.5.3")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "prov" version))
      (sha256
        (base32
          "1a9h406laclxalmdny37m0yyw7y17n359akclbahimdggq853jd0"))))
  (build-system python-build-system)
  (home-page "https://github.com/trungdong/prov")
  (synopsis
    "A library for W3C Provenance Data Model supporting PROV-JSON, PROV-XML and PROV-O (RDF)")
  (description
    "A library for W3C Provenance Data Model supporting PROV-JSON, PROV-XML and PROV-O (RDF)")
  (license license:expat))

The import fetches the latest version and even the software license! Note also how high-level the definition is. Not even a download URL to specify.