[Package Issue Report]: [Question]: get python311Packages.django python311Packages.ipython working together

[dephiros]

Are you requesting a new package or reporting an issue on an existing one?
Existing

If reporting an issue on an existing one, what is the name of the package?
python311Packages.django python311Packages.ipython

What changes are you requesting?
I am trying to test out options 2(explained in the context section below)
When installing any python package like django from nix, ipython or regular python interpreter cannot seem to see the package. This makes sense because the package is probably not linked with PYTHON_PATH
Nix seems to use withPackages as the solution

Is there a way to do something similar with devbox?

We can also create a plugin but then it will bypass devbox's benefits of being able to install packages from different nixpkgs version

Additional context
I am trying to evaluate a few option of adopting nix/devbox for an internal project. Two ways that we are thinking of:

1. Use devbox to install python. Then continue to use pip/pip-compile to manage package
2. Use devbox to install python and its related dependencies

(still learning about the nix ecosystems so any feedbacks or recommendations are appreciated)

Really leaning toward options 2 since we can start using things like caching from the nix eco system

[Lagoja]

Hey @dephiros, you're right that Nix expects developers to use the withPackages or withExtensions method that most of these packages are bundled with. Unfortunately, a lot of those methods aren't very compatible with setting the version of the subpackages (like django or ipython) that you want to include. We're working on some ways to expose these methods while also letting developers pin or select the versions they need.

For now, I'd recommend leaning towards option 1 -- installing python, then using pip to manage the packages in a virtual environment with venv. Our plugin for Python sets up the virtual environment automatically to support this.

[dephiros]

Thanks for the response @Lagoja .
I have a follow-up question.
I am trying to install psycopg2 with pip and it requires openssl.
I tried installing openssl but psycopg2 still error pip install psycopg2==2.9.9 --no-binary=psycopg2 and cannot find openssl.

      ld: library not found for -lssl
      clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
      error: command '/nix/store/sa6hywsm1mqfyd1xakyzv4ljjsb3hawh-clang-wrapper-11.1.0/bin/clang' failed with exit code 1

Checking ./.devbox/nix/profile/default/lib which is included in ~LDPATH and I found other lib that I installed like jansson but not openssl
Would you happen to know why this issue happen?

[gcurtis]

@dephiros could you try adding "flake:nixpkgs#openssl^bin,dev" to the list of packages in your devbox.json and then installing psycopg2 again? For example:

devbox.json

{
  "packages": [
    "python@3.11",
    "python311Packages.pip",
    "postgresql@latest",
    "flake:nixpkgs#openssl^bin,dev"
  ]
}

$ devbox shell
(devbox) $ . $VENV_DIR/bin/activate
(.venv) (devbox) $ pip install psycopg2==2.9.9 --no-binary=psycopg2
(.venv) (devbox) $ python
Python 3.11.6 (main, Oct  2 2023, 13:45:54) [Clang 16.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
>>> psycopg2
<module 'psycopg2' from '/var/folders/79/1yc1ywp10w9f2xnr_rpp_ff00000gn/T/tmp.FFwpDXdvJi/.devbox/virtenv/python311Packages.pip/.venv/lib/python3.11/site-packages/psycopg2/__init__.py'>

[dephiros]

@gcurtis , thank you for the suggestion
I tried "flake:nixpkgs#openssl^bin,dev" but devbox does not seem to like the flake: syntax
Based on the doc, I tried github:NixOS/nixpkgs/23.11#openssl^bin,dev but devbox does not seem to like the ^`

Error: error searching for pkg github:NixOS/nixpkgs/23.11#openssl%5Edev,bin: exit status 1

2024/01/10 13:08:43 Command stderr: error: flake 'github:NixOS/nixpkgs/23.11' does not provide attribute 'packages.aarch64-darwin.openssl^dev,bin', 'legacyPackages.aarch64-darwin.openssl^dev,bin' or 'openssl^dev,bin'

Ended up trying:

    "github:NixOS/nixpkgs/23.11#openssl.dev",
    "github:NixOS/nixpkgs/23.11#openssl.bin"

and psycopg2 seems to install now. Do you think it is a bug in devbox to not support github:NixOS/nixpkgs/23.11#openssl^bin,dev. I could not find a documentation on the ^ but this comment makes me think it is a valid flake url syntax

[dephiros]

Sorry to keep piling on this issue(let me know if I should create a separate one)
I am trying to install pycurl next. Using the same strategy to install curl, I get the errors:

      clang -bundle -undefined dynamic_lookup -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.4/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib -L/nix/store/jxgh20m58s1nwzficc9h8iz9rbaszzbl-libxcrypt-4.4.36/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.43.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-apple-framework-SystemConfiguration-11.0.0/lib -L/nix/store/my3y0ixd8sy1g6vslrhj3i609vvhz5wp-tzdata-2023c/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.4/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib -L/nix/store/jxgh20m58s1nwzficc9h8iz9rbaszzbl-libxcrypt-4.4.36/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.43.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-apple-framework-SystemConfiguration-11.0.0/lib -L/nix/store/my3y0ixd8sy1g6vslrhj3i609vvhz5wp-tzdata-2023c/lib build/temp.macosx-11.0-arm64-cpython-311/src/docstrings.o build/temp.macosx-11.0-arm64-cpython-311/src/easy.o build/temp.macosx-11.0-arm64-cpython-311/src/easycb.o build/temp.macosx-11.0-arm64-cpython-311/src/easyinfo.o build/temp.macosx-11.0-arm64-cpython-311/src/easyopt.o build/temp.macosx-11.0-arm64-cpython-311/src/easyperform.o build/temp.macosx-11.0-arm64-cpython-311/src/module.o build/temp.macosx-11.0-arm64-cpython-311/src/multi.o build/temp.macosx-11.0-arm64-cpython-311/src/oscompat.o build/temp.macosx-11.0-arm64-cpython-311/src/pythoncompat.o build/temp.macosx-11.0-arm64-cpython-311/src/share.o build/temp.macosx-11.0-arm64-cpython-311/src/stringcompat.o build/temp.macosx-11.0-arm64-cpython-311/src/threadsupport.o build/temp.macosx-11.0-arm64-cpython-311/src/util.o -L/nix/store/c39qm57grkavw8a4hkramahpnspm1inq-curl-8.4.0/lib -L/nix/store/ai4hd8f1xhr0rfjdr17bxx1rwi42sx97-python3-3.11.6/lib -lssl -lcrypto -lcurl -o build/lib.macosx-11.0-arm64-cpython-311/pycurl.cpython-311-darwin.so -flat_namespace
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.4/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.43.2/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p1/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-apple-framework-SystemConfiguration-11.0.0/lib'
      ld: warning: directory not found for option '-L/nix/store/my3y0ixd8sy1g6vslrhj3i609vvhz5wp-tzdata-2023c/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.4/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.43.2/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p1/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib'
      ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-apple-framework-SystemConfiguration-11.0.0/lib'
      ld: warning: directory not found for option '-L/nix/store/my3y0ixd8sy1g6vslrhj3i609vvhz5wp-tzdata-2023c/lib'
      ld: warning: directory not found for option '-L/Users/annguyen/.local/share/devbox/global/default/.devbox/nix/profile/default/lib'
      ld: file not found: /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices for architecture arm64
      clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
      error: command '/nix/store/sa6hywsm1mqfyd1xakyzv4ljjsb3hawh-clang-wrapper-11.1.0/bin/clang' failed with exit code 1
      [end of output]

Any insight here is appreciated 🙏

[gcurtis]

@dephiros which version of Devbox are you on? You'll need Devbox 0.8.3 or higher for the flake ^ syntax to work. The dot syntax works too, so what you have looks good.

The second error looks like the linker can't find the CoreServices system library. I'm not sure why it's needed, but maybe try installing Xcode command line tools to see if that helps. You can do that by running xcode-select --install or installing and opening Xcode once.

Here are some additional debugging tips for build errors with Nix on macOS:

• The clang being invoked is separate from the macOS default clang. Nix wraps it in a shell script. That script injects a bunch of flags so that clang can find the other Nix-installed libraries. If you set NIX_DEBUG=1 (I think it goes up to 7 for more verbosity) the clang wrapper script will print a bunch of debugging info describing the flags it's setting.
• The ld warnings with the eeeeeees in them are misleading and you can ignore them. The only real error is the last one with "ld: file not found: ...".

[kalbasit]

Thanks for your help @gcurtis, I work with @dephiros at the same company.

I was able to take it further with the following devbox.json but now I'm stuck at install xmlsec with pip install xmlsec==1.3.13.

{
  "packages": [
    "python@3.11",
    "nodePackages.pnpm@latest",
    "nodejs@20.10.0",
    "postgresql@latest",
    "jq@latest",
    "uwsgi@2.0.23",
    "zlib@1.3",
    "expat@2.5",
    "jansson@2.14",
    "pcre@8.45",
    "libxcrypt@4.4.36",
    "openssl@3.0.12",
    "pkg-config@latest",
    "flake:nixpkgs#xmlsec^out,dev",
    "flake:nixpkgs#openssl^bin,dev",
    "flake:nixpkgs#curl^dev",
    "flake:nixpkgs#cairo^dev"
  ],
  "shell": {
    "init_hook": [". $VENV_DIR/bin/activate"],
    "scripts": {
      "test": ["echo \"Error: no test specified\" && exit 1"]
    }
  }
}

🔋 99% at 15:40:37 ❯ pip install xmlsec==1.3.13
Collecting xmlsec==1.3.13
  Using cached xmlsec-1.3.13.tar.gz (64 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting lxml>=3.8 (from xmlsec==1.3.13)
  Using cached lxml-5.1.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (3.5 kB)
Using cached lxml-5.1.0-cp311-cp311-macosx_11_0_arm64.whl (4.5 MB)
Building wheels for collected packages: xmlsec
  Building wheel for xmlsec (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for xmlsec (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.0-arm64-cpython-311
      creating build/lib.macosx-11.0-arm64-cpython-311/xmlsec
      copying src/xmlsec/tree.pyi -> build/lib.macosx-11.0-arm64-cpython-311/xmlsec
      copying src/xmlsec/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-311/xmlsec
      copying src/xmlsec/constants.pyi -> build/lib.macosx-11.0-arm64-cpython-311/xmlsec
      copying src/xmlsec/template.pyi -> build/lib.macosx-11.0-arm64-cpython-311/xmlsec
      copying src/xmlsec/py.typed -> build/lib.macosx-11.0-arm64-cpython-311/xmlsec
      running build_ext
      error: xmlsec1 is not installed or not in path.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for xmlsec
Failed to build xmlsec
ERROR: Could not build wheels for xmlsec, which is required to install pyproject.toml-based projects