"Jupyter is not available in this Python installation" when using pipx

[venpopov]

Bug description

I use pipx to install python stuff (https://github.com/pypa/pipx). Jupyterlab works just fine, and in VScode I am able to select kernels installed this way no problem. However, when I run quarto render on a jupyter notebook, I get the following error:

quarto render r-notebook.ipynb --execute

Starting ir kernel...Traceback (most recent call last):
  File "/Applications/quarto/share/jupyter/jupyter.py", line 21, in <module>
    from notebook import notebook_execute, RestartKernel
  File "/Applications/quarto/share/jupyter/notebook.py", line 15, in <module>
    from yaml import safe_load as parse_string
ModuleNotFoundError: No module named 'yaml'
Python 3 installation:
  Version: 3.13.1
  Path: /opt/homebrew/opt/python@3.13/bin/python3.13
  Jupyter: (None)

Jupyter is not available in this Python installation.
Install with python3 -m pip install jupyter

Jupyter is globally accessible to all environments:

> pipx list
venvs are in /Users/venpopov/.local/pipx/venvs
apps are exposed on your $PATH at /Users/venpopov/.local/bin
manual pages are exposed at /Users/venpopov/.local/share/man
   package jupyterlab 4.3.4, installed using Python 3.13.1
    - debugpy
    - httpx
    - ipython
    - ipython3
    - jlpm
    - jsonpointer
    - jsonschema
    - jupyter
    - jupyter-dejavu
    - jupyter-events
    - jupyter-execute
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-lab
    - jupyter-labextension
    - jupyter-labhub
    - jupyter-migrate
    - jupyter-nbconvert
    - jupyter-run
    - jupyter-server
    - jupyter-troubleshoot
    - jupyter-trust
    - normalizer
    - pybabel
    - pygmentize
    - pyjson5
    - send2trash
    - wsdump
    - man1/ipython.1
   package notebook 7.3.2, installed using Python 3.13.1
    - jupyter-notebook
   package radian 0.6.13, installed using Python 3.13.1
    - radian

commands like jupyter work just fine in any terminal and VScode. It seems that quarto CLI tool is making erroneous assumptions about how to check for the presence of jupyter.

Steps to reproduce

the following installs jupyterlab via pipx

pipx install jupyterlab --include-deps

Expected behavior

quarto CLI should recognize the globally accessible jupyter installation

Actual behavior

No response

Your environment

• zsh terminal
• MacOS Sonoma

Quarto check output

Quarto 1.6.39
[✓] Checking environment information...
Quarto cache location: /Users/venpopov/Library/Caches/quarto
[✓] Checking versions of quarto binary dependencies...
Pandoc version 3.4.0: OK
Dart Sass version 1.70.0: OK
Deno version 1.46.3: OK
Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
Version: 1.6.39
Path: /Applications/quarto/bin

[✓] Checking tools....................OK
TinyTeX: (not installed)
Chromium: (not installed)

[✓] Checking LaTeX....................OK
Tex: (not detected)

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
Version: 3.13.1
Path: /opt/homebrew/opt/python@3.13/bin/python3.13
Jupyter: (None)

  Jupyter is not available in this Python installation.
  Install with python3 -m pip install jupyter

[✓] Checking R installation...........OK
Version: 4.4.2
Path: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources
LibPaths:
- /Users/venpopov/Library/R/arm64/4.4/library
- /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
knitr: 1.49
rmarkdown: 2.29

[✓] Checking Knitr engine render......OK

[cderv]

Can you share result of

# short version
jupyter kernelspec list
# long version 
jupyter kernelspec list --json

Also can you run quarto check jupyter after opting in for debug logging ?
See https://quarto.org/docs/troubleshooting/#verbose-mode to set the environment variable.

This should give us some trace information about kernel detection.

Maybe Quarto automatic detection does not account for a pipx installation and this should be improved. 🤔

What is done exactly by Quarto is not really looking for jupyter executable. Quarto will look for a python installation and check for some packages to detect capabilities. Among jupyter_core package available in the Python installation. if not available then you get the message you see.

So, in your situation

• Python found is /opt/homebrew/opt/python@3.13/bin/python3.13
• This version of python does not have jupyter_core available.

So there is something related to using pipx and Quarto together. 🤔

QUARTO_PYTHON can be set to a path to the python installation associated to the jupyter install. Possibly one in /Users/venpopov/.local/pipx/venvs

Thanks for the feedback. Waiting for your additional input now. Thanks !

[cderv]

So there is something related to using pipx and Quarto together. 🤔

I don't know exactly how pipx works, but I think it is meant to be execute some provided executable.

What quarto do is equivalent to python -m jupyter to detect kernel spec for find available kernel and then calls something like python jupyter.py with a internal python script that do all the work for executing notebook.

Quarto does not call Jupyter.exe you have on PATH. So it is not expected to work with a pipx jupyter install. It needs to use a python version that have Jupyter and dependencies available.

pipx does create a venv for each tool installed, so possibly setting QUARTO_PYTHON to the directory path where python.exe is in $PIPX_LOCAL_VENVS/jupyterlab could work.

What drives the use of pipx to use with Quarto ? Is this global install of jupyter, and then multiple python kernel for each project ? I assume the notebook used with Quarto will require some python packages that needs to be available within the python environment. How does this work if you use pipx ?

Hope it helps understand the situation here.

[venpopov]

Thanks for looking into it. Here is the output you requested:

Available kernels:
  ir         /Users/venpopov/Library/Jupyter/kernels/ir
  python3    /Users/venpopov/.local/pipx/venvs/jupyterlab/share/jupyter/kernels/python3

and

{
  "kernelspecs": {
    "ir": {
      "resource_dir": "/Users/venpopov/Library/Jupyter/kernels/ir",
      "spec": {
        "argv": [
          "/Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/bin/R",
          "--slave",
          "-e",
          "IRkernel::main()",
          "--args",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "R",
        "language": "R",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "python3": {
      "resource_dir": "/Users/venpopov/.local/pipx/venvs/jupyterlab/share/jupyter/kernels/python3",
      "spec": {
        "argv": [
          "python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3 (ipykernel)",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "debugger": true
        }
      }
    }
  }
}

from quarto check jupyter with debug mode:

Quarto version: 1.6.40
Quarto 1.6.40
[execProcess] python /Applications/quarto/share/capabilities/jupyter.py
[execProcess] python3 /Applications/quarto/share/capabilities/jupyter.py
[execProcess] Success: true, code: 0
[✓] Checking Python 3 installation....OK
      Version: 3.13.1
      Path: /opt/homebrew/opt/python@3.13/bin/python3.13
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[NotebookContext]: Starting Cleanup

[venpopov]

I recently reinstalled my system and started using pipx to install python-based applications that are globally accessible. For Jupyter for example from this tutorial. I would still need to create virtual environments for projects where I would install specific packages to use in the code, but applications such as jupyter become globally accessible.

pipx puts applications into separate venvs:

> ls -la /Users/venpopov.local/pipx/venvs
drwxr-xr-x@ 10 venpopov  staff  320 Jan  6 21:39 jupyterlab
drwxr-xr-x@ 10 venpopov  staff  320 Jan  6 21:41 notebook
drwxr-xr-x   8 venpopov  staff  256 Dec 25 16:30 radian

and them symlinks their executables to a folder on the path:

total 32
lrwxr-xr-x@ 1 venpopov  staff    33B Jan  1 15:15 colors.sh -> /Users/venpopov/scripts/colors.sh
lrwxr-xr-x@ 1 venpopov  staff    56B Jan  6 21:52 debugpy -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/debugpy
lrwxr-xr-x@ 1 venpopov  staff    54B Jan  6 21:52 httpx -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/httpx
lrwxr-xr-x@ 1 venpopov  staff    56B Jan  6 21:52 ipython -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/ipython
lrwxr-xr-x@ 1 venpopov  staff    57B Jan  6 21:52 ipython3 -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/ipython3
lrwxr-xr-x@ 1 venpopov  staff    53B Jan  6 21:52 jlpm -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jlpm
lrwxr-xr-x@ 1 venpopov  staff    60B Jan  6 21:52 jsonpointer -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jsonpointer
lrwxr-xr-x@ 1 venpopov  staff    59B Jan  6 21:52 jsonschema -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jsonschema
lrwxr-xr-x@ 1 venpopov  staff    56B Jan  6 21:52 jupyter -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter
lrwxr-xr-x@ 1 venpopov  staff    63B Jan  6 21:52 jupyter-dejavu -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-dejavu
lrwxr-xr-x@ 1 venpopov  staff    63B Jan  6 21:52 jupyter-events -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-events
lrwxr-xr-x@ 1 venpopov  staff    64B Jan  6 21:52 jupyter-execute -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-execute
lrwxr-xr-x@ 1 venpopov  staff    63B Jan  6 21:52 jupyter-kernel -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-kernel
lrwxr-xr-x@ 1 venpopov  staff    67B Jan  6 21:52 jupyter-kernelspec -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-kernelspec
lrwxr-xr-x@ 1 venpopov  staff    60B Jan  6 21:52 jupyter-lab -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-lab
lrwxr-xr-x@ 1 venpopov  staff    69B Jan  6 21:52 jupyter-labextension -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-labextension
lrwxr-xr-x@ 1 venpopov  staff    63B Jan  6 21:52 jupyter-labhub -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-labhub
lrwxr-xr-x@ 1 venpopov  staff    64B Jan  6 21:52 jupyter-migrate -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-migrate
lrwxr-xr-x@ 1 venpopov  staff    66B Jan  6 21:52 jupyter-nbconvert -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-nbconvert
lrwxr-xr-x@ 1 venpopov  staff    63B Jan  6 21:41 jupyter-notebook -> /Users/venpopov/.local/pipx/venvs/notebook/bin/jupyter-notebook
lrwxr-xr-x@ 1 venpopov  staff    60B Jan  6 21:52 jupyter-run -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-run
lrwxr-xr-x@ 1 venpopov  staff    63B Jan  6 21:52 jupyter-server -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-server
lrwxr-xr-x@ 1 venpopov  staff    69B Jan  6 21:52 jupyter-troubleshoot -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-troubleshoot
lrwxr-xr-x@ 1 venpopov  staff    62B Jan  6 21:52 jupyter-trust -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/jupyter-trust
lrwxr-xr-x@ 1 venpopov  staff    45B Jan  2 17:03 lsd -> /Users/venpopov/.local/opt/lsd-v1.1.5/bin/lsd
lrwxr-xr-x@ 1 venpopov  staff    59B Jan  6 21:52 normalizer -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/normalizer
lrwxr-xr-x@ 1 venpopov  staff    56B Jan  6 21:52 pybabel -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/pybabel
lrwxr-xr-x@ 1 venpopov  staff    59B Jan  6 21:52 pygmentize -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/pygmentize
lrwxr-xr-x@ 1 venpopov  staff    56B Jan  6 21:52 pyjson5 -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/pyjson5
lrwxr-xr-x  1 venpopov  staff    51B Dec 25 16:30 radian -> /Users/venpopov/.local/pipx/venvs/radian/bin/radian
lrwxr-xr-x@ 1 venpopov  staff    59B Jan  6 21:52 send2trash -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/send2trash
-rwxr--r--@ 1 venpopov  staff    14K Jan  2 17:03 webi
lrwxr-xr-x@ 1 venpopov  staff    55B Jan  6 21:52 wsdump -> /Users/venpopov/.local/pipx/venvs/jupyterlab/bin/wsdump

[venpopov]

To be honest, this is a new computer setup, so I was just starting to see how it would work with Quarto. For now, I have installed an R kernel, and I am able to set it up just fine in VSCode as the kernel for a notebook, and I can execute code and run the notebook. The problem occurs only when I try to use quarto to render the notebook or convert it. I have previously used quarto only with .qmd files and have worked with jupyter notebooks without quarto a few times before - it's the first time I'm trying to use the two together.

[cderv]

I would still need to create virtual environments for projects where I would install specific packages to use in the code

Did you try use the python version of those virtual env with Quarto ? Activate this env for your doc, and then quarto render ?

To execute the code in the cell, the virtual env with associated package will need to be accessible.

[venpopov]

I have not yet. My workflow is almost entirely R-based and I use ‘renv’ instead of virtual environments with quarto projects. My attempts to shoehorn in Jupyter notebooks in this workflow is because of this thread: posit-dev/positron#1533 (comment) where @juliasilge says that:

If you want an inline experience, use .ipynb
If you want a send-to-console experience, use .qmd

and that “converting is quite straightforward” which in my experience trying to set it up now doesn’t seem to be

My workflow is almost entire R based. I vastly prefer the editing experience of .qmd files, but inline outputs of jupyter notebooks is a big quality of life improvement during rapid prototyping. Is there a way to continue using renv for dependency management without adding redundant virtual environments on top just to use jupyter?

[juliasilge]

As far my statement about converting being straightforward, that turned out to be not true upon deeper experience; for example, see #9463.

[venpopov]

As far my statement about converting being straightforward, that turned out to be not true upon deeper experience; for example, see #9463.

Yes, I just ran into related issues try to convert existing R notebooks: #11787

[venpopov]

I would still need to create virtual environments for projects where I would install specific packages to use in the code

Did you try use the python version of those virtual env with Quarto ? Activate this env for your doc, and then quarto render ?

To execute the code in the cell, the virtual env with associated package will need to be accessible.

I tried it now and running into issues with packages. I did the following:

python -m venv venv
source venv/bin/activate
python -m pip install --upgrade pip wheel setuptools

then

quarto check jupyter detects the kernel, but gives an error that yaml package is missing:

quarto check jupyter
Quarto 1.6.40
[✓] Checking Python 3 installation....OK
      Version: 3.13.1
      Path: /Users/venpopov/venv/bin/python3
      Jupyter: 5.7.2
      Kernels: ir, python3

(/) Checking Jupyter engine render....Traceback (most recent call last):
  File "/Applications/quarto/share/jupyter/jupyter.py", line 21, in <module>
    from notebook import notebook_execute, RestartKernel
  File "/Applications/quarto/share/jupyter/notebook.py", line 15, in <module>
    from yaml import safe_load as parse_string
ModuleNotFoundError: No module named 'yaml'
[✓] Checking Jupyter engine render....OK

so I install pyyaml via python -m pip install pyyaml, and run the check again and this time I get an error about package nbformat

(-) Checking Jupyter engine render....Traceback (most recent call last):
  File "/Applications/quarto/share/jupyter/jupyter.py", line 21, in <module>
    from notebook import notebook_execute, RestartKernel
  File "/Applications/quarto/share/jupyter/notebook.py", line 19, in <module>
    import nbformat
ModuleNotFoundError: No module named 'nbformat'
[✓] Checking Jupyter engine render....OK

when I install that, i get another error about missing nbclient. Install that, and it finally clears:

quarto check jupyter
Quarto 1.6.40
[✓] Checking Python 3 installation....OK
      Version: 3.13.1
      Path: /Users/venpopov/venv/bin/python3
      Jupyter: 5.7.2
      Kernels: ir, python3

[✓] Checking Jupyter engine render....OK

At that point rendering works fine. So I will need to install pyyaml, nbformat and nbclient for each environment?

[cscheid]

(disclosure: I don't know how pipx works)

Note that pip install quarto-cli does install all required dependencies like pyyaml, nbformat and nbclient. This combination of issues makes me think that this isn't something that quarto is doing wrong.

[cderv]

So I will need to install pyyaml, nbformat and nbclient for each environment?

As I said, quarto will run something like python jupyter.py and this script requires dependencies that are the one from jupyter.

Current workflow is

• Create virtual env: python -m venv venv
• Install Jupyter in it so that Quarto can work with all necessary python dependencies.

This is our currently documented way with virtualenv: https://quarto.org/docs/projects/virtual-environments.html#using-venv

Another path is to install Quarto inside the virtual env (as mentioned just above: #11804 (comment)) and this will install all necessary dependencies.

pipx being a tool to install in isolated environment some executable provided in python package, maybe using pipx and quarto is installing Quarto with pipx pipx install quarto-cli 🤔

Not sure if Quarto will use the same python version as the isolated venv it would have been installed in though.

[venpopov]

Thanks for the detailed suggestions. I will do some experimentation and report back