Install fails on M1 macs

[tessadgreen]

Describe the bug

Unable to install using pip on M1 mac (Tessa and @stefanpeidli)
To Reproduce

mamba create -n pertpy_env python=3.9 conda activate pertpy_env pip install pertpy

(pertpy_env) stefanpeidli@Stefans-MBP ~ % pip install pertpy Collecting pertpy Using cached pertpy-0.3.0-py3-none-any.whl (82 kB) Collecting muon>=0.1.2 Using cached muon-0.1.2-py3-none-any.whl (287 kB) Collecting pypi-latest>=0.1.1 Using cached pypi_latest-0.1.2-py3-none-any.whl (10 kB) Collecting scipy<2.0.0,>=1.9.3 Downloading scipy-1.10.0-cp39-cp39-macosx_12_0_arm64.whl (28.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━28.9/28.9 MB 12.3 MB/s eta 0:00:00
Collecting PyYAML>=5.4.1
Downloading PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl (173 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
174.0/174.0 kB 8.3 MB/s eta 0:00:00
Collecting arviz<0.15.0,>=0.14.0
Using cached arviz-0.14.0-py3-none-any.whl (1.7 MB)
Collecting ipywidgets>=7.6.5
Downloading ipywidgets-8.0.4-py3-none-any.whl (137 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.8/137.8 kB 8.5 MB/s eta 0:00:00
Collecting scanpy>=1.8.1
Downloading scanpy-1.9.1-py3-none-any.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.1 MB/s eta 0:00:00 Collecting pyqt5<6.0.0,>=5.15.7 Using cached PyQt5-5.15.9.tar.gz (3.2 MB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
Traceback (most recent call last):
File "/Users/stefanpeidli/miniconda3/envs/pertpy_env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 144, in prepare_metadata_for_build_wheel
hook = backend.prepare_metadata_for_build_wheel
AttributeError: module 'sipbuild.api' has no attribute 'prepare_metadata_for_build_wheel'

  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/Users/stefanpeidli/miniconda3/envs/pertpy_env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
      main()
    File "/Users/stefanpeidli/miniconda3/envs/pertpy_env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/stefanpeidli/miniconda3/envs/pertpy_env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 148, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
    File "/private/var/folders/fr/z238tpm931s466144gcdpdj40000gn/T/pip-build-env-hvo51gpt/overlay/lib/python3.9/site-packages/sipbuild/api.py", line 46, in build_wheel
      project = AbstractProject.bootstrap('wheel',
    File "/private/var/folders/fr/z238tpm931s466144gcdpdj40000gn/T/pip-build-env-hvo51gpt/overlay/lib/python3.9/site-packages/sipbuild/abstract_project.py", line 87, in bootstrap
      project.setup(pyproject, tool, tool_description)
    File "/private/var/folders/fr/z238tpm931s466144gcdpdj40000gn/T/pip-build-env-hvo51gpt/overlay/lib/python3.9/site-packages/sipbuild/project.py", line 585, in setup
      self.apply_user_defaults(tool)
    File "project.py", line 68, in apply_user_defaults
      super().apply_user_defaults(tool)
    File "/private/var/folders/fr/z238tpm931s466144gcdpdj40000gn/T/pip-build-env-hvo51gpt/overlay/lib/python3.9/site-packages/pyqtbuild/project.py", line 70, in apply_user_defaults
      super().apply_user_defaults(tool)
    File "/private/var/folders/fr/z238tpm931s466144gcdpdj40000gn/T/pip-build-env-hvo51gpt/overlay/lib/python3.9/site-packages/sipbuild/project.py", line 236, in apply_user_defaults
      self.builder.apply_user_defaults(tool)
    File "/private/var/folders/fr/z238tpm931s466144gcdpdj40000gn/T/pip-build-env-hvo51gpt/overlay/lib/python3.9/site-packages/pyqtbuild/builder.py", line 69, in apply_user_defaults
      raise PyProjectOptionException('qmake',
  sipbuild.pyproject.PyProjectOptionException
  [end of output]

` note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.`

Expected behavior

Installs and loads

System [please complete the following information]:

• OS: Mac OS Ventura 13.01 [Tessa had the same problem in 13.02]
• Language Version: Python 3.9 [Tessa had same in python 3.10]
• Virtual environment: miniconda (stefan), miniforge (tessa)

Additional context

brew install of Qt5 seems like it might be necessary, but we're still having trouble with it freezing on the license step for pyqt5 install even after that

[Zethson]

@xinyuejohn you're using a M1 right? Got any idea?

[tessadgreen]

Prior to installing pertpy, install Qt5 via brew (and follow the directions to add to path), restart terminal, then run:
pip install pyqt5 --config-settings --confirm-license= --verbose

Then pip install pertpy works

[stefanpeidli]

Note: conda install -c anaconda pyqt will lead to a version below the one required by pertpy (5.5.12 < 5.5.17).

[tessadgreen]

Accidentally closed when trying to add additional issue:

Setting up developer environment as described here

At the make install step I get

` • Installing pyqt5-qt5 (5.15.2): Failed

RuntimeError

Unable to find installation candidates for pyqt5-qt5 (5.15.2)
`

Installing pyqt5 separately as described above does not fix.

`make install
rm -fr build/
rm -fr dist/
rm -fr .eggs/
find . -name '.egg-info' -exec rm -fr {} +
find . -name '.egg' -exec rm -f {} +
find . -name '.pyc' -exec rm -f {} +
find . -name '.pyo' -exec rm -f {} +
find . -name '*~' -exec rm -f {} +
find . -name 'pycache' -exec rm -fr {} +
poetry install
Installing dependencies from lock file

Package operations: 51 installs, 0 updates, 0 removals

• Installing pyqt5-qt5 (5.15.2): Failed

RuntimeError

Unable to find installation candidates for pyqt5-qt5 (5.15.2)

at ~/miniforge3/envs/pertpy-distances/lib/python3.9/site-packages/poetry/installation/chooser.py:105 in choose_for
101│
102│ links.append(link)
103│
104│ if not links:
→ 105│ raise RuntimeError(f"Unable to find installation candidates for {package}")
106│
107│ # Get the best link
108│ chosen = max(links, key=lambda link: self._sort_key(package, link))
109│

make: *** [install] Error 1`

[tessadgreen]

pip install PyQt5-Qt5 ERROR: Could not find a version that satisfies the requirement PyQt5-Qt5 (from versions: none) ERROR: No matching distribution found for PyQt5-Qt5

Not sure where to go from here @Zethson -- we considered removing the requirement from our branch but we're not sure where it's coming from in the first place

[Zethson]

It's from ipykernel I think

[xinyuejohn]

@xinyuejohn you're using a M1 right? Got any idea?

Sorry for the late reply. I am using Mac with intel i5.

[Moomboh]

The problematic pyqt5 dependency is required by ete3, see https://github.com/etetoolkit/ete/blob/master/setup.py#L81
It is required by pertpy here, probably because ete3 depends on it. I believe for ipykernel it is just an optional dependency.

[Zethson]

@Moomboh Indeed. ete3 requires it for plotting. So I'll need to make these dependencies optional urgh.

[Zethson]

If I did things correctly, ete3 and pyqt5 should now be optional dependencies. @tessadgreen can you please test the latest development branch?

[Zethson]

Apparently people are working on it etetoolkit/ete#656

[tessadgreen]

Installing from the latest development branch following the directions in the Contributor Guide works, and I'm seeing the 'ete3' warning on import. Nox runs successfully and I'm able to import pertpy into python.

Should note that the Qt5 brew install was system wide so it is available in the new environment for me/I don't know if it now works without that step.

[Moomboh]

Can confirm that setting up dev environment according to the Contributor Guide also works for me on Apple Silicon.
I also have qt5 installed system wide via brew install qt5 which might be an requirement.

Though to get nox to run I had to do:

poetry run pip install nox nox-poetry
poetry run nox

I also managed to get pertpy running including rtools support and ete3 running on Apple Silicon:

1. Install qt5 via homebrew:

   brew install qt5

   Make sure to add all the environment variables to your ~/.zshrc as described at the end of the output of the brew install command then reload your shell:

   zsh

2. Create environment with mamba (conda should work as well, but I haven't tested it):

   # environment.yaml
   name: pertpy-apple-silicon
   channels:
     - conda-forge
   dependencies:
     - python=3.9
     - rpy2

   mamba create -f environment.yaml -n pertpy-apple-silicon
   mamba activate pertpy-apple-silicon

3. Install pyqt5 via pip in conda enviroment:

   pip install pyqt5 --config-settings --confirm-license= --verbose

   This needs to be compiled from sources so this might take quite a while

4. Install ete3 via pip:

   pip install ete3

5. Finally install pertpy nightly via pip:

   pip install git+https://github.com/theislab/pertpy.git

(As of this writing nightly was at theislab/pertpy@3e6a702)

Now pertpy should be working fine and you should be able to use all features 🎉

import pertpy as pt

---

I still get the ete3 warning on import, but I think this is due to a bug in pertpy/tools/init.py#L10
There are faulty import paths getting masked by the catching of the ImportError for the ete3 warning.
from pertpy.tools.coda._sccoda import Sccoda works fine.

I think pertpy/tools/init.py#L10 should be instead:

try:
    from pertpy.tools.coda._sccoda import Sccoda
    from pertpy.tools.coda._tasccoda import Tasccoda
except ImportError as e:
    if "ete3" in str(e):
        print(
            "[bold yellow]To use sccoda or tasccoda please install ete3 with `pip install ete3`"
        )
    else:
        raise e

@Zethson should I open a separate issue for this?

[Zethson]

@Moomboh thank you very much for your in depth response.

Yes, please open a separate issue for this. I'd also be happy to get a PR in with your suggested improvement and we can discuss it directly there!

[tessadgreen]

New brew install required by the _dialogue.py. The new linear programming solver requires system-wise ipopt install on Mac, which can be done with brew install ipopt

[Zethson]

New brew install required by the _dialogue.py. The new linear programming solver requires system-wise ipopt install on Mac, which can be done with brew install ipopt

It should only be required if you try to run DIALOGUE though, right?

Also, ipopt is on conda

[tessadgreen]

Oh, yes sorry this came up when trying to run Dialogue, the install was fine/didn't error and I just had the problem when I tried to run the multifactor PMD with the linear programming addition. I got the error ApplicationError: No executable found for solver 'ipopt' and followed the directions here. Assumed that was the problem I was having but I'm going back through the log now and it looks like ipopt isn't a dependency/wasn't installed by pip and the brew was overkill. Sorry for the confusion!

[Zethson]

Yeah, I'll write up proper installation instructions soon, sorry.

[tessadgreen]

Update: we're failing on M3 mac now, even when following the directions in the documentation to the letter.

brew install --cask mambaforge

Run the command as directed by brew to add to your path
(At top of directions)

brew install gcc

https://scipy.github.io/devdocs/building/index.html#system-level-dependencies
Follow the directions here

brew install gfortran openblas pkg-config

Run whatever brew tells you to run
export LDFLAGS="-L/opt/homebrew/opt/openblas/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openblas/include"

brew info openblas | grep PKG_CONFIG_PATH

and then run the command that spits out.

Create a new environment using mamba (here with python 3.11) and activate it [note: current docs have conflicting python version numbers]

mamba create -n pertpy-env python=3.11
mamba activate pertpy-env


git clone https://github.com/scverse/pertpy.git
pip install . 

That works, and pytorch is successfully interfacing with MPS (yay)

To make pertpy loadable, then need to run

pip install arviz toytree formulaic pydeseq2

And then we're able to import, though I do get the following warning.

OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.

[tessadgreen]

update: although this installs and runs, Jax is in CPU-only mode. Trying to figure out if that's fixable via some combination of this and the actual directions