Errors in installing the example plugin

[ka-sarthak]

When the instructions on the readme are followed, it can happen that the virtual environment one installs has Python>3.9. A higher python version can run into some errors while installing nomad and its dependency.

For example, running python3 -m venv .pyenv, I ended up installing a virtual environment with Python 3.11.

When I ran pip install '.[dev]' --index-url https://gitlab.mpcdf.mpg.de/api/v4/projects/2187/packages/pypi/simple , I got the error with phonopy package, and nomad wasn't installed.

      c/_phonopy.c:35:10: fatal error: Python.h: No such file or directory
         35 | #include <Python.h>
            |          ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for phonopy
Successfully built nomad-schema-plugin-example
Failed to build numpy phonopy
ERROR: Could not build wheels for numpy, phonopy, which is required to install pyproject.toml-based projects

I think it's better to specify the version of python to be used for creating virtual environment. In my experience, python3.9 works nicely.

So I suggest that rather than having python3 -m venv .pyenv in readme, have python3.9 -m venv .pyenv.

@blueraft

[blueraft]

Thanks for filing the issue.

It works fine for me using py3.11.

The error message is due to missing build dependencies for compiling python programs. You could try installing them following the instructions here or here and retry installing package.

You could also try installing the dependencies using pre-built binaries instead by using the following command:

pip install '.[dev]' --index-url https://gitlab.mpcdf.mpg.de/api/v4/projects/2187/packages/pypi/simple --prefer-binary

[ka-sarthak]

Thanks @blueraft for looking into this and sorry for the delay. It seems that the issue lies with my Python3.11 installation then. I reinstalled it following the stackoverflow link you shared and running the following:

sudo dnf install python3-devel

Creating a virtual env over this and then building the nomad-package with the example plugin is a success, gladly!

However, when I run the pytest after this, it gives some issues related to numpy. Can you check if you also encounter this?

================================================================ ERRORS =================================================================
_________________________________________________ ERROR collecting tests/test_schema.py _________________________________________________
ImportError while importing test module '/home/kapoor/repositories/nomad-schema-plugin-example/tests/test_schema.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
__init__.pxd:942: in numpy.import_array
    ???
E   RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf

During handling of the above exception, another exception occurred:
/usr/lib64/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_schema.py:3: in <module>
    from nomad.client import normalize_all, parse
.pyenv/lib64/python3.11/site-packages/nomad/client/__init__.py:19: in <module>
    from .archive import ArchiveQuery
.pyenv/lib64/python3.11/site-packages/nomad/client/archive.py:28: in <module>
    from nomad import config, metainfo as mi
.pyenv/lib64/python3.11/site-packages/nomad/metainfo/__init__.py:33: in <module>
    from .metainfo import (
.pyenv/lib64/python3.11/site-packages/nomad/metainfo/metainfo.py:36: in <module>
    import pandas as pd
.pyenv/lib64/python3.11/site-packages/pandas/__init__.py:22: in <module>
    from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401
.pyenv/lib64/python3.11/site-packages/pandas/compat/__init__.py:18: in <module>
    from pandas.compat.numpy import (
.pyenv/lib64/python3.11/site-packages/pandas/compat/numpy/__init__.py:4: in <module>
    from pandas.util.version import Version
.pyenv/lib64/python3.11/site-packages/pandas/util/__init__.py:2: in <module>
    from pandas.util._decorators import (  # noqa:F401
.pyenv/lib64/python3.11/site-packages/pandas/util/_decorators.py:14: in <module>
    from pandas._libs.properties import cache_readonly
.pyenv/lib64/python3.11/site-packages/pandas/_libs/__init__.py:13: in <module>
    from pandas._libs.interval import Interval
pandas/_libs/interval.pyx:1: in init pandas._libs.interval
    ???
pandas/_libs/hashtable.pyx:1: in init pandas._libs.hashtable
    ???
pandas/_libs/missing.pyx:1: in init pandas._libs.missing
    ???
.pyenv/lib64/python3.11/site-packages/pandas/_libs/tslibs/__init__.py:36: in <module>
    from pandas._libs.tslibs import dtypes
pandas/_libs/tslibs/dtypes.pyx:1: in init pandas._libs.tslibs.dtypes
    ???
pandas/_libs/tslibs/np_datetime.pyx:28: in init pandas._libs.tslibs.np_datetime
    ???
__init__.pxd:944: in numpy.import_array
    ???
E   ImportError: numpy.core.multiarray failed to import
======================================================== short test summary info ========================================================
ERROR tests/test_schema.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=========================================================== 1 error in 0.54s ============================================================

Perhaps a bit related, few months back when I was installing the nomad-lab locally, I found that Python3.11 was giving similar issues. I then reverted my Python version to 3.9, and everything worked like charm. I came to believe that building of nomad package was more robustly tested with Python3.9. Maybe this is different now.

[blueraft]

Can you check if you also encounter this?

Yep, I'm getting two different versions of pandas, 1.5.3 for py3.9 and 1.5.1 for py3.11. I think the correct fix here is to use a newer version of numpy but that will have to be done in the main nomad-FAIR repo.

The conflict is caused by:
    pandas 1.5.3 depends on numpy>=1.21.0; python_version >= "3.10"
    pandas 1.5.3 depends on numpy>=1.23.2; python_version >= "3.11"
    nomad-lab 1.2.1 depends on numpy~=1.22.4
    pandas 1.5.3 depends on numpy>=1.21.0; python_version >= "3.10"
    pandas 1.5.3 depends on numpy>=1.23.2; python_version >= "3.11"
    nomad-lab 1.2.0 depends on numpy~=1.22.4

I came to believe that building of nomad package was more robustly tested with Python3.9.

This is true, ideally we would also test compatibility with other versions of python and run matrix tests but that doesn't seem feasible at the moment.

[ka-sarthak]

@blueraft I see. In this case, I think it's better to specify in the README to create the virtual env with Python3.9.x. That's at least free of conflicts as far as we know.