Tight Version Constraints on thinc in spaCy 3.8.2 Causing Dependency Resolution Issues for AutoGluon Users

[tonyhoo]

How to reproduce the behaviour

When attempting to install autogluon==1.1.1, the installation fails due to dependency resolution issues involving spaCy<4.0 and its tight version constraints on thinc. This leads pip to select incompatible versions of packages like onnx, resulting in installation failures.

Steps to reproduce:

1. Create a new virtual environment:

   python -m venv test_env
   source test_env/bin/activate  # On Windows, use: test_env\Scripts\activate

2. Upgrade pip to the latest version:

   pip install --upgrade pip

3. Attempt to install autogluon==1.1.1:

   pip install autogluon==1.1.1

4. Observe the installation failure, particularly with the onnx package.

Error Message:

Collecting onnx
  Using cached onnx-1.10.0.tar.gz (10.0 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [11 lines of output]
      /tmp/pip-install-xxxxxx/onnx/setup.py:36: DeprecationWarning: Use shutil.which instead of find_executable
        CMAKE = find_executable('cmake3') or find_executable('cmake')
      /tmp/pip-install-xxxxxx/onnx/setup.py:37: DeprecationWarning: Use shutil.which instead of find_executable
        MAKE = find_executable('make')
      fatal: not a git repository (or any of the parent directories): .git
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-xxxxxx/onnx/setup.py", line 318, in <module>
          raise FileNotFoundError("Unable to find " + requirements_file)
      FileNotFoundError: Unable to find requirements.txt
      [end of output]

Additional Information:

• The issue started occurring after the release of spaCy==3.8.2 on 2024-10-01.
• Downgrading spaCy to version 3.7.5 resolves the installation issue.
• The tight version constraints on thinc specified by spaCy<4.0 (e.g., thinc<8.4.0,>=8.3.0) appear to be causing dependency resolution conflicts.
• This affects users who are installing packages that depend on spaCy<4.0, such as autogluon.

Explanation:

• spaCy<4.0 specifies strict version constraints on thinc, requiring versions between >=8.3.0 and <8.4.0.
• This tight constraint causes pip to spend a long time resolving dependencies and ultimately selects older, incompatible versions of other packages (like onnx), which fail to install.
• Relaxing the version constraints on thinc could allow pip to find a compatible set of package versions, preventing the installation failure.

Your Environment

• Operating System: Linux (Ubuntu 20.04)
• Python Version Used: 3.10
• spaCy Version Used: spaCy==3.8.2
• Environment Information:
  • pip Version: 24.2
  • Other Relevant Packages:
    • autogluon==1.1.1
    • thinc versions being considered: 8.3.0, 8.3.1, 8.3.2

---

Our Request:

We kindly request the spaCy team to consider adjusting the version constraints on thinc in spaCy<4.0 to be less restrictive, or to release a patch version that relaxes these constraints. This adjustment could help prevent dependency resolution issues for users installing packages that depend on spaCy<4.0.

Is it feasible to modify these constraints without impacting spaCy's functionality? Any guidance or assistance on this matter would be greatly appreciated.

Thank you for your time and support.

---

Note: If this issue is better suited for a discussion, please let me know, and I'll be happy to open a discussion thread instead.

[tonyhoo]

kind reminder

[Goldziher]

Also happening in 3.12.

[jmaniuvc]

hi. I keep getting the same error on my Linux environment python 3.10.12. How can I solve this?

[kimlizette]

I'm experiencing the same
numpy==2.1.3
thinc==9.1.1
spacy==3.8.2

[honnibal]

The version constraints on thinc accurately reflect the compatibility range. The issue is that numpy v2 splits the ecosystem for now, and so there's a number of irreconcilable version conflicts.

It's impossible for us to have a version that is compatible across numpy v1 and numpy v2, because we compile against numpy and the binary API has changed. This means there will be some package combinations that are impossible to install together. If package A needs numpy v1 and package B needs numpy v2, you won't be able to find a version of spaCy that works with both packages.

[Goldziher]

The version constraints on thinc accurately reflect the compatibility range. The issue is that numpy v2 splits the ecosystem for now, and so there's a number of irreconcilable version conflicts.

It's impossible for us to have a version that is compatible across numpy v1 and numpy v2, because we compile against numpy and the binary API has changed. This means there will be some package combinations that are impossible to install together. If package A needs numpy v1 and package B needs numpy v2, you won't be able to find a version of spaCy that works with both packages.

is it possible to split numpy into a peer dependency and require the user to install it?

maybe its a silly question, but worth asking.

Its possible to be defensive about this in python, dunno about bindings etc. that you guys have.