Cannot import name '__version__' from 'tokenizers.tokenizers'

[ArthurAardvark]

My admittedly complex setup for ComfyUI has been on the fritz. This is just the latest error.

But I've tried everything under the sun.

I uninstalled Tokenizers/Transformers and installed with pip install transformers -U. Nope. (Purged cache between installs, too).

I sourced the Git Repo, built it in Cargo, installed the compiled bindings to my virtual instance via poetry run pip install -e path/to/bindings. Nope.

Then I tried to add version='0.21.0' to the init.py and still a no-go.

I tried different versions of Tokenizers/Transformers, too.

The full error logs if for some reason it matters:

Traceback (most recent call last):
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1863, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zack/.home/local/mise/installs/python/3.11.9/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/models/clip/tokenization_clip.py", line 25, in <module>
    from ...tokenization_utils import AddedToken, PreTrainedTokenizer, _is_control, _is_punctuation, _is_whitespace
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 27, in <module>
    from .tokenization_utils_base import (
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 91, in <module>
    from tokenizers import AddedToken
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/__init__.py", line 78, in <module>
    from .tokenizers import (
ImportError: cannot import name '__version__' from 'tokenizers.tokenizers' (/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-darwin.so)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/main.py", line 136, in <module>
    import execution
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/execution.py", line 13, in <module>
    import nodes
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/nodes.py", line 22, in <module>
    import comfy.diffusers_load
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/diffusers_load.py", line 3, in <module>
    import comfy.sd
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/sd.py", line 24, in <module>
    from . import model_detection
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/model_detection.py", line 1, in <module>
    import comfy.supported_models
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/supported_models.py", line 5, in <module>
    from . import sd1_clip
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/comfy/sd1_clip.py", line 3, in <module>
    from transformers import CLIPTokenizer
  File "<frozen importlib._bootstrap>", line 1229, in _handle_fromlist
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1852, in __getattr__
    value = getattr(module, name)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1851, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1865, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.models.clip.tokenization_clip because of the following error (look up to see its traceback):
cannot import name '__version__' from 'tokenizers.tokenizers' (/Users/zack/.home/gitrepos/Stable-Diff/ComfyUI/.venv/lib/python3.11/site-packages/tokenizers/tokenizers.cpython-311-darwin.so)

And I'm using a Mac M1. Any help is much appreciated.