Loading Transformers models with an empty token fails

[anakin87]

To Reproduce

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2", token="")

UPDATE: this issue also affects other transformers models (not only Sentence Transformers).

This issue is not specific to Haystack but affects multiple components and
is causing integration tests to fail when run from forks (example).

Error Message

Traceback (most recent call last):
  File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 406, in hf_raise_for_status
    response.raise_for_status()
  File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/adapter_config.json
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/transformers/utils/hub.py", line 403, in cached_file

resolved_file = hf_hub_download(

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn

return fn(*args, **kwargs)

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 860, in hf_hub_download

return _hf_hub_download_to_cache_dir(

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 967, in _hf_hub_download_to_cache_dir

_raise_on_head_call_error(head_call_error, force_download, local_files_only)

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1482, in _raise_on_head_call_error

raise head_call_error

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1374, in _get_metadata_or_catch_error

metadata = get_hf_file_metadata(

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn

return fn(*args, **kwargs)

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1294, in get_hf_file_metadata

r = _request_wrapper(

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 278, in _request_wrapper

response = _request_wrapper(

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 302, in _request_wrapper

hf_raise_for_status(response)

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 454, in hf_raise_for_status

raise _format(RepositoryNotFoundError, message, response) from e

huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-67a23a84-3082c753624cfc0b683fbbb7;108dba11-1266-42b8-bb28-d3a7b3eb745f)
Repository Not Found for url: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/adapter_config.json.

Please make sure you specified the correct repo_id and repo_type.

If you are trying to access a private or gated repo, make sure you are authenticated.

Invalid credentials in Authorization header
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/home/anakin87/apps/hfork/try.py", line 3, in 

model = SentenceTransformer("all-MiniLM-L6-v2", token="")

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 308, in init

modules, self.module_kwargs = self._load_sbert_model(

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 1728, in _load_sbert_model

module = module_class(model_name_or_path, cache_dir=cache_folder, backend=self.backend, **kwargs)

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 77, in init

config = self._load_config(model_name_or_path, cache_dir, backend, config_args)

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/sentence_transformers/models/Transformer.py", line 105, in _load_config

find_adapter_config_file(

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/transformers/utils/peft_utils.py", line 88, in find_adapter_config_file

adapter_cached_filename = cached_file(

File "/home/anakin87/apps/hfork/.hatch/test/lib/python3.10/site-packages/transformers/utils/hub.py", line 426, in cached_file

raise EnvironmentError(

OSError: sentence-transformers/all-MiniLM-L6-v2 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'

If this is a private repository, make sure to pass a token having permission to this repo either by logging in with huggingface-cli login or by passing token=<your_token>

Describe the bug
Loading Transformers models with an empty token now fails.
Previously, providing an invalid or empty token did not cause issues with public models.

For tests running from forks, an empty environment variable is being set for the token instead of leaving it unset, leading to failures.

Additional context

• I reported this to maintainers: Issues loading model to SentenceTransformer without passing HF token UKPLab/sentence-transformers#3212 (comment).
• While we understand whether this change is intentional, we can adjust the tests to work around it.

[anakin87]

It seems that also Transformers models are affected. https://github.com/deepset-ai/haystack/actions/runs/13140649366/job/36667079770?pr=8809