Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transformers cannot load ModernBERT for sequence classification #35362

Open
2 of 4 tasks
eneko-caf opened this issue Dec 20, 2024 · 1 comment · May be fixed by #35370
Open
2 of 4 tasks

Transformers cannot load ModernBERT for sequence classification #35362

eneko-caf opened this issue Dec 20, 2024 · 1 comment · May be fixed by #35370
Labels
bug Usage General questions about the library

Comments

@eneko-caf
Copy link

System Info

I am trying to test the new ModernBER, following this notebook from the official documentation: https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/finetune_modernbert_on_glue.ipynb model for sequence classification but I am getting the following error:

Traceback (most recent call last):
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1038, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 740, in __getitem__
    raise KeyError(key)
KeyError: 'modernbert'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/train_full_fine_tuning.py", line 75, in <module>
    train_model_ft(tmp_folder_dataset.name, args)
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/train_full_fine_tuning.py", line 39, in train_model_ft
    orchestrate_training_with_epoch_artifacts(dataset=dataset, args=args)
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/utils/training/training_utils.py", line 153, in orchestrate_training_with_epoch_artifacts
    tokenizer, model = get_model_tokenizer(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/utils/training/training_utils.py", line 46, in get_model_tokenizer
    model = AutoModelForSequenceClassification.from_pretrained(training_model_name,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1040, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `modernbert` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

I am using:

  • Python version: 3.12.7
  • Tranformers version: 4.47.1
  • Tranformers information:
- `transformers` version: 4.47.1
- Platform: Linux-6.8.0-1018-aws-x86_64-with-glibc2.35
- Python version: 3.12.7
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:  no
- Using GPU in script?: yes
- GPU type: NVIDIA A10G

The code snippet used is this:

model = AutoModelForSequenceClassification.from_pretrained("answerdotai/ModernBERT-base", cache_dir=model_saving_path,
                                                               num_labels=12, compile=False)

Thank you very much!

Who can help?

@ArthurZucker
@stevhliu

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

Just executing

model = AutoModelForSequenceClassification.from_pretrained("answerdotai/ModernBERT-base", cache_dir=model_saving_path,
                                                         num_labels=12, compile=False)

the problem will arise

Expected behavior

To load the model normally
@eneko-caf eneko-caf added the bug label Dec 20, 2024
@seanfarr788
Copy link

Currently need to compile the latest transformers

pip install git+https://github.com/huggingface/transformers

source: https://huggingface.co/answerdotai/ModernBERT-base/discussions/3

@ArthurZucker ArthurZucker added the Usage General questions about the library label Dec 20, 2024
@Rocketknight1 Rocketknight1 linked a pull request Dec 20, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Usage General questions about the library
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update missing model error message huggingface/transformers
3 participants
@seanfarr788 @ArthurZucker @eneko-caf