When Attempting to loadSavedModel, I Encountered 'java.lang.Exception: Could Not Retrieve the SavedModelBundle + ()

[TerryLaw535]

Is there an existing issue for this?

• I have searched the existing issues and did not find a match.

Who can help?

No response

What are you working on?

I fine-tuned a T5 model from Hugging Face and wanted to import it into Spark NLP.

Current Behavior

I followed the instructions from [https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/HuggingFace%20in%20Spark%20NLP%20-%20BERT.ipynb], and as I tried to loadSavedModel in Spark NLP, it reported "Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.annotators.seq2seq.BartTransformer.loadSavedModel: java.lang.Exception: Could not retrieve the SavedModelBundle + ()". Afterward, I attempted to import several other models like t5-base and bart from Hugging Face, and I encountered the same problem.

Expected Behavior

I hope that my fine-tuned T5 model can be successfully loaded in the spark nlp. Or if possible, could anyone make a similar instruction that how to import seq2seq model such as T5 and Bart from huggingface to sparknlp?

Steps To Reproduce

MODEL_NAME = 'google-t5/t5-small'

tokenizer = T5Tokenizer.from_pretrained(MODEL_NAME)
tokenizer.save_pretrained('./{}_tokenizer/'.format(MODEL_NAME))

from transformers import TFAutoModelForSeq2SeqLM, TFT5ForConditionalGeneration
model = TFT5ForConditionalGeneration.from_pretrained(MODEL_NAME)

@tf. function(
input_signature=(
tf.TensorSpec(name="input_ids", shape=(None, None), dtype=tf.int32),
tf.TensorSpec(name="attention_mask", shape=(None, None), dtype=tf.int32)
),
jit_compile=False,
)

def serving_fn(input_ids, attention_mask):
outputs = model.generate(
input_ids=input_ids,
attention_mask=attention_mask,
max_length=512,
min_length=10,
# length_penalty=0.9,
# repetition_penalty=2.0,
# num_beams=4,
# early_stopping=True,
return_dict_in_generate=True,
)
return {"sequences": outputs["sequences"]}

model.save_pretrained("./{}".format(MODEL_NAME), saved_model=True, signatures={'serving_default':serving_fn})

asset_path = '{}/saved_model/1/assets'.format(MODEL_NAME)
!cp {MODEL_NAME}_tokenizer/spiece.model {asset_path}
labels = model.config.id2label
labels = [value for key, value in sorted(labels.items(), reverse=False)]
with open(asset_path+'/labels.txt', 'w') as f:
f.write('\n'.join(labels))

spark = sparknlp.start()
from sparknlp.annotator import *
T5 = T5Transformer.loadSavedModel('{}/saved_model/1'.format(MODEL_NAME), spark)

Spark NLP version and Apache Spark

spark nlp version: '5.3.2'
spark version: '3.5.1'
tensorflow version: '2.15.0'
transformer version: '4.39.1'

Type of Spark Application

Python Application

Java Version

No response

Java Home Directory

No response

Setup and installation

pip

Operating System and Version

Lunix

Link to your project (if available)

No response

Additional Information

No response

[maziyarpanahi]

Hi @TerryLaw535

To import T5 models (fine-tuned or already on HF) you can use one of these 2 notebooks:

• TensorFlow: https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/HuggingFace%20in%20Spark%20NLP%20-%20T5.ipynb
• ONNX: https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/onnx/HuggingFace_ONNX_in_Spark_NLP_T5.ipynb

Depending on which one you have saved (exported) your T5 model, you can follow the instructions to import your model into Spark NLP. (I personally recommend first trying ONNX, it has a faster inference time)

[TerryLaw535]

Hi @maziyarpanahi

Thank you so much for your reply. I tried ONNX and it showed that [ONNXRuntimeError] : 1 : FAIL : Loading the model from onnx_models/google-t5/t5-small/decoder_model_merged.onnx failed:/onnxruntime_src/onnxruntime/core/graph/model.cc:179 onnxruntime::Model::Model(onnx::ModelProto&&, the onnxruntime::PathString&, the onnxruntime::IOnnxRuntimeOpSchemaRegistryList*, the onnxruntime::logging::Logger&, the onnxruntime::ModelOptions&) Unsupported model IR version: 10, max supported IR version: 9.

I also tried the TensorFlow method. I completely followed the instructions except for setting tensorflow == 2.8, since this version is too old and no longer available. I tried the code:
try:
model = T5ExportModel.from_pretrained(MODEL_NAME)
except:
model = T5ExportModel.from_pretrained(MODEL_NAME, from_pt=True)

model.export(EXPORT_PATH, use_cache=True)

and it report:
TypeError: in user code:

File "/tmp/ipykernel_70993/430537825.py", line 50, in decoder_init_serving  *
    logits = self.shared(sequence_output, mode="linear")
File "/home/weichengyu/.local/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler  **
    raise e.with_traceback(filtered_tb) from None

TypeError: Embedding.call() got an unexpected keyword argument 'mode'

Do you know how to deal with these problems? I think these problems are all related to version issues.

Thank you!

[Alexwangziyu]

Hi @TerryLaw535

To import T5 models (fine-tuned or already on HF) you can use one of these 2 notebooks:

• TensorFlow: https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/HuggingFace%20in%20Spark%20NLP%20-%20T5.ipynb
• ONNX: https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/onnx/HuggingFace_ONNX_in_Spark_NLP_T5.ipynb

Depending on which one you have saved (exported) your T5 model, you can follow the instructions to import your model into Spark NLP. (I personally recommend first trying ONNX, it has a faster inference time)

hi maziyarpanahi, thanks for your resource, however, I tried different transformers, tensorflow versions but none of them worked, basically there are all kinds of errors. And the code doesn't work on colab correctly either. Could you please check it out? Thank you so much!

[maziyarpanahi]

Hi @Alexwangziyu

I will assign someone to have a look at the TensorFlow to Spark NLP, something might have changed in Hugging Face. For the T5, I will ask someone to make a quick test to see if the notebooks are up to date.

[TerryLaw535]

Hi @Alexwangziyu

I will assign someone to have a look at the TensorFlow to Spark NLP, something might have changed in Hugging Face. For the T5, I will ask someone to make a quick test to see if the notebooks are up to date.

Hi @maziyarpanahi

Thank you so much for your help. Apart from that, is it possible for us to directly fine-tune the T5 or BART model through Spark NLP?

[maziyarpanahi]

Hi @Alexwangziyu
I will assign someone to have a look at the TensorFlow to Spark NLP, something might have changed in Hugging Face. For the T5, I will ask someone to make a quick test to see if the notebooks are up to date.

Hi @maziyarpanahi

Thank you so much for your help. Apart from that, is it possible for us to directly fine-tune the T5 or BART model through Spark NLP?

You are welcome. Unfortunately, the fine-tuning is not available on Java bindings or neither TensorFlow, PyTorch, and ONNX. When any of these libraries make the fine-tuning available, we will also implement it. Until then, we usually fine-tune outside and easily scale inside Spark NLP.

[TerryLaw535]

Hi @Alexwangziyu
I will assign someone to have a look at the TensorFlow to Spark NLP, something might have changed in Hugging Face. For the T5, I will ask someone to make a quick test to see if the notebooks are up to date.

Hi @maziyarpanahi
Thank you so much for your help. Apart from that, is it possible for us to directly fine-tune the T5 or BART model through Spark NLP?

You are welcome. Unfortunately, the fine-tuning is not available on Java bindings or neither TensorFlow, PyTorch, and ONNX. When any of these libraries make the fine-tuning available, we will also implement it. Until then, we usually fine-tune outside and easily scale inside Spark NLP.

Hi, @maziyarpanahi

Thank you so much for your help previously! I have encountered a couple of issues while following the method shared in the https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/HuggingFace%20in%20Spark%20NLP%20-%20T5.ipynb. I would appreciate your guidance on these:

1. Unexpected Keyword Argument 'mode':

Issue: While attempting to save the vanilla T5 model using the T5ExportModel class from the notebook (originally used for flan-t5), I encountered a TypeError: Embedding.call() got an unexpected keyword argument 'mode'. I modified the code from logits = self.shared(sequence_output, mode="linear") to logits = self.shared(sequence_output) to bypass this error.
Resulting Problem: After importing the model into Spark NLP, the outputs are repetitive and nonsensical (e.g., [ab ab ab ab... ab]).
Question: Could this issue be related to the signature of the model? How can we fix this?

2. IndexError in the flan-T5-large Model:

Issue: When trying to import a fine-tuned flan-T5-large model in the same manner, I receive an error: "../transformers/utils/generic.py" line 273, in getitem return self.to_tuple()[k] IndexError: tuple index out of range.
Question: Is there a known method to resolve this error?

We are working under a tight deadline, and your prompt assistance would be greatly appreciated to help us resolve these issues swiftly. Thank you so much!

[maziyarpanahi]

Hi @DevinTDHa
I think you have tested the T5 notebook in ONNX recently, could you please have a look at this? You can select any T5 model first to be sure it works first/

[TerryLaw535]

Hi @maziyarpanahi
Thank you so much for replying. Apart from T5, do you have the BART model imported into the Spark NLP documentation (specifically BartForConditionalGeneration, not BartForSequenceClassification)? I believe the decoder for BART differs from that of T5. I would greatly appreciate it if you could share it with me!
By the way, for the ONNX method, the error of the code you shared in the notebook is 'Unsupported model IR version: 10, max supported IR version: 9', I think it is a version problem.

[maziyarpanahi]

Hi @TerryLaw535
I wasn't aware we forgot to add notebook for BartForConditionalGeneration. I will create a task for it to be added to the repo soon.

[TerryLaw535]

Hi @maziyarpanahi
Thank you so much! Is it possible for me to know when I can expect to see the BART notebook? We are currently working with both BART and T5 and are under a tight schedule. The T5 notebook you provided seems to have a problem with any T5 models other than the Flan T5 that you chose in the example. As a result, we sincerely ask if you can provide notebooks to transfer both T5 and BART models (for example, T5 base and BART base).

[khanhvy31]

Hi @maziyarpanahi I tried the flan T5 notebook for T5 and I also got the problem above. Please advice.

[DevinTDHa]

Hi Everyone,

I looked into this and the TF export seems to indeed have some issues. I believe there are some implementation differences in the transformers library between the older (plain) t5 models. I will look into this.

Regarding the ONNX export, the issue above should be fixed with #14234. For the particular issue Unsupported model IR version: 10, max supported IR version: 9, we need to pin the version of ONNX with onnx==1.14.0.

So for example, while installing the dependencies we need to run

! pip install {OTHER_DEPENDENCIES} onnx==1.14.0

The PR above is not merged yet, but if you pin the onnx version manually, it should work. If possible, we also suggest to use ONNX instead. Let me know if this helps!

[khanhvy31]

@DevinTDHa Do you have a ONNX example for Bart? Thank you so much for the tip, I got T5 run!!

[TerryLaw535]

@maziyarpanahi @DevinTDHa
Thank you so much for your help. I got T5 run successfully. Is it possible for me to know when could I expect to get the Bart notebook?

[williambarteck]

I have also recieved this same error message when trying to fine tune a BertForSequenceClassification model then loading it into SparkNLP

I followed the example notebook exactly and have all of my necessary files saved in
{my_model}/saved_model/1

but when it loads I get
Py4JJavaError: An error occurred while calling z:com.johnsnowlabs.nlp.annotators.classifier.dl.BertForSequenceClassification.loadSavedModel.
: java.lang.Exception: Could not retrieve the SavedModelBundle + ()