[2025.0.0.0rc2] Meta-Llama-3-8B-Instruct chatting with itself (not producing / detecting EOS token properly)

[RyanMetcalfeInt8]

Using the chat sample, it seems like instead of producing / detecting EOS token, it generates 'assistant' string/token, which causes model to continually chat with itself until max tokens is reached:

I am using openvino-genai==2025.0.0.0rc2, and the LLama3 model I am using was generated using:

optimum-cli export openvino -m meta-llama/Meta-Llama-3-8B-Instruct --weight-format int4 --sym --group-size -1 --ratio 1.0 Meta-Llama-3-8B-Instruct

Optimum version used for conversion is from: pip install -r export-requirements.txt

optimum                       1.24.0.dev0
optimum-intel                 1.22.0.dev0+b49fcbb

[RyanMetcalfeInt8]

By the way, issue occurs with all devices -- CPU, GPU, NPU.

[Aznie-Intel]

Hi @RyanMetcalfeInt8 Have you tried the TinyLlama-1.1B-Chat-v1.0 model? I didn't encounter any issue with the model.

[RyanMetcalfeInt8]

Hi @Aznie-Intel -- No, I didn't try TinyLlama. Perhaps the issue is model-dependent, then.

[RyanMetcalfeInt8]

Although it does seem that even in your screenshot, the chatbot is chatting with itself, right?