TemplateError: Prompting with images is incompatible with system messages.

[hessaAlawwad]

System Info

Hello,

I am trying the follwoing code and it throughs an error message:
TemplateError: Prompting with images is incompatible with system messages.

The code is (Ref):

# Load the standard llama 3.2 11B Vision Instruct model/folder
model = MllamaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
       device_map="auto",
    # attn_implementation="flash_attention_2", NOT AVAILABLE YET FOR MLLAMA CONDITIONALGENERATION
    local_files_only=True
)

processor = AutoProcessor.from_pretrained(model_id)

messages = [
    [
        {
            "role": "system",
            "content": "ALWAYS start your sentence with an emoji, no matter what. Even if it does not make sense. If you can, use a stop sign emoji."
        },
        {
            "role": "user", 
            "content": [
                {"type": "image"},
                {"type": "text", "text": "Do you see that person?"}
            ]
        }
    ],
]

text = processor.apply_chat_template(messages, add_generation_prompt=True)

image_path = "https://llava-vl.github.io/static/images/view.jpg" #URL
image = Image.open(requests.get(image_path, stream=True).raw)

inputs = processor(text=text, images=image, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=30)
print(processor.decode(output[0]))

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

Not accepting the system role message when sending text+image

Error logs

TemplateError: Prompting with images is incompatible with system messages.

Expected behavior

as I am adding the image to the user role. it should work fine

[wukaixingxp]

@TQA-task Thanks for this bug report. We have confirmed that it is a unexpected bug and will work with Hugging face team to get system role supported for our vision models. Will keep you updated once it is done.

[yann-trustlab]

@wukaixingxp any update regarding this bug? I ran into the same problem with transformers==4.46.3. Thank you!

[AgatheBauer]

I get the same error message while using Llama-3.2-90b via Groq.