VLM generate: tests can't generate image/video tokens

[gante]

What does this PR do?

Our VLM generate mixin tests, introduced in #33533, are flaky. Because they use randomly initialized models, nothing prevents the models from generating image/video tokens, which a) shouldn't happen b) crash the forward pass.

This PR ensure our generation tests don't generate those tokens.

Commands ran to ensure the issue is fixed:
✅ (the test is no longer flaky) py.test tests/models/video_llava/test_modeling_video_llava.py::VideoLlavaForConditionalGenerationModelTest::test_sample_generate_dict_output --flake-finder --flake-runs=1000
✅ (we can run generation tests on all models after these changes) py.test tests/models/ -k test_sample_generate_dict_output

cc @zucchini-nlp -- when you're back from holidays, plz review even if it is already merged, in case you want to change things 🤗

[ydshieh]

works for me, although I think we probably don't need this condition

image_token_index < config.get_text_config().vocab_size

i.e. just always add those 2 token to bad word

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gante]

@ydshieh we do, llava_onevision fails without it :o (because the token in question is outside than the vocab size in the test settings)