-
Notifications
You must be signed in to change notification settings - Fork 26.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VLMs: enable generation tests #33533
Conversation
@@ -98,9 +98,16 @@ class GenerationTesterMixin: | |||
|
|||
def _get_input_ids_and_config(self, batch_size=2): | |||
config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common() | |||
input_ids = inputs_dict[self.input_name] | |||
input_ids = inputs_dict.pop(self.input_name)[:batch_size, :] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd day we use model.main_input_name
here but let's leave it for the next PR. Seems it will cause many tests to fail 😓
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's add a todo then! :)
@@ -80,7 +81,7 @@ def __init__( | |||
"initializer_range": 0.02, | |||
"num_labels": 3, | |||
"num_choices": 4, | |||
"pad_token_id": 0, | |||
"pad_token_id": 1, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I chaneg this so that it doesn't clash with special image token, so when we artificially pad input sequence it won't get extra image placeholders than needed
tests/generation/test_utils.py
Outdated
# we don't want encoder-decoder models to start from filled decoder ids | ||
_ = inputs_dict.pop("decoder_input_ids", None) | ||
_ = inputs_dict.pop("decoder_attention_mask", None) | ||
|
||
# we'll set cache use in each test differently | ||
_ = inputs_dict.pop("use_cache", None) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hope it's okay this way, otherwise we can override this in each model which returns decoder inputs
. I don't think we need a new helper like get_input_for_generation
as the current one works good already
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔥🔥
(having tests makes me happy!)
@@ -98,9 +98,16 @@ class GenerationTesterMixin: | |||
|
|||
def _get_input_ids_and_config(self, batch_size=2): | |||
config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common() | |||
input_ids = inputs_dict[self.input_name] | |||
input_ids = inputs_dict.pop(self.input_name)[:batch_size, :] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's add a todo then! :)
tests/generation/test_utils.py
Outdated
if any(name in model.__class__.__name__.lower() for name in ("blip", "llava", "paligemma")): | ||
self.skipTest( | ||
"For VLMs inputs embeds won't match input ids unless images are encoded and merged with ids properly" | ||
) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This tags the whole test as skipped, which doesn't seem to be the goal (if the skip is reached, then the test above passes).
Let's simply run the rest of the test if a if
condition is satisfied, and leave a TODO to work on these models
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, btw, just so you know, the below part is not running right now because it should be cache_position
in singular. I didn't fix it here because fixing -> many hidden failures. I can work on it as separate PR or feel free to fix it if you have bandwidth 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh no -- let's work it on a separate PR
outputs_from_ids = model.generate(input_ids) | ||
self.assertEqual(outputs_from_ids.shape, (2, 20)) | ||
outputs_from_ids = model.generate(input_ids, max_new_tokens=5) | ||
self.assertEqual(outputs_from_ids.shape, (input_ids.shape[0], input_ids.shape[1] + 5)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, let's not rely on defaults 👍
Co-authored-by: Joao Gante <[email protected]>
Will try to find the root reason for flaky tests, otherwise we'll need to override and add |
@zucchini-nlp the PR still missed a review from a core maintainer :P (cc @LysandreJik for a quick look at the PR) |
Omg! O_o you're right, sorry, completely missed that |
* add tests * fix whisper * update * nit * add qwen2-vl * more updates! * better this way * fix this one * fix more tests * fix final tests, hope so * fix led * Update tests/generation/test_utils.py Co-authored-by: Joao Gante <[email protected]> * pr comments * not pass pixels and extra for low-mem tests, very flaky because of visio tower --------- Co-authored-by: Joao Gante <[email protected]>
* add tests * fix whisper * update * nit * add qwen2-vl * more updates! * better this way * fix this one * fix more tests * fix final tests, hope so * fix led * Update tests/generation/test_utils.py Co-authored-by: Joao Gante <[email protected]> * pr comments * not pass pixels and extra for low-mem tests, very flaky because of visio tower --------- Co-authored-by: Joao Gante <[email protected]>
What does this PR do?
Part of #33374. This PR adds GenerationTesterMixin in all VLMs (except BLIP) and modifies tests to accept all input kwargs, not only
input_ids
. That way we can test the whole logic, including merging image and text embeddings.I also ran a few tests for other models, and locally nothing seemed broken
cc @amyeroberts if you want to take a look :)