[Tests]: Adding dummy causal models for testing in regular CI run #427

abukhoy · 2025-05-29T11:42:38Z

Purpose of this PR:

This update aims to reduce test execution time for causal language model inference. Previously, tests were run using full-scale models with one or two layers, which was inefficient and time-consuming. Refactoring CLI api testing for independent testing and redundant conftest code.

What’s Changed:

Introduced dummy models with significantly smaller configurations by adjusting parameters such as max_position_embeddings, num_hidden_layers, num_attention_heads, hidden_size, intermediate_size, vocab_size and additional_params.
These lightweight models are used exclusively for testing purposes to ensure faster execution without compromising test coverage.

And CLI testing has two test scripts one is for export, compile, and execute, another is for infer cli api.

Note: This optimization is applied only to causal language models.

Signed-off-by: Abukhoyer Shaik <[email protected]>

quic-rishinr · 2025-07-03T11:02:56Z

tests/transformers/models/test_causal_lm_models.py

    "hpcai-tech/grok-1",
 ]

+test_dummy_model_configs = [


Can we move this outside this file? may be we can maintain a CSV file for better readability.

Yes, made a json file for dummy configs.

quic-rishinr · 2025-07-03T11:19:36Z

tests/transformers/models/test_causal_lm_models.py

    "hpcai-tech/grok-1",
 ]

+test_dummy_model_configs = [
+    # model_name, model_type, max_position_embeddings, num_hidden_layers, num_attention_heads, hidden_size, intermediate_size, vocab_size, additional_params
+    ("TinyLlama/TinyLlama-1.1B-Chat-v1.0", "llama", 128, 1, 2, 64, 256, 32000, {"num_key_value_heads": 1}),


are we following any criteria for selecting these configs?

No criteria, but some of the models has constraint for some of the config params with specific value.

quic-rishinr · 2025-07-03T11:55:48Z

tests/transformers/models/test_causal_lm_models.py

-
+    if model_hf is None:
+        model_hf, _ = load_causal_lm_model(model_config)
+    model_hf_cb = copy.deepcopy(model_hf)


why do we need this?

Because, later for CB testing we don't need to call the load_causal_lm_model again. And it reduces the code for dummy model execution. If you don't want this, please let me know.

quic-rishinr · 2025-07-15T05:21:38Z

tests/cloud/test_export_compile_execute.py

+@pytest.mark.cli
+@pytest.mark.parametrize("config", configs)
+def test_export_compile_execute_qnn_fb(mocker, config):
+    # testing export -> compile -> infer with full_batch_size in QNN enviroment


Typo in "enviroment"

quic-rishinr · 2025-07-15T05:25:24Z

tests/cloud/test_export_compile_execute.py

+@pytest.mark.qnn
+@pytest.mark.cli
+@pytest.mark.parametrize("config", configs)
+def test_export_compile_execute_qnn(mocker, config):


Both test_export_compile_execute_qnn and test_export_compile_execute_qnn_fb is currently having same configs right? Ideally in test_export_compile_execute_qnn we should be providing BS and in test_export_compile_execute_qnn_fb we should be providing FBS.

Rename test_export_compile_execute_qnn_fb -> test_export_compile_execute_qnn_fbs for better readability

Typo in 'enviroment'

quic-rishinr · 2025-07-15T05:52:13Z

tests/cloud/test_infer.py

    )
+    check_infer(mocker=mocker, generation_len=20, **local_config)


Can we have a vlm qnn test as well?

I tried adding this test but got error, so removed from the test.

quic-rishinr · 2025-07-15T05:54:03Z

tests/cloud/test_infer.py

        mxfp6=ms.mxfp6,
        mxint8=ms.mxint8,
        full_batch_size=ms.full_batch_size,
        enable_qnn=ms.enable_qnn,
+        image_url=kwargs["image_url"],
+    )


how can we make sure the infer is running as expected? Please include proper asset for checking, export, compile and generation is running proper.

Added checks and assertions which were possible.

tests/transformers/models/test_causal_lm_models.py

Signed-off-by: Abukhoyer Shaik <[email protected]>

asmigosw · 2025-07-22T10:15:22Z

tests/transformers/models/test_causal_lm_models.py

+    filtered_configs = [(config, name) for name, config in model_config_dict.items() if name in selected_model_names]
+    config_objects = [item for item in filtered_configs]
+    model_names = [name for _, name in filtered_configs]
+    return config_objects, model_names


This code loops over filtered_configs multiple times to extract config_objects and model_names, which is redundant. Can we reduce the number of loops?

Signed-off-by: Abukhoyer Shaik <[email protected]>

quic-hemagnih · 2025-07-23T08:30:34Z

tests/cloud/test_export_compile_execute.py

+
+    data = []
+
+    for i in range(12):


Please avoid using magic numbers, we should define constants for 12

This is model specific, so we cannot generalize now. This 12 is number of layers of the model like gpt2 model in this case. Actually, we are creating a custom IO file here and later it will be no longer needed when the compile CLI Api would be backed by HL Api.

quic-hemagnih · 2025-07-23T08:32:35Z

tests/cloud/test_export_compile_execute.py

+        mxint8=model_setup.mxint8,
+        full_batch_size=model_setup.full_batch_size,
+        enable_qnn=model_setup.enable_qnn,
+    )


Yes we should add a check for custom IO file too

quic-hemagnih · 2025-07-23T08:40:20Z

tests/transformers/models/test_causal_lm_models.py

@@ -5,13 +5,15 @@
 #
 # -----------------------------------------------------------------------------

+import copy
+import json
 import os


Can you add changes of the file https://github.com/quic/efficient-transformers/pull/314/files#diff-fcc05ce2e83110153682cf6f488ad15825018072c3efd5e48266119cebe4e77a in this PR. I think some how we missed the swiftkv Unit test from our repo. I think since you are making changes in this file, we can add these changes too,

I will add it then.

Adding dummy causal models for testing in regular CI run

87c253f

Signed-off-by: Abukhoyer Shaik <[email protected]>

abukhoy requested review from quic-rishinr, ochougul, quic-hemagnih and quic-amitraj as code owners May 29, 2025 11:42

abukhoy added 9 commits May 30, 2025 06:34

Test config modification

366a0f4

Signed-off-by: Abukhoyer Shaik <[email protected]>

modification

4c01d13

Signed-off-by: Abukhoyer Shaik <[email protected]>

remove randomness in pytorch output

c217780

Signed-off-by: Abukhoyer Shaik <[email protected]>

cloud json fixed

9b1d9f9

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

ed840d1

Signed-off-by: Abukhoyer Shaik <[email protected]>

Linter Fixed

e9812fc

Signed-off-by: Abukhoyer Shaik <[email protected]>

cloud tests changed

4f54c39

Signed-off-by: Abukhoyer Shaik <[email protected]>

CLI tests Single thread

1981c60

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

1aabae0

quic-rishinr added the 1.21.0 label Jun 24, 2025

abukhoy added 5 commits June 25, 2025 09:56

duplicate models are added

951c242

Signed-off-by: Abukhoyer Shaik <[email protected]>

QNN Cli test fixed

73e4ff9

Signed-off-by: Abukhoyer Shaik <[email protected]>

qnn config path disabled

d1cc91e

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

5a48f92

CLI tests are refactored

b47f518

Signed-off-by: Abukhoyer Shaik <[email protected]>

quic-amitraj assigned abukhoy Jul 8, 2025

quic-amitraj added the ready for review label Jul 8, 2025

abukhoy added 6 commits July 9, 2025 10:50

Merge branch 'main' into tests-optim

204592c

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

26e6469

Merge branch 'main' into tests-optim

eafe1d7

Merge branch 'main' into tests-optim

f6c10e6

Merge branch 'main' into tests-optim

3c6680d

Merge branch 'main' into tests-optim

d2a53ee

quic-rishinr requested changes Jul 15, 2025

View reviewed changes

comments are addressing

bdf96e4

Signed-off-by: Abukhoyer Shaik <[email protected]>

abukhoy added 7 commits July 16, 2025 11:50

comments are addressing

174d33e

Signed-off-by: Abukhoyer Shaik <[email protected]>

created dummy models config for fast testing

27a7b2f

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

f0cef88

comments are addressing

85f13ac

Signed-off-by: Abukhoyer Shaik <[email protected]>

comments are addressing

78a4dcf

Signed-off-by: Abukhoyer Shaik <[email protected]>

comments are addressing

af2f825

Signed-off-by: Abukhoyer Shaik <[email protected]>

Merge branch 'main' into tests-optim

668ab2c

asmigosw reviewed Jul 22, 2025

View reviewed changes

comments addressing

d631fbd

Signed-off-by: Abukhoyer Shaik <[email protected]>

quic-hemagnih reviewed Jul 23, 2025

View reviewed changes

		)
		check_infer(mocker=mocker, generation_len=20, **local_config)

[Tests]: Adding dummy causal models for testing in regular CI run #427

Are you sure you want to change the base?

[Tests]: Adding dummy causal models for testing in regular CI run #427

Uh oh!

Conversation

abukhoy commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this PR:

What’s Changed:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abukhoy commented May 29, 2025 •

edited

Loading