feature(testing): --test-changed-only on plan and --select-model on test command

Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: cmgoffena13

docs/concepts/overview.md

@@ -52,7 +52,7 @@ Bad data is worse than no data. The best way to keep bad data out of your system
 ### [Tests](./tests.md)
 SQLMesh "tests" are similar to unit tests in software development, where the unit is a single model. SQLMesh tests validate model *code* — you specify the input data and expected output, then SQLMesh runs the test and compares the expected and actual output.
 
-SQLMesh automatically runs tests for models included in a `plan` (added, modified, or restated). Plans with no such models skip unit tests by default. Use `--all-tests` for the full suite, `--skip-tests` to skip, or run tests on demand with the [`test` command](../reference/cli.md#test).
+SQLMesh automatically runs all unit tests when a `plan` is created. Use `--test-changed-only` to run tests only for models included in the plan (added, modified, or restated), `--skip-tests` to skip, run tests for specific models with [`sqlmesh test --select-model`](../reference/cli.md#test), or run the full suite on demand with the [`test` command](../reference/cli.md#test).
 
 Learn more in the [testing guide](../guides/testing.md).
 

docs/concepts/tests.md

@@ -2,7 +2,7 @@
 
 Testing allows you to protect your project from regression by continuously verifying that the output of each model matches your expectations. Unlike [audits](audits.md), tests are executed either on demand (for example, as part of a CI/CD job or via [`sqlmesh test`](../reference/cli.md#test)) or when a new [plan](plans.md) is created.
 
-By default, `sqlmesh plan` runs unit tests only for models included in the plan (added, modified, or restated). Plans with no such models skip unit tests. Use `--all-tests` to run the full suite, or `--skip-tests` to run none.
+By default, `sqlmesh plan` runs all unit tests. Use `--test-changed-only` to run tests only for models included in the plan (added, modified, or restated), or `--skip-tests` to run none. With both `--select-model` and `--test-changed-only`, tests run only for selected models that changed.
 
 Similar to unit testing in software development, SQLMesh evaluates the model's logic against predefined inputs and then compares the output to expected outcomes provided as part of each test.
 

docs/faq/faq.md

@@ -102,7 +102,7 @@
     SQLMesh's default behavior is appropriate for most deployments, but you can override where SQLMesh creates physical tables and views with [schema configuration options](../guides/configuration.md#environment-schemas).
 
 ??? question "What's the difference between a `test` and an `audit`?"
-    A SQLMesh [`test`](../concepts/tests.md) is analogous to a "unit test" in software engineering. It tests *code* based on known inputs and outputs. In SQLMesh, the inputs and outputs are specified in a YAML file, and SQLMesh runs tests for models included in the plan when `sqlmesh plan` is executed (use `--all-tests` for the full suite).
+    A SQLMesh [`test`](../concepts/tests.md) is analogous to a "unit test" in software engineering. It tests *code* based on known inputs and outputs. In SQLMesh, the inputs and outputs are specified in a YAML file, and SQLMesh runs all unit tests when `sqlmesh plan` is executed (use `--test-changed-only` to run only tests for models in the plan).
 
     Writing YAML is annoying and error-prone, so SQLMesh's [`create_test` command](../concepts/tests.md#automatic-test-generation) allows you to automatically generate YAML test files based on queries of existing data tables.
 

docs/guides/models.md

@@ -161,7 +161,7 @@ Reverting to a previous model version is a quick operation since no additional w
 
 SQLMesh automatically validates your models in order to ensure the quality and accuracy of your data. This is done via the following:
 
-* Running unit tests for models in the plan when you execute the `plan` command (use `--all-tests` for the full suite, or `--skip-tests` to skip). This ensures changes applied to any environment are logically validated. Refer to [testing](../concepts/tests.md) for more information.
+* Running all unit tests when you execute the `plan` command (use `--test-changed-only` to run only tests for models in the plan, or `--skip-tests` to skip). This ensures changes applied to any environment are logically validated. Refer to [testing](../concepts/tests.md) for more information.
 * Running audits whenever data is loaded to a table (either for backfill or loading on a cadence). This way you know all data present in any table has passed all defined audits. Refer to [auditing](../concepts/audits.md) for more information.
 
 SQLMesh also provides automatic validation via CI/CD by automatically creating a preview environment.

docs/guides/testing.md

@@ -12,13 +12,19 @@ OK
 ```
 As the unit tests run, SQLMesh will identify any that fail.
 
-By default, `sqlmesh plan` runs unit tests only for models included in the plan (added, modified, or restated). Plans with no such models skip unit tests. Use `--all-tests` to run the full suite, or `--skip-tests` to run none.
+By default, `sqlmesh plan` runs all unit tests. Use `--test-changed-only` to run tests only for models included in the plan (added, modified, or restated), or `--skip-tests` to run none. With both `--select-model` and `--test-changed-only`, tests run only for selected models that changed.
 
 For more information about tests, refer to [testing](../concepts/tests.md).
 
 ### Test changes to a specific model
 
-To run a specific model test, pass in the suite file name followed by `::` and the name of the test; for example: `sqlmesh test tests/test_suite.yaml::test_example_full_model`.
+To run unit tests for a specific model, use `--select-model`:
+
+```bash
+$ sqlmesh test --select-model sqlmesh_example.full_model
+```
+
+Alternatively, pass in the suite file name followed by `::` and the name of the test; for example: `sqlmesh test tests/test_suite.yaml::test_example_full_model`.
 
 ### Run a subset of tests
 

docs/reference/cli.md

@@ -357,8 +357,8 @@ Options:
                                   Default: prod.
   --skip-tests                    Skip tests prior to generating the plan if
                                   they are defined.
-  --all-tests                     Run all unit tests instead of only tests for
-                                  models included in the plan.
+  --test-changed-only             Run unit tests only for models included in
+                                  the plan instead of all tests.
   --skip-linter                   Skip linting prior to generating the plan if
                                   the linter is enabled.
   -r, --restate-model TEXT        Restate data for specified models and models
@@ -628,6 +628,8 @@ Options:
   -v, --verbose        Verbose output.
   --preserve-fixtures  Preserve the fixture tables in the testing database,
                        useful for debugging.
+  --select-model TEXT  Select specific models to run unit tests for. Can be
+                       specified multiple times.
   --help               Show this message and exit.
 ```
 

docs/reference/notebook.md

@@ -96,7 +96,7 @@ options:
 #### plan
 ```
 %plan [--start START] [--end END] [--execution-time EXECUTION_TIME]
-            [--create-from CREATE_FROM] [--skip-tests] [--all-tests]
+            [--create-from CREATE_FROM] [--skip-tests] [--test-changed-only]
             [--restate-model [RESTATE_MODEL ...]] [--no-gaps]
             [--skip-backfill, --dry-run] [--forward-only]
             [--effective-from EFFECTIVE_FROM] [--no-prompts] [--auto-apply]
@@ -120,8 +120,8 @@ options:
                         The environment to create the target environment from
                         if it doesn't exist. Default: prod.
   --skip-tests, -t      Skip the unit tests defined for the model.
-  --all-tests           Run all unit tests instead of only tests for models
-                        included in the plan.
+  --test-changed-only   Run unit tests only for models included in the plan
+                        instead of all tests.
   --restate-model <[RESTATE_MODEL ...]>, -r <[RESTATE_MODEL ...]>
                         Restate data for specified models (and models
                         downstream from the one specified). For production
@@ -433,7 +433,8 @@ options:
 
 #### run_test
 ```
-%run_test [--pattern [PATTERN ...]] [--verbose] [--preserve-fixtures] [tests ...]
+%run_test [--pattern [PATTERN ...]] [--verbose] [--preserve-fixtures]
+          [--select-model [SELECT_MODEL ...]] [tests ...]
 
 Run unit test(s).
 
@@ -446,6 +447,8 @@ options:
   --verbose, -v         Verbose output.
   --preserve-fixtures   Preserve the fixture tables in the testing database,
                         useful for debugging.
+  --select-model <[SELECT_MODEL ...]>
+                        Select specific models to run unit tests for.
 ```
 
 #### audit

sqlmesh/cli/main.py

@@ -424,9 +424,9 @@ def diff(ctx: click.Context, environment: t.Optional[str] = None) -> None:
     default=None,
 )
 @click.option(
-    "--all-tests",
+    "--test-changed-only",
     is_flag=True,
-    help="Run all unit tests instead of only tests for models included in the plan.",
+    help="Run unit tests only for models included in the plan instead of all tests.",
     default=None,
 )
 @click.option(
@@ -801,6 +801,12 @@ def create_test(
     default=False,
     help="Preserve the fixture tables in the testing database, useful for debugging.",
 )
+@click.option(
+    "--select-model",
+    type=str,
+    multiple=True,
+    help="Select specific models to run unit tests for.",
+)
 @click.argument("tests", nargs=-1)
 @click.pass_obj
 @error_handler
@@ -810,14 +816,19 @@ def test(
     k: t.List[str],
     verbose: int,
     preserve_fixtures: bool,
+    select_model: t.List[str],
     tests: t.List[str],
 ) -> None:
     """Run model unit tests."""
+    model_names = (
+        obj._new_selector().expand_model_selections(select_model) if select_model else None
+    )
     result = obj.test(
         match_patterns=k,
         tests=tests,
         verbosity=Verbosity(verbose),
         preserve_fixtures=preserve_fixtures,
+        model_names=model_names,
     )
     if not result.wasSuccessful():
         exit(1)

sqlmesh/cli/project_init.py

@@ -286,6 +286,25 @@ def _gen_example_objects(schema_name: str) -> ExampleObjects:
         num_orders: 1
   """
 
+    tests["test_incremental_model"] = f"""test_example_incremental_model:
+  model: {incremental_model_name}
+  vars:
+    start: 2020-01-01
+    end: 2020-01-02
+  inputs:
+    {seed_model_name}:
+      rows:
+      - id: 1
+        item_id: 1
+        event_date: 2020-01-01
+  outputs:
+    query:
+      rows:
+      - id: 1
+        item_id: 1
+        event_date: 2020-01-01
+  """
+
     return ExampleObjects(
         sql_models=sql_models,
         python_models=python_models,

sqlmesh/core/console.py

@@ -2232,6 +2232,9 @@ def log_test_results(self, result: ModelTextTestResult, target_dialect: str) ->
         message = (
             f"Ran {result.testsRun} tests against {target_dialect} in {result.duration} seconds."
         )
+        if result.tests_skipped:
+            message = f"{message}\nSkipped {result.tests_skipped} tests"
+
         if result.wasSuccessful():
             self._print("=" * divider_length)
             self._print(
@@ -3160,6 +3163,8 @@ def log_test_results(self, result: ModelTextTestResult, target_dialect: str) ->
         message = (
             f"Ran {result.testsRun} tests against {target_dialect} in {result.duration} seconds."
         )
+        if result.tests_skipped:
+            message = f"{message}\nSkipped {result.tests_skipped} tests"
 
         if result.wasSuccessful():
             success_color = {"color": "#008000"}
@@ -3600,6 +3605,8 @@ def log_test_results(self, result: ModelTextTestResult, target_dialect: str) ->
             return
 
         message = f"Ran `{result.testsRun}` Tests Against `{target_dialect}`"
+        if result.tests_skipped:
+            message = f"{message}\n**Skipped `{result.tests_skipped}` Tests**"
 
         if result.wasSuccessful():
             self._print(f"**Successfully {message}**\n\n")