feature: adding smarter unit test execution

Signed-off-by: Cortland Goffena <30168413+cmgoffena13@users.noreply.github.com>

Author: cmgoffena13

docs/concepts/overview.md

@@ -52,7 +52,9 @@ Bad data is worse than no data. The best way to keep bad data out of your system
 ### [Tests](./tests.md)
 SQLMesh "tests" are similar to unit tests in software development, where the unit is a single model. SQLMesh tests validate model *code* — you specify the input data and expected output, then SQLMesh runs the test and compares the expected and actual output.
 
-SQLMesh automatically runs tests when you apply a `plan`, or you can run them on demand with the [`test` command](../reference/cli.md#test).
+SQLMesh automatically runs tests for models included in a `plan` (added, modified, or restated). Plans with no such models, and `--skip-backfill` / `--dry-run` plans, skip unit tests by default. Use `--all-tests` for the full suite, `--skip-tests` to skip, or run tests on demand with the [`test` command](../reference/cli.md#test).
+
+Learn more in the [testing guide](../guides/testing.md).
 
 ### [Audits](./audits.md)
 In contrast to tests, SQLMesh "audits" validate the results of model code applied to your actual data.

docs/concepts/tests.md

@@ -1,6 +1,12 @@
 # Testing
 
-Testing allows you to protect your project from regression by continuously verifying that the output of each model matches your expectations. Unlike [audits](audits.md), tests are executed either on demand (for example, as part of a CI/CD job) or every time a new [plan](plans.md) is created.
+Testing allows you to protect your project from regression by continuously verifying that the output of each model matches your expectations. Unlike [audits](audits.md), tests are executed either on demand (for example, as part of a CI/CD job or via [`sqlmesh test`](../reference/cli.md#test)) or when a new [plan](plans.md) is created.
+
+By default, `sqlmesh plan` runs unit tests only for models included in the plan (added, modified, or restated). Plans with no such models skip unit tests. Use `--all-tests` to run the full suite, or `--skip-tests` to run none.
+
+!!! important
+
+    `--skip-backfill` / `--dry-run` plans skip unit tests by default. Pass `--all-tests` if you still want the full suite to run.
 
 Similar to unit testing in software development, SQLMesh evaluates the model's logic against predefined inputs and then compares the output to expected outcomes provided as part of each test.
 

docs/faq/faq.md

@@ -66,7 +66,7 @@
 
     However, the commands to create and apply a plan are different. In Terraform, the "plan" command generates a plan and saves it to file. The "apply" command reads a plan file and applies it.
 
-    In SQLMesh, the `sqlmesh plan` command generates a plan, runs any unit tests, and prompts the user to apply the plan. There is no "apply" command in SQLMesh.
+    In SQLMesh, the `sqlmesh plan` command generates a plan, runs unit tests for models included in the plan, and prompts the user to apply the plan. There is no "apply" command in SQLMesh.
 
 ## Getting Started
 
@@ -102,7 +102,7 @@
     SQLMesh's default behavior is appropriate for most deployments, but you can override where SQLMesh creates physical tables and views with [schema configuration options](../guides/configuration.md#environment-schemas).
 
 ??? question "What's the difference between a `test` and an `audit`?"
-    A SQLMesh [`test`](../concepts/tests.md) is analogous to a "unit test" in software engineering. It tests *code* based on known inputs and outputs. In SQLMesh, the inputs and outputs are specified in a YAML file, and SQLMesh automatically runs them when `sqlmesh plan` is executed.
+    A SQLMesh [`test`](../concepts/tests.md) is analogous to a "unit test" in software engineering. It tests *code* based on known inputs and outputs. In SQLMesh, the inputs and outputs are specified in a YAML file, and SQLMesh runs tests for models included in the plan when `sqlmesh plan` is executed (use `--all-tests` for the full suite).
 
     Writing YAML is annoying and error-prone, so SQLMesh's [`create_test` command](../concepts/tests.md#automatic-test-generation) allows you to automatically generate YAML test files based on queries of existing data tables.
 
@@ -126,7 +126,7 @@
 ??? question "What's the difference between `sqlmesh plan` and `sqlmesh run`?"
     During project development, there are two things in play: the current state of your project files and the existing states of each environment you have.
 
-    SQLMesh’s `plan` command is the primary tool for understanding the effects of changes you make to your project. If your project files have changed or are different from the state of an environment, you execute `sqlmesh plan [environment name]` to synchronize the environment's state with your project files. `sqlmesh plan` will generate a summary of the actions needed to implement the changes, automatically run unit tests, and prompt you to `apply` the plan and implement the changes.
+    SQLMesh’s `plan` command is the primary tool for understanding the effects of changes you make to your project. If your project files have changed or are different from the state of an environment, you execute `sqlmesh plan [environment name]` to synchronize the environment's state with your project files. `sqlmesh plan` will generate a summary of the actions needed to implement the changes, run unit tests for models included in the plan, and prompt you to `apply` the plan and implement the changes.
 
     If your project files have not changed, you execute `sqlmesh run` to run your project's models and audits.
 
@@ -210,7 +210,7 @@
     - Configure your project and set up a project database (using DuckDB locally or a database connection)
     - Create, configure, and modify models, audits, tests, and other project components
     - Execute `sqlmesh plan [environment name]` to:
-        - Generate a summary of the differences between your project files and the environment and whether each change is `breaking`. The `plan` includes a list of the actions needed to implement the changes and automatically runs the project's unit `test`s.
+        - Generate a summary of the differences between your project files and the environment and whether each change is `breaking`. The `plan` includes a list of the actions needed to implement the changes and runs unit `test`s for models included in the plan.
         - Optionally apply the plan to implement the actions and run the project's `audit`s.
     - Execute `sqlmesh run` on a schedule to ingest and transform new data
 

docs/guides/models.md

@@ -161,7 +161,11 @@ Reverting to a previous model version is a quick operation since no additional w
 
 SQLMesh automatically validates your models in order to ensure the quality and accuracy of your data. This is done via the following:
 
-* Running unit tests by default when you execute the `plan` command. This ensures all changes to applied to any environment are logically validated. Refer to [testing](../concepts/tests.md) for more information.
+* Running unit tests for models in the plan when you execute the `plan` command (use `--all-tests` for the full suite, or `--skip-tests` to skip). This ensures changes applied to any environment are logically validated. Refer to [testing](../concepts/tests.md) for more information.
+
+    !!! important
+
+        `--skip-backfill` / `--dry-run` plans skip unit tests by default. Pass `--all-tests` if you still want tests to run.
 * Running audits whenever data is loaded to a table (either for backfill or loading on a cadence). This way you know all data present in any table has passed all defined audits. Refer to [auditing](../concepts/audits.md) for more information.
 
 SQLMesh also provides automatic validation via CI/CD by automatically creating a preview environment.

docs/guides/testing.md

@@ -12,6 +12,12 @@ OK
 ```
 As the unit tests run, SQLMesh will identify any that fail.
 
+By default, `sqlmesh plan` runs unit tests only for models included in the plan (added, modified, or restated). Plans with no such models skip unit tests. Use `--all-tests` to run the full suite, or `--skip-tests` to run none.
+
+!!! important
+
+    `--skip-backfill` / `--dry-run` plans skip unit tests by default. Pass `--all-tests` if you still want the full suite to run.
+
 For more information about tests, refer to [testing](../concepts/tests.md).
 
 ### Test changes to a specific model

docs/reference/cli.md

@@ -357,6 +357,8 @@ Options:
                                   Default: prod.
   --skip-tests                    Skip tests prior to generating the plan if
                                   they are defined.
+  --all-tests                     Run all unit tests instead of only tests for
+                                  models included in the plan.
   --skip-linter                   Skip linting prior to generating the plan if
                                   the linter is enabled.
   -r, --restate-model TEXT        Restate data for specified models and models
@@ -371,7 +373,8 @@ Options:
                                   when comparing to existing snapshots for
                                   matching models in the target environment.
   --skip-backfill, --dry-run      Skip the backfill step and only create a
-                                  virtual update for the plan.
+                                  virtual update for the plan. Unit tests are
+                                  also skipped unless --all-tests is passed.
   --empty-backfill                Produce empty backfill. Like --skip-backfill
                                   no models will be backfilled, unlike --skip-
                                   backfill missing intervals will be recorded

docs/reference/notebook.md

@@ -96,7 +96,7 @@ options:
 #### plan
 ```
 %plan [--start START] [--end END] [--execution-time EXECUTION_TIME]
-            [--create-from CREATE_FROM] [--skip-tests]
+            [--create-from CREATE_FROM] [--skip-tests] [--all-tests]
             [--restate-model [RESTATE_MODEL ...]] [--no-gaps]
             [--skip-backfill, --dry-run] [--forward-only]
             [--effective-from EFFECTIVE_FROM] [--no-prompts] [--auto-apply]
@@ -120,6 +120,8 @@ options:
                         The environment to create the target environment from
                         if it doesn't exist. Default: prod.
   --skip-tests, -t      Skip the unit tests defined for the model.
+  --all-tests           Run all unit tests instead of only tests for models
+                        included in the plan.
   --restate-model <[RESTATE_MODEL ...]>, -r <[RESTATE_MODEL ...]>
                         Restate data for specified models (and models
                         downstream from the one specified). For production
@@ -131,7 +133,9 @@ options:
                         comparing to existing snapshots for matching models in
                         the target environment.
   --skip-backfill, --dry-run
-                        Skip the backfill step and only create a virtual update for the plan.
+                        Skip the backfill step and only create a virtual
+                        update for the plan. Unit tests are also skipped
+                        unless --all-tests is passed.
   --forward-only        Create a plan for forward-only changes.
   --effective-from EFFECTIVE_FROM
                         The effective date from which to apply forward-only

sqlmesh/cli/main.py

@@ -423,6 +423,12 @@ def diff(ctx: click.Context, environment: t.Optional[str] = None) -> None:
     help="Skip tests prior to generating the plan if they are defined.",
     default=None,
 )
+@click.option(
+    "--all-tests",
+    is_flag=True,
+    help="Run all unit tests instead of only tests for models included in the plan.",
+    default=None,
+)
 @click.option(
     "--skip-linter",
     is_flag=True,
@@ -446,7 +452,7 @@ def diff(ctx: click.Context, environment: t.Optional[str] = None) -> None:
     "--skip-backfill",
     "--dry-run",
     is_flag=True,
-    help="Skip the backfill step and only create a virtual update for the plan.",
+    help="Skip the backfill step and only create a virtual update for the plan. Unit tests are also skipped unless --all-tests is passed.",
     default=None,
 )
 @click.option(

sqlmesh/core/context.py

@@ -115,6 +115,7 @@
     ModelTestMetadata,
     generate_test,
     run_tests,
+    filter_tests_by_model_names,
     filter_tests_by_patterns,
 )
 from sqlmesh.core.user import User
@@ -1347,6 +1348,7 @@ def plan(
         execution_time: t.Optional[TimeLike] = None,
         create_from: t.Optional[str] = None,
         skip_tests: t.Optional[bool] = None,
+        all_tests: t.Optional[bool] = None,
         restate_models: t.Optional[t.Iterable[str]] = None,
         no_gaps: t.Optional[bool] = None,
         skip_backfill: t.Optional[bool] = None,
@@ -1384,6 +1386,7 @@ def plan(
             create_from: The environment to create the target environment from if it
                 doesn't exist. If not specified, the "prod" environment will be used.
             skip_tests: Unit tests are run by default so this will skip them if enabled
+            all_tests: Run every loaded unit test instead of only tests for models in the plan
             restate_models: A list of either internal or external models, or tags, that need to be restated
                 for the given plan interval. If the target environment is a production environment,
                 ALL snapshots that depended on these upstream tables will have their intervals deleted
@@ -1430,6 +1433,7 @@ def plan(
             execution_time=execution_time,
             create_from=create_from,
             skip_tests=skip_tests,
+            all_tests=all_tests,
             restate_models=restate_models,
             no_gaps=no_gaps,
             skip_backfill=skip_backfill,
@@ -1484,6 +1488,7 @@ def plan_builder(
         execution_time: t.Optional[TimeLike] = None,
         create_from: t.Optional[str] = None,
         skip_tests: t.Optional[bool] = None,
+        all_tests: t.Optional[bool] = None,
         restate_models: t.Optional[t.Iterable[str]] = None,
         no_gaps: t.Optional[bool] = None,
         skip_backfill: t.Optional[bool] = None,
@@ -1518,6 +1523,7 @@ def plan_builder(
             create_from: The environment to create the target environment from if it
                 doesn't exist. If not specified, the "prod" environment will be used.
             skip_tests: Unit tests are run by default so this will skip them if enabled
+            all_tests: Run every loaded unit test instead of only tests for models in the plan
             restate_models: A list of either internal or external models, or tags, that need to be restated
                 for the given plan interval. If the target environment is a production environment,
                 ALL snapshots that depended on these upstream tables will have their intervals deleted
@@ -1559,6 +1565,7 @@ def plan_builder(
             "execution_time": execution_time,
             "create_from": create_from,
             "skip_tests": skip_tests,
+            "all_tests": all_tests,
             "restate_models": list(restate_models) if restate_models is not None else None,
             "no_gaps": no_gaps,
             "skip_backfill": skip_backfill,
@@ -1588,6 +1595,9 @@ def plan_builder(
         }
 
         skip_tests = explain or skip_tests or False
+        all_tests = all_tests or False
+        if skip_tests and all_tests:
+            raise PlanError("Cannot combine --all-tests with --skip-tests.")
         no_gaps = no_gaps or False
         skip_backfill = skip_backfill or False
         empty_backfill = empty_backfill or False
@@ -1596,6 +1606,10 @@ def plan_builder(
         skip_linter = skip_linter or False
         min_intervals = min_intervals or 0
 
+        # Virtual-only plans (--skip-backfill / --dry-run) skip unit tests unless --all-tests.
+        if skip_backfill and not all_tests:
+            skip_tests = True
+
         environment = environment or self.config.default_target_environment
         environment = Environment.sanitize_name(environment)
         is_dev = environment != c.PROD
@@ -1614,8 +1628,6 @@ def plan_builder(
         if not skip_linter:
             self.lint_models()
 
-        self._run_plan_tests(skip_tests=skip_tests)
-
         environment_ttl = (
             self.environment_ttl if environment not in self.pinned_environments else None
         )
@@ -1698,6 +1710,15 @@ def plan_builder(
             *[s.name for s in context_diff.added],
         }
 
+        self._run_plan_tests(
+            skip_tests=skip_tests,
+            all_tests=all_tests,
+            model_names={
+                *modified_model_names,
+                *(expanded_restate_models or set()),
+            },
+        )
+
         if (
             is_dev
             and not include_unmodified
@@ -2314,14 +2335,15 @@ def test(
         verbosity: Verbosity = Verbosity.DEFAULT,
         preserve_fixtures: bool = False,
         stream: t.Optional[t.TextIO] = None,
+        model_names: t.Optional[t.Collection[str]] = None,
     ) -> ModelTextTestResult:
         """Discover and run model tests"""
         if verbosity >= Verbosity.VERBOSE:
             import pandas as pd
 
             pd.set_option("display.max_columns", None)
 
-        test_meta = self.select_tests(tests=tests, patterns=match_patterns)
+        test_meta = self.select_tests(tests=tests, patterns=match_patterns, model_names=model_names)
 
         result = run_tests(
             model_test_metadata=test_meta,
@@ -2781,15 +2803,24 @@ def _run_tests(
         result = self.test(stream=test_output_io, verbosity=verbosity)
         return result, test_output_io.getvalue()
 
-    def _run_plan_tests(self, skip_tests: bool = False) -> t.Optional[ModelTextTestResult]:
-        if not skip_tests:
-            result = self.test()
-            if not result.wasSuccessful():
-                raise PlanError(
-                    "Cannot generate plan due to failing test(s). Fix test(s) and run again."
-                )
-            return result
-        return None
+    def _run_plan_tests(
+        self,
+        skip_tests: bool = False,
+        all_tests: bool = False,
+        model_names: t.Optional[t.Collection[str]] = None,
+    ) -> t.Optional[ModelTextTestResult]:
+        if skip_tests:
+            return None
+
+        if not all_tests and model_names is not None and not model_names:
+            return None
+
+        result = self.test(model_names=None if all_tests else model_names)
+        if not result.wasSuccessful():
+            raise PlanError(
+                "Cannot generate plan due to failing test(s). Fix test(s) and run again."
+            )
+        return result
 
     def _warn_if_virtual_catalog_rematerialization(self, plan: "Plan") -> None:
         """Warn when ClickHouse models appear as new snapshots solely because a virtual catalog
@@ -3465,6 +3496,7 @@ def select_tests(
         self,
         tests: t.Optional[t.List[str]] = None,
         patterns: t.Optional[t.List[str]] = None,
+        model_names: t.Optional[t.Collection[str]] = None,
     ) -> t.List[ModelTestMetadata]:
         """Filter pre-loaded test metadata based on tests and patterns."""
 
@@ -3488,6 +3520,14 @@ def select_tests(
         if patterns:
             test_meta = filter_tests_by_patterns(test_meta, patterns)
 
+        if model_names is not None:
+            test_meta = filter_tests_by_model_names(
+                test_meta,
+                set(model_names),
+                default_catalog=self.default_catalog,
+                dialect=self.default_dialect,
+            )
+
         return test_meta
 
 

sqlmesh/core/test/__init__.py

@@ -3,6 +3,7 @@
 from sqlmesh.core.test.definition import ModelTest as ModelTest, generate_test as generate_test
 from sqlmesh.core.test.discovery import (
     ModelTestMetadata as ModelTestMetadata,
+    filter_tests_by_model_names as filter_tests_by_model_names,
     filter_tests_by_patterns as filter_tests_by_patterns,
 )
 from sqlmesh.core.test.result import ModelTextTestResult as ModelTextTestResult