Perf: add opt-in project-index loading for lint (#5913)

Signed-off-by: Andreas Fredhøi <andreas.fredhoi@fresio.no>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: fresioAS

docs/guides/configuration.md

@@ -329,6 +329,16 @@ By default, the SQLMesh cache is stored in a `.cache` directory within your proj
 
 The cache directory is automatically created if it doesn't exist. You can clear the cache using the `sqlmesh clean` command.
 
+#### Project index
+
+The `--use-project-index` option on supported commands maintains a persistent model dependency index in the cache directory. Each project writes a file named `<project>_<hash>_model_index.json`.
+
+A full project load with the option enabled creates or refreshes the index. SQLMesh invalidates it when relevant configuration, gateway, macro, audit, or signal metadata changes, or when the set of model files changes. If the index is missing, invalid, or stale, SQLMesh safely falls back to a full project load and rebuilds it.
+
+For operations targeting selected models, the index allows SQLMesh to load only those models and their upstream dependencies.
+
+In multi-repository projects, dependencies that cross project boundaries may not be represented by an individual project's index. SQLMesh detects incomplete scoped loads and falls back to loading the full configured project set.
+
 ### Table/view storage locations
 
 SQLMesh creates schemas, physical tables, and views in the data warehouse/engine. Learn more about why and how SQLMesh creates schema in the ["Why does SQLMesh create schemas?" FAQ](../faq/faq.md#schema-question).

docs/guides/linter.md

@@ -135,6 +135,21 @@ $ sqlmesh lint --local
 
 This can make linting faster in repositories where all referenced models are loaded from local files. In multi-repository setups, or when linting only a subset of projects, `--local` may cause additional linting errors because SQLMesh will not resolve references or schemas from models that exist only in remote state.
 
+For faster targeted linting, enable the persistent project index with `--use-project-index`. When
+models are selected with `--model`, SQLMesh loads, resolves, and validates only those models and
+their transitive upstream dependencies. The same behavior can be enabled by default for the
+Python API and CLI with the `linter.use_project_index` configuration option:
+
+```yaml
+linter:
+  enabled: true
+  use_project_index: true
+```
+
+`Context.lint_models` uses this configuration value when `use_project_index` is omitted. Passing
+`use_project_index=False` explicitly disables it for that call. If a context was already loaded,
+an indexed lint of selected models reloads the context so the requested scope is applied.
+
 
 ## Applying linting rules
 

docs/reference/cli.md

@@ -650,9 +650,12 @@ Usage: sqlmesh lint [OPTIONS]
 
 Options:
   --model TEXT           A model to lint. Multiple models can be linted.  If no models are specified, every model will be linted.
+  --use-project-index    Use the persistent project index. With --model, only the selected models and their upstream dependencies
+                         are loaded, resolved, and validated, so errors in unrelated models are not reported. Without --model,
+                         every model is still loaded and linted.
   --local                Lint using only locally loaded project files without loading state. In multi-repository setups, or when
                          linting only a subset of projects, this may cause additional linting errors because SQLMesh will not resolve
                          references or schemas from models that exist only in remote state.
   --help                 Show this message and exit.
 
-```
+```

docs/reference/configuration.md

@@ -60,6 +60,13 @@ The `model_defaults` key is **required** and must contain a value for the `diale
 
 See all the keys allowed in `model_defaults` at the [model configuration reference page](./model_configuration.md#model-defaults).
 
+### Linter
+
+| Option                    | Description                                                                                                                                        | Type    | Required |
+|---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|---------|----------|
+| `linter.enabled`          | Whether linting is enabled (Default: `False`)                                                                                                      | boolean | N        |
+| `linter.use_project_index` | Whether to use the persistent project index for linting. Targeted linting loads selected models and their upstream dependencies. (Default: `False`) | boolean | N        |
+
 ### Variables
 
 The `variables` key can be used to provide values for user-defined variables, accessed using the [`@VAR` macro function](../concepts/macros/sqlmesh_macros.md#global-variables) in SQL model definitions, [`context.var` method](../concepts/models/python_models.md#global-variables) in Python model definitions, and [`evaluator.var` method](../concepts/macros/sqlmesh_macros.md#accessing-global-variable-values) in Python macro functions.

sqlmesh/cli/main.py

@@ -141,6 +141,10 @@ def cli(
         if ctx.invoked_subcommand in SKIP_LOAD_COMMANDS:
             load = False
 
+    # Unlike the other commands above, lint can scope its own load for multi-project contexts.
+    if ctx.invoked_subcommand == "lint":
+        load = False
+
     configs = load_configs(config, Context.CONFIG_TYPE, paths, dotenv_path=dotenv)
     log_limit = list(configs.values())[0].log_limit
 
@@ -1209,6 +1213,12 @@ def environments(obj: Context) -> None:
     multiple=True,
     help="A model to lint. Multiple models can be linted. If no models are specified, every model will be linted.",
 )
+@click.option(
+    "--use-project-index",
+    is_flag=True,
+    default=None,
+    help="Use the persistent project index. With --model, only the selected models and their upstream dependencies are loaded, resolved, and validated, so errors in unrelated models are not reported. Without --model, every model is still loaded and linted. Can also be enabled with linter.use_project_index.",
+)
 @click.option(
     "--local",
     is_flag=True,
@@ -1221,9 +1231,18 @@ def environments(obj: Context) -> None:
 def lint(
     obj: Context,
     models: t.Iterator[str],
+    use_project_index: t.Optional[bool],
 ) -> None:
     """Run the linter for the target model(s)."""
-    obj.lint_models(models)
+    obj.lint_models(
+        models,
+        use_project_index=use_project_index,
+    )
+
+    if not obj.models:
+        raise click.ClickException(
+            f"`{obj.path}` doesn't seem to have any models... cd into the proper directory or specify the path(s) with -p."
+        )
 
 
 @cli.group(no_args_is_help=True)

sqlmesh/core/config/linter.py

@@ -16,13 +16,16 @@ class LinterConfig(BaseConfig):
     Args:
         enabled: Flag indicating whether the linter should run
 
+        use_project_index: Whether to use the persistent project index when linting.
+
         rules: A list of error rules to be applied on model
         warn_rules: A list of rules to be applied on models but produce warnings instead of raising errors.
         ignored_rules: A list of rules to be excluded/ignored
 
     """
 
     enabled: bool = False
+    use_project_index: bool = False
 
     rules: t.Set[str] = set()
     warn_rules: t.Set[str] = set()

sqlmesh/core/context.py

@@ -418,6 +418,7 @@ def __init__(
         self._linters: t.Dict[str, Linter] = {}
         self._loaded: bool = False
         self._load_state: bool = load_state
+        self._uncached_model_names: t.Set[str] = set()
         self._selector_cls = selector or NativeSelector
 
         self.path, self.config = t.cast(t.Tuple[Path, C], next(iter(self.configs.items())))
@@ -641,11 +642,31 @@ def refresh(self) -> None:
         if any(loader.reload_needed() for loader in self._loaders):
             self.load()
 
-    def load(self, update_schemas: bool = True) -> GenericContext[C]:
-        """Load all files in the context's path."""
+    def load(
+        self,
+        update_schemas: bool = True,
+        model_fqns: t.Optional[t.Set[str]] = None,
+        use_project_index: bool = False,
+    ) -> GenericContext[C]:
+        """Load files in the context's path, optionally scoped to specific models.
+
+        Args:
+            update_schemas: Whether to update model schemas and validate model definitions.
+            model_fqns: If provided with ``use_project_index=True``, only the selected models
+                and their transitive upstream dependencies are loaded.
+            use_project_index: Whether to use and maintain the persistent project model index.
+                When ``model_fqns`` is not provided, all models are loaded and the index is
+                refreshed for future scoped loads.
+        """
         load_start_ts = time.perf_counter()
 
-        loaded_projects = [loader.load() for loader in self._loaders]
+        loaded_projects = [
+            loader.load(
+                model_fqns=model_fqns,
+                use_project_index=use_project_index,
+            )
+            for loader in self._loaders
+        ]
 
         self.dag = DAG()
         self._standalone_audits.clear()
@@ -688,6 +709,27 @@ def load(self, update_schemas: bool = True) -> GenericContext[C]:
                 BUILTIN_RULES.union(project.user_rules), config.linter
             )
 
+        indexed_model_fqns = {
+            fqn for project in loaded_projects for fqn in (project.indexed_model_fqns or set())
+        }
+        if model_fqns and (
+            not model_fqns <= self._models.keys()
+            or any(
+                dependency in indexed_model_fqns and dependency not in self._models
+                for model in self._models.values()
+                for dependency in model.depends_on
+            )
+        ):
+            # A missing or stale index, a new model, or a dependency crossing project
+            # boundaries requires a full load to preserve existing behavior.
+            self.load(
+                update_schemas=False,
+                use_project_index=use_project_index,
+            )
+            if update_schemas:
+                self._update_model_schemas_and_validate(model_fqns)
+            return self
+
         # Load environment statements from state for projects not in current load
         if self._load_state and any(self._projects):
             prod = self.state_reader.get_environment(c.PROD)
@@ -713,34 +755,13 @@ def load(self, update_schemas: bool = True) -> GenericContext[C]:
                         else:
                             local_store[snapshot.name] = snapshot.node  # type: ignore
 
+        self._uncached_model_names = uncached
+
         for model in self._models.values():
             self.dag.add(model.fqn, model.depends_on)
 
         if update_schemas:
-            for fqn in self.dag:
-                model = self._models.get(fqn)  # type: ignore
-
-                if not model or fqn in uncached:
-                    continue
-
-                # make a copy of remote models that depend on local models or in the downstream chain
-                # without this, a SELECT * FROM local will not propogate properly because the downstream
-                # model will get mutated (schema changes) but the object is the same as the remote cache
-                if any(dep in uncached for dep in model.depends_on):
-                    uncached.add(fqn)
-                    self._models.update({fqn: model.copy(update={"mapping_schema": {}})})
-                    continue
-
-            update_model_schemas(
-                self.dag,
-                models=self._models,
-                cache_dir=self.cache_dir,
-            )
-
-            models = self.models.values()
-            for model in models:
-                # The model definition can be validated correctly only after the schema is set.
-                model.validate_definition()
+            self._update_model_schemas_and_validate(model_fqns or None)
 
         duplicates = set(self._models) & set(self._standalone_audits)
         if duplicates:
@@ -767,6 +788,53 @@ def load(self, update_schemas: bool = True) -> GenericContext[C]:
         self._loaded = True
         return self
 
+    def _update_model_schemas_and_validate(self, model_fqns: t.Optional[t.Set[str]] = None) -> None:
+        """Updates the mapping schemas of the given models (all models by default) and validates their definitions.
+
+        Args:
+            model_fqns: If provided, only these models and their transitive upstream
+                dependencies are processed.
+        """
+        if model_fqns is not None:
+            model_fqns = {
+                fqn for target in model_fqns for fqn in (target, *self.dag.upstream(target))
+            }
+
+        uncached = set(self._uncached_model_names)
+
+        for fqn in self.dag:
+            if model_fqns is not None and fqn not in model_fqns:
+                continue
+
+            model = self._models.get(fqn)
+
+            if not model or fqn in uncached:
+                continue
+
+            # make a copy of remote models that depend on local models or in the downstream chain
+            # without this, a SELECT * FROM local will not propogate properly because the downstream
+            # model will get mutated (schema changes) but the object is the same as the remote cache
+            if any(dep in uncached for dep in model.depends_on):
+                uncached.add(fqn)
+                self._models.update({fqn: model.copy(update={"mapping_schema": {}})})
+                continue
+
+        models = self._models
+        if model_fqns is not None:
+            models = UniqueKeyDict(
+                "models", {fqn: model for fqn, model in self._models.items() if fqn in model_fqns}
+            )
+
+        update_model_schemas(
+            self.dag,
+            models=models,
+            cache_dir=self.cache_dir,
+        )
+
+        for model in models.values():
+            # The model definition can be validated correctly only after the schema is set.
+            model.validate_definition()
+
     @python_api_analytics
     def run(
         self,
@@ -3435,7 +3503,42 @@ def lint_models(
         self,
         models: t.Optional[t.Iterable[t.Union[str, Model]]] = None,
         raise_on_error: bool = True,
+        use_project_index: t.Optional[bool] = None,
     ) -> t.List[AnnotatedRuleViolation]:
+        """Lint the selected models.
+
+        Args:
+            models: Models to lint. If omitted, all loaded models are linted.
+            raise_on_error: Whether to raise when an error-level violation is found.
+            use_project_index: Whether to use the persistent project index. If omitted, the
+                value of ``linter.use_project_index`` is used. Indexed linting of selected
+                models reloads an already-loaded context so the requested scope is applied.
+        """
+        models = list(models) if models is not None else []
+        use_project_index = (
+            self.config.linter.use_project_index if use_project_index is None else use_project_index
+        )
+
+        target_fqns = (
+            {
+                normalize_model_name(
+                    model,
+                    default_catalog=self.default_catalog,
+                    dialect=self.default_dialect,
+                )
+                if isinstance(model, str)
+                else model.fqn
+                for model in models
+            }
+            if models and use_project_index
+            else None
+        )
+
+        # An already-loaded context does not otherwise enter the loading path. Reload when
+        # indexed linting is requested for specific models so the scope is actually applied.
+        if not self._loaded or target_fqns is not None:
+            self.load(model_fqns=target_fqns, use_project_index=use_project_index)
+
         found_error = False
 
         model_list = (