Perf: add opt-in project-index planning

docs/examples/sqlmesh_cli_crash_course.md

@@ -738,6 +738,11 @@ You'll use these commands as needed to validate that your changes are behaving a
 
 This is a great way to verify that your model's SQL is looking as expected before applying the changes. It is especially important if you're migrating from one query engine to another (ex: postgres to databricks).
 
+In large projects, add `--use-project-index` to load only the model being rendered and its upstream
+dependencies. To enable this behavior by default, set
+[`render.use_project_index`](../reference/configuration.md#render) to `true` in the project
+configuration.
+
 === "SQLMesh"
 
     ```bash
@@ -1254,4 +1259,4 @@ If you notice you have a lot of old development schemas/data, you can clean them
 
     ```bash
     tcloud sqlmesh janitor
-    ```
+    ```

docs/guides/configuration.md

@@ -90,6 +90,7 @@ The `config` sub-module API documentation describes the individual classes used
     - [Connection configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/config/connection.html) (separate classes for each supported database/engine)
     - [Scheduler configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/config/scheduler.html) (separate classes for each supported scheduler)
 - [Plan change categorization configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/config/categorizer.html#CategorizerConfig): `CategorizerConfig()`
+- [Render configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/config/render.html): `RenderConfig()`
 - [User configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/user.html#User): `User()`
 - [Notification configuration](https://sqlmesh.readthedocs.io/en/latest/_readthedocs/html/sqlmesh/core/notification_target.html) (separate classes for each notification target)
 
@@ -331,14 +332,41 @@ The cache directory is automatically created if it doesn't exist. You can clear
 
 #### Project index
 
-The `--use-project-index` option on supported commands maintains a persistent model dependency index in the cache directory. Each project writes a file named `<project>_<hash>_model_index.json`.
+The `--use-project-index` option on the `lint`, `plan`, and `render` commands maintains a persistent model dependency index in the cache directory. Each project writes a file named `<project>_<hash>_model_index.json`. The option can be enabled by default for each command with `linter.use_project_index`, `plan.use_project_index`, or `render.use_project_index`, respectively.
 
 A full project load with the option enabled creates or refreshes the index. SQLMesh invalidates it when relevant configuration, gateway, macro, audit, or signal metadata changes, or when the set of model files changes. If the index is missing, invalid, or stale, SQLMesh safely falls back to a full project load and rebuilds it.
 
 For operations targeting selected models, the index allows SQLMesh to load only those models and their upstream dependencies.
 
 In multi-repository projects, dependencies that cross project boundaries may not be represented by an individual project's index. SQLMesh detects incomplete scoped loads and falls back to loading the full configured project set.
 
+#### Indexed planning
+
+Indexed planning also reuses snapshot state already loaded while building the plan and scopes graph
+work to changed or selected model lineage. It does not change the resulting plan. Enable it by
+default for the CLI and Python API with `plan.use_project_index`:
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    plan:
+      use_project_index: true
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, PlanConfig
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect="duckdb"),
+        plan=PlanConfig(use_project_index=True),
+    )
+    ```
+
+`Context.plan` and `Context.plan_builder` use this configuration value when `use_project_index` is
+omitted. Passing `use_project_index=False` explicitly disables indexed planning for that API call.
+
 ### Table/view storage locations
 
 SQLMesh creates schemas, physical tables, and views in the data warehouse/engine. Learn more about why and how SQLMesh creates schema in the ["Why does SQLMesh create schemas?" FAQ](../faq/faq.md#schema-question).
@@ -1503,6 +1531,35 @@ SQLMesh provides a linter that checks for potential issues in your models' code.
 
 Learn more about linting configuration in the [linting guide](./linter.md).
 
+### Rendering
+
+By default, `sqlmesh render` loads every model in the project. In large projects, you can use the
+persistent project index to load only the model being rendered and its transitive upstream
+dependencies. Enable indexed rendering for an individual command with `--use-project-index`, or
+make it the project default with the `render.use_project_index` configuration option.
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    render:
+      use_project_index: true
+    ```
+
+=== "Python"
+
+    ```python linenums="1"
+    from sqlmesh.core.config import Config, ModelDefaultsConfig, RenderConfig
+
+    config = Config(
+        model_defaults=ModelDefaultsConfig(dialect="duckdb"),
+        render=RenderConfig(use_project_index=True),
+    )
+    ```
+
+`Context.render` uses the configured value when `use_project_index` is omitted. Passing
+`use_project_index=False` explicitly disables indexed rendering for that API call. See the
+[`render` CLI reference](../reference/cli.md#render) for the other rendering options.
+
 ### Debug mode
 
 To enable debug mode set the `SQLMESH_DEBUG` environment variable to one of the following values: "1", "true", "t", "yes" or "y".

docs/reference/cli.md

@@ -405,6 +405,17 @@ Options:
                                   versions of the models and standalone
                                   audits.
   --explain                       Explain the plan instead of applying it.
+  --ignore-cron                   Run all missing intervals, ignoring
+                                  individual cron schedules. Only applies if
+                                  --run is set.
+  --min-intervals TEXT            For every model, ensure at least this many
+                                  intervals are covered by a missing intervals
+                                  check regardless of the plan start date
+  --use-project-index             Refresh the persistent project index, reuse
+                                  loaded snapshot state, and scope plan graph
+                                  work to changed or selected model lineage
+                                  without changing the plan result. Can also
+                                  be enabled with plan.use_project_index.
   -v, --verbose                   Verbose output. Use -vv for very verbose
                                   output.
   --help                          Show this message and exit.
@@ -447,6 +458,10 @@ Options:
                               only they will be expanded as raw queries.
   --dialect TEXT              The SQL dialect to render the query as.
   --no-format                 Disable fancy formatting of the query.
+  --use-project-index         Use the persistent project index to load and
+                              render only the target model and its upstream
+                              dependencies. Can also be enabled with
+                              render.use_project_index.
   --max-text-width INTEGER    The max number of characters in a segment before
                               creating new lines in pretty mode.
   --leading-comma             Determines whether or not the comma is leading

docs/reference/configuration.md

@@ -67,6 +67,12 @@ See all the keys allowed in `model_defaults` at the [model configuration referen
 | `linter.enabled`          | Whether linting is enabled (Default: `False`)                                                                                                      | boolean | N        |
 | `linter.use_project_index` | Whether to use the persistent project index for linting. Targeted linting loads selected models and their upstream dependencies. (Default: `False`) | boolean | N        |
 
+### Render
+
+| Option                     | Description                                                                                                                                                       | Type    | Required |
+|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------|----------|
+| `render.use_project_index` | Whether to use the persistent project index when rendering. Only the target model and its upstream dependencies are loaded. (Default: `False`)                    | boolean | N        |
+
 ### Variables
 
 The `variables` key can be used to provide values for user-defined variables, accessed using the [`@VAR` macro function](../concepts/macros/sqlmesh_macros.md#global-variables) in SQL model definitions, [`context.var` method](../concepts/models/python_models.md#global-variables) in Python model definitions, and [`evaluator.var` method](../concepts/macros/sqlmesh_macros.md#accessing-global-variable-values) in Python macro functions.
@@ -102,6 +108,7 @@ Configuration for the `sqlmesh plan` command.
 | `no_diff`                 | Don't show diffs for changed models (Default: False)                                                                                                                                                                                                    | boolean              | N        |
 | `no_prompts`              | Disables interactive prompts in CLI (Default: True)                                                                                                                                                                                                     | boolean              | N        |
 | `always_recreate_environment`              | Always recreates the target environment from the environment specified in `create_from` (by default `prod`) (Default: False)                                                                                                                                                                                                     | boolean              | N        |
+| `use_project_index`                        | Whether to refresh the persistent project index, reuse loaded snapshot state, and scope plan graph work to changed or selected model lineage without changing the plan result. (Default: `False`)                                                                                                                                | boolean              | N        |
 
 ## Run
 

sqlmesh/cli/main.py

@@ -64,6 +64,13 @@ def _sqlmesh_version() -> str:
         return "0.0.0"
 
 
+def _raise_if_no_models(context: Context, path: t.Any) -> None:
+    if not context.models:
+        raise click.ClickException(
+            f"`{path}` doesn't seem to have any models... cd into the proper directory or specify the path(s) with -p."
+        )
+
+
 @click.group(cls=_SQLMeshGroup, no_args_is_help=True)
 @click.version_option(version=_sqlmesh_version(), message="%(version)s")
 @opt.paths
@@ -141,8 +148,8 @@ def cli(
         if ctx.invoked_subcommand in SKIP_LOAD_COMMANDS:
             load = False
 
-    # Unlike the other commands above, lint can scope its own load for multi-project contexts.
-    if ctx.invoked_subcommand == "lint":
+    # These commands can scope their own load for multi-project contexts.
+    if ctx.invoked_subcommand in ("lint", "plan", "render"):
         load = False
 
     configs = load_configs(config, Context.CONFIG_TYPE, paths, dotenv_path=dotenv)
@@ -163,10 +170,8 @@ def cli(
             logger.exception("Failed to initialize SQLMesh context")
         raise
 
-    if load and not context.models:
-        raise click.ClickException(
-            f"`{paths}` doesn't seem to have any models... cd into the proper directory or specify the path(s) with -p."
-        )
+    if load:
+        _raise_if_no_models(context, paths)
 
     ctx.obj = context
 
@@ -284,6 +289,12 @@ def init(
     help="The SQL dialect to render the query as.",
 )
 @click.option("--no-format", is_flag=True, help="Disable fancy formatting of the query.")
+@click.option(
+    "--use-project-index",
+    is_flag=True,
+    default=None,
+    help="Use the persistent project index to load and render only the target model and its upstream dependencies. Can also be enabled with render.use_project_index.",
+)
 @opt.format_options
 @click.pass_context
 @error_handler
@@ -297,19 +308,20 @@ def render(
     expand: t.Optional[t.Union[bool, t.Iterable[str]]] = None,
     dialect: t.Optional[str] = None,
     no_format: bool = False,
+    use_project_index: t.Optional[bool] = None,
     **format_kwargs: t.Any,
 ) -> None:
     """Render a model's query, optionally expanding referenced models."""
-    model = ctx.obj.get_model(model, raise_if_missing=True)
-
     rendered = ctx.obj.render(
         model,
         start=start,
         end=end,
         execution_time=execution_time,
         expand=expand,
+        use_project_index=use_project_index,
     )
 
+    model = ctx.obj.get_model(model, raise_if_missing=True)
     format_config = ctx.obj.config_for_node(model).format
     format_kwargs = {
         **format_config.generator_options,
@@ -561,6 +573,12 @@ def diff(ctx: click.Context, environment: t.Optional[str] = None) -> None:
     default=None,
     help="For every model, ensure at least this many intervals are covered by a missing intervals check regardless of the plan start date",
 )
+@click.option(
+    "--use-project-index",
+    is_flag=True,
+    default=None,
+    help="Refresh the persistent project index, reuse loaded snapshot state, and scope plan graph work to changed or selected model lineage without changing the plan result. Can also be enabled with plan.use_project_index.",
+)
 @opt.verbose
 @click.pass_context
 @error_handler
@@ -579,8 +597,18 @@ def plan(
     allow_additive_models = kwargs.pop("allow_additive_model") or None
     backfill_models = kwargs.pop("backfill_model") or None
     ignore_cron = kwargs.pop("ignore_cron") or None
+    use_project_index = kwargs.pop("use_project_index")
     setattr(get_console(), "verbosity", Verbosity(verbose))
 
+    context.load(
+        use_project_index=(
+            context.config.plan.use_project_index
+            if use_project_index is None
+            else use_project_index
+        )
+    )
+    _raise_if_no_models(context, context.path)
+
     context.plan(
         environment,
         restate_models=restate_models,
@@ -589,6 +617,7 @@ def plan(
         allow_additive_models=allow_additive_models,
         backfill_models=backfill_models,
         ignore_cron=ignore_cron,
+        use_project_index=use_project_index,
         **kwargs,
     )
 
@@ -1239,10 +1268,7 @@ def lint(
         use_project_index=use_project_index,
     )
 
-    if not obj.models:
-        raise click.ClickException(
-            f"`{obj.path}` doesn't seem to have any models... cd into the proper directory or specify the path(s) with -p."
-        )
+    _raise_if_no_models(obj, obj.path)
 
 
 @cli.group(no_args_is_help=True)

sqlmesh/core/config/__init__.py

@@ -37,6 +37,7 @@
 from sqlmesh.core.config.naming import NameInferenceConfig as NameInferenceConfig
 from sqlmesh.core.config.linter import LinterConfig as LinterConfig
 from sqlmesh.core.config.plan import PlanConfig as PlanConfig
+from sqlmesh.core.config.render import RenderConfig as RenderConfig
 from sqlmesh.core.config.root import Config as Config, DbtConfig as DbtConfig
 from sqlmesh.core.config.run import RunConfig as RunConfig
 from sqlmesh.core.config.scheduler import BuiltInSchedulerConfig as BuiltInSchedulerConfig

sqlmesh/core/config/plan.py

@@ -21,6 +21,8 @@ class PlanConfig(BaseConfig):
         use_finalized_state: Whether to compare against the latest finalized environment state, or to use
             whatever state the target environment is currently in.
         always_recreate_environment: Whether to always recreate the target environment from the `create_from` environment.
+        use_project_index: Whether to use the persistent project index and related planning
+            optimizations.
     """
 
     forward_only: bool = False
@@ -32,3 +34,4 @@ class PlanConfig(BaseConfig):
     auto_apply: bool = False
     use_finalized_state: bool = False
     always_recreate_environment: bool = False
+    use_project_index: bool = False

sqlmesh/core/config/render.py

@@ -0,0 +1,13 @@
+from __future__ import annotations
+
+from sqlmesh.core.config.base import BaseConfig
+
+
+class RenderConfig(BaseConfig):
+    """Configuration for rendering model queries.
+
+    Args:
+        use_project_index: Whether to use the persistent project index when rendering.
+    """
+
+    use_project_index: bool = False

sqlmesh/core/config/root.py

@@ -35,6 +35,7 @@
 from sqlmesh.core.config.naming import NameInferenceConfig as NameInferenceConfig
 from sqlmesh.core.config.linter import LinterConfig as LinterConfig
 from sqlmesh.core.config.plan import PlanConfig
+from sqlmesh.core.config.render import RenderConfig
 from sqlmesh.core.config.run import RunConfig
 from sqlmesh.core.config.dbt import DbtConfig
 from sqlmesh.core.config.scheduler import (
@@ -141,6 +142,7 @@ class Config(BaseConfig):
         format: The formatting options for SQL code.
         ui: The UI configuration for SQLMesh.
         plan: The plan configuration.
+        render: The render configuration.
         migration: The migration configuration.
         variables: A dictionary of variables that can be used in models / macros.
         disable_anonymized_analytics: Whether to disable the anonymized analytics collection.
@@ -183,6 +185,7 @@ class Config(BaseConfig):
     format: FormatConfig = FormatConfig()
     ui: UIConfig = UIConfig()
     plan: PlanConfig = PlanConfig()
+    render: RenderConfig = RenderConfig()
     migration: MigrationConfig = MigrationConfig()
     model_naming: NameInferenceConfig = NameInferenceConfig()
     variables: t.Dict[str, t.Any] = {}
@@ -208,6 +211,7 @@ class Config(BaseConfig):
         "ui": UpdateStrategy.NESTED_UPDATE,
         "loader_kwargs": UpdateStrategy.KEY_UPDATE,
         "plan": UpdateStrategy.NESTED_UPDATE,
+        "render": UpdateStrategy.NESTED_UPDATE,
         "before_all": UpdateStrategy.EXTEND,
         "after_all": UpdateStrategy.EXTEND,
         "linter": UpdateStrategy.NESTED_UPDATE,