[components][rfc] @component_loader decorator

[OwenKephart]

Summary & Motivation

Adds an @component_loader decorator and the corresponding infrastructure to allow users to write arbitrary python code to load their Components.

Note that the convention I've gone with here has the user manually call XComponent.load(), rather than going through the __init__. My thought here was basically just that this feels like better practice as it provides a common shared format between the python and yaml interfaces.

In short, the way to use this is to have a component.py file instead of a component.yaml file, which will look something like:

from dagster_components import ComponentLoadContext, component_loader
from ... import MyComponentClass

@component_loader
def load(context: ComponentLoadContext) -> MyComponentClass:
    return MyComponentClass.load(
        params=...,
        context=context,
    )

And again, there's nothing stopping you from using the regular constructor, I just think that might be harder to use.

How I Tested These Changes

Changelog

NOCHANGELOG

[OwenKephart]

• [components][rfc] @component_loader decorator #27182 👈 (View in Graphite)
• [components] Refactor component load behavior #27181
• [components] Explicitly pass params to Component.load #27174
• [components] Rename get_params_schema_type #27163
• [components] Update docs commenting logic #27162
• [components] Update resolution deferral logic #27136
• [bug] Fix issue with source position yaml loading #27160
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[schrockn]

We could go all the way and do this?

@component_decl
def load(context: ComponentLoadContext) -> PipesSubprocessScriptCollectionParams:
   return PipesSubprocessScriptCollectionParams(
        scripts=[
            PipesSubprocessScriptParams(
                path="cool_script.py",
                assets=[
                    AssetAttributesModel(
                        key="cool_script",
                        automation_condition="{{ automation_condition.eager() }}",
                    ),
                ],
            ),
        ]
    )

[OwenKephart]

I think I prefer leaving the init method accessible to the user here. This gives the user more control over what actually happens, and means that if we do come up with a good theory about how to craft a nice set of init args for a given component then users are able to access that with the same interface

[OwenKephart]

Merge activity

• Jan 23, 7:00 PM EST: A user started a stack merge that includes this pull request via Graphite.
• Jan 23, 7:00 PM EST: A user merged this pull request with Graphite.