Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Top level definitions config is not merged with job level config #14865

Open
ignaski opened this issue Jun 20, 2023 · 1 comment
Open

Top level definitions config is not merged with job level config #14865

ignaski opened this issue Jun 20, 2023 · 1 comment
Labels
area: config Related to Configuration type: bug Something isn't working

Comments

@ignaski
Copy link

ignaski commented Jun 20, 2023

Dagster version

1.3.10

What's the issue?

I am trying to use the .configure_at_launch() feature. The example code at https://docs.dagster.io/concepts/resources#configuring-resources-at-launch-time works, however if config is added at the job level, then the code location fails to build with an error:

dagster._core.errors.DagsterInvalidConfigError: Error in config when building job 'my_job' 
    Error 1: Missing required config entry "resources" at the root. Sample config for missing entry: {'resources': {'db_conn': {'config': {'table': '...'}}}}

What did you expect to happen?

I expect it not to ask for the default config, because the resource is defined to be configured at launch

How to reproduce?

The code which I used to reproduce the error

from dagster import ConfigurableResource, Definitions, asset, job, op


class DatabaseResource(ConfigurableResource):
    table: str

    def read(self):
        pass


@op
def data_from_database(db_conn: DatabaseResource):
    return db_conn.read()


@job(config={"execution": {
    "config": {
        "multiprocess": {
            "max_concurrent": 10,
        }
    },
}})
def my_job():
    data_from_database()


defs = Definitions(
    jobs=[my_job],
    resources={"db_conn": DatabaseResource.configure_at_launch()},
)

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

@ignaski ignaski added the type: bug Something isn't working label Jun 20, 2023
@yuhan yuhan added the area: config Related to Configuration label Jun 20, 2023
@garethbrickman
Copy link
Contributor

This will potentially fix: #12870

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: config Related to Configuration type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants