-
Notifications
You must be signed in to change notification settings - Fork 184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Shared defaults for load_yaml_dags #297
Comments
Something like this? ./dags/default.yml
./dags/bi.yml
./dags/ds.yml
./dags/ml.yml
... |
@cmarteepants, only thing I think I'd add here is referencing the default values in |
@cmarteepants So basically I had been mainly thinking of this only for Docker compose does something similar, but the merge rules are kind of adhoc-yet-sensible |
@wearpants If everything is contained in the same yaml today, yes anything in As for the how? I'll be honest: I haven't delved much into the source code to understand how this was implemented. Could be something we are getting "for free" from pyyaml, but never looked into it as the capability was around from before Astronomer took over the project. I opened up issue #295 so we can we document this properly. The examples in the issue are for extending, overriding and even generating the exact same dag structure with different task ids, and they all work. I really like your idea about splitting up the definitions into different files though, and allowing for different defaults per folders. I'd even go so far as push that as a best practice. We'd need to allow for an order of precedence, but assuming we can pull it off (and I don't see why not, but I'm the PM :D) I agree, I think it would be really powerful. I'll have someone on the engineering team start looking into this within the next few sprints. Do you want to be kept to update as things progress? |
@cmarteepants yes please keep me in the loop, happy to hop on a quick design brainstorming session call as well if that be helpful |
Sounds like a perfect use for https://hydra.cc |
Description
It'd be nice to pass some shared
default_args
for a directory, either via a python object or adefaults.yml
file in the directory.Use case/motivation
One DAG per file is easier for users IMO, and as a system administrator I'd like to be able to give them a shared set of pre-baked defaults (env vars, etc.)
Related issues
#289, #290
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: