-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add AWS ECS task run execution mode #1507
Add AWS ECS task run execution mode #1507
Conversation
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1507 +/- ##
==========================================
+ Coverage 97.31% 97.33% +0.02%
==========================================
Files 79 80 +1
Lines 4616 4689 +73
==========================================
+ Hits 4492 4564 +72
- Misses 124 125 +1 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @CarlosGitto, this is a great contribution, thank you very much!
Some early feedback:
Code coverage
- Please, feel free to use
# pragma: no cover
in the import statements that codecov mentioned that are untested. This is one of the few exceptions we open to using# pragma: no cover
. This should increase the current 85.50% of diff hit (target 97.05%).
Pre-commit
There were some minor linting details that pre-commit complained:
ix end of files..............................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook
Fixing docs/getting_started/aws-container-run-job.rst
trim trailing whitespace......................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook
Fixing docs/getting_started/aws-container-run-job.rst
ruff..........................................................................Failed
- hook id: ruff
- files were modified by this hook
Found 1 error (1 fixed, 0 remaining).
black.........................................................................Failed
- hook id: black
- files were modified by this hook
reformatted cosmos/__init__.py
To learn more about how we use pre-commit, please, check https://github.com/astronomer/astronomer-cosmos/blob/c344eb4750f0269ecd267326373814abf56ec18e/docs/contributing.rst#pre-commit
I'm optimistic we'll be able to ship this as part of Cosmos 1.9 - we'll do a more thorough review in the upcoming days
…inside `DbtAwsEcsBaseOperator`
…overage. Add `DbtSourceAwsEcsOperator` to cosmos `__init__` file
b6d678d
to
49ed127
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great, @CarlosGitto , thank you very much for the contribution! We'll cut an alpha release tomorrow so you can try it out. The stable release is planned for the 14th
Description
This PR introduces an AWS ECS operator that allows Cosmos to run dbt tasks in AWS ECS. This ensures that dbt executions are consistent with the existing Airflow DAGs running in ECS.
Similar to the Azure Container Instance execution mode, this enhancement provides an ECS-native execution mode for Cosmos users.
What changes?
AwsEcsOperator
classes (inheriting fromAbstractDbtBaseOperator
) - Based on the originalEcsRunTaskOperator
Related Issue(s)
resolves #1355
Breaking Change?
No, just an added feature
Checklist