Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AWS ECS task run execution mode #1507

Merged

Conversation

CarlosGitto
Copy link
Contributor

@CarlosGitto CarlosGitto commented Feb 4, 2025

Description

This PR introduces an AWS ECS operator that allows Cosmos to run dbt tasks in AWS ECS. This ensures that dbt executions are consistent with the existing Airflow DAGs running in ECS.

Similar to the Azure Container Instance execution mode, this enhancement provides an ECS-native execution mode for Cosmos users.

What changes?

  • New AwsEcsOperator classes (inheriting from AbstractDbtBaseOperator) - Based on the original EcsRunTaskOperator
  • Tests
  • Adjusted documentation

Related Issue(s)

resolves #1355

Breaking Change?

No, just an added feature

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@dosubot dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Feb 4, 2025
Copy link

netlify bot commented Feb 4, 2025

Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name Link
🔨 Latest commit 9fa54a5
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/67a50c73350f2300087a79d5

@dosubot dosubot bot added the area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc label Feb 4, 2025
Copy link

codecov bot commented Feb 4, 2025

Codecov Report

Attention: Patch coverage is 98.63014% with 1 line in your changes missing coverage. Please review.

Project coverage is 97.33%. Comparing base (24108f0) to head (9fa54a5).
Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
cosmos/__init__.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1507      +/-   ##
==========================================
+ Coverage   97.31%   97.33%   +0.02%     
==========================================
  Files          79       80       +1     
  Lines        4616     4689      +73     
==========================================
+ Hits         4492     4564      +72     
- Misses        124      125       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @CarlosGitto, this is a great contribution, thank you very much!

Some early feedback:

Code coverage

  • Please, feel free to use # pragma: no cover in the import statements that codecov mentioned that are untested. This is one of the few exceptions we open to using # pragma: no cover . This should increase the current 85.50% of diff hit (target 97.05%).

Pre-commit

There were some minor linting details that pre-commit complained:

ix end of files..............................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing docs/getting_started/aws-container-run-job.rst

trim trailing whitespace......................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing docs/getting_started/aws-container-run-job.rst

ruff..........................................................................Failed
- hook id: ruff
- files were modified by this hook

Found 1 error (1 fixed, 0 remaining).

black.........................................................................Failed
- hook id: black
- files were modified by this hook

reformatted cosmos/__init__.py

To learn more about how we use pre-commit, please, check https://github.com/astronomer/astronomer-cosmos/blob/c344eb4750f0269ecd267326373814abf56ec18e/docs/contributing.rst#pre-commit

I'm optimistic we'll be able to ship this as part of Cosmos 1.9 - we'll do a more thorough review in the upcoming days

@CarlosGitto CarlosGitto force-pushed the feat/aws-ecs-task-run-execution-mode branch from b6d678d to 49ed127 Compare February 5, 2025 21:33
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, @CarlosGitto , thank you very much for the contribution! We'll cut an alpha release tomorrow so you can try it out. The stable release is planned for the 14th

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Feb 6, 2025
@tatiana tatiana added this to the Cosmos 1.9.0 milestone Feb 6, 2025
@tatiana tatiana merged commit 03358bf into astronomer:main Feb 6, 2025
65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc lgtm This PR has been approved by a maintainer size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Enable execution mode for AWS ECS tasks
3 participants