Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

812 eoy table #818

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

812 eoy table #818

wants to merge 16 commits into from

Conversation

chmnata
Copy link
Collaborator

@chmnata chmnata commented Jan 2, 2024

What this pull request accomplishes:

  • taskflowify-ish the dag (definitely more things can be done on the bluetooth side, but can wait after declarative partitioning switch)
  • update path from etc to data
  • make it run again (?)

Issue(s) this solves:

What, in particular, needs to reviewed:

  • Theoretically the start time, interval time, date interval end are all correct so it should run at 2024-12-01.
  • reconfirm the interval time 🤔

What needs to be done by a sysadmin after this PR is merged

git pull to data_script

def yearly_task():
"""Task group to create yearly tables and triggers."""
bt_replace_trigger = PythonOperator(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you forgot to taskflow-ify this one! 🤔

@gabrielwol
Copy link
Collaborator

As discussed, here is the template I've used within the ETL DAGs which uses generic short circuit operators short circuit operators to check the date.
In the future we could wrap this all inside a generic TaskGroup which Islam has an example of in VFH!


return task_msg
bt_bot = PostgresHook('bt_bot')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can go inside bt_replace_trigger once you taskflow that one.

@gabrielwol gabrielwol linked an issue Jul 15, 2024 that may be closed by this pull request
@gabrielwol
Copy link
Collaborator

I think the new @task.run_if decorator in 2.10 will be helpful for simplifying EOY/EOM tasks. Something like
@task.run_if(lambda context: check_jan_1st(context["ds"]))
https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html#conditionally-skipping-tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

End of Year Dag did not run
2 participants