Dynamic Tasks at Runtime

[pramod-prabhakar-kapase-db]

Hi All,

I need help to generate dynamic task at runtime.

DAG get trigger by other tag and it send dynamic list, sometime 3, 4, 5 etc, to generate tasks next DAG to perform.

Branching is needed between tasks, if task 1 success then task 2 will execute else it will move file to failed location and so on. Finally file should move to success location

List will be something like

tasks = [{name:'stage_1", data: 'data'}, {name:'stage_2", data: 'data'}, {name:'stage_3", data: 'data'}]

DAG should generate like

Task_Stage_1 -> Branch -> [Task_Stage_2, move_file_to_failed_location] -> Branch -> [Task_stage_3, move_file_to_failed_location] -> move_file_to_success_location

I tried using Dynamic Task - In first python task I created a list and passed to Dynamic group there I tried iterating it to generate task but I got error saying that Mapped Attribute can't be interable.

I will create a dummy sample and attach it in sometime.

I went through the stackoverflow and it was mentioned that we can't create Dynamic task at runtime.

Is it possible to get Context before DAG render so that I can generate Dynamic task before DAG render.
context = get_current_context()
params = context['params']
params['configuration']['stages']

[potiuk]

Is it possible to get Context before DAG render so that I can generate Dynamic task before DAG render.

No. DAG parsing prepares structure (no context). Dag Execution has context (but cannot change structure - beyond changing number of tasks or tasks groups based on context, resuls of previous tasks).

You likely should look at dynamic tasks groups (which were added to dynamic tasks somewhat recently - if I understand your problem correctly).

[raphaelauv]

@pramod-prabhakar-kapase-db https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dynamic-task-mapping.html