Dynamic Pool Allocation Based on Runtime Parameters

[BigNerd]

Current Limitation

In Airflow, pool assignment for tasks must be done at DAG definition time. This static allocation doesn't allow for dynamic pool selection based on parameters that are only known at runtime.

Use Case

There are scenarios where the appropriate pool for a task can only be determined when the DAG is triggered, based on input parameters or runtime conditions. Currently, to work around this limitation, users must create multiple pre-defined tasks (potentially hundreds) to cover all possible pool assignments, and then use branching to select the appropriate task at runtime.

Desired Functionality

The ability to dynamically assign a task to a pool based on runtime parameters or conditions. This would allow for more flexible resource allocation and simplify DAG definitions in cases where pool selection depends on runtime information.

Benefits

Simplified DAG definitions
Reduced number of pre-defined tasks
More flexible and efficient resource allocation
Improved scalability for workflows with many potential pools

Example Scenario

Imagine a DAG that processes data for different clients. Each client's data should be processed using a specific resource pool, but the client is only known when the DAG is triggered. Currently, this requires creating separate tasks for each possible client/pool combination, leading to bloated DAG definitions.
Questions to Consider

How can Airflow be extended to support dynamic pool allocation without compromising its core design principles?
What would be the implications for the Airflow scheduler and executor?
How might this feature interact with existing Airflow concepts like task instances and XComs?

We welcome the Airflow community's thoughts on how best to implement this functionality while maintaining Airflow's robustness and ease of use.

[potiuk]

Currently not possible because pools are used as selection criteria when you select which tasks to schedule. If you would like to invest your time and effort in figuring out what to change, you would have to dig deep in how scheduler works and probably figure another mechanism that could be used more dynamically, but this is rather serious architectural change and would require somoene (you or maybe you can find and encourage someone to attempt it) - to write an AIP (Airflow Improvement Proposal) - https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals and likely a POC (Proof of Concept implementing it).

This is open-source and it means that whoever wants might attempt it and contribute it.