Ability to tag tasks?

[p-hebert]

I'm currently building a dag that self-populates its tasks based on the files (py, ipynb) of a directory.
I would like to access the xcom of all of those tasks once they are executed in order to retrieve their result from S3/GCS and then combine them in memory to assemble a finalized file.

Currently since these tasks are all of the same Operator type, I use the Operator type to filter the list of upstream tasks, but I plan to expand to different Operator types. While I could expand the list of Operator types that are filtered, I figured task tags would be cleaner.

I looked into the impl of BaseOperator, PythonOperator and some provider operators, and I couldn't find anything that looked like tags. The closest I found was the use of op_kwargs/op_args for PythonOperator, but not all operators have these parameters.
Hence:

1. Is there any way to tag operators? Alternatively I can monkey-patch tags on my operators, but that sounds hacky and dangerous.
2. Is this something that was considered in the past and decided against?
3. Any better solution?

[potiuk]

Tags are available at dag, not at task levels. But what I can suggest is a naming convention for task_ids. That should do pretty well.

…

On Wed, Aug 18, 2021 at 9:15 PM Philippe Hebert ***@***.***> wrote: I'm currently building a dag that self-populates its tasks based on the files (py, ipynb) of a directory. I would like to access the xcom of all of those tasks once they are executed in order to retrieve their result from S3/GCS and then combine them in memory to assemble a finalized file. Currently since these tasks are all of the same Operator type, I use the Operator type to filter the list of upstream tasks, but I plan to expand to different Operator types. While I could expand the list of Operator types that are filtered, I figured task tags would be cleaner. I looked into the impl of BaseOperator, PythonOperator and some provider operators, and I couldn't find anything that looked like tags. The closest I found was the use of op_kwargs/op_args for PythonOperator, but not all operators have these parameters. Hence: 1. Is there any way to tag operators? Alternatively I can monkey-patch tags on my operators, but that sounds hacky and dangerous. 2. Is this something that was considered in the past and decided against? 3. Any better solution? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#17697>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAERMI7CY6BUBUCC2KQLXPLT5QBDXANCNFSM5CMVVEUQ> .

-- +48 660 796 129

[p-hebert]

@potiuk, after some extra hours going through the implementation for other stuff, I noticed that BaseOperator.__init__ argument "params" seems to support usage as tags:

1. It's exposed as an attribute on the BaseOperator task:

airflow/airflow/models/baseoperator.py

Line 617 in c4d043d

self.params = params or {} # Available in templates!

2. It's directly read from the task copy in get_template_context

airflow/airflow/models/taskinstance.py

Lines 1751 to 1757 in 9922287

	params = {} # type: Dict[str, Any]
	with contextlib.suppress(AttributeError):
	params.update(task.dag.params)
	if task.params:
	params.update(task.params)
	if conf.getboolean('core', 'dag_run_conf_overrides_params'):
	self.overwrite_params_with_dag_run_conf(params=params, dag_run=dag_run)

3. Bar overrides from other configurations, params is exposed in context:

airflow/airflow/models/taskinstance.py

Line 1975 in 9922287

'params': params,

Hence if I understood correctly:

1. I could call the operator with params={"task_tags": ("tag:one", "tag:two")} and be able to access those tags directly in a templated string
2. I could instanciate the other operators with params as in (1) and then filter them through their params since it is an attribute on the task.

Hence I think tagging is actually already supported :)

[potiuk]

If you can make it works for you - that's cool. I think the para,s are not in any way addressed in UI, but if it's not needed for you that might do the job!

[p-hebert]

I confirm, it does :)

[potiuk]

Cool. I love new features, that reqiuire 0 new lines of code to maintain 😃

[jose-padron-momox]

i would like to have that in v3... sometimes is not enough to have task names templated, a nice tag/label helps too