Weird error with TriggerDagRunOperator with Airflow 2.9.3

[Kenny1217]

I'm trying to track down what is causing this weird issue with the TriggerDagRunOperator. We have a dag that calls another dag with the following code:
t1 = TriggerDagRunOperator( task_id="dag_1", trigger_dag_id="dag_2", wait_for_completetion=True, poke_interval=10 )

Running the "dag_2" manually runs fine with out any issues, but running the dag that calls the TriggerDagRunOperator has a tendency to fail with the error AttributeError: 'list' object has no attribute 'items'. Clearing the task works once in a while but still sometimes fails. We run our Airflow instance in a k8s cluster on-prem. We just upgraded from 2.9.2 to 2.9.3. So I'm wondering if the upgrade changed something with how the TriggerDagRunOperator works because this is the first time seeing this type of error. Like I said sometimes it works and sometimes it doesn't. Below is the error I get from the log on the failed task.

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 465, in _execute_task result = _execute_callable(context=context, **execute_callable_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 432, in _execute_callable return execute_callable(context=context, **execute_callable_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 401, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/operators/trigger_dagrun.py", line 177, in execute dag_run = trigger_dag( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/api/common/trigger_dag.py", line 126, in trigger_dag triggers = _trigger_dag( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/api/common/trigger_dag.py", line 53, in _trigger_dag dag = dag_bag.get_dag(dag_id) # prefetch dag if it is stored serialized ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/utils/session.py", line 79, in wrapper return func(*args, session=session, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 200, in get_dag self._add_dag_from_db(dag_id=dag_id, session=session) File "/usr/local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 282, in _add_dag_from_db dag = row.dag ^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/models/serialized_dag.py", line 232, in dag return SerializedDAG.from_dict(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/serialization/serialized_objects.py", line 1552, in from_dict return cls.deserialize_dag(serialized_obj["dag"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/serialization/serialized_objects.py", line 1484, in deserialize_dag v = cls._deserialize_params_dict(v) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/airflow/serialization/serialized_objects.py", line 802, in _deserialize_params_dict for k, v in encoded_params.items(): ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'list' object has no attribute 'items'

[Tirke]

We have the same error running airflow on Kube on 2.9.3.

[ignashkin]

Hello!
have you fixed the issue?

[Kenny1217]

I just stuck with 2.9.2 for now. One thing I wanted to try but didn't get a chance to was maybe try downgrading the kubernetes provider to see if that had any effect with it.