Replies: 5 comments 2 replies
-
|
Thanks for opening your first issue here! Be sure to follow the issue template! |
Beta Was this translation helpful? Give feedback.
-
|
Can you see what kind of configuration you have for database/sql_engine_collation_for_ids or core/sql_engine_collation_for_ids you have and what's your collation for the task_id columns? |
Beta Was this translation helpful? Give feedback.
-
|
More information here: https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html?highlight=mysql#setting-up-a-mysql-database |
Beta Was this translation helpful? Give feedback.
-
|
Also would be useful if you let us know what is the collation of your database and your "task_instance.task_id" column in case they are different. |
Beta Was this translation helpful? Give feedback.
-
|
In my case, I have the same issue while upgrading 2.1.0 to 2.3.2 (mysql migration) |
Beta Was this translation helpful? Give feedback.
-
|
@dryshliak @socar-rudy -> Can you please follow this chapter "Handling migration problems" specifically about teh MySQL Collation problems? https://github.com/apache/airflow/blob/main/docs/apache-airflow/installation/upgrading.rst#handling-migration-problems This is documentation about handling similar issues that we are going to have releaed in Airflow 2.4.0. Can you please follow it and report what have been your problems and whether the process described there worked for you? Thanks in advance - I am looking forward to your reports asap, we are about to relaase 2.4.0 and the docs so confirmation from you might help 100s of other users. Also any feedback to the documntation is most welcome. |
Beta Was this translation helpful? Give feedback.
-
|
Converted to discussion until more information provided. |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
2.3.1 (latest released)
What happened
On "airflow db upgrade" I'm getting next error.
and a
task_maptable to track mapping values from XCom.Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/default.py", line 716, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.9/dist-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/usr/local/lib/python3.9/dist-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/usr/local/lib/python3.9/dist-packages/MySQLdb/connections.py", line 254, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (3780, "Referencing column 'task_id' and referenced column 'task_id' in foreign key constraint 'task_map_task_instance_fkey' are incompatible.")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/dist-packages/airflow/main.py", line 38, in main
args.func(args)
File "/usr/local/lib/python3.9/dist-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py", line 82, in upgradedb
db.upgradedb(to_revision=to_revision, from_revision=from_revision, show_sql_only=args.show_sql_only)
File "/usr/local/lib/python3.9/dist-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/airflow/utils/db.py", line 1446, in upgradedb
command.upgrade(config, revision=to_revision or 'heads')
File "/usr/local/lib/python3.9/dist-packages/alembic/command.py", line 320, in upgrade
script.run_env()
File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 563, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "", line 850, in exec_module
File "", line 228, in _call_with_frames_removed
File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/env.py", line 107, in
run_migrations_online()
File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/env.py", line 101, in run_migrations_online
context.run_migrations()
File "", line 8, in run_migrations
File "/usr/local/lib/python3.9/dist-packages/alembic/runtime/environment.py", line 851, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.9/dist-packages/alembic/runtime/migration.py", line 620, in run_migrations
step.migration_fn(**kw)
File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/versions/0100_2_3_0_add_taskmap_and_map_id_on_taskinstance.py", line 75, in upgrade
op.create_table(
File "", line 8, in create_table
File "", line 3, in create_table
File "/usr/local/lib/python3.9/dist-packages/alembic/operations/ops.py", line 1254, in create_table
return operations.invoke(op)
File "/usr/local/lib/python3.9/dist-packages/alembic/operations/base.py", line 392, in invoke
return fn(self, operation)
File "/usr/local/lib/python3.9/dist-packages/alembic/operations/toimpl.py", line 114, in create_table
operations.impl.create_table(table)
File "/usr/local/lib/python3.9/dist-packages/alembic/ddl/impl.py", line 352, in create_table
self._exec(schema.CreateTable(table))
File "/usr/local/lib/python3.9/dist-packages/alembic/ddl/impl.py", line 193, in _exec
return conn.execute(construct, multiparams)
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/base.py", line 1200, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/sql/ddl.py", line 77, in _execute_on_connection
return connection._execute_ddl(
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/base.py", line 1290, in _execute_ddl
ret = self._execute_context(
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/base.py", line 1748, in _execute_context
self.handle_dbapi_exception(
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/base.py", line 1929, in handle_dbapi_exception
util.raise(
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/engine/default.py", line 716, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.9/dist-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/usr/local/lib/python3.9/dist-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/usr/local/lib/python3.9/dist-packages/MySQLdb/connections.py", line 254, in query
_mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (3780, "Referencing column 'task_id' and referenced column 'task_id' in foreign key constraint 'task_map_task_instance_fkey' are incompatible.")
[SQL:
CREATE TABLE task_map (
dag_id VARCHAR(250) COLLATE utf8mb3_bin NOT NULL,
task_id VARCHAR(250) COLLATE utf8mb3_bin NOT NULL,
run_id VARCHAR(250) COLLATE utf8mb3_bin NOT NULL,
map_index INTEGER NOT NULL,
length INTEGER NOT NULL,
keysJSON,PRIMARY KEY (dag_id, task_id, run_id, map_index),
CONSTRAINT task_map_length_not_negative CHECK (length >= 0),
CONSTRAINT task_map_task_instance_fkey FOREIGN KEY(dag_id, task_id, run_id, map_index) REFERENCES task_instance (dag_id, task_id, run_id, map_index) ON DELETE CASCADE
)
What you think should happen instead
The upgrade process should finish without errors
How to reproduce
Install the latest version of Python 3.9 on Ubuntu 20.04 lts
Install Airflow steps
AIRFLOW_VERSION=2.3.1 PYTHON_VERSION="$(python3 --version | cut -d " " -f 2 | cut -d "." -f 1-2)" CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-no-providers-${PYTHON_VERSION}.txt" pip3 install "apache-airflow[celery,redis,mysql,amazon]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"Operating System
Ubuntu 20.04 lts
Versions of Apache Airflow Providers
No response
Deployment
Other
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions