Alembic migration failed when updating from 2.8.3 to 2.10.0 (version file ec3471c1e067)

[Timelessprod]

Apache Airflow version

2.10.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When upgrading Airflow from version 2.8.3 to 2.10.0, Airflow fails to restart. Indeed the Alembic migration fails with the error:

alembic.util.exc.CommandError: Can't locate revision identified by 'ec3471c1e067'

The missing version file is located here : https://github.com/apache/airflow/blob/e001b88f5875cfd7e295891a0bbdbc75a3dccbfb/airflow/migrations/versions/0148_2_10_0_dataset_alias_dataset_event.py

It's no longer on the master branch but it is on the last commit with tag 2.10.0

In a attempt to fix the problem I tried to revert the version back to 2.8.3 but Alembic is still looking for the ec3471c1e067 version file when restarting the instance, and thus Airflow is unable to start anymore.

What you think should happen instead?

The migration should have go through without error and Airflow start.

How to reproduce

• Install Airflow 2.8.3 with a persistent database
• Update the Airflow version to 2.10.0
• Restart the instance
• See the error

Operating System

Debian 12.6

Versions of Apache Airflow Providers

From the official constraints file at https://raw.githubusercontent.com/apache/airflow/constraints-2.10.0/constraints-3.11.txt

apache-airflow-providers-airbyte==3.8.1
apache-airflow-providers-alibaba==2.8.1
apache-airflow-providers-amazon==8.27.0
apache-airflow-providers-apache-beam==5.7.2
apache-airflow-providers-apache-cassandra==3.5.1
apache-airflow-providers-apache-drill==2.7.3
apache-airflow-providers-apache-druid==3.10.2
apache-airflow-providers-apache-flink==1.4.2
apache-airflow-providers-apache-hdfs==4.4.2
apache-airflow-providers-apache-hive==8.1.2
apache-airflow-providers-apache-iceberg==1.0.0
apache-airflow-providers-apache-impala==1.4.2
apache-airflow-providers-apache-kafka==1.5.0
apache-airflow-providers-apache-kylin==3.6.2
apache-airflow-providers-apache-livy==3.8.1
apache-airflow-providers-apache-pig==4.4.1
apache-airflow-providers-apache-pinot==4.4.2
apache-airflow-providers-apache-spark==4.9.0
apache-airflow-providers-apprise==1.3.2
apache-airflow-providers-arangodb==2.5.1
apache-airflow-providers-asana==2.5.1
apache-airflow-providers-atlassian-jira==2.6.1
apache-airflow-providers-celery==3.7.3
apache-airflow-providers-cloudant==3.5.2
apache-airflow-providers-cncf-kubernetes==8.3.4
apache-airflow-providers-cohere==1.2.1
apache-airflow-providers-common-compat==1.1.0
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.15.0
apache-airflow-providers-databricks==6.8.0
apache-airflow-providers-datadog==3.6.1
apache-airflow-providers-dbt-cloud==3.9.0
apache-airflow-providers-dingding==3.5.1
apache-airflow-providers-discord==3.7.1
apache-airflow-providers-docker==3.12.3
apache-airflow-providers-elasticsearch==5.4.2
apache-airflow-providers-exasol==4.5.3
apache-airflow-providers-fab==1.2.2
apache-airflow-providers-facebook==3.5.2
apache-airflow-providers-ftp==3.10.1
apache-airflow-providers-github==2.6.2
apache-airflow-providers-google==10.21.1
apache-airflow-providers-grpc==3.5.2
apache-airflow-providers-hashicorp==3.7.1
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-influxdb==2.6.0
apache-airflow-providers-jdbc==4.4.0
apache-airflow-providers-jenkins==3.6.1
apache-airflow-providers-microsoft-azure==10.3.0
apache-airflow-providers-microsoft-mssql==3.8.0
apache-airflow-providers-microsoft-psrp==2.7.1
apache-airflow-providers-microsoft-winrm==3.5.1
apache-airflow-providers-mongo==4.1.2
apache-airflow-providers-mysql==5.6.3
apache-airflow-providers-neo4j==3.6.1
apache-airflow-providers-odbc==4.6.3
apache-airflow-providers-openai==1.2.2
apache-airflow-providers-openfaas==3.5.1
apache-airflow-providers-openlineage==1.10.0
apache-airflow-providers-opensearch==1.3.0
apache-airflow-providers-opsgenie==5.6.1
apache-airflow-providers-oracle==3.10.3
apache-airflow-providers-pagerduty==3.7.2
apache-airflow-providers-papermill==3.7.2
apache-airflow-providers-pgvector==1.2.2
apache-airflow-providers-pinecone==2.0.1
apache-airflow-providers-postgres==5.11.3
apache-airflow-providers-presto==5.5.2
apache-airflow-providers-qdrant==1.1.2
apache-airflow-providers-redis==3.7.1
apache-airflow-providers-salesforce==5.7.2
apache-airflow-providers-samba==4.7.1
apache-airflow-providers-segment==3.5.1
apache-airflow-providers-sendgrid==3.5.1
apache-airflow-providers-sftp==4.10.3
apache-airflow-providers-singularity==3.5.1
apache-airflow-providers-slack==8.8.0
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-snowflake==5.6.1
apache-airflow-providers-sqlite==3.8.2
apache-airflow-providers-ssh==3.12.0
apache-airflow-providers-tableau==4.5.2
apache-airflow-providers-tabular==1.5.1
apache-airflow-providers-telegram==4.5.2
apache-airflow-providers-teradata==2.5.0
apache-airflow-providers-trino==5.7.2
apache-airflow-providers-vertica==3.8.2
apache-airflow-providers-weaviate==2.0.0
apache-airflow-providers-yandex==3.11.2
apache-airflow-providers-ydb==1.2.0
apache-airflow-providers-zendesk==4.7.1

Deployment

Other 3rd-party Helm chart

Deployment details

Custom Helm chart deployment.

Anything else?

Currently my instance of Airflow is unable to start, even after trying to revert to the last used version (2.8.3).

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[tirkarthi]

main branch is now under development for Airflow 3 where migration files were pruned. You should probably try using v2-10-test if fixes for 2.10 are needed and use 2.10.0 tag.

https://github.com/apache/airflow/tree/v2-10-test

[Timelessprod]

@tirkarthi Why have those version files been removed from the 2.10.0 release ? Currently Alembic cannot find them so there's something wrong with the release I think.

Is this "v2-10-test" version among the pre-releases available on PIP ? My libraries are managed by PIP and I'd prefer no to manually clone the repo if possible.

Thank you.

[potiuk]

You have not mentioned it but my guess is that you are dynamically installing packages and after this dynamic installation - ikely your image actually runs lower version of Airlfow.

The revision is in the alembic table so the migration worked - but the airflow command you run uses a downgraded version of airflow after your dynamic pip install command.

Please take a look on what version of the image you are using and whether your customization / extending the image has some problems. For example when you are installing dynamically new packages - the pip install command might automatically downgrade airflow. It solely depends on packages you install and resolution of pip which might decide that downgrading airflow fulfills other dependencies better.

That's why in the offiical installation description of upgrade scenarios, you will find that you always have to specify apache-airflow=${AIRFLOW_VERSION} https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-and-upgrade-scenarios - so that pip won't downgrade airflow.

This is one of the reasons why we always discouraged installing packages dynamically when your image starts - because for example airlfow version might change - but also other packages might and recommend users to build their own image - this way they can be sure what version of packages they have there. With dynamic package

[potiuk]

I think you need to look closer and look at the logs of the command that failed. It's almost certain that whatever command is run uses different version of Airflow than 2.10 - look at that command and entire output of it, usually it should print airflow version there. Could be different image your command is using (if this is not dynamic installation)

[potiuk]

So just to be sure that we are at the same page - t's often the case that various part of the upgrade mechanism in 3rd-party charts use different images for migration than the one to run airflow. I'd suggest you to look closer into that - and post some finding - that might help.

And the reason why I am writing it, is that I just tested it back and forth (and it's also automatically tested in our CI) - and there is completely no problem with running migrations up/down when airflow version is "good". And you can see the ec3471c1e067 migration there. So something is wrong in your system, not in airflow.

root@62cc4511f446:/opt/airflow# airflow db downgrade --to-version 2.8.3
Performing downgrade with database postgresql+psycopg2://postgres:***@postgres/airflow

Warning: About to reverse schema migrations for the airflow metastore. Please ensure you have backed up your database before any upgrade or downgrade operation. Proceed? (y/n)
y
[2024-08-29T15:16:55.031+0000] {db.py:1754} INFO - Attempting downgrade to revision 88344c1d9134
[2024-08-29T15:16:55.034+0000] {db.py:1765} INFO - Applying downgrade migrations.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running downgrade 22ed7efa9da2 -> 8684e37832e6, Add dag_schedule_dataset_alias_reference table.
INFO  [alembic.runtime.migration] Running downgrade 8684e37832e6 -> 41b3bc7c0272, Add dataset_alias_dataset association table.
INFO  [alembic.runtime.migration] Running downgrade 41b3bc7c0272 -> ec3471c1e067, Add try_number to audit log.
INFO  [alembic.runtime.migration] Running downgrade ec3471c1e067 -> 05e19f3176be, Add dataset_alias_dataset_event.
INFO  [alembic.runtime.migration] Running downgrade 05e19f3176be -> d482b7261ff9, Add dataset_alias.
INFO  [alembic.runtime.migration] Running downgrade d482b7261ff9 -> c4602ba06b4b, Add task_instance_history.
INFO  [alembic.runtime.migration] Running downgrade c4602ba06b4b -> 677fdbb7fc54, Added DagPriorityParsingRequest table.
INFO  [alembic.runtime.migration] Running downgrade 677fdbb7fc54 -> 0fd0c178cbe8, add new executor field to db.
INFO  [alembic.runtime.migration] Running downgrade 0fd0c178cbe8 -> 686269002441, Add indexes on dag_id column in referencing tables.
INFO  [alembic.runtime.migration] Running downgrade 686269002441 -> bff083ad727d, Fix inconsistency between ORM and migration files.
INFO  [alembic.runtime.migration] Running downgrade bff083ad727d -> 1949afb29106, Remove ``idx_last_scheduling_decision`` index on last_scheduling_decision in dag_run table.
INFO  [alembic.runtime.migration] Running downgrade 1949afb29106 -> ee1467d4aa35, update trigger kwargs type and encrypt.
INFO  [alembic.runtime.migration] Running downgrade ee1467d4aa35 -> b4078ac230a1, add display name for dag and task instance.
INFO  [alembic.runtime.migration] Running downgrade b4078ac230a1 -> 8e1c784a4fc7, Change value column type to longblob in xcom table for mysql.
INFO  [alembic.runtime.migration] Running downgrade 8e1c784a4fc7 -> ab34f260b71c, Adding max_consecutive_failed_dag_runs column to dag_model table.
INFO  [alembic.runtime.migration] Running downgrade ab34f260b71c -> d75389605139, add dataset_expression in DagModel.
INFO  [alembic.runtime.migration] Running downgrade d75389605139 -> 1fd565369930, Add run_id to (Audit) log table and increase event name length.
INFO  [alembic.runtime.migration] Running downgrade 1fd565369930 -> 88344c1d9134, Add rendered_map_index to TaskInstance.
Downgrade complete
root@62cc4511f446:/opt/airflow# airflow db migrate
DB: postgresql+psycopg2://postgres:***@postgres/airflow
Performing upgrade to the metadata database postgresql+psycopg2://postgres:***@postgres/airflow
[2024-08-29T15:17:03.874+0000] {migration.py:215} INFO - Context impl PostgresqlImpl.
[2024-08-29T15:17:03.875+0000] {migration.py:218} INFO - Will assume transactional DDL.
[2024-08-29T15:17:03.876+0000] {migration.py:215} INFO - Context impl PostgresqlImpl.
[2024-08-29T15:17:03.876+0000] {migration.py:218} INFO - Will assume transactional DDL.
[2024-08-29T15:17:03.877+0000] {db.py:1674} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 88344c1d9134 -> 1fd565369930, Add rendered_map_index to TaskInstance.
INFO  [alembic.runtime.migration] Running upgrade 1fd565369930 -> d75389605139, Add run_id to (Audit) log table and increase event name length.
INFO  [alembic.runtime.migration] Running upgrade d75389605139 -> ab34f260b71c, add dataset_expression in DagModel.
INFO  [alembic.runtime.migration] Running upgrade ab34f260b71c -> 8e1c784a4fc7, Adding max_consecutive_failed_dag_runs column to dag_model table.
INFO  [alembic.runtime.migration] Running upgrade 8e1c784a4fc7 -> b4078ac230a1, Change value column type to longblob in xcom table for mysql.
INFO  [alembic.runtime.migration] Running upgrade b4078ac230a1 -> ee1467d4aa35, add display name for dag and task instance.
INFO  [alembic.runtime.migration] Running upgrade ee1467d4aa35 -> 1949afb29106, update trigger kwargs type and encrypt.
INFO  [alembic.runtime.migration] Running upgrade 1949afb29106 -> bff083ad727d, Remove ``idx_last_scheduling_decision`` index on last_scheduling_decision in dag_run table.
INFO  [alembic.runtime.migration] Running upgrade bff083ad727d -> 686269002441, Fix inconsistency between ORM and migration files.
INFO  [alembic.runtime.migration] Running upgrade 686269002441 -> 0fd0c178cbe8, Add indexes on dag_id column in referencing tables.
INFO  [alembic.runtime.migration] Running upgrade 0fd0c178cbe8 -> 677fdbb7fc54, add new executor field to db.
INFO  [alembic.runtime.migration] Running upgrade 677fdbb7fc54 -> c4602ba06b4b, Added DagPriorityParsingRequest table.
INFO  [alembic.runtime.migration] Running upgrade c4602ba06b4b -> d482b7261ff9, Add task_instance_history.
INFO  [alembic.runtime.migration] Running upgrade d482b7261ff9 -> 05e19f3176be, Add dataset_alias.
INFO  [alembic.runtime.migration] Running upgrade 05e19f3176be -> ec3471c1e067, Add dataset_alias_dataset_event.
INFO  [alembic.runtime.migration] Running upgrade ec3471c1e067 -> 41b3bc7c0272, Add try_number to audit log.
INFO  [alembic.runtime.migration] Running upgrade 41b3bc7c0272 -> 8684e37832e6, Add dataset_alias_dataset association table.
INFO  [alembic.runtime.migration] Running upgrade 8684e37832e6 -> 22ed7efa9da2, Add dag_schedule_dataset_alias_reference table.
/usr/local/lib/python3.8/site-packages/airflow/migrations/versions/0151_2_10_0_dag_schedule_dataset_alias_reference.py:46 SAWarning: Table 'dag_schedule_dataset_alias_reference' specifies columns 'dag_id' as primary_key=True, not matching locally specified columns 'alias_id', 'dag_id'; setting the current primary key columns to 'alias_id', 'dag_id'. This warning may become an exception in a future release
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
/usr/local/lib/python3.8/site-packages/airflow/serialization/serialized_objects.py:1863 DeprecationWarning: This module is deprecated. The `cncf.kubernetes` provider before version 7.4.0 uses this module - you should migrate to a newer version of `cncf.kubernetes` to get rid of this warning. If you import the module via `airflow.kubernetes` import, please use `cncf.kubernetes' provider 7.4.0+ and switch all your imports to use `apache.airflow.providers.cncf.kubernetes` to get rid of the warning.
Database migrating done!
root@62cc4511f446:/opt/airflow# pip freeze | grep apache-airflow
apache-airflow==2.10.0

But I got similar error when I use different airflow version and run upgrade or downgrade:

alembic.util.exc.CommandError: Can't locate revision identified by '22ed7efa9da2'
root@62cc4511f446:/opt/airflow# pip freeze | grep apache-airflow
apache-airflow==2.9.0

The interesting thing is that your revision is somewhere in the middle of the revisions for 2.10.0 - which indicates that either one of your earlier migrations has been abruptly stopped, or that you used some in-development version of 2.10.0 (alpha/beta maybe for example to get to that revision.

[potiuk]

So again - you need to look deeper - we won't be able to help you if we can't reproduce the exact problem that you have, unfortunately.

[Timelessprod]

The version I used is the one available on pip and tagged 2.10.0. I have in my requirements.txt the line apache-airflow==2.10.0. We build a docker image from these requirements and our DAGs etc. and then use Helm chart to deploy it as a container. So there's nothing affecting the airflow version outside this requirement and indeed when booting the service shows the airflow version as 2.10.0.

[potiuk]

Then I have no other hypothesis what happened I your case. Can't reproduce it with the released version. You stillhavenot pro used full logs and explanation of which commandfails for you to aid the research. Maybe if you explain it and provide the full logs we could help - because the only way this might happen is where you use airflow that has this migration missing. So the only way it can happenix that sometbing, somewhere in your deployment downgrades airflow. Since this your chart and 'some' yet unnamed airflow command, you are the only person that has enough details to make detailed investigation.

As you saw vfrom the full logs provided - standard airflow 2.10.0 works just fine with upgrade/downgrade. Similar complete logs and evidences (rather than just explanations of assumptions you have with requirements) might be helpful in helping you to uncover what happens, but 'plain' upgrade/downgrade - works with 'vanilla' airflow as you saw. Maybe you have some typos in your scripts that makes you think you don't downgrade airflow incidentally. Or maybe you look at wrong part of your chart /image - hard to say. There are many possibilities there and only you can investigate it. We have not enough information/time to spend at looking at your scripts/charts/images. Maybe for example somewhere in your image you also have another airflow version installed In a different path and it takes precedence when you run your failing command. And so on, so on.

Without you digging deeper and finding what else happens but hypothesis that the airflow that you use to run the comand that fails for you is not the 'plain airflow 2.10.0'. By trying what you explained on plain airflow version and failing to reproduce it we'vedo e what we could to help you to solve your problem. Take all the helpful information you have here and try to dig deeper.

[potiuk]

Converting to discussion - this is unlikely to be airflow issue.

[amrit2196]

I am facing a similar issue. I upgraded from airflow version 2.5.3 to 2.10.0, but when I am try to downgrade it again to 2.5.3 it fails with error message stating alembic.util.exc.CommandError: Can't locate revision identified by '22ed7efa9da2' . This is the alembic version for 2.10.3. I am not sure why it is trying to look for this value in the first place. Thus I am not able to downgrade at all.

[potiuk]

Again. Likely your version of airflow after dynamically updating packages in your image is not what you think it is. Likely some dynamically installed packages are upgrading Airflow to 2.10.3. This is one of the issues when you install some package and pip decides to upgrade your Airflow. If you are using dynamically installed packages, this is one of the problems that you can have - unless you also pin airlfow (as explained in our documentation https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-and-upgrade-scenarios

Generally speaking, you have to always be in control of which version of airlfow you use at each point of time when you do migration.

[Timelessprod]

Dynamic install has nothing to do with the issue here as well as for me before. Once again the problem relates to the way Airflow maintainers deal with migration files when releasing new versions.

@amrit2196 Your issue is that Airflow don't provide all the Alembic revision files when you are in the version 2.10.X. Indeed when you are in version 2.10.X there are only the revision files up to 2.9.0, any version prior this (2.8, etc.) have there migration files removed from the library. In your case I presume the revision 22ed7efa9da2 is the last revision of Airflow 2.8.X which has been deleted when they packaged the version 2.10.0, thus when you downgrade from 2.10.3 to 2.5.3, Alembic can't locate the revision and throw this error. To downgrade back to 2.5.3, you should downgrade progressively from 2.10.3 to 2.9.3 then to 2.8.4 then to 2.7.3 then to 2.6.3 and finally to 2.5.3.

I have to admit that removing revision files like this is not a very clean way of handling Alembic. Most project I have seen are squasing all the changes in the revision files of a medium version (2.9, 2.8, 2.7 for example) into a single revision file and manually adjust the revision ids to make everything link perfectly (not super hard to do). Doing so allow their users to jump versions without error. However with Airflow you need to upgrade medium-version by medium-version (2.5 to 2.6 to 2.7 etc.)