First authenticated request after MySQL idle disconnect returns 500: RevokedToken.is_revoked runs on a stale session before FAB's recovery (#62919) can kick in

[uplsh580]

Under which category would you file this issue?

Providers

Apache Airflow version

3.2.2

What happened and how to reproduce it?

With MySQL as the metadata DB and the FAB auth manager, after the api-server sits idle longer than MySQL's wait_timeout, the first authenticated request (any UI or REST endpoint) fails with HTTP 500. The exception is an OperationalError (MySQL error 4031, "The client was disconnected by the server because of inactivity") raised from RevokedToken.is_revoked inside BaseAuthManager.get_user_from_token:

https://github.com/apache/airflow/blob/3.2.2/airflow-core/src/airflow/api_fastapi/auth/managers/base_auth_manager.py#L152

The immediately following request succeeds, because the failed query poisons and effectively resets the pooled connection. In production this shows up as intermittent 500s on the first request after overnight idle periods.

Traceback (trimmed)

  File ".../airflow/api_fastapi/auth/managers/base_auth_manager.py", line 152, in get_user_from_token
    if (jti := payload.get("jti")) and RevokedToken.is_revoked(jti):
  File ".../airflow/utils/session.py", line 100, in wrapper
    return func(*args, session=session, **kwargs)
  File ".../airflow/models/revoked_token.py", line 61, in is_revoked
    return bool(session.scalar(select(exists().where(cls.jti == jti))))
  ...
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (4031,
  'The client was disconnected by the server because of inactivity.
   See wait_timeout and interactive_timeout for configuring this behavior.')

How to reproduce

1. Run Airflow 3.2.x with MySQL as the metadata DB and the FAB auth manager.
2. Lower the idle timeout to make the repro fast: SET GLOBAL wait_timeout = 60;
3. Make any authenticated API request (this warms up the pooled connection).
4. Wait longer than wait_timeout (e.g. 90 s) without touching the api-server.
5. Repeat the same request → HTTP 500 with OperationalError 4031 raised from
   RevokedToken.is_revoked. Repeat once more → succeeds.

What you think should happen instead?

Root cause

1. The FAB auth manager shares core's scoped settings.Session. FabAuthManager.deserialize_user leaves a transaction open after the request is served, so the connection stays checked out across requests. Once it idles past MySQL's wait_timeout, the server drops it (error 4031). Because the connection is never returned to the pool, there is no checkout event — pool_pre_ping / pool_recycle cannot help.
2. This exact failure mode was reported in apache-airflow-providers-fab 3.4.0: Intermittent OperationalError (4031) when FAB session reuses a MySQL connection dropped due to idle timeout #62903 and fixed by fix(fab): recover from first idle MySQL disconnect in token auth #62919, which added a discard-and-retry recovery to FabAuthManager.deserialize_user.
3. However, 3.2.0 introduced the JWT revocation check (AIP-84 : Add JWT token revokation for logout invalidation (#47952) #61339 / AIP-84): RevokedToken.is_revoked in get_user_from_token now runs before deserialize_user, so it is the first thing to touch the poisoned scoped session — and it has no recovery logic. The 500 that fix(fab): recover from first idle MySQL disconnect in token auth #62919 fixed is back, just raised one step earlier in the auth path.

What you think should happen instead?

The first request after an idle disconnect should recover transparently, as it does in deserialize_user since #62919. The revoked-token check should apply the same recovery: on SQLAlchemyError, discard the scoped session (settings.Session.remove()) and retry once on a fresh connection. Something like:

if jti := payload.get("jti"):
    try:
        revoked = RevokedToken.is_revoked(jti)
    except SQLAlchemyError:
        log.warning(
            "Revoked-token check failed on a stale DB session; "
            "discarding the session and retrying once.",
            exc_info=True,
        )
        with suppress(Exception):
            settings.Session.remove()
        revoked = RevokedToken.is_revoked(jti)
    if revoked:
        raise InvalidTokenError("Token has been revoked")

Alternatively, the recovery could be hoisted so it covers the whole get_user_from_token path instead of being duplicated per call site.

Operating System

No response

Deployment

None

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

apache-airflow-providers-fab==3.6.4 (already includes the #62919 fix — it is bypassed, not missing)

Official Helm Chart version

1.22.0 (latest released)

Kubernetes Version

1.25

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

• Occurs deterministically whenever the first request after an idle period touches a connection older than wait_timeout; looks intermittent in production.
• Related prior work on the same class of failure:
  • apache-airflow-providers-fab 3.4.0: Intermittent OperationalError (4031) when FAB session reuses a MySQL connection dropped due to idle timeout #62903 / fix(fab): recover from first idle MySQL disconnect in token auth #62919 — same disconnect, recovery added to deserialize_user (this check runs earlier and bypasses it)
  • providers-fab: Unhandled MySQLdb.OperationalError in cleanup_session_middleware Session.remove() causes 500 on api-server #62335 / providers-fab: Handle database errors in cleanup_session_middleware Session.remove() #62336 — Session.remove() hardening in FAB's cleanup middleware
  • [3.2] API server pool exhaustion from per-request RevokedToken.is_revoked DB query #66493 / Cache RevokedToken.is_revoked to avoid per-request DB roundtrip #66494 — same call site, different concern (per-request DB roundtrip / pool exhaustion)
• We have been running the retry patch above in production for several weeks (applied at image build time) and it eliminated these 500s entirely.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct