Database migration issue with catalog migration (catalog perm)

[otrebor]

Bug description

We are trying to upgrade our superset instance to 410rc2-py310

when we start superset, we get this error while the migration scripts are trying to update the catalog:

Loaded your LOCAL configuration at [/app/superset_config.py]
WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
/usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
INFO:superset.utils.screenshots:No PIL installation found
INFO:superset.utils.pdf:No PIL installation found
INFO  [alembic.env] Starting the migration scripts.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade ec54aca4c8a2 -> 317970b4400c, Added always_filter_main_dttm to datasource
INFO  [alembic.runtime.migration] Running upgrade 317970b4400c -> 4b85906e5b91, add on delete cascade for dashboard_roles
INFO  [alembic.runtime.migration] Running upgrade 4b85906e5b91 -> b7851ee5522f, replay 317970b4400c
INFO  [alembic.runtime.migration] Running upgrade b7851ee5522f -> 06dd9ff00fe8, add_percent_calculation_type_funnel_chart
INFO  [alembic.runtime.migration] Running upgrade 06dd9ff00fe8 -> 65a167d4c62e, add indexes to report models
INFO  [alembic.runtime.migration] Running upgrade 65a167d4c62e -> 59a1450b3c10, drop_filter_sets_table
INFO  [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> a32e0c4d8646, migrate-sunburst-chart
INFO  [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> 96164e3017c6
INFO  [alembic.runtime.migration] Running upgrade 96164e3017c6, a32e0c4d8646 -> 15a2c68a2e6b, merging two heads
INFO  [alembic.runtime.migration] Running upgrade a32e0c4d8646 -> 214f580d09c9, migrate_filter_boxes_to_native_filters
INFO  [alembic.runtime.migration] Running upgrade 214f580d09c9 -> e863403c0c50, drop_url_table
INFO  [alembic.runtime.migration] Running upgrade e863403c0c50, 15a2c68a2e6b -> 1cf8e4344e2b, merging
INFO  [alembic.runtime.migration] Running upgrade 1cf8e4344e2b -> 87d38ad83218, Migrate can_view_and_drill permission
INFO  [alembic.runtime.migration] Running upgrade 87d38ad83218 -> 17fcea065655, change_text_to_mediumtext
INFO  [alembic.runtime.migration] Running upgrade 17fcea065655 -> be1b217cd8cd, big_number_kpi_single_metric
INFO  [alembic.runtime.migration] Running upgrade be1b217cd8cd -> 678eefb4ab44, Add access token table
INFO  [alembic.runtime.migration] Running upgrade 678eefb4ab44 -> c22cb5c2e546, empty message
INFO  [alembic.runtime.migration] Running upgrade c22cb5c2e546 -> 5ad7321c2169, mig new csv upload perm
INFO  [alembic.runtime.migration] Running upgrade 5ad7321c2169 -> d60591c5515f, mig new excel upload perm
INFO  [alembic.runtime.migration] Running upgrade d60591c5515f -> 5f57af97bc3f, Add catalog column
INFO  [alembic.runtime.migration] Running upgrade 5f57af97bc3f -> 3dfd0e78650e, add_query_sql_editor_id_index
INFO  [alembic.runtime.migration] Running upgrade 5f57af97bc3f -> 4a33124c18ad, mig new columnar upload perm
INFO  [alembic.runtime.migration] Running upgrade 4a33124c18ad -> 58d051681a3b, Add catalog_perm to tables
    26/26
    7/7
Traceback (most recent call last):
  File "/usr/local/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 358, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.10/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/superset/extensions/../migrations/env.py", line 142, in <module>
    run_migrations_online()
  File "/app/superset/extensions/../migrations/env.py", line 133, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.10/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.10/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
    step.migration_fn(**kw)
  File "/app/superset/migrations/versions/2024-05-01_10-52_58d051681a3b_add_catalog_perm_to_tables.py", line 47, in upgrade
    upgrade_catalog_perms(engines={"postgresql"})
  File "/app/superset/migrations/shared/catalogs.py", line 373, in upgrade_catalog_perms
    for database in session.query(Database).all():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2773, in all
    return self._iter().all()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1476, in all
    return self._allrows()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 401, in _allrows
    rows = self._fetchall_impl()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1389, in _fetchall_impl
    return self._real_result._fetchall_impl()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1813, in _fetchall_impl
    return list(self.iterator)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks
    fetch = cursor._raw_all_rows()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 393, in _raw_all_rows
    return [make_row(row) for row in rows]
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 393, in <listcomp>
    return [make_row(row) for row in rows]
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/type_api.py", line 1716, in process
    return process_value(impl_processor(value), dialect)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 479, in process_result_value
    value = super().process_result_value(value=value, dialect=dialect)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 423, in process_result_value
    self._update_key()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 392, in _update_key
    self.engine._update_key(key)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 52, in _update_key
    digest.update(key)
TypeError: argument 'data': a bytes-like object is required, not 'NoneType'
WARNING:root:Applying Monkey Patching for gevent

I can't really understand if it's a configuration problem or a bug. The connection string seems fine and previous tables are updated.

we see other issues like
#29492
#29728
but i don't think that's the problem for us.

Screenshots/recordings

No response

Superset version

master / latest-dev

Python version

3.10

Node version

16

Browser

Chrome

Additional context

Loaded your LOCAL configuration at [/app/superset_config.py]
WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `FILTER_STATE_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
WARNING:superset.utils.cache_manager:Falling back to the built-in cache, that stores data in the metadata database, for the following cache: `EXPLORE_FORM_DATA_CACHE_CONFIG`. It is recommended to use `RedisCache`, `MemcachedCache` or another dedicated caching backend for production deployments
/usr/local/lib/python3.10/site-packages/flask_limiter/extension.py:333: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
INFO:superset.utils.screenshots:No PIL installation found
INFO:superset.utils.pdf:No PIL installation found
INFO  [alembic.env] Starting the migration scripts.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade ec54aca4c8a2 -> 317970b4400c, Added always_filter_main_dttm to datasource
INFO  [alembic.runtime.migration] Running upgrade 317970b4400c -> 4b85906e5b91, add on delete cascade for dashboard_roles
INFO  [alembic.runtime.migration] Running upgrade 4b85906e5b91 -> b7851ee5522f, replay 317970b4400c
INFO  [alembic.runtime.migration] Running upgrade b7851ee5522f -> 06dd9ff00fe8, add_percent_calculation_type_funnel_chart
INFO  [alembic.runtime.migration] Running upgrade 06dd9ff00fe8 -> 65a167d4c62e, add indexes to report models
INFO  [alembic.runtime.migration] Running upgrade 65a167d4c62e -> 59a1450b3c10, drop_filter_sets_table
INFO  [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> a32e0c4d8646, migrate-sunburst-chart
INFO  [alembic.runtime.migration] Running upgrade 59a1450b3c10 -> 96164e3017c6
INFO  [alembic.runtime.migration] Running upgrade 96164e3017c6, a32e0c4d8646 -> 15a2c68a2e6b, merging two heads
INFO  [alembic.runtime.migration] Running upgrade a32e0c4d8646 -> 214f580d09c9, migrate_filter_boxes_to_native_filters
INFO  [alembic.runtime.migration] Running upgrade 214f580d09c9 -> e863403c0c50, drop_url_table
INFO  [alembic.runtime.migration] Running upgrade e863403c0c50, 15a2c68a2e6b -> 1cf8e4344e2b, merging
INFO  [alembic.runtime.migration] Running upgrade 1cf8e4344e2b -> 87d38ad83218, Migrate can_view_and_drill permission
INFO  [alembic.runtime.migration] Running upgrade 87d38ad83218 -> 17fcea065655, change_text_to_mediumtext
INFO  [alembic.runtime.migration] Running upgrade 17fcea065655 -> be1b217cd8cd, big_number_kpi_single_metric
INFO  [alembic.runtime.migration] Running upgrade be1b217cd8cd -> 678eefb4ab44, Add access token table
INFO  [alembic.runtime.migration] Running upgrade 678eefb4ab44 -> c22cb5c2e546, empty message
INFO  [alembic.runtime.migration] Running upgrade c22cb5c2e546 -> 5ad7321c2169, mig new csv upload perm
INFO  [alembic.runtime.migration] Running upgrade 5ad7321c2169 -> d60591c5515f, mig new excel upload perm
INFO  [alembic.runtime.migration] Running upgrade d60591c5515f -> 5f57af97bc3f, Add catalog column
INFO  [alembic.runtime.migration] Running upgrade 5f57af97bc3f -> 3dfd0e78650e, add_query_sql_editor_id_index
INFO  [alembic.runtime.migration] Running upgrade 5f57af97bc3f -> 4a33124c18ad, mig new columnar upload perm
INFO  [alembic.runtime.migration] Running upgrade 4a33124c18ad -> 58d051681a3b, Add catalog_perm to tables
    26/26
    7/7
Traceback (most recent call last):
  File "/usr/local/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 358, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.10/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.10/site-packages/alembic/command.py", line 403, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.10/site-packages/alembic/script/base.py", line 583, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/superset/extensions/../migrations/env.py", line 142, in <module>
    run_migrations_online()
  File "/app/superset/extensions/../migrations/env.py", line 133, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.10/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.10/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
    step.migration_fn(**kw)
  File "/app/superset/migrations/versions/2024-05-01_10-52_58d051681a3b_add_catalog_perm_to_tables.py", line 47, in upgrade
    upgrade_catalog_perms(engines={"postgresql"})
  File "/app/superset/migrations/shared/catalogs.py", line 373, in upgrade_catalog_perms
    for database in session.query(Database).all():
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2773, in all
    return self._iter().all()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1476, in all
    return self._allrows()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 401, in _allrows
    rows = self._fetchall_impl()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1389, in _fetchall_impl
    return self._real_result._fetchall_impl()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1813, in _fetchall_impl
    return list(self.iterator)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks
    fetch = cursor._raw_all_rows()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 393, in _raw_all_rows
    return [make_row(row) for row in rows]
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 393, in <listcomp>
    return [make_row(row) for row in rows]
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/type_api.py", line 1716, in process
    return process_value(impl_processor(value), dialect)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 479, in process_result_value
    value = super().process_result_value(value=value, dialect=dialect)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 423, in process_result_value
    self._update_key()
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 392, in _update_key
    self.engine._update_key(key)
  File "/usr/local/lib/python3.10/site-packages/sqlalchemy_utils/types/encrypted/encrypted_type.py", line 52, in _update_key
    digest.update(key)
TypeError: argument 'data': a bytes-like object is required, not 'NoneType'
WARNING:root:Applying Monkey Patching for gevent

Checklist

• I have searched Superset docs and Slack and didn't find a solution to my problem.
• I have searched the GitHub issue tracker and didn't find a similar bug report.
• I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot]

Hey @otrebor! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together!

The error you're encountering seems to be related to the upgrade_catalog_perms function in the migration script, which is trying to update catalog permissions. The specific TypeError indicates that a NoneType is being passed where a bytes-like object is expected, likely during the encryption key update process in encrypted_type.py.

Here are a few steps you can take to troubleshoot and potentially resolve the issue:

1. Check Encryption Key Configuration:
   Ensure that your encryption keys are correctly configured in your superset_config.py. A missing or misconfigured key could lead to a NoneType being passed.

2. Database Integrity:
   Verify the integrity of your database. Ensure that all necessary tables and columns exist and are correctly populated, especially those related to catalogs and encryption.

3. Dependencies:
   Make sure all dependencies are up to date, especially those related to SQLAlchemy and encryption libraries.

4. Debugging:
   Add logging or debugging statements in the encrypted_type.py file to trace the value of the key being passed to digest.update. This might help identify why a NoneType is being encountered.

5. Review Related Issues:
   Although you mentioned that other issues might not be directly related, reviewing them could provide additional insights or workarounds.

If these steps don't resolve the issue, consider isolating the problem by creating a minimal setup to reproduce the error, which might help in identifying the root cause.

To continue talking to Dosu, mention @dosu.