Upgrade from 0.38.0 to 1.0.0 fails

[grafke]

A clear and concise description of what the bug is.

Expected results

A successfull upgrade from 0.38.0 to 1.0.0

Actual results.

superset db upgrade command has failed with:
pkg_resources.ContextualVersionConflict: (PyJWT 2.0.1 (/usr/local/lib/python3.8/site-packages), Requirement.parse('PyJWT<2.0,>=1.6.4'), {'Flask-JWT-Extended'})

After manually downgrading the pyjwt to 1.7.1 I ran the superset db upgrade again, and this time another error popped up:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: column "database_id" contains null values


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/cli.py", line 136, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/__init__.py", line 96, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/__init__.py", line 271, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 294, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 481, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.8/site-packages/alembic/util/compat.py", line 182, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/superset/migrations/env.py", line 124, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.8/site-packages/superset/migrations/env.py", line 116, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py", line 560, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.8/site-packages/superset/migrations/versions/af30ca79208f_collapse_alerting_models_into_a_single_.py", line 120, in upgrade
    op.add_column(
  File "<string>", line 8, in add_column
  File "<string>", line 3, in add_column
  File "/usr/local/lib/python3.8/site-packages/alembic/operations/ops.py", line 1758, in add_column
    return operations.invoke(op)
  File "/usr/local/lib/python3.8/site-packages/alembic/operations/base.py", line 354, in invoke
    return fn(self, operation)
  File "/usr/local/lib/python3.8/site-packages/alembic/operations/toimpl.py", line 134, in add_column
    operations.impl.add_column(table_name, column, schema=schema, **kw)
  File "/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py", line 256, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py", line 146, in _exec
    return conn.execute(construct, multiparams)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1068, in _execute_ddl
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) column "database_id" contains null values

[SQL: ALTER TABLE alerts ADD COLUMN database_id INTEGER NOT NULL]
(Background on this error at: http://sqlalche.me/e/13/gkpj)```

what actually happens.

#### How to reproduce the bug

1. Install 0.38.0 using this Dockerfile: https://github.com/amancevice/docker-superset/blob/main/Dockerfile
2. Then change the superset version to 1.0.0 (https://github.com/apache/superset/archive/${SUPERSET_VERSION}.tar.gz)
3. Install
4. Run superset db upgrade inside a docker container

### Environment

(please complete the following information):

- superset version:  1.0.0
- python version:  3.8.6
- node.js version: 12

### Checklist

Make sure to follow these steps before submitting your issue - thank you!

- [x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [x] I have reproduced the issue with at least the latest released version of superset.
- [x] I have checked the issue tracker for the same issue and I haven't found one similar.

[grafke]

Looks like it's related to https://medium.com/the-andela-way/alembic-how-to-add-a-non-nullable-field-to-a-populated-table-998554003134

[grafke]

I worked around this by updating a portion of the file af30ca79208f_collapse_alerting_models_into_a_single_.py:

       else:  # mysql does not support server_default for text fields
        op.add_column(
            "alerts",
            sa.Column("validator_config", sa.Text(), default="", nullable=True),
        )
        op.add_column(
            "alerts", sa.Column("database_id", sa.Integer(), default=0, nullable=True),
        )
        op.execute("UPDATE alerts SET database_id = 1")
        op.alter_column('alerts', 'database_id', nullable=False)

        op.add_column("alerts", sa.Column("sql", sa.Text(), default="", nullable=True))
        op.execute("UPDATE alerts SET sql = '' ")
        op.alter_column('alerts', 'sql', nullable=False)

        op.add_column(
            "alerts",
            sa.Column(
                "validator_type", sa.String(length=100), default="", nullable=True
            ),
        )
        op.execute("UPDATE alerts SET validator_type = '' ")
        op.alter_column('alerts', 'validator_type', nullable=False)

Judging by the comment, this section should apply to mysql only, however, my superset instance is backed by the postgres db.

[lamielle]

I am investigating a 0.38.0 -> 1.0.0 upgrade as well. We are deployed into k8s using a Helm chart based on the one provided in the Superset repo. I'm also running on Postgres. I haven't yet tried an upgrade but plan to sometime this week. I will see if I am able to repro the bug you're reporting here.

[vnourdin]

We have the same issue (column "database_id" contains null values) coming from 0.37.
Deployed in K8S (EKS) + PostgreSQL (RDS) through similar Helm chart too.

[vnourdin]

The problem was coming from the alerts table, but I checked and there were no null database_id in this table, so I don't get it. 🤷
Our alerts were just an old test, so I truncated the table, and we achieved to finish the init_db 🎉
But for people already using alerts it is still a blocker !