Improve error messages when given an empty database object

[schmidma]

Description

When starting the dashboard for an empty or non-existing database storage, the dashboard crashes when trying to determine the version information. The storage is empty and there is thus no table called version_info.

Traceback (most recent call last):
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: version_info

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna/storages/_rdb/storage.py", line 69, in _create_scoped_session
    yield session
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna/storages/_rdb/storage.py", line 1070, in _init_version_info_model
    version_info = models.VersionInfoModel.find(session)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna/storages/_rdb/models.py", line 596, in find
    version_info = session.query(cls).one_or_none()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 2773, in one_or_none
    return self._iter().one_or_none()  # type: ignore
           ^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 2846, in _iter
    result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                  ^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/orm/session.py", line 2232, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/orm/session.py", line 2127, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1413, in execute
    return meth(
           ^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1987, in _exec_single_context
    self._handle_dbapi_exception(
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 2344, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1968, in _exec_single_context
    self.dialect.do_execute(
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: version_info
[SQL: SELECT version_info.version_info_id AS version_info_version_info_id, version_info.schema_version AS version_info_schema_version, version_info.library_version AS version_info_library_version
FROM version_info]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/maximilian/.venv/optuna-dashboard/bin/optuna-dashboard", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna_dashboard/_cli.py", line 114, in main
    storage = get_storage(args.storage, storage_class=args.storage_class)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna_dashboard/_storage_url.py", line 56, in get_storage
    return guess_storage_from_url(storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna_dashboard/_storage_url.py", line 67, in guess_storage_from_url
    return get_rdb_storage(storage_url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna_dashboard/_storage_url.py", line 74, in get_rdb_storage
    return RDBStorage(storage_url, skip_compatibility_check=True, skip_table_creation=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna/storages/_rdb/storage.py", line 229, in __init__
    self._version_manager = _VersionManager(self.url, self.engine, self.scoped_session)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna/storages/_rdb/storage.py", line 1064, in __init__
    self._init_version_info_model()
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna/storages/_rdb/storage.py", line 1069, in _init_version_info_model
    with _create_scoped_session(self.scoped_session, True) as session:
  File "/usr/lib64/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/maximilian/.venv/optuna-dashboard/lib64/python3.11/site-packages/optuna/storages/_rdb/storage.py", line 87, in _create_scoped_session
    raise optuna.exceptions.StorageInternalError(message) from e
optuna.exceptions.StorageInternalError: An exception is raised during the commit. This typically happens due to invalid data in the commit, e.g. exceeding max length.

How to Reproduce

1. run optuna dashboard optuna-dashboard sqlite:///db.sqlite3
2. optuna fails with sqlite3.OperationalError: no such table: version_info

Python version

3.11.3

Optuna version

3.1.1

optuna-dashboard version or git revision

0.9.2

Web browser

none

[c-bata]

Thank you for your report.

It is actually an intended behavior, not a bug. Do you think we should improve error messages?

[c-bata]

Refs #263

[schmidma]

I was not aware that this is intended behavior. But I fully understand that perspective. I have some thoughts/comments on that.

If optuna-dashboard is not intended to create storages, it should also not create the file for the respective storage.
Starting the dashboard with optuna-dashboard sqlite:///db.sqlite3 when there was no db.sqlite3 file before, it is created (and empty).

Additionally, as you suggested, the error message could hint, that the storage is completely empty? Or hint the user, that optuna-dashboard can only work on initialized storages?

I was confused that this is an intended behavior, as optuna-dashboard is able to create new studies in the dashboard, and can also work on initialized storages without any studies. I expected to be able to also start optuna-dashboard on empty storages to create my first study via the dashboard as well.
Is there a optuna command to only initialize the storage without creating any studies? This would be helpful to kickstart a new setup.

[c-bata]

I see. Thank you for your opinion. Let me change the issue title and remove the bug label.

Is there a optuna command to only initialize the storage without creating any studies? This would be helpful to kickstart a new setup.

optuna command does not provide such a feature. If it's useful for many users, we might be able to consider introducing it to Optuna. As a workaround, you can create an empty database like the following.

$ python -c 'import optuna; optuna.storages.RDBStorage("sqlite:///not-found.db", skip_table_creation=False)'
$ ls not-found.db
not-found.db