export/import crash

[nicolasaunai]

Description

Export a catalogue to json (about 100k events)
import the catalog from that json file
the import takes forever (much longer than the creation) and eventually crashes

log :

sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.IntegrityError) UNIQUE constraint failed: catalogues.uuid
[SQL: INSERT INTO catalogues (uuid, name, author, predicate, tags, removed, attributes) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('a09e4b86-21a6-4cfb-b59c-579f3e54fb4b', 'THEMIS_MPcrossings_2007-2016_V1', 'toto', None, '', 0, '{}')]
(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)
Traceback (most recent call last):
  File "/home/jeandet/Documents/prog/GH-stuff/tscat_gui/tscat_gui/model.py", line 172, in data
    return item.text()
  File "/home/jeandet/Documents/prog/GH-stuff/tscat_gui/tscat_gui/model.py", line 65, in text
    return get_entity_from_uuid_safe(self._uuid).name
  File "/home/jeandet/Documents/prog/GH-stuff/tscat_gui/tscat_gui/utils/helper.py", line 14, in get_entity_from_uuid_safe
    catalogues = tscat.get_catalogues(tscat.filtering.UUID(uuid))
  File "/home/jeandet/.local/lib/python3.10/site-packages/tscat/__init__.py", line 302, in get_catalogues
    for cat in backend().get_catalogues(base_dict):
  File "/home/jeandet/.local/lib/python3.10/site-packages/tscat/orm_sqlalchemy/__init__.py", line 229, in get_catalogues
    for c in self._create_query(base, orm.Catalogue, 'events', removed=base['removed']):
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2900, in __iter__
    return self._iter().__iter__()
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2907, in _iter
    result = self.session.execute(
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute
    conn = self._connection_for_bind(bind)
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
    return self._transaction._connection_for_bind(
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 721, in _connection_for_bind
    self._assert_active()
  File "/home/jeandet/.local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 601, in _assert_active
    raise sa_exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.IntegrityError) UNIQUE constraint failed: catalogues.uuid
[SQL: INSERT INTO catalogues (uuid, name, author, predicate, tags, removed, attributes) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('a09e4b86-21a6-4cfb-b59c-579f3e54fb4b', 'THEMIS_MPcrossings_2007-2016_V1', 'toto', None, '', 0, '{}')]
(Background on this error at: https://sqlalche.me/e/14/gkpj) (Background on this error at: https://sqlalche.me/e/14/7s2a)

[pboettch]

Could you upload a compressed version of this catalogue here?

[jeandet]

test_cat.zip

Or to create the base catalog from python:

import speasy as spz
from tscat import create_catalogue, create_event, add_events_to_catalogue, save, Session

cat=spz.get_data(spz.inventories.tree.amda.Catalogs.SharedCatalogs.EARTH.THEMIS_MPcrossings_2007_2016_V1)

with Session() as s:
    events = [ s.create_event(*e, "bob") for e in list(cat)+list(cat)+list(cat) ]

ccat = create_catalogue(cat.name, "bob")
add_events_to_catalogue(ccat, events)
save()

[pboettch]

FTM I'm unable to reproduce. I imported and exported the catalogue several times, it takes some time, but it works..

[nicolasaunai]

You have done it via the gui not just the backend, right? (Don’t know if it matters)

[pboettch]

Yes.