in future. clustering gorjun's database

[etursunbaev]

I think in future it should be implemented database clustering. For now there is one master node and no backups. It is dangerous.

[emli]

@etursunbaev can you write some details and proves? As I know we backup databases in prod CDN. Do we do backups of artifacts?

[etursunbaev]

As you know only one master prod works so imagine in some situations this master will fail than all related systems will fail. From infra side we always ready help to you. Even if backups exist it might take several hours to restore system.

[dilshat]

As a quick solution we can consider Kurjun db and artifacts replication to some standby node that would be turned into master when there is a problem with current master. Of course, a proper monitoring setup is crucial here. Monitoring will make the transition seamless

As a long term solution we should separately consider Kurjun database and Kurjun artifacts:

1. Kurjun artifacts can be only replicated to some standby or active node
2. Kurjun database contains metadata: template and user information, sharing and permissions.
   It should be based upon a database implementation that supports clustering (afaik BoltDB does not support it)
   For solving (2) we can either extend BoltDB with additional functionality, e.g. see https://github.com/vincent-petithory/rbolt
   Or better use another database that supports clusterization out-of-the-box, e.g. see https://github.com/tidwall/summitdb or https://github.com/siddontang/ledisdb

[emli]

@etursunbaev can you report the current state of backups? Do we do backups of database and files?