Upgrading KubeCF triggers multiple database-seeder qjob runs

[f0rmiga]

Describe the bug

Upgrading KubeCF gets multiple database-seeder runs. They have a race condition that usually sets the wrong values in the database. E.g. the database password secrets get rotated and what the KubeCF jobs get (the latest revision) is not necessarily the last database-seeder errand to update the passwords in the database. Manually triggering the database-seeder at a later stage seems to help.

To Reproduce

Give KubeCF an upgrade from 2.2.3 to 2.5.4.

Expected behavior

Only one database-seeder run.

Environment

quarks-operator version 6.1.8+0.g2821133a.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175135667

The labels on this github issue will be updated when the story is started.

[manno]

The seeder job was configured to run on 'config changes', so every referenced secret update could trigger the job again. The job is idempotent.

[f0rmiga]

@manno You just described the bug. We don't want the job to be triggered multiple times concurrently. Fine if they are executed in the correct order, though.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175194113

The labels on this github issue will be updated when the story is started.

[manno]

@f0rmiga I see, but that's not how "update on config change" works. We'll have to discuss if it's possible to implement this on the operator side.
So executing in any order is fine, if the jobs are not run in parallel?

[f0rmiga]

@manno I see. We can perhaps do a session to further understand the KubeCF needs and how we can leverage the current cf-operator capabilities? Or even get to the drawing board yet again?

[manno]

Yes, we should :)
We briefly discussed smarter scripts in kubecf (they might be able to know if the database needs a migration) and we discussed making qjobs (optionally) run exclusive.