Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Full Flyway integration #262

Closed
Tracked by #257
benedeki opened this issue Sep 4, 2024 · 3 comments · Fixed by #276
Closed
Tracked by #257

Full Flyway integration #262

benedeki opened this issue Sep 4, 2024 · 3 comments · Fixed by #276
Assignees
@benedeki
Labels
DB Issues touching the Database part of the project infrastructure Project setup and deployment

Comments

@benedeki
Copy link
Contributor

benedeki commented Sep 4, 2024 •
edited
Loading

Background

Steps to ensure Atum 0.3.0 release and deploy
For now, we stick to Flyway migration tool (despite its drawbacks). It's infinitely better then doing DB changes manually.

Feature

Ensure that Flyway can be used in CD and then we stick to it.

Proposed Solution [Optional]

Solution Ideas:

  1. Identify a basic level of Flyway scripts (somewhere where tables are already deployed)
  2. Get the database manually to that state
  3. Set Flyway watermark to that level (version of script last deployed)
@benedeki benedeki mentioned this issue Sep 4, 2024
Closed
@benedeki benedeki added DB Issues touching the Database part of the project infrastructure Project setup and deployment labels Sep 4, 2024
@benedeki
Copy link
Contributor Author

benedeki commented Sep 19, 2024 •
edited
Loading

Solution steps:

  • figure out SBT Flyway plugin configuration - configuration can be overridden by environment variables.
  • Reorder scripts - first the utility functions, then tables, then all other functions
  • set the flyway base - at the last table creation
  • create drop script for all non-utility functions

@benedeki
Copy link
Contributor Author

benedeki commented Sep 23, 2024 •
edited
Loading

Solution for upgrade from v0.1.0/v0.2.0 to v0.3.0

  • call manually sbt -Dflyway.password=<postgres user password> -Dflyway.url=<target db url> flywayBaseline= // TODO
  • add to the deployment script sbt -Dflyway.password=<postgres user password> -Dflyway.url=<target db url> flywayMigrate

benedeki added a commit that referenced this issue Sep 24, 2024
@benedeki
#262: Full Flyway integration
1e85ef7 
* numbered the SQL scripts for Flyway
* changed `PartitioningForDB` class to contain `keysToValuesMap` instead of `keysToValues` according to validation
* added sbt alias `testAll` for running all tests
* database test for validation function moved to proper folder
* set `flywayBaselineVersion` for upgrade purposes
@benedeki benedeki mentioned this issue Sep 24, 2024
Merged
benedeki added a commit that referenced this issue Sep 30, 2024
@benedeki
#262: Full Flyway integration (#276)
2c1a107 
* #262: Full Flyway integration
* numbered the SQL scripts for Flyway
* changed `PartitioningForDB` class to contain `keysToValuesMap` instead of `keysToValues` according to validation
* added sbt alias `testAll` for running all tests
* database test for validation function moved to proper folder
* set `flywayBaselineVersion` for upgrade purposes
* README.md update
@github-project-automation github-project-automation bot moved this from 🆕 To groom to ✅ Done in CPS small repos project Sep 30, 2024
@github-actions GitHub Actions
Copy link

Release notes:
Reordered and fixed database structure scripts to allow Flyway usage for incremental migration of existing database (version 0.1.x). Known issues: there might be needed some manual steps if upgrading from database of version 0.2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benedeki benedeki

Labels
DB Issues touching the Database part of the project infrastructure Project setup and deployment
Projects
CPS small repos project
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#262: Full Flyway integration AbsaOSS/atum-service
1 participant
@benedeki