Migrating the apps from vintage MC to CAPA MC

[teemow]

No description provided.

[teemow]

@weatherhog @piontec we would need to refine this story. So for the CAPA migration we need to ensure that all apps from an vintage MC can be migrated to a new CAPA MC.

For the migration we will spin up a new CAPI MC and then migrate the clusters over from the vintage MC to the new controllers on the CAPI MC. We need to do something similar for the apps.

This has a high priority. Happy to chat and bring in other people.

[teemow]

please talk to @calvix

[teemow]

The migration process is documented here: #2826

[teemow]

Goal is to have an MVP for how we can migrate apps seemless from one MC to the other.

[piontec]

Things/questions to check on honey badger's side:

• which app-operator versions are being used right now and which versions of Chart CR they produce (do we have a case where Chart CR's version will differ)?
• to which version of app-operator are we upgrading to?
• what's the realistic ETA to finish the migration?
• what about changes needed in the MCB+CMC flux repos? How to handle them? Is the new MC provisioned well in advance?
• what will change between the old and new MC?
• do we get it right that we will migrate 1 WC at a time?

[piontec]

From Vaclav:
Q: which app-operator versions are being used right now and which versions of Chart CR they produce (do we have a case where Chart CR's version will differ)?
A: for vintage version it depends on what release will be used for the migration atm we use the 19.0.1 as base for testing, it might change depending on the release which will be used for migration
https://github.com/giantswarm/releases/blob/master/aws/v19.0.1/release.yaml#L133-L134

Q: to which version of app-operator are we upgrading to?|
A: again depends when will this happens, currently this si used https://github.com/giantswarm/cluster-apps-operator/blob/master/helm/cluster-apps-operator/values.yaml#L1-L3
the version of app operator is probably up to you to decide, if you say we need to adjust to some specific version than so be it, but i since this will be quite long process I would expect that at some point vintage will have some older version and we will be migrating to new version currently used for CAPI, if there is a breaking change that would cause issues we should monitor this and make notes, most likely we wont have a chance to keep vintage up to date after the initial migration start

Q: what's the realistic ETA to finish the migration?
A: I cannot answer that as this is not only on my or my team, other teams also will have stuf to do and we will be relying on customers, rather ask in the #wg-capa-migration or on the bi-weekly sync
My guess is that the infra part could be in some preview state in end of OCtober and maybe customer ready by end of November, but customers wont migrate until 2024 Q1

Q: what about changes needed in the MCB+CMC flux repos? How to handle them?
A: This is out of scope of my knowledge and up to you to figure out, you should see the migration as migration from one MC to another MC, the name is different, the management fleet will be empty and so on, you might need to prepare migration for the apps that would be deployed by flux

Q: what will change between the old and new MC?
A: From App perspective it will be another MC, not new and old, it will just run different provision controller for the cluster

Q: do we get it right that we will migrate 1 WC at a time?
A: Yes, it will be triggered manually, one by one

[piontec]

Notes from a meeting with Vaclav:

• stuff from release-operator is already handled
• WC nodes will be replaced, all of them
• we need to check out-of-band is used for Flux to target WCs
• DNS: we will switch the DNS of the MC cluster - every pod will be restarted anyway
• vintage is not gone next year
• WC IDs will stay the same
• kube.config Secret name and structure - we have to check if it stays the same
• we should be migrating from PSS (not PSP) vintage

[alex-dabija]

@weatherhog & @piontec do you know when you will be ready to let the other teams test the migration of their apps?

[weatherhog]

@nce and @calvix are working on this. But there are several small hurdles that need to be solved. @nce tried to get it work from gaia but is now changing to gauss as it was not working on gaia

[alex-dabija]

How are things going on this? Do you still have problems?

[nce]

No, i'm currently implementing the apps migration. Currently i'm not able to estimate a timeframe, i'll update here asap

[alex-dabija]

No, i'm currently implementing the apps migration. Currently i'm not able to estimate a timeframe, i'll update here asap

Thanks!

@nce Could you list what still needs to be done? I would like to understand how much effort is still required.

[nce]

Not a complete list, but what comes to my mind:

• currently i'm debugging around app-operator, might just be a "ordering" thing of migration items
• admission webhooks are blocking bc/ of missing cm/secrets the apps (which is related to 1) )
• refactoring/polishing
• testing complex apps // having someone more proficient look over implementation details
• monitor behaviour of migrated apps

Unfortunately the roundtrip for testing the migration is a bit long

I hope to solve most of above tasks with @mproffitt together now, as we put more manpower behind this issue

[nce]

Current state:

• we have a solid migration scheme right now, allowing us to run successful migrations
• sorted the injection timing of the apps on the new MC

We lack:

• proper testing of all fields in the app
• complex app migrations
• refactoring of related cm/secret migrations

We face sometimes:

• issues with golem iam
• switch to gaia bc gauss isn't available anymore

[nce]

Everything is consolidated here: https://github.com/giantswarm/app-migration-cli

The tool is ready to be tested by other teams for the whole migration project.