[Flaky Test] CAPM3 deployment is not ready

[mboukhalfa]

Which jobs are flaking?

Only :
Clusterctl upgrade main

Which tests are flaking?

metal3-periodic-e2e-clusterctl-upgrade-test-main When testing cluster upgrade from releases (v1.7=>current) in the STEP: [0] Upgrading providers to the latest version available

Since when has it been flaking?

First seen on Jul 10, 2024, 10:10:00 PM

Jenkins link

https://jenkins.nordix.org/view/Metal3%20Periodic/job/metal3-periodic-e2e-clusterctl-upgrade-test-main/77/consoleFull

Reason for failure (if possible)

Not sure just pasting the error logs

16:51:00    < Exit [AfterEach] When testing cluster upgrade from releases (v1.7=>current) [clusterctl-upgrade] @ 07/18/24 13:50:54.088 (5m1.138s)
16:51:00  • [FAILED] [3413.643 seconds]
16:51:00  When testing cluster upgrade from releases (v1.7=>current) [clusterctl-upgrade] [It] Should create a management cluster and then upgrade all the providers
16:51:00  /home/metal3ci/go/pkg/mod/sigs.k8s.io/cluster-api/[email protected]/e2e/clusterctl_upgrade.go:253
16:51:00  
16:51:00    [FAILED] failed to run clusterctl upgrade
16:51:00    Unexpected error:
16:51:00        <*errors.withStack | 0xc001bb3848>: 
16:51:00        deployment "capm3-controller-manager" is not ready after 5m0s: failed to connect to the management cluster: action failed after 0 attempts: context deadline exceeded
16:51:00        {
16:51:00            error: <*errors.withMessage | 0xc001fecde0>{
16:51:00                cause: <*errors.withStack | 0xc001bb3818>{
16:51:00                    error: <*errors.withMessage | 0xc001fecdc0>{
16:51:00                        cause: <*errors.withStack | 0xc001bb37e8>{
16:51:00                            error: <*errors.withMessage | 0xc001fecda0>{
16:51:00                                cause: <context.deadlineExceededError>{},
16:51:00                                msg: "action failed after 0 attempts",
16:51:00                            },
16:51:00                            stack: [0x3113ac5, 0x3132326, 0x31198b0, 0x24afe72, 0x24afccd, 0x24b0185, 0x31196f3, 0x3119673, 0x3119507, 0x314108b, 0x313ec45, 0x315aa8d, 0x326dc48, 0x3272d08, 0x34d627a, 0x192c193, 0x194036d, 0x148fe21],
16:51:00                        },
16:51:00                        msg: "failed to connect to the management cluster",
16:51:00                    },
16:51:00                    stack: [0x313233c, 0x31198b0, 0x24afe72, 0x24afccd, 0x24b0185, 0x31196f3, 0x3119673, 0x3119507, 0x314108b, 0x313ec45, 0x315aa8d, 0x326dc48, 0x3272d08, 0x34d627a, 0x192c193, 0x194036d, 0x148fe21],
16:51:00                },
16:51:00                msg: "deployment \"capm3-controller-manager\" is not ready after 5m0s",
16:51:00            },
16:51:00            stack: [0x31197e8, 0x3119507, 0x314108b, 0x313ec45, 0x315aa8d, 0x326dc48, 0x3272d08, 0x34d627a, 0x192c193, 0x194036d, 0x148fe21],
16:51:00        }
16:51:00    occurred
16:51:00    In [It] at: /home/metal3ci/go/pkg/mod/sigs.k8s.io/cluster-api/[email protected]/framework/clusterctl/client.go:202 @ 07/18/24 13:35:57.588
16:51:00  
16:51:00    Full Stack Trace
16:51:00      sigs.k8s.io/cluster-api/test/framework/clusterctl.Upgrade({_, _}, {{0xc000de2300, 0x53}, {0xc00063a35d, 0x47}, 0x0, {0xc001a7a300, 0x24}, {0xc00193b640, ...}, ...})
16:51:00      	/home/metal3ci/go/pkg/mod/sigs.k8s.io/cluster-api/[email protected]/framework/clusterctl/client.go:202 +0x63d
16:51:00      sigs.k8s.io/cluster-api/test/framework/clusterctl.UpgradeManagementClusterAndWait({_, _}, {{0x41f4178, 0xc000fb1560}, {0xc00063a35d, 0x47}, 0x0, {0x3d27c5b, 0x7}, {0x0, ...}, ...}, ...)
16:51:00      	/home/metal3ci/go/pkg/mod/sigs.k8s.io/cluster-api/[email protected]/framework/clusterctl/clusterctl_helpers.go:212 +0x9e8
16:51:00      sigs.k8s.io/cluster-api/test/e2e.ClusterctlUpgradeSpec.func2()
16:51:00      	/home/metal3ci/go/pkg/mod/sigs.k8s.io/cluster-api/[email protected]/e2e/clusterctl_upgrade.go:582 +0x38ba

Anything else we need to know?

clusterctl was failing because of other issue the got fixed recently and this flake can be also because of infra also it is frequent

Label(s) to be applied

/kind flake

[mboukhalfa]

triage/accepted

[metal3-io-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[mquhuy]

@mboukhalfa is this still an issue?

[mboukhalfa]

seems no the clusterctl did not fail for long

[adilGhaffarDev]

we are not seeing this flake anymore. Closing this issue.