fix(controllers): extend crash-loop self-heal to memory members

[K.J. Valencik (kjvalencik)]

The crash-loop self-heal (#336) excluded memory-medium members on the theory that the pod-loss self-heal already covers them. That check only fires when the Pod is gone or replaced (Status.PodUID mismatch); a memory member whose Pod is alive but whose etcd can never start keeps the same Pod UID forever and is invisible to it. The operator therefore left such a member crash-looping unrecovered forever.

The live failure mode: a replacement learner is created with --initial-cluster baked into its immutable Pod spec; if membership changes again before the learner's first successful boot, etcd fatals with error validating peerURLs ...: member count is unequal on every restart. With max-learners=1 the wedged learner also blocks all further member replacement, so the cluster stays degraded until a human (or a node rotation that happens to kill the Pod) intervenes.

Drop the medium exclusion. Every other guard stays: non-bootstrap, etcdContainerStuck (not ready, restart threshold, not OOMKilled, Pod not terminating) and the quorum gate. Replacing a stuck memory member is strictly milder than the already-covered PVC case — its data dies with the Pod anyway.

Includes a regression test (TestUpdateStatus_ReplacesStuckMemoryMember) and doc updates in docs/concepts.md / docs/operations.md.

Related: #351

Summary by CodeRabbit

• Bug Fixes

  • Extended crash-loop self-healing to memory-backed members, enabling automatic replacement when cluster quorum is retained.

• Documentation

  • Clarified that crash-loop replacement applies to both memory-backed and PVC-backed members.
  • Documented behavior for stale membership, lost data directories, and memory members without PVC cleanup.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 781ab2c1-2a76-46cb-81ec-4bd0c7aea594

📥 Commits

Reviewing files that changed from the base of the PR and between fe2b15e and 0ecaf81.

📒 Files selected for processing (4)

• controllers/etcdmember_controller.go
• controllers/etcdmember_controller_test.go
• docs/concepts.md
• docs/operations.md

🚧 Files skipped from review as they are similar to previous changes (4)

• controllers/etcdmember_controller.go
• controllers/etcdmember_controller_test.go
• docs/operations.md
• docs/concepts.md

---

📝 Walkthrough

Walkthrough

Crash-loop self-healing now applies to eligible memory-backed members as well as PVC-backed members. A regression test covers replacement when quorum remains available. Documentation describes the expanded behavior.

Changes

Crash-loop recovery

Layer / File(s)	Summary
Memory member self-healing and regression coverage controllers/etcdmember_controller.go, controllers/etcdmember_controller_test.go	The controller allows replacement of persistently crash-looping non-bootstrap members without requiring PVC storage. The test verifies deletion of a stuck memory-backed member when the cluster retains quorum.
Recovery documentation docs/concepts.md, docs/operations.md	The documentation covers memory-backed and PVC-backed members, quorum protection, replacement learners, and PVC cleanup only when a PVC exists.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• cozystack/etcd-operator#347: Changes related PVC lifecycle and reclamation behavior.

Suggested labels: bugfix

Suggested reviewers: androndo, lllamnyp

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: extending crash-loop self-healing to memory members.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/concepts.md`:
- Around line 160-166: Document that crash-loop self-healing applies only when
EtcdMember.Spec.Bootstrap is false: qualify the trigger and replacement flow in
docs/concepts.md lines 160-166, and qualify the crash-loop replacement exception
in both storage rows at docs/concepts.md lines 130-131. Keep bootstrap members
excluded from this recovery behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 88795c74-1f40-4e63-95a9-ba3ac4d66c3f

📥 Commits

Reviewing files that changed from the base of the PR and between 9c5d896 and ffe7e26.

📒 Files selected for processing (4)

• controllers/etcdmember_controller.go
• controllers/etcdmember_controller_test.go
• docs/concepts.md
• docs/operations.md

[Timofei Larkin (lllamnyp)]

Hi K.J. Valencik (@kjvalencik) , thanks for your amazing work here and on #351. Could you please run git commit --amend --signoff --no-edit && git push --force-with-lease on this PR so the DCO check passes?

[K.J. Valencik (kjvalencik)]

Done, thanks!