xfs corruption, but no xfs_repair

[smira]

Bug Report

XFS partition got corrupted, but Talos didn't run xfs-repair.

Description

Logs

09/02/2024 11:41:47 [talos] task mountEphemeralPartition (1/1): starting
09/02/2024 11:41:47 XFS (sda6): Mounting V5 Filesystem
09/02/2024 11:41:47 XFS (sda6): totally zeroed log
09/02/2024 11:41:48 XFS (sda6): Corruption warning: Metadata has LSN (90:781352) ahead of current LSN (1:0). Please unmount and run xfs_repair (>= v4.3) to resolve.
09/02/2024 11:41:48 XFS (sda6): log mount/recovery failed: error -22
09/02/2024 11:41:48 XFS (sda6): log mount failed
09/02/2024 11:41:48 [talos] task mountEphemeralPartition (1/1): failed: error mounting: 1 error(s) occurred:
09/02/2024 11:41:49  invalid argument
09/02/2024 11:41:50 [talos] phase ephemeral (8/17): failed
09/02/2024 11:41:50 [talos] boot sequence: failed

Environment

• Talos version: [talosctl version --nodes <problematic nodes>]
• Kubernetes version: [kubectl version --short]
• Platform:

[smira]

Filesystem Corruption Detection

1. Errors EUCLEAN, EINVAL from mount syscall (more errors?).
2. If META key 'needs_repair' is set

• set this key early on boot, and remove it once machine enters running & ready
• user can set this key manually and reboot

Scenario: Talos boots up, mount() finishes successfully, but the filesystem is corrupted, so containerd fails to start, so the META key needs_repair is not removed, and on next reboot Talos will run xfs_repair.

Filesystem Repair

1. Try mounting the filesystem (temporarily) (to replay the XFS log) [ignore errors].
2. Run xfs_repair.
3. If it fails, go to step 1, but next time add -L.

[smira]

Two PRs:

1. Adds EINVAL to EUCLEAN (backport to 1.6)
2. Which adds needs_repair flag - 1.7 only.

[frezbo]

Add EIO (-5) also