feat(machined): add panic/force mode reboot

[laurazard]

Pull Request

closes #11782

What? (description)

Add a forceful mode to the reboot sequence (talosctl reboot --mode force) that bypasses graceful userspace teardown.

Why? (reasoning)

In certain situations, Talos's shutdown/reboot sequence hangs while waiting for services/mounts to be gracefully stopped (see: #11775).

---

I left a few questions as TODOs, PTAL at them if you review! Mainly I'm wondering about *machine.RebootRequest vs. runtime.XxxOptions, and whether to error out if we don't get a *machine.RebootRequest or just silently continue (which is fine for this, since any hypothetical older client wouldn't support force-reboot anyway, and the default is "graceful reboot".

Acceptance

Please use the following checklist:

• you linked an issue (if applicable)
• you included tests (if applicable)
• you ran conformance (make conformance)
• you formatted your code (make fmt)
• you linted your code (make lint)
• you generated documentation (make docs)
• you ran unit-tests (make unit-tests)

[shanduur]

Do we need to log here, or can we just wrap the returned error with the typeURL?

[laurazard]

Yeah, I was going to ask about this – I'm not sure about in general what we tend to do in talosctl, and (from what I saw) logging seems kind of inconsistent – I had to add this to even figure out what was going wrong because I was just getting very generic error output.

I'm happy to change the error returned, (after checking whether we're using ErrEventNotSupported isn't being used as a sentinel anywhere/compared to directly).

(As a sidenote, I wish we had a talosctl --debug ... and debug level logging.)

[laurazard]

It feels weird to wrap ErrEventNotSupported with the typeURL since the typeURL is a part of why that error happened, so I pushed a commit changing ErrEventNotSupported into a

type ErrEventNotSupported struct {
  TypeURL string
}
...

However, in practice we can't/shouldn't do this outside of a major version, since it changes a public API (hence me leaving it separate). In the meantime, we can either keep it as-is, or wrap it like suggested.

[laurazard]

Oh, I didn't realize having an extra commit meant CI would complain/tests won't run 😭 I'll remove it then and just skip it for now.

[smira]

Looks like "regular" reboot is broken now (I was looking at integration-qemu logs), @laurazard please let me know if you need help to troubleshoot/read logs.