Cluster recovery improvements

[MggMuggins]

Fixes #13524

A detailed description of the problem this resolves is in the issue. This PR:

• lxd cluster edit:
  • Prompts the user with a warning & link to the docs prior to cluster edit
  • Includes more information about member roles in the yaml during editing
  • Generates a tarball (/var/snap/lxd/common/lxd/database/recovery_db.tar.gz) with the contents of the database & the new raft configuration as yaml
  • Creates a patch.global.sql to update the addresses of any nodes that were changed in the global nodes table
• On Daemon startup, looks for & loads a recovery tarball at /var/snap/lxd/common/lxd/database/recovery_db.tar.gz and:
  • Replaces the existing database contents with the incoming contents
  • Updates cluster member addresses in the local DB's raft_nodes table

Currently the unpack code also creates the global sql patch on each node; in general this should only be done on one node. Since the patch is idempotent (it's just a couple of UPDATE), it works fine, but it's unideal. See my comment below.

LXD-1194

[github-actions]

Heads up @mionaalex - the "Documentation" label was applied to this issue.

[ru-fu]

Thanks, the docs look good now!

[MggMuggins]

When applying patch.global.sql on only one cluster member, occasionally database startup fails for one or (usually) more members (patch applied).

Error: Failed to initialize global database: failed to ensure schema: Failed to ensure schema: Failed to update cluster member version info: updated 0 rows instead of 1 with address "10.1.1.102:9393"

That error tracks back to https://github.com/canonical/lxd/blob/main/lxd/db/cluster/query.go#L15-L17

That query selects WHERE address=?; when we load the recovery tarball, we change the value of cluster.https_address, which is used as the address parameter in that query. If patch.global.sql has not been committed yet, the query affects zero rows instead of one (as it is looking for the old address). I inserted a select * from nodes before that query to test and it returns the old addresses when the failure happens.

While schema.Ensure (which calls updateNodeVersion) is wrapped in query.Retry, somehow this failure sometimes persists through the retry and causes the daemon to fail to start.

This isn't an issue with the current cluster edit because patch.global.sql is created on every node (Reconfigure is called on each cluster member per the docs). This ensures that the address changes are visible to each schema.Ensure transaction before updateNodeVersion is called.

The easy solution/workaround here is simply to create patch.global.sql on each cluster member. The patch is idempotent so there's no harm in it beyond doing more work than needed on startup.

Alternatively, it may be possible to use the node ID instead of address to update the schema/api_extensions fields in the nodes table. Since we don't have access to the global database before ensuring the schema, this would require getting the ID from the local DB. I'm not sure if we can rely on the raft_nodes.id in the local DB being the same as nodes.id in the global DB.

@tomponline Let me know what you think is the more reasonable approach or if this doesn't sound right. Thanks!

[MggMuggins]

I renamed the recovery tarball to lxd_recovery_db.tar.gz so as to prevent confusion with other recovery tarballs for other Micro* daemons. I assume that LXD would not be the only service to lose quorum in a cluster recovery situation.