LXD 5.20: rsync errors with lxc copy and files with NTACLs #13707
Comments
|
Hi, Do you still experience this issue using latest/stable (5.21.1) as LXD 5.20 isn't supported anymore? |
|
I had a similar problem on other servers copying a nextcloud container (but no NTACLs there), and upgraded the destination server to 5.21.1 LTS. As this didn't help, I didn't upgrade the other ones. But in that case, deleting the files with rsync error on the destination did help, no more error with the next lxc copy. |
|
I now upgraded both servers to lxd 5.21.1 LTS, but the problem persists. |
|
Hi @MaxRower apologies ive not had chance to investigate this yet |
|
@boltmark please can you look into this when you get a chance. Thanks! |
|
@boltmark hows this going? |
Hi @tomponline, this is next on my list after I get through my current bugs. |
|
Hi @MaxRower, Can you please check your system logs for any AppArmor denials when you get the I believe what is happening here is that rsync is being blocked on the target by the AppArmor profile used in lxd. Since lxd does not provide the I imagine you'll see something like this: |
|
Any thoughts on this @mihalicyn ? |
|
@MaxRower what is the target storage pool driver in this case? |
tomponline
added
Bug
|
LXD grew an Apparmor profile for rsync in LXD 5.14 (#11510) which is probably why you didn't run into the issue with 5.12. NT ACLs are saved under Considering that our invocation of
An alternative that should work (I still have to test) would be to use optimized refresh between identical storage backends (btrfs to btrfs or zfs to zfs) as that would entirely bypass |
btrfs for source and target. |
With btrfs send and receive it will probably copy the entire subvolume, not only the changes, like rsync? That's why I can only use rsync-based transfers, copying hundreds of GB every day for a backup would be very time consuming, and for remote copies impossible. Sometimes in the past, I had a ticket for exactly that, so that --instance-only uses rsync. |
I will do that, when the error occurs again. |
I theory, the Now, since your backup system (the receiving side of the copy) has been "tainted" with rsync, there is no longer any commonality at the FS level between the source and the destination. This means that the refresh has to always happen with rsync due to that accidental/unexpected tainting. We've had a couple of bugs in our One way to confirm the above explanation would be to start with a fresh copy/destination and see if LXD stops trying to use rsync when doing subsequent refreshes. That's what I'd like to test but haven't got the time to get around doing it just yet. |
|
To set So, to write a But there is a trick. Some LSMs can override that behavior and lift that specific requirement. For example, SELinux have a pretty complex logic (ref https://github.com/torvalds/linux/blob/c763c43396883456ef57e5e78b64d3c259c4babc/security/selinux/hooks.c#L3197) and everything depends on the current SELinux setup on the machine. |
|
Hi @MaxRower! Please, can you show: |
architecture: x86_64
|
Required information
Issue description
After upgrading lxd from 5.12 (24643) to 5.20, there are errors with "lxc copy" using rsync with changed files using NTACLs for samba server.
The remote host has the same configuration (OS and file system)
lxc copy remotehost:samba samba --refresh --instance-only -c boot.autostart=false -q
rsync: [receiver] rsync_xal_set: lsetxattr("/var/snap/lxd/common/lxd/storage-pools/lxd/containers/samba/rootfs/*******","security.NTACL") failed: Operation not permitted (1)
Running the rsync command manually with a dedicated snapshot for rsync does work, e.g.
rsync -e "ssh -i keyfile" -ar --devices --numeric-ids --partial --sparse --xattrs --filter="-x security.selinux" --delete --compress --compress-level=2 root@remotehost:/srv/lxd/containers-snapshots/samba/rsync/ /srv/lxd/containers/samba/
If I run rsync manually, it copies all changed files successfully, and lxc copy has no errors afterwards, until there are changed files with xattrs again (or if I delete some of those files manually on the target).
Steps to reproduce
Information to attach
dmesg)lxc info NAME --show-log)lxc config show NAME --expanded)time="2024-07-07T13:32:50+02:00" level=error msg="Failed migration on target" clusterMoveSourceName= err="Failed creating instance on target: Rsync receive failed: /var/snap/lxd/common/lxd/storage-pools/lxd/containers/samba/: [exit status 23] (rsync: [receiver] rsync_xal_set: lsetxattr(\"/var/snap/lxd/common/lxd/storage-pools/lxd/containers/samba/rootfs/srv/shared/daten/Kinder/Johannes/Java/Java-Projekte/ServerStop\",\"security.NTACL\") failed: Operation not permitted (1)\nrsync: [receiver] rsync_xal_set: lsetxattr(\"/var/snap/lxd/common/lxd/storage-pools/lxd/containers/samba/rootfs/*********\",\"security.NTACL\") failed: Operation not permitted (1) and many more...lxc monitorwhile reproducing the issue)The text was updated successfully, but these errors were encountered: