rd.live.overlay.overlayfs=1 is broken on Fedora 39 LiveOS

[mahdiaqallal]

Describe the bug
A clear and concise description of what the error is.

rd.live.overlay.overlayfs=1 is supposed to provide a non-persistent (i.e. deleted after reboot of Live Image) and temporary storage in RAM at /run/overlayfs which is by default 32GiB unless another parameter is supplied : rd.live.overlay.size=<size_MiB> . See dracut documentation #Booting live images

Using rd.live.overlay.overlayfs=1 as kernel command line parameter has two issues

1. sudo dmesg | grep overlayfs includes an error:
   [ 13.317280] overlayfs: failed to resolve '/run/overlayfs': -2

2. Installing any rpm package once Fedora 39 LiveOS is running, results with the same errors
   error: lsetfilecon: (33 /usr/bin/make-dummy-cert;66130180, system_u:object_r:bin_t:s0) Permission denied
   error: Plugin selinux: hook fsm_file_prepare failed
   [...]
   Error unpacking rpm package
   Failed:
   Error: Transaction failed

Below an example with openssl

 sudo dnf install -y openssl`
Last metadata expiration check: 1:44:39 ago on Sun 07 Apr 2024 08:41:58 PM CEST.
Dependencies resolved.
===============================================================================
 Package                  Architecture            Version                            Repository               Size
===============================================================================
Installing:
 openssl                  x86_64                  1:3.1.1-4.fc39                     fedora                  1.0 M

Transaction Summary
===============================================================================
Install  1 Package

Total download size: 1.0 M
Installed size: 1.6 M
Downloading Packages:
openssl-3.1.1-4.fc39.x86_64.rpm                                                    1.4 MB/s | 1.0 MB     00:00    
-------------------------------------------------------------------------------------------------------------------
Total                                                                              802 kB/s | 1.0 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                           1/1 
  Installing       : openssl-1:3.1.1-4.fc39.x86_64                                                             1/1 
error: lsetfilecon: (33 /usr/bin/make-dummy-cert;66130180, system_u:object_r:bin_t:s0) Permission denied
error: Plugin selinux: hook fsm_file_prepare failed

Error unpacking rpm package openssl-1:3.1.1-4.fc39.x86_64
  Verifying        : openssl-1:3.1.1-4.fc39.x86_64                                                             1/1 

Failed:
  openssl-1:3.1.1-4.fc39.x86_64                                                                                    

Error: Transaction failed

Distribution used
Which distribution was this behaviour seen in?
Fedora 39 LiveOS

Dracut version
Which dracut version was this behaviour seen in?
Fedora 39 dracut-059-16.fc39
Init system
Which init system is being used?
systemd
To Reproduce
Steps or code to reproduce the behavior.
Insert alongside your kernel command line parameter on Fedora 39 Live ISO (for example in a grub.cfg menuentry) rd.live.overlay.overlayfs=1 after the parameter rd.live.image
Expected behavior
A clear and concise description of what you expected to happen.

We should have with this parameter :

1. a root live filesystem with 32GiB of available space instead of 4.9G is used and 1.5G is available space out of a total size of 6.4G

2. the ability to install rpm packages with sudo dnf install <package> until the available RAM is depleted

Additional context
Add any other context you like about the problem here.

a. Users of Fedora Linux have been reporting related bugs here: https://discussion.fedoraproject.org/t/fedora-liveos-root-system-and-available-ram/82531/1

CC @gregory-lee-bartholomew

(Cherry-picked commit from #2604)

b. The Booting Live Images documentation section could need be more "verbose" with more explanations, examples, use-cases. IMHO

[mahdiaqallal]

Food for thoughts: couldn't it be related with SELinux rules because of the folders where /run/overlayfs stores all modified files and directories after thoses unpacked from squashfs in the live boot ?

I found this thread in Fedora Discussion and this and that bugzilla report

Can't be sure if it helps or not, it is well beyond my understanding

I didn't use any of the livecd-tools to reproduce this issue. The Fedora 39 LiveOS was directly written into an usb using following command
sudo dd if=/<path-to-Fedora 39 LiveOS.iso> of=/dev/sdb

Then the usb drive was booted and it's grub.cfg modified according to my description above

FYI: livecd-tools can't be used as of Fedora 39 and since Fedora 35+ because the structure of the .iso files has been changed and an appropriate release still needs to be available from the official Fedora repos. see this issues livecd-tools/livecd-tools#244 and livecd-tools/livecd-tools#253 in livecd-tools repository

[FGrose]

The 2 problems you identified at the top are fixed in an update of Fedora 39 such as in the live-respin, https://dl.fedoraproject.org/pub/alt/live-respins/F39-WORK-x86_64-LIVE-20240401.iso
Those problems are not related to the rd.live.overlay.overlayfs option.

The dracut documentation #Booting live images is incomplete. It should rather say:

SquashFS (read-only) base filesystem image
Note — There are 3 separate overlay types available:

• Device-mapper snapshots (the original offering),
• Device-mapper thin provisioning snapshots (see rd.live.overlay.thin,
  a later offering), and
• OverlayFS based overlay mounts (a more recent offering).

Using one of these technologies, the system will provide a writable overlay for
the base, read-only SquashFS root filesystem. These methods enable a relatively
fast boot and lower RAM usage.

With the original Device-mapper snapshot overlay, users must be careful to
avoid writing too many blocks to the snapshot device. Once the blocks of the
snapshot overlay are exhausted, the whole root filesystem becomes read-only
leading to application failures. The snapshot overlay device is marked
'Overflow', and a difficult recovery is required to repair and enlarge the
overlay offline.

When rd.live.overlay= is not specified for persistent overlay storage, or
the specified file is not found or writable, a Device-mapper snapshot based
non-persistent or temporary overlay is automatically created as a sparse file
in RAM of the initramfs. This file will only consume content space as required
blocks are allocated. This snapshot based overlay defaults to an apparent size
of 32 GiB in RAM, and can be adjusted with the rd.live.overlay.size= kernel
command line option. This file is hidden (and appears deleted) when the boot
process switches out of the initramfs to the main root filesystem but its loop
device remains connected to the Device-mapper snapshot.

Even with large Device-mapper overlay files for write space, the available root
filesystem capacity is limited by the total allocated size of the base root
filesystem, which often provide only a small number of gigabytes of free space.

This shortage could be remedied by building the root filesystem with more
allocated free space, or the OverlayFS based overlay mount method can be used.

When the rd.live.overlay.overlayfs option is specified or when
rd.live.overlay= points to an appropriate directory with a sister at
/../ovlwork, then an OverlayFS based overlay mount is employed. Such a
persistent OverlayFS overlay can extend the available root filesystem storage
up to the capacity of the LiveOS disk device.

For non-persistent OverlayFS overlays, the /run/overlayfs directory in the
/run tmpfs is used for temporary storage. This filesystem is typically sized
to one half of the RAM total in the system.
The command: mount -o remount,size=<nbytes> /run will resize this virtual
filesystem after booting.

Cross reference link dracut-ng/dracut-ng#137

[mahdiaqallal]

@FGrose : thanks for your reply .
It might need some elaboration though:

Which kernel command line parameter is needed to activate on a Fedora Worsktation Live .ISO, the "Device-mapper snapshot based non-persistent overlays":

Device-mapper snapshot based non-persistent overlays are
sparse files in RAM that only consume content space as required blocks are
allocated. These snapshot based overlays default to an apparent size of 32 GiB
in RAM.

Regarding live-respins

The 2 problems you identified at the top are fixed in an update of Fedora 39 such as in the live-respin, https://dl.fedoraproject.org/pub/alt/live-respins/F39-WORK-x86_64-LIVE-20240401.iso

Are live-respins natively corrected for rd.live.overlay.overlayfs=1 ?
Does this mean that on Fedora 40 Worsktation Live ISO the bug for rd.live.overlay.overlayfs=1 is fixed too ?

I'll try soon sudo mount -o remount,size=70% /run (on 32GiB RAM laptop) on Fedora 40 Workstation LIve ISO and will provide feedback

[mahdiaqallal]

@FGrose Thanks for updating the man page.
Please it needs some enhancement like examples for the two use cases below so I can test it on Fedora 40beta Workstation LIve ISO command line parameters and provide feedback

What are the minimal kernel command line parameters to activate (on a freshly created Fedora 40beta Workstation Live USB):

1. Temporary in memory ove
   OR
2. Permanent boot reristant on disk overlay (provided a dedicated vfat or ext4 or btrfs partition has been created)