Putting a user in the group "systemd-journal" or "kvm" does not work anymore from Ignition

[pothos]

Description

I have an old config where a new user gets created and added to the systemd-journal group from Ignition and another one with the kvm group. This fails to boot because Ignition does not find the group.

Impact

Regression, instance fails to boot (initrd emergency console)

Environment and steps to reproduce

The 2.2.0 config has an entry passwd → users → "name": "myuser", "groups": ["sudo", "docker", "systemd-journal", "kvm"]

Expected behavior

It works again

Additional information

Maybe we need to add the group in the initramfs, at least I see it in /usr/share/baselayout/group already and we could concatenate these entries to the initramfs passwd if that helps

[pothos]

I hit this again with the kvm group, this is a regression, maybe we can work around it by creating the group from ignition using the same ID but it means to have a duplicate entry in /etc/groups

[pothos]

Adding the group in Ignition doesn't work because it already exists… I think we should prioritize this if there is no easy workaround.

Original error [failed] creating or modifying user "build": exit status 6: Cmd: "useradd" "--root" "/sysroot" "--create-home" "--password" "*" "--groups" "docker,sudo,kvm" "build" Stdout: "" Stderr: "useradd: group 'kvm' does not exist\n"

Error after trying to add the group via Ignition: Ignition failed: failed to create users/groups: failed to configure groups: failed to create group "kvm": exit status 9: Cmd: "groupadd" "--root" "/sysroot" "--gid" "78" "--password" "*" "--system" "kvm" Stdout: "" Stderr: "groupadd: group 'kvm' already exists\n"

[pothos]

As @tormath1 found out, useradd looks for the group manually in /etc/groups instead of only looking up the known groups. I think in our case this even makes sense because we pass --root /sysroot but it's not consistent with the behavior of groupadd and I think it should be controllable by flag whether the known groups are resolved or explicitly looked up in /etc/groups and only one behavior should be chosen by default and be consistent for both tools.

[pothos]

As workaround we could add more special groups:
https://github.com/flatcar/baselayout/blob/flatcar-master/scripts/flatcar-tmpfiles#L13
Eventually upstream fixes would be nice of course…

[pothos]

Talked with @tormath1 and realized that this is not really something upstream could do as we want it to happen because of possible library compatibility problems and our nss plugin would also not know that it should look up under /sysroot/usr/share/baselayout/. The best solution would be to run chroot /sysroot /usr/bin/useradd and chroot /sysroot /usr/bin/groupadd from Ignition and drop the --root switch.

[pothos]

It's apparently not working (anymore?) to modify an nss-provided group. A workaround could be to check if defining a systemd-userdbd user json works with a memberOf field to extend the nss-provided group: https://systemd.io/USER_RECORD/ and https://www.freedesktop.org/software/systemd/man/systemd-userdbd.service.html

[tormath1]

gave a try with systemd-userdb, no more luck. To summarize, usermod -aG $GROUP $USER does not work if $GROUP is defined in nss-usrfiles (/usr/share/baselayout/group).

EDIT: I tried with lts-3033.3.9 which still has ignition-0.36, this is the config:

passwd:
  users:
    - name: tormath1
      groups:
        - systemd-journal
        - kvm

Indeed there is not error during the boot, but the user tormath1 is not in groups:

tormath1@localhost ~ $ groups
tormath1
tormath1@localhost ~ $

I think the error has always been there but silently ignored.

[pothos]

There is probably a bug in systemd-userdb when trying to create a new user with it, we can't add it to a group nor does it work to define a group and add members to it.

First, /etc/nsswitch.conf was created as follows (delete the symlink):

# /etc/nsswitch.conf:

passwd:      files usrfiles sss systemd
shadow:      files usrfiles sss systemd
group:       files [SUCCESS=merge] usrfiles [SUCCESS=merge] sss [SUCCESS=merge] systemd
gshadow:     files [SUCCESS=merge] usrfiles [SUCCESS=merge] sss [SUCCESS=merge] systemd

hosts:       files usrfiles resolve [!UNAVAIL=return] myhostname dns
networks:    files usrfiles dns

services:    files usrfiles
protocols:   files usrfiles
rpc:         files usrfiles

ethers:      files
netmasks:    files
netgroup:    files
bootparams:  files
automount:   files
aliases:     files

Then there are the user and group files in /etc/userdb/.
core2.group:

{
	"groupName" : "core2",
	"gid" : 502,
        "disposition" : "system",
        "members" : []        
}

core2.user:

{
        "userName" : "core2",
        "uid" : 502,
        "gid" : 502,
        "shell" : "/bin/bash",
        "homeDirectory" : "/home/core2",
        "memberOf": ["kvm"],
        "disposition" : "system",
        "locked" : false
}

You also need the symlinks:
502.user→core2.user
502.group→core2.group

Then the problem was that kvm wasn't added:

localhost /home/core # userdbctl user core2
   User name: core2
 Disposition: system
    Login OK: yes
 Password OK: no (none set)
         UID: 502
         GID: 502 (core2)
   Directory: /home/core2
     Storage: classic
       Shell: /bin/bash
      Locked: no
   Passwords: none
     Service: io.systemd.NameServiceSwitch

localhost /home/core # userdbctl groups-of-user core2
No memberships.
localhost /home/core # userdbctl users-in-group core2
No memberships.
localhost /home/core # userdbctl users-in-group kvm
No memberships.
localhost /home/core # sudo -u core2 groups
core2

[pothos]

@tormath1 found systemd/systemd#24381

Can confirm that the .membership file works. It also works for regular users like core:

localhost /etc/userdb # echo > core:kvm.membership  #  touch does not work, it can't be empty
localhost /etc/userdb # sudo -u core groups
wheel kvm docker systemd-journal portage core

So, as a workaround to add users to a group, Ignition could create this membership file. (coreos/ignition#1593)

[pothos]

The systemd-journal group is now a group like docker, so that should work out of the box in future Alphas.
For kvm and others the workaround will still be needed (see docs PR), but maybe we can also include this workaround in Ignition as automatic fallback.

[pothos]

For kvm and others the workaround will still be needed (see docs PR), but maybe we can also include this workaround in Ignition as automatic fallback.

Or rather use wrappers for useradd and friends in Flatcar that detect when userdbd should be used because the group is defined under /usr.

[chewi]

I have fixed the userdb issue in systemd/systemd#32871, pending review.

[chewi]

Apparently, systemd is actually working as designed (but not documented). 😞