[ubutnu-22-lts] Issues with setting the timeserver

[sdwilsh]

I'm not sure if this is a bug or if I'm just doing something wrong. I've followed the documentation for timesyncd, and only changed the NTP line to point to my internal time server. If I ssh into the machine, I see that /etc/system/timesyncd.conf contains the right data based on the config I supplied. However, systemctl status systemd-timesyncd shows this:

● systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-01-27 03:57:52 UTC; 2min 37s ago
       Docs: man:systemd-timesyncd.service(8)
   Main PID: 1441 (systemd-timesyn)
     Status: "Initial synchronization to time server 185.125.190.56:123 (ntp.ubuntu.com)."
      Tasks: 2 (limit: 2273)
     Memory: 1.3M
        CPU: 62ms
     CGroup: /system.slice/systemd-timesyncd.service
             └─1441 /lib/systemd/systemd-timesyncd

That is not what is configured in /etc/system/timesyncd.conf. If I sudo systemctl restart systemd-timesyncd, the right time server is used, however.

[paynejacob]

Can you describe your steps in a little more detail? There is no mechanism to restart the service when configuration is changed so if you applied this setting in an after install stage the configuration would be updated but the changed would not apply until after a system reboot. If this is a normal boot and the time sync settings are applied but the process is not using them there may be a race condition with when the settings are applied and when timesyncd starts.

[paynejacob]

Yeah the reboot after install would have picked it up, if you still have issues you can try moving it to the rootfs stage but I think boot is the correct stage.

[sdwilsh]

I'm not sure where would be a good spot for this in the documentation, but it might be worthwhile to call out that a number of settings via the config might not be active after the first boot, and a reboot is needed to pick up the changes.

[paynejacob]

Did you add your cloud configuration after the device had started with the installer or was it mounted as a cloud config volume or added to your iso some how?

[sdwilsh]

The former, but let me be clearer on my steps:

1. New VM with kairos-ubuntu-22-lts-v1.4.0-k3sv1.26.0+k3s1.iso (I see 1.5.1 is out today, so I'll switch to testing that in the future)
2. Select manual install at the prompt
3. Write a config with with the timesyncd settings to point to my local ntp server
4. Run sudo kairos-agent manual-install --device "auto" --reboot config.yml
5. See the machine reboot
6. ssh into the machine, and run systemctl status systemd-timesyncd

[sdwilsh]

Confirmed that this is still an issue with kairos-ubuntu-22-lts-v1.5.1-k3sv1.26.1+k3s1.iso.

[paynejacob]

Ah thank you for the detailed steps in that case this sounds like a race condition try moving to the rootfs stage. The reboot on step 5 should have applied your settings. If all else fails you could add a command to your cloud config to reload sytemd-timesyncd after the setting is changed.

[sdwilsh]

This didn't work either, unfortunately. Here's the full config that I'm passing to sudo kairos-agent manual-install --device "auto" --reboot:

#cloud-config

hostname: test-controlplan01

users:
  - name: kairos
    groups: admin
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMiRn+dJyIkJ22qrkuMlNC33xxS7VUwkYRY/55Wf4ryq

k3s:
  enabled: true
  args:
  - --disable=traefik,servicelb

stages:
  rootfs:
    - name: Setup NTP
      systemctl:
        enable:
          - systemd-timesyncd
      timesyncd:
        NTP: pool.ntp.hogs.tswn.us
        FallbackNTP: ""
  initramfs:
    - files:
        - path: /etc/systemd/network/01-man.network
          permissions: 0644
          content: |
            [Match]
            Name=enp2s1

            [Network]
            Address=10.10.0.100/24
            Gateway=10.10.0.1
            DNS=10.10.0.1

Running it in rootfs didn't even update /etc/systemd/timesyncd.conf

[sdwilsh]

I'm happy to report that initramfs appears to work. I'm guessing this might Ubuntu-specific though, so I'm not sure what we want to do with the docs.