[noble] syslog release breaks rsyslog

[ramonskie]

in syslog_storer in the syslog release
it overwrite rsyslog see: https://github.com/cloudfoundry/syslog-release/blob/main/jobs/syslog_storer/templates/rsyslog.conf.erb

this causes rsyslog to fail.
it seems to have to do with write permissions.

as we had also problems with writing the bosh-agent logs to the /var/vcap/bosh/log from systemd due to permissions.
and we agreed that logs should be in the logs directory.

more investigation is needed

[ramonskie]

noticed when ci tests where failing https://bosh.ci.cloudfoundry.org/teams/stemcell/pipelines/stemcells-ubuntu-noble/jobs/test-stemcells-master-ipv4/builds/47#L667487cc:926

systemctl status rsyslog output:

un 28 09:05:38 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension.>
Jun 28 09:05:38 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: action 'action-8-builtin:omfile' resumed (module 'builtin:omfile') [v8.2312.0 try https://www.rsyslog.com/e/2359 ]
Jun 28 09:05:38 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: action 'action-8-builtin:omfile' suspended (module 'builtin:omfile'), retry 0. There should be messages before this one giving the reason for suspension.>
Jun 28 09:05:50 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: rsyslogd: file '/var/vcap/data/syslog_storer/buffered/agg_backlog.00000001': open error: Permission denied [v8.2312.0 try https://www.rsyslog.com/e/2433 ]
Jun 28 09:05:50 b1518270-e300-444b-a56f-e2f356433f53 rsyslogd[5399]: file '/var/vcap/data/syslog_storer/buffered/agg_backlog.00000001': open error: Permission denied [v8.2312.0 try https://www.rsyslog.com/e/2433 ]

[ctlong]

This reads like rsyslog does not have access to /var/vcap/data/syslog_storer in Noble. Was there a permissions change from Jammy to Noble related to this directory?

[rkoster]

Nobel stemcells can be found here:

• storage.googleapis.com/bosh-core-stemcells-candidate/google/bosh-stemcell-0.59-google-kvm-ubuntu-noble-go_agent.tgz
• storage.googleapis.com/bosh-core-stemcells-candidate/aws/bosh-stemcell-0.59-aws-xen-hvm-ubuntu-noble-go_agent.tgz
• storage.googleapis.com/bosh-core-stemcells-candidate/azure/bosh-stemcell-0.59-azure-hyperv-ubuntu-noble-go_agent.tgz

Source: https://bosh.ci.cloudfoundry.org/teams/stemcell/pipelines/stemcells-ubuntu-noble/

[ctlong]

Sorry, I lost track of this for a while.

❓ Is this error still occurring?
❓ Has the new rsyslog package for Noble been pulled in?