Description
After a minor package update is released to Debian, puppetserver can sometimes get into a stuck state, which typically looks like this on the side of individual server clients:
krinkle@wp-03:~$ sudo run-puppet-agent
Info: Using environment 'staging'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error:
Exception while executing '/etc/puppet/code/environments/staging/bin/config-version.sh': Cannot run program
The fix is simply to restart the puppet server:
krinkle@puppet-04:~$ sudo systemctl status puppetserver
Details about the root cause in the comments at c08df02.
@supertassu wrote
At Wikimedia we disabled automatic JRE updates on Puppetservers for this exact reason. We could do that here, but I would prefer to avoid this option as our server is open to the internet. (I guess that'd be relatively trivial to fix, except figuring out how to provision new nodes.) The other option is to hook up some logic to automatically restart the puppetserver service when needed and figure out the systemd magic to run that after the automatic updates finish.