stdout and stderr from failed commands are sent to /dev/null

[Cynerva]

I recently ran juju-crashdump with:

juju-crashdump -s -a debug-layer -a config -m mymodel

and received this output:

2021-08-09 18:04:51,930 - juju-crashdump started.
2021-08-09 18:05:51,103 - command juju ssh {} -- "cd /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/addons; for unit in $(cat debug-layer-units); do sudo juju-run $unit actions/debug; done; cp /home/ubuntu/debug-*.tar.gz /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/addon_output || true" failed
2021-08-09 18:05:51,104 - command juju ssh {} -- "cd /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/addons; for unit in $(cat debug-layer-units); do sudo juju-run $unit actions/debug; done; cp /home/ubuntu/debug-*.tar.gz /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/addon_output || true" failed
2021-08-09 18:06:01,741 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'mkdir -p /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output;sudo netstat -taupn | grep LISTEN 2>/dev/null | tee /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output/listening.txt || true'" failed
2021-08-09 18:06:01,743 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'mkdir -p /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output;sudo netstat -taupn | grep LISTEN 2>/dev/null | tee /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output/listening.txt || true'" failed
2021-08-09 18:06:04,813 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'mkdir -p /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output;sudo ps aux | tee /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output/psaux.txt || true'" failed
2021-08-09 18:06:04,815 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'mkdir -p /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output;sudo ps aux | tee /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output/psaux.txt || true'" failed
2021-08-09 18:06:07,885 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'mkdir -p /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa'" failed
2021-08-09 18:06:07,885 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'mkdir -p /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa'" failed
2021-08-09 18:06:10,957 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'find /etc/alternatives /etc/ceilometer /etc/ceph /etc/cinder /etc/cloud /etc/glance /etc/gnocchi /etc/keystone /etc/netplan /etc/network /etc/neutron /etc/nova /etc/quantum /etc/swift /etc/udev/rules.d /lib/udev/rules.d /opt/nedge/var/log /run/cloud-init /usr/share/lxc/config /var/lib/charm /var/lib/libvirt/filesystems/plumgrid-data/log /var/lib/libvirt/filesystems/plumgrid/var/log /var/lib/cloud/seed /var/log /var/snap/simplestreams/common/sstream-mirror-glance.log /var/crash /var/snap/juju-db/common/logs/ /var/lib/mysql/*-mysql-router /tmp/juju-exec*/script.sh /var/lib/lxd/containers/*/rootfs/etc/alternatives /var/lib/lxd/containers/*/rootfs/etc/ceilometer /var/lib/lxd/containers/*/rootfs/etc/ceph /var/lib/lxd/containers/*/rootfs/etc/cinder /var/lib/lxd/containers/*/rootfs/etc/cloud /var/lib/lxd/containers/*/rootfs/etc/glance /var/lib/lxd/containers/*/rootfs/etc/gnocchi /var/lib/lxd/containers/*/rootfs/etc/keystone /var/lib/lxd/containers/*/rootfs/etc/netplan /var/lib/lxd/containers/*/rootfs/etc/network /var/lib/lxd/containers/*/rootfs/etc/neutron /var/lib/lxd/containers/*/rootfs/etc/nova /var/lib/lxd/containers/*/rootfs/etc/quantum /var/lib/lxd/containers/*/rootfs/etc/swift /var/lib/lxd/containers/*/rootfs/etc/udev/rules.d /var/lib/lxd/containers/*/rootfs/lib/udev/rules.d /var/lib/lxd/containers/*/rootfs/opt/nedge/var/log /var/lib/lxd/containers/*/rootfs/run/cloud-init /var/lib/lxd/containers/*/rootfs/usr/share/lxc/config /var/lib/lxd/containers/*/rootfs/var/lib/charm /var/lib/lxd/containers/*/rootfs/var/lib/libvirt/filesystems/plumgrid-data/log /var/lib/lxd/containers/*/rootfs/var/lib/libvirt/filesystems/plumgrid/var/log /var/lib/lxd/containers/*/rootfs/var/lib/cloud/seed /var/lib/lxd/containers/*/rootfs/var/log /var/lib/lxd/containers/*/rootfs/var/snap/simplestreams/common/sstream-mirror-glance.log /var/lib/lxd/containers/*/rootfs/var/crash /var/lib/lxd/containers/*/rootfs/var/snap/juju-db/common/logs/ /var/lib/lxd/containers/*/rootfs/var/lib/mysql/*-mysql-router /var/lib/lxd/containers/*/rootfs/tmp/juju-exec*/script.sh -mount -type f -size -5000000c -o -size 5000000c 2>/dev/null | tar -pcf /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar --files-from - 2>/dev/null'" failed
2021-08-09 18:06:10,959 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'find /etc/alternatives /etc/ceilometer /etc/ceph /etc/cinder /etc/cloud /etc/glance /etc/gnocchi /etc/keystone /etc/netplan /etc/network /etc/neutron /etc/nova /etc/quantum /etc/swift /etc/udev/rules.d /lib/udev/rules.d /opt/nedge/var/log /run/cloud-init /usr/share/lxc/config /var/lib/charm /var/lib/libvirt/filesystems/plumgrid-data/log /var/lib/libvirt/filesystems/plumgrid/var/log /var/lib/cloud/seed /var/log /var/snap/simplestreams/common/sstream-mirror-glance.log /var/crash /var/snap/juju-db/common/logs/ /var/lib/mysql/*-mysql-router /tmp/juju-exec*/script.sh /var/lib/lxd/containers/*/rootfs/etc/alternatives /var/lib/lxd/containers/*/rootfs/etc/ceilometer /var/lib/lxd/containers/*/rootfs/etc/ceph /var/lib/lxd/containers/*/rootfs/etc/cinder /var/lib/lxd/containers/*/rootfs/etc/cloud /var/lib/lxd/containers/*/rootfs/etc/glance /var/lib/lxd/containers/*/rootfs/etc/gnocchi /var/lib/lxd/containers/*/rootfs/etc/keystone /var/lib/lxd/containers/*/rootfs/etc/netplan /var/lib/lxd/containers/*/rootfs/etc/network /var/lib/lxd/containers/*/rootfs/etc/neutron /var/lib/lxd/containers/*/rootfs/etc/nova /var/lib/lxd/containers/*/rootfs/etc/quantum /var/lib/lxd/containers/*/rootfs/etc/swift /var/lib/lxd/containers/*/rootfs/etc/udev/rules.d /var/lib/lxd/containers/*/rootfs/lib/udev/rules.d /var/lib/lxd/containers/*/rootfs/opt/nedge/var/log /var/lib/lxd/containers/*/rootfs/run/cloud-init /var/lib/lxd/containers/*/rootfs/usr/share/lxc/config /var/lib/lxd/containers/*/rootfs/var/lib/charm /var/lib/lxd/containers/*/rootfs/var/lib/libvirt/filesystems/plumgrid-data/log /var/lib/lxd/containers/*/rootfs/var/lib/libvirt/filesystems/plumgrid/var/log /var/lib/lxd/containers/*/rootfs/var/lib/cloud/seed /var/lib/lxd/containers/*/rootfs/var/log /var/lib/lxd/containers/*/rootfs/var/snap/simplestreams/common/sstream-mirror-glance.log /var/lib/lxd/containers/*/rootfs/var/crash /var/lib/lxd/containers/*/rootfs/var/snap/juju-db/common/logs/ /var/lib/lxd/containers/*/rootfs/var/lib/mysql/*-mysql-router /var/lib/lxd/containers/*/rootfs/tmp/juju-exec*/script.sh -mount -type f -size -5000000c -o -size 5000000c 2>/dev/null | tar -pcf /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar --files-from - 2>/dev/null'" failed
2021-08-09 18:06:11,442 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'find /etc/alternatives /etc/ceilometer /etc/ceph /etc/cinder /etc/cloud /etc/glance /etc/gnocchi /etc/keystone /etc/netplan /etc/network /etc/neutron /etc/nova /etc/quantum /etc/swift /etc/udev/rules.d /lib/udev/rules.d /opt/nedge/var/log /run/cloud-init /usr/share/lxc/config /var/lib/charm /var/lib/libvirt/filesystems/plumgrid-data/log /var/lib/libvirt/filesystems/plumgrid/var/log /var/lib/cloud/seed /var/log /var/snap/simplestreams/common/sstream-mirror-glance.log /var/crash /var/snap/juju-db/common/logs/ /var/lib/mysql/*-mysql-router /tmp/juju-exec*/script.sh /var/lib/lxd/containers/*/rootfs/etc/alternatives /var/lib/lxd/containers/*/rootfs/etc/ceilometer /var/lib/lxd/containers/*/rootfs/etc/ceph /var/lib/lxd/containers/*/rootfs/etc/cinder /var/lib/lxd/containers/*/rootfs/etc/cloud /var/lib/lxd/containers/*/rootfs/etc/glance /var/lib/lxd/containers/*/rootfs/etc/gnocchi /var/lib/lxd/containers/*/rootfs/etc/keystone /var/lib/lxd/containers/*/rootfs/etc/netplan /var/lib/lxd/containers/*/rootfs/etc/network /var/lib/lxd/containers/*/rootfs/etc/neutron /var/lib/lxd/containers/*/rootfs/etc/nova /var/lib/lxd/containers/*/rootfs/etc/quantum /var/lib/lxd/containers/*/rootfs/etc/swift /var/lib/lxd/containers/*/rootfs/etc/udev/rules.d /var/lib/lxd/containers/*/rootfs/lib/udev/rules.d /var/lib/lxd/containers/*/rootfs/opt/nedge/var/log /var/lib/lxd/containers/*/rootfs/run/cloud-init /var/lib/lxd/containers/*/rootfs/usr/share/lxc/config /var/lib/lxd/containers/*/rootfs/var/lib/charm /var/lib/lxd/containers/*/rootfs/var/lib/libvirt/filesystems/plumgrid-data/log /var/lib/lxd/containers/*/rootfs/var/lib/libvirt/filesystems/plumgrid/var/log /var/lib/lxd/containers/*/rootfs/var/lib/cloud/seed /var/lib/lxd/containers/*/rootfs/var/log /var/lib/lxd/containers/*/rootfs/var/snap/simplestreams/common/sstream-mirror-glance.log /var/lib/lxd/containers/*/rootfs/var/crash /var/lib/lxd/containers/*/rootfs/var/snap/juju-db/common/logs/ /var/lib/lxd/containers/*/rootfs/var/lib/mysql/*-mysql-router /var/lib/lxd/containers/*/rootfs/tmp/juju-exec*/script.sh -mount -type f -size -5000000c -o -size 5000000c 2>/dev/null | tar -pcf /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar --files-from - 2>/dev/null'" failed
2021-08-09 18:06:11,553 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'find /etc/alternatives /etc/ceilometer /etc/ceph /etc/cinder /etc/cloud /etc/glance /etc/gnocchi /etc/keystone /etc/netplan /etc/network /etc/neutron /etc/nova /etc/quantum /etc/swift /etc/udev/rules.d /lib/udev/rules.d /opt/nedge/var/log /run/cloud-init /usr/share/lxc/config /var/lib/charm /var/lib/libvirt/filesystems/plumgrid-data/log /var/lib/libvirt/filesystems/plumgrid/var/log /var/lib/cloud/seed /var/log /var/snap/simplestreams/common/sstream-mirror-glance.log /var/crash /var/snap/juju-db/common/logs/ /var/lib/mysql/*-mysql-router /tmp/juju-exec*/script.sh /var/lib/lxd/containers/*/rootfs/etc/alternatives /var/lib/lxd/containers/*/rootfs/etc/ceilometer /var/lib/lxd/containers/*/rootfs/etc/ceph /var/lib/lxd/containers/*/rootfs/etc/cinder /var/lib/lxd/containers/*/rootfs/etc/cloud /var/lib/lxd/containers/*/rootfs/etc/glance /var/lib/lxd/containers/*/rootfs/etc/gnocchi /var/lib/lxd/containers/*/rootfs/etc/keystone /var/lib/lxd/containers/*/rootfs/etc/netplan /var/lib/lxd/containers/*/rootfs/etc/network /var/lib/lxd/containers/*/rootfs/etc/neutron /var/lib/lxd/containers/*/rootfs/etc/nova /var/lib/lxd/containers/*/rootfs/etc/quantum /var/lib/lxd/containers/*/rootfs/etc/swift /var/lib/lxd/containers/*/rootfs/etc/udev/rules.d /var/lib/lxd/containers/*/rootfs/lib/udev/rules.d /var/lib/lxd/containers/*/rootfs/opt/nedge/var/log /var/lib/lxd/containers/*/rootfs/run/cloud-init /var/lib/lxd/containers/*/rootfs/usr/share/lxc/config /var/lib/lxd/containers/*/rootfs/var/lib/charm /var/lib/lxd/containers/*/rootfs/var/lib/libvirt/filesystems/plumgrid-data/log /var/lib/lxd/containers/*/rootfs/var/lib/libvirt/filesystems/plumgrid/var/log /var/lib/lxd/containers/*/rootfs/var/lib/cloud/seed /var/lib/lxd/containers/*/rootfs/var/log /var/lib/lxd/containers/*/rootfs/var/snap/simplestreams/common/sstream-mirror-glance.log /var/lib/lxd/containers/*/rootfs/var/crash /var/lib/lxd/containers/*/rootfs/var/snap/juju-db/common/logs/ /var/lib/lxd/containers/*/rootfs/var/lib/mysql/*-mysql-router /var/lib/lxd/containers/*/rootfs/tmp/juju-exec*/script.sh -mount -type f -size -5000000c -o -size 5000000c 2>/dev/null | tar -pcf /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar --files-from - 2>/dev/null'" failed
2021-08-09 18:06:14,029 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'tar --append -f /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar -C /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output . || true;tar --append -f /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar -C /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/ journalctl || true;tar --append -f /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar -C /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/addon_output . || true'" failed
2021-08-09 18:06:14,036 - Command "timeout 45s ssh -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected] 'tar --append -f /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar -C /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/cmd_output . || true;tar --append -f /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar -C /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/ journalctl || true;tar --append -f /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar -C /tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/addon_output . || true'" failed
2021-08-09 18:06:17,102 - Command "scp -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected]:/tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar a4cb1336-c7c6-4e1d-b238-ed7478c40f1f.tar" failed
2021-08-09 18:06:17,105 - Command "scp -o StrictHostKeyChecking=no -i ~/.local/share/juju/ssh/juju_id_rsa [email protected]:/tmp/ce11e8be-cb51-4b5b-8555-7b5ab370f4fa/juju-dump-ce11e8be-cb51-4b5b-8555-7b5ab370f4fa.tar f6b8e6df-e43e-43fc-9506-c4f5294c0e5f.tar" failed
2021-08-09 18:06:23,904 - juju-crashdump finished.

Important information is missing from the crashdump; the debug-layer and config addons did not add their information to the crashdump like they were supposed to. I'm unable to troubleshoot this because all I can see from the output of juju-crashdump itself is that the commands "failed".

Command output is clearly being sent to /dev/null here, here, and here. Additionally, in the failure case here, stdout has been captured but is never logged. It would be really helpful to be able to see the command output so I can troubleshoot further.