Test driver crashes if special characters are present #84

dz0ny · 2025-02-03T13:56:53Z

Describe the bug

A simple → character in the output crashes the driver.

vm # qemu-kvm: terminating on signal 15 from pid 1046 (/nix/store/y027d3bvlaizbri04c1bzh28hqd6lj01-python3-3.11.7/bin/python3.11)
(finished: cleanup, in 0.03 seconds)
Traceback (most recent call last):
additionally exposed symbols:
    vm,
    vlan1,
    start_all, test_script, machines, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, Machine
  File "/nix/store/ifkgyf5n4i9sb23ic96l1g1r33zf0fag-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/ifkgyf5n4i9sb23ic96l1g1r33zf0fag-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/__init__.py", line 126, in main
    driver.run_tests()
  File "/nix/store/ifkgyf5n4i9sb23ic96l1g1r33zf0fag-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/driver.py", line 159, in run_tests
    self.test_script()
  File "/nix/store/ifkgyf5n4i9sb23ic96l1g1r33zf0fag-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/driver.py", line 151, in test_script
    exec(self.tests, symbols, None)
  File "<string>", line 3, in <module>
  File "/nix/store/ifkgyf5n4i9sb23ic96l1g1r33zf0fag-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/machine.py", line 703, in succeed
    (status, out) = self.execute(command, timeout=timeout)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ifkgyf5n4i9sb23ic96l1g1r33zf0fag-nixos-test-driver-1.1/lib/python3.11/site-packages/test_driver/machine.py", line 642, in execute
    rc = int(self._next_newline_closed_block_from_shell().strip())
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: 'Created symlink /etc/systemd/system/sockets.target.wants/pareto-core.socket → /etc/systemd/system/pareto-core.socket.'

To Reproduce

Activate systemd unit under Ubuntu VM.

Expected behavior

Should be able to parse and pass any string via test driver.

System information

Operating System: Linux
Machine Type: 4x16
Architecture: amd64

Additional context

https://github.com/ParetoSecurity/pareto-core/actions/runs/13114941839/job/36586854338?pr=77#step:5:1636

The text was updated successfully, but these errors were encountered:

picnoir · 2025-02-06T09:12:59Z

Related to #5 to some extent.

When running the test-driver on other distros (e.g., via [https://github.com/numtide/nix-vm-test/](nix-vm-test)) executing a command can fail if the driver receives both stdout and stderr. The test-driver’s logic for reading output currently assumes it will only read base64 encoded stdout, so any extra stderr content leads to unexpected failures. This commit fixes the issue by redirecting the stderr to /dev/null, ensuring the driver only sees stdout. The tradeoff is losing stderr messages in the machine logs. Users can work around this by redirecting stderr to stdout in the command they are sending to the machine: ```python machine.execute("some_command 2>&1") ``` Related issues: - numtide/nix-vm-test#84 - numtide/nix-vm-test#5

When running tests on Ubuntu VMs, the `test-driver` can crash if the executed command calls `sudo` and writes to stderr. For example: ```python vm.execute("sudo bash -c \"echo 'Created foo → bar.\n' >&2 && echo 'foo' \"") ``` `sudo` spawns a new TTY on Ubuntu, merging both `stdout` and `stderr` into a single stream. However, the `test-driver` only expects base64-encoded `stdout` and fails when stderr data appears. - Normal command (note `1` and `2` are different consoles) ``` > bash -c 'echo $$; ls -l /proc/$$/fd' lr-x------ 1 root root 64 Feb 18 06:10 0 -> /dev/hvc0 l-wx------ 1 root root 64 Feb 18 06:10 1 -> pipe:[20942] l-wx------ 1 root root 64 Feb 18 06:10 2 -> /dev/ttyS0 lr-x------ 1 root root 64 Feb 18 06:10 3 -> /proc/664/fd ``` - Using `sudo` (note both `1` and `2` use the same console `/dev/pts/0`) ``` > sudo bash -c 'echo $$; ls -l /proc/$$/fd lrwx------ 1 root root 64 Feb 18 06:10 0 -> /dev/pts/0 l-wx------ 1 root root 64 Feb 18 06:10 1 -> pipe:[20943] lrwx------ 1 root root 64 Feb 18 06:10 2 -> /dev/pts/0 lr-x------ 1 root root 64 Feb 18 06:10 3 -> /proc/670/fd ``` Because `stderr` is now mixed with `stdout` on `/dev/pts/0`, the `test-driver`'s base64 decoding code crashes on unexpected data: ([encoding step](https://github.com/NixOS/nixpkgs/blob/8a24fbd0f3b4f47f01c897a95b1b65d6a5576c01/nixos/lib/test-driver/src/test_driver/machine.py#L578), [decoding step](https://github.com/NixOS/nixpkgs/blob/8a24fbd0f3b4f47f01c897a95b1b65d6a5576c01/nixos/lib/test-driver/src/test_driver/machine.py#L588)). A minimal reproducible example is provided in [am-on/nix-test-driver-ubuntu-bug](https://github.com/am-on/nix-test-driver-ubuntu-bug). Related issues: - numtide#84 - numtide#5

When running tests on Ubuntu VMs, the `test-driver` can crash if the executed command calls `sudo` and writes to `stderr`. For example: ```python vm.execute("sudo bash -c \"echo 'Created foo → bar.\n' >&2 && echo 'foo' \"") ``` `sudo` spawns a new TTY for `stderr` on Ubuntu that gets read by the `test-driver`'s output parsing. However, the `test-driver` only expects base64-encoded `stdout` and fails when `stderr` data is read. - Normal command (note `stderr` usees `/dev/ttyS0`) ``` > bash -c 'echo $$; ls -l /proc/$$/fd' lr-x------ 1 root root 64 Feb 18 06:10 0 -> /dev/hvc0 l-wx------ 1 root root 64 Feb 18 06:10 1 -> pipe:[20942] l-wx------ 1 root root 64 Feb 18 06:10 2 -> /dev/ttyS0 lr-x------ 1 root root 64 Feb 18 06:10 3 -> /proc/664/fd ``` - Using `sudo` (note `stderr` uses `/dev/pts/0`) ``` > sudo bash -c 'echo $$; ls -l /proc/$$/fd lrwx------ 1 root root 64 Feb 18 06:10 0 -> /dev/pts/0 l-wx------ 1 root root 64 Feb 18 06:10 1 -> pipe:[20943] lrwx------ 1 root root 64 Feb 18 06:10 2 -> /dev/pts/0 lr-x------ 1 root root 64 Feb 18 06:10 3 -> /proc/670/fd ``` Because `stderr` is now read along with the `stdout`, the `test-driver`'s base64 decoding code crashes on unexpected data: - [encoding step](https://github.com/NixOS/nixpkgs/blob/8a24fbd0f3b4f47f01c897a95b1b65d6a5576c01/nixos/lib/test-driver/src/test_driver/machine.py#L578), - [receiving the data](https://github.com/NixOS/nixpkgs/blob/8a24fbd0f3b4f47f01c897a95b1b65d6a5576c01/nixos/lib/test-driver/src/test_driver/machine.py#L515), - [decoding step](https://github.com/NixOS/nixpkgs/blob/8a24fbd0f3b4f47f01c897a95b1b65d6a5576c01/nixos/lib/test-driver/src/test_driver/machine.py#L588). A minimal reproducible example is provided in [am-on/nix-test-driver-ubuntu-bug](https://github.com/am-on/nix-test-driver-ubuntu-bug). Related issues: - numtide#84 - numtide#5

When running tests on Ubuntu VMs, the `test-driver` can crash if the executed command calls `sudo` and writes to `stderr`. For example: ```python vm.execute("sudo bash -c \"echo 'Created foo → bar.\n' >&2 && echo 'foo' \"") ``` `sudo` spawns a new TTY for `stderr` on Ubuntu that gets read by the `test-driver`'s output parsing. However, the `test-driver` only expects base64-encoded `stdout` and fails when `stderr` data is read. - Normal command (note `stderr` usees `/dev/ttyS0`) ``` > bash -c 'echo $$; ls -l /proc/$$/fd' lr-x------ 1 root root 64 Feb 18 06:10 0 -> /dev/hvc0 l-wx------ 1 root root 64 Feb 18 06:10 1 -> pipe:[20942] l-wx------ 1 root root 64 Feb 18 06:10 2 -> /dev/ttyS0 lr-x------ 1 root root 64 Feb 18 06:10 3 -> /proc/664/fd ``` - Using `sudo` (note `stderr` uses `/dev/pts/0`) ``` > sudo bash -c 'echo $$; ls -l /proc/$$/fd lrwx------ 1 root root 64 Feb 18 06:10 0 -> /dev/pts/0 l-wx------ 1 root root 64 Feb 18 06:10 1 -> pipe:[20943] lrwx------ 1 root root 64 Feb 18 06:10 2 -> /dev/pts/0 lr-x------ 1 root root 64 Feb 18 06:10 3 -> /proc/670/fd ``` Because `stderr` is now read along with the `stdout`, the `test-driver`'s base64 decoding code crashes on unexpected data: - [encoding step](https://github.com/NixOS/nixpkgs/blob/8a24fbd0f3b4f47f01c897a95b1b65d6a5576c01/nixos/lib/test-driver/src/test_driver/machine.py#L578), - [receiving the data](https://github.com/NixOS/nixpkgs/blob/8a24fbd0f3b4f47f01c897a95b1b65d6a5576c01/nixos/lib/test-driver/src/test_driver/machine.py#L515), - [decoding step](https://github.com/NixOS/nixpkgs/blob/8a24fbd0f3b4f47f01c897a95b1b65d6a5576c01/nixos/lib/test-driver/src/test_driver/machine.py#L588). A minimal reproducible example is provided in [am-on/nix-test-driver-ubuntu-bug](https://github.com/am-on/nix-test-driver-ubuntu-bug). Related issues: - #84 - #5

dz0ny added the bug Something isn't working label Feb 3, 2025

dz0ny mentioned this issue Feb 3, 2025

feat: add integration testing workflow and scripts for Debian, Fedora, and Ubuntu ParetoSecurity/agent#77

Merged

am-on mentioned this issue Feb 15, 2025

lib/test-driver: fix reading command output for other distros NixOS/nixpkgs#382260

Open

13 tasks

am-on mentioned this issue Feb 18, 2025

Fix test-driver output parsing crash on Ubuntu #87

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test driver crashes if special characters are present #84

Test driver crashes if special characters are present #84

dz0ny commented Feb 3, 2025

picnoir commented Feb 6, 2025

Test driver crashes if special characters are present #84

Test driver crashes if special characters are present #84

Comments

dz0ny commented Feb 3, 2025

picnoir commented Feb 6, 2025