sandbox: serialize the lifecycle transitions of a state directory by ppenna · Pull Request #476 · microsoft/nvx

Pedro Henrique Penna (ppenna) · 2026-10-10T15:10:26Z

Summary

provision, start, stop, and deprovision now hold an exclusive lock on lifecycle.lock in the state directory for their whole duration: fcntl.flock() on Linux, and msvcrt.locking() on Windows, on a handle that shares deletion. Overlapping transitions of one sandbox therefore take effect one at a time. Of two overlapping starts, exactly one launches OpenVMM, and the other then fails with sandbox is already running or has stale runtime state. exec does not take the lock, so a long workload cannot delay stop, and OpenVMM does not inherit it.
Each transition polls for the lock for at most --timeout seconds and then fails with another lifecycle transition of the sandbox is still in progress. No retry sleeps past that deadline or starts after it. The CLI now also passes --timeout to provision and deprovision, and rejects a value that is not finite and greater than 0.
Lock file lifetime:
- A provisioned state directory keeps lifecycle.lock, and a release removes it from any other directory, so a command aimed at a directory without a sandbox leaves nothing behind.
- A caller that acquires a lock file that its holder removed in the meantime retries only if the sandbox is still provisioned. Otherwise it fails with sandbox is not provisioned, so it never recreates the lock file in a directory that deprovision is removing.
- deprovision writes a complete tombstone into the lock file before removing it. Before releasing the lock, it clears the tombstone again if it has removed the directory, or failed to remove it or the lock file. The tombstone therefore remains only while lingering lock files keep the directory. A provision that finds it waits while only lingering lock files keep that directory, and then retries. It stops waiting early if the path names another directory or the directory holds anything else. Without the tombstone, a waiting provision retries at once, so it reports sandbox is already provisioned when another provision recreated the directory first.
- provision also recreates a state directory that deprovision removed after provision prepared it, and on Windows it waits out a lock file pending deletion.
- Because deprovision writes to the lock file, it must be a regular file that no other name links. Windows opens it without following a reparse point.
Runtime files that no record vouches for:
- start and deprovision both refuse a control capability or control socket that remains without a runtime record, as after a killed start whose OpenVMM may still run. deprovision's diagnostic says how to recover.
- start creates its capability exclusively with mode 0600, so a failed start's cleanup only ever removes its own files.
- stop and a failed start remove the runtime record last, so an interrupted cleanup leaves a record that shows OpenVMM gone.
- A failed start removes its runtime files only once it has ended its OpenVMM process. If it cannot end it, the files stay, so the sandbox does not look stopped while that VM may still run.
deprovision removes the lock file and then the directory while it holds the lock. On NFS, and on Windows file systems without POSIX deletion such as ReFS, a removed lock file stays in the directory until every waiter has closed it. In that case deprovision releases the lock and retries the removal for up to --timeout seconds, as long as only lingering lock files remain. It reports any other failure to remove the directory at once. A provision that waits for that removal completes it itself once nothing keeps the directory, in case deprovision has given up, and deprovision treats a path that names no directory, or another one, as removed. On POSIX it holds the old directory open, so the new one cannot reuse its inode number.
Tests:
- A regression test races two starts through a stand-in OpenVMM that serves the managed control protocol on a Unix socket or a named pipe. It then checks that the next transition can take the lock while the VM runs, and stops and deprovisions the sandbox.
- New tests cover:
  - leftover runtime files for start and deprovision, and the cleanup order;
  - lock exclusion, bounded waits, deadlines that reject retries once they pass, and timeout validation;
  - waiters during deprovision, waiters that keep the removed lock file, and transitions that start during deprovision;
  - deprovision retrying the directory removal only while lock files linger, and reporting any other failure at once;
  - a waiting provision that retries at once when another provision recreated the directory first or deprovision failed to remove it or the lock file, and whose wait for a lingering lock file ends once the directory is replaced or provisioned again;
  - tombstone writes and reads that transfer only part of the tombstone at a time;
  - a waiting provision that completes a removal that deprovision gave up, and deprovision accepting a removal that a waiting provision completed;
  - a failed start that cannot end its OpenVMM process, which keeps the runtime files;
  - the provision handoff, which fails without the tombstone, and a provision that starts in the residual window;
  - a provision whose directory deprovision removed before it opened the lock file, a lock file pending deletion, and a hard or symbolic link in place of the lock file;
  - which directories keep the lock file.
- The sandbox-lifecycle acceptance now restarts the sandbox with two overlapping public starts, expects lifecycle.lock in its state listings, and removes unvouched runtime files during cleanup once no fixture OpenVMM remains. Its simulation gains an unserialized-starts fault that the acceptance must detect.
doc/run.md and doc/usage.md document all of the above.

Notes

State layout: a provisioned state directory now also holds lifecycle.lock. A sandbox that an earlier NVX version provisioned gains the file at its next transition, and deprovision removes it with NVX's other files.
Crash leftovers: start used to overwrite a capability or control socket that had no runtime record, and deprovision used to remove it. Both now refuse it, because its OpenVMM may still run on the same scratch image.
Failed starts: a failed start used to remove its runtime files even when it could not end its OpenVMM process. They now stay until OpenVMM is gone, and start and deprovision refuse the sandbox meanwhile.
Residual window: a provision that starts between deprovision's removal of the lock file and its removal of the emptied directory can still provision the directory first. On file systems with POSIX deletion, that gap is two consecutive syscalls. deprovision then fails with sandbox was provisioned again while deprovision removed it, and nothing is orphaned.
- No marker inside the directory can close that gap, because rmdir needs the directory empty.
- A rename handoff isn't possible either: Windows refuses to rename a directory while a file inside it is open, even with FILE_SHARE_DELETE.
- Closing it would need a lock outside --state-dir, which is documented in doc/run.md.
Rebase: rebased onto dev at 942e298. The only conflict, in an earlier rebase, was the import block of scripts/test_microvm_tests.py, which test-microvm: draw host-loopback port pairs independently #477 also changed. The new dev commits change no guest, kernel, or OpenVMM input, so the bare-metal artifacts still match.
--timeout stays a per-step bound. start already bounds the connect, the ping, and the failed-start grace period by it separately, and the lock wait is one more bounded step.
CI on the first revision: the only failure was NVX microVM tests / Linux / KVM, in the filesystem-snapshot scenario, which this change does not touch. The capture log ended mid-marker at NVX-FI, as when the snapshot froze the guest before its console line reached the host. The scenario passed in 10 of 10 bare-metal KVM runs both with this change and on dev, and the next CI run, for 12745d8, was fully green.

Validation

All results are for the current head, ed60feb, rebased onto dev at 942e298.

Local (Windows 11, Python 3.14.7):
- ruff check, ruff format --check, compileall, and pyright for Linux and Windows pass.
- The full validate-nvx suite passes (918 tests, 108 skipped).
- The new race test fails in 3 of 3 runs against dev's sandbox_lifecycle.py, with two OpenVMM launches, and passes with the fix.
- The lock tests and the acceptance simulation pass with six copies running concurrently to load the machine.
Remote bare metal: each host ran a detached worktree of this head. No dev release covers the current guest inputs yet (the newest predates filesystem: map host paths from any volume, each with its own access #459), so each host used the release package that CI built for e054b2a (KVM and MSHV, run 38057714971) or 009f05b (WHP, run 38024829470). The packages were installed with checksum verification, and their guest, kernel, and OpenVMM inputs equal this commit's.
- On prometheus32 (Linux/KVM, Python 3.14), prometheus30 (Linux/MSHV, Azure Linux 3.0, Python 3.12), and prometheus28 (Windows/WHP, Windows Server build 26100, Python 3.14):
  - The validate-nvx unit suites pass (918 tests).
  - test-microvm --scenario managed-exec-config --scenario sandbox-lifecycle passes in 3 of 3 runs on each backend, and each overlapping restart returned statuses 0 and 1.
  - Racing public starts on a freshly provisioned sandbox, followed by exec, stop, and deprovision, are clean in 10 of 10 iterations with two starts and in 5 of 5 with three on each backend. One start succeeds, the others fail with the diagnostic, one OpenVMM runs, and the runtime record names it.
- The same harness on the pre-fix code (218ad09fd), with the same artifacts, went wrong in 7 of 10 iterations on KVM, 8 of 10 on MSHV, and 10 of 10 on WHP:
  - On Linux, the second OpenVMM failed with failed to bind microVM control console listener .../control.sock: Address already in use (os error 98). Its start's cleanup left only config.json, openvmm.log, and the second OpenVMM's outcome.json. The first VM kept running while exec and stop reported sandbox is not running. This is the Linux variant that the issue had only traced.
  - On WHP, both starts succeeded and two VMs ran. In 9 iterations, stop ended only the recorded VM. In the tenth, exec and stop failed with managed control endpoint closed, and both VMs kept running.
- Every host run checked for other OpenVMM processes first. The harness ended only OpenVMM processes whose arguments named its own fixtures, and no OpenVMM process remained on any host.

Fixes #464

Copilot

🟡 Changes recommended

Lock waits are unbounded, and the lingering-file fallback remains racy with existing waiters.

2 open findings

What changed in this PR

Serializes sandbox lifecycle transitions to prevent concurrent starts from orphaning OpenVMM processes.

Changes:

Adds cross-platform lifecycle locking and exclusive capability creation.
Handles stale runtime artifacts and deprovision cleanup.
Expands unit, acceptance, simulation, and documentation coverage.

File	Description
`scripts/nvx_tools/sandbox_lifecycle.py`	Implements lifecycle locking and safer runtime cleanup.
`scripts/test_nvx_tools.py`	Adds lock, race, and stale-file tests.
`scripts/test_microvm_tests.py`	Simulates serialized and faulty concurrent starts.
`scripts/nvx_tools/sandbox_lifecycle_tests.py`	Adds overlapping-start acceptance coverage.
`doc/run.md`	Documents locking and runtime-file behavior.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

Concurrent reprovisioning can bypass teardown serialization, and non-finite timeouts can cause an unbounded lock wait.

2 open findings

2 resolved since last review

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Deprovision can still orphan an OpenVMM process when a killed start leaves runtime files without runtime.json.

3 open findings

2 resolved since last review

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Lock-file links can cause deprovisioning to overwrite data outside the sandbox state directory.

1 open finding

Lock file links can overwrite arbitrary target files · New

3 resolved since last review

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Provisioning can still race with directory removal before acquiring the lifecycle lock.

1 open finding

Retry lifecycle lock acquisition after state directory removal · New

1 resolved since last review

Lock file links can overwrite arbitrary target files

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Lock and tombstone polling can still complete after the configured timeout deadline.

1 open finding

Enforce deadline before tombstone handoff retries · New

1 resolved since last review

Retry lifecycle lock acquisition after state directory removal

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

The directory-removal fallback unnecessarily waits for the full timeout on unrelated filesystem errors.

1 open finding

Avoid retrying rmdir permission errors when the directory is empty · New

1 resolved since last review

Enforce deadline before tombstone handoff retries

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Directory-generation and partial-tombstone races can still break the intended deprovision handoff.

2 open findings

1 resolved since last review

Avoid retrying rmdir permission errors when the directory is empty

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

An unlink failure can leave a tombstoned lock that makes an overlapping provision wait unnecessarily until timeout.

1 open finding

Clear tombstone when unlink fails · New

2 resolved since last review

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Failed-start cleanup can orphan a live VM, and failed directory-removal fallback can strand tombstoned waiters.

2 open findings

1 resolved since last review

Clear tombstone when unlink fails

🧠 Review effort: Balanced

Fixes #464. sandbox start rejected an existing runtime record with a plain exists() check, but wrote the record only after it had launched and identified OpenVMM. Two overlapping starts of one state directory, such as an orchestrator's retry of a slow start, both passed the check and both launched OpenVMM, which on Linux also opened the same scratch image. Each start overwrote control.capability and runtime.json, so the two files could name different VMs, and a start that failed for any reason removed runtime.json, control.capability, and control.sock whether or not they were its own. The sandbox then ran a VM that no sandbox command tracked or could stop, or, when the record and the capability named different VMs, two VMs that neither stop nor deprovision could end. Reproduction ------------ The issue reproduced the race on Windows with a stand-in OpenVMM and traced the Linux variant. With real OpenVMM on bare-metal hosts and the release packages that CI built for e054b2a and, for WHP, 009f05b, whose guest, kernel, and OpenVMM inputs equal this commit's, two public starts that raced on a freshly provisioned sandbox went wrong in 7 of 10 iterations on Linux/KVM, 8 of 10 on Linux/MSHV, and 10 of 10 on Windows/WHP: - On Linux, where every start listens on control.sock, the second OpenVMM failed with "failed to bind microVM control console listener .../control.sock: Address already in use", and its start's cleanup removed the first start's runtime record, capability, and socket. The first VM kept running while exec and stop reported that the sandbox was not running. - On WHP, both starts succeeded and two OpenVMM processes ran. In 9 iterations, stop ended the one that the runtime record named and left the other running. In the tenth, exec and stop failed because the managed control endpoint closed, and both VMs kept running. The new unit test races two starts through a stand-in OpenVMM that serves the managed control protocol on a Unix socket or a named pipe. Against the previous sandbox_lifecycle.py, it launched two OpenVMM processes in each of 3 runs on a Windows workstation. Fix --- - provision, start, stop, and deprovision hold an exclusive lock on lifecycle.lock in the state directory from their first look at the state until they return: flock() on Linux, and msvcrt.locking() on Windows, on a handle that shares deletion. Overlapping transitions of one sandbox therefore take effect one at a time, and of two overlapping starts, exactly one launches OpenVMM while the other then fails because the sandbox is already running. Each transition polls for the lock for at most --timeout seconds and then fails, so a transition that hangs cannot hold up the others indefinitely. No retry sleeps past that deadline or starts after it. The CLI now also passes --timeout to provision and deprovision, and rejects a value that is not finite and greater than 0. exec does not take the lock, so a long workload cannot delay stop, and OpenVMM does not inherit it. - A caller that acquires a lock file that its holder removed meanwhile holds no lock. It retries with the file that the state directory holds only if the sandbox is still provisioned or it provisions the sandbox itself, and otherwise fails because the sandbox is not provisioned, so it creates no lock file in a directory that deprovision is removing. Deprovision writes the whole of a mark into the lock file before it removes it, and clears the mark again before it releases the lock if it has removed the directory, or failed to remove it or the lock file. The mark therefore remains only while lingering lock files keep the directory. A provision that finds it waits until deprovision has removed that directory, or the path names another directory or a provisioned sandbox, and then retries, as one that finds no mark does at once. A provision that finds the state directory itself gone, or, on Windows, its lock file pending deletion, provisions it again. Since deprovision writes to the lock file, it must be a regular file that no other name links, and Windows opens it without following a reparse point. A provisioned state directory keeps the lock file, and a release removes it from any other directory, so a command aimed at a directory that holds no sandbox leaves nothing behind. - start also refuses a control capability or control socket that remains without a runtime record, as after a killed start whose OpenVMM may still run, and creates its capability exclusively with mode 0600. Every runtime file that a failed start removes is then its own. deprovision refuses such files too, with a diagnostic that says how to recover, rather than remove the files of a VM that may still run. A failed start and stop remove the runtime record last, so that an interrupted cleanup leaves a record that shows OpenVMM gone. A failed start removes its runtime files only once it has ended its OpenVMM process, so that one that it could not end keeps the sandbox from looking stopped. - deprovision removes the lock file and the directory while it holds the lock, so a waiting transition cannot create a new lock file in the directory. NFS, and Windows file systems without POSIX deletion such as ReFS, keep a removed file in its directory until every waiter has closed it, and a transition that starts meanwhile holds a new lock file there until it finds the sandbox gone. While only such lock files keep the directory, deprovision releases the lock and retries its removal for up to --timeout seconds, and it reports any other failure to remove the directory at once. A provision that waits for the removal stops waiting once the directory holds anything else, and once nothing keeps the directory, removes it itself, in case deprovision has given up. Deprovision in turn takes a path that names no directory, or another one, as the removal done. On POSIX systems, it holds the old directory open meanwhile, so that a new one cannot reuse its inode number. Only a provision that starts after deprovision removed the lock file, and before it removed the directory, can provision the directory again first, and deprovision then fails because the sandbox was provisioned again. Tests ----- - Two starts that race through the serving stand-in launch one OpenVMM, and the other fails with "sandbox is already running or has stale runtime state". The runtime record names the launched OpenVMM, another transition takes the lock while that OpenVMM runs, and stop and deprovision then end it and remove the sandbox. - start and deprovision refuse a capability or control socket without a runtime record and leave the file intact, start launches no OpenVMM, and creating a capability fails when one exists. stop and a failed start remove the runtime record after the other runtime files, and a failed start that cannot end its OpenVMM process keeps them, so that start and deprovision then refuse the sandbox. - Lock holders exclude each other, and start, stop, and deprovision queued behind a held lock fail after their timeout and change nothing. A stop that waits while deprovision removes the sandbox fails because the sandbox is not provisioned, without opening the lock file again. deprovision outlasts a lingering lock file, waiters that keep the removed one, and a transition that starts meanwhile, and only a provisioned state directory keeps the lock file. It retries removing the emptied directory only while the directory is not empty, at once if the last lingering lock file is gone by then, and reports a denied removal at once rather than after its timeout. A provision that waits while deprovision removes the sandbox provisions the directory again afterwards, and both succeed, as does a provision whose directory deprovision removed before it opened the lock file, while a provision that starts in between makes deprovision report that the sandbox was provisioned again. A provision that waits while deprovision removes the directory retries at once when another provision recreates the directory first, and then reports that the sandbox is already provisioned, or when deprovision fails to remove the directory or the lock file, and then provisions the directory. Its wait for a lingering lock file ends once the path names another directory, or the directory holds anything but lock files. Once nothing keeps the directory, it removes the directory itself, and if that removal is denied, it provisions the directory in place. It completes a removal that deprovision gave up, and deprovision accepts a removal that it completed and followed with a new sandbox. The mark survives writes and reads that transfer only part of it at a time. The waits sleep at most until their deadline and retry nothing after it, so neither a directory that deprovision removes nor a lock file that another transition replaces only once the deadline has passed counts. The CLI and the lifecycle functions reject a timeout that is not finite and greater than 0, and a hard or symbolic link in place of the lock file is refused and its target left intact. - The sandbox-lifecycle acceptance restarts the sandbox with two overlapping public starts and requires one success, the other's diagnostic, and a single OpenVMM process. Its expected state directories include lifecycle.lock, its cleanup removes runtime files that no record names once no fixture OpenVMM remains, and its simulation detects starts that are not serialized. Validation ---------- On a Windows workstation, ruff, ruff format, and pyright for Linux and Windows pass, and so do the validate-nvx unit suites. On each of the three bare-metal hosts, with the same artifacts: - the validate-nvx unit suites pass; - managed-exec-config and sandbox-lifecycle pass in 3 of 3 runs, and each overlapping restart launched one OpenVMM; and - the racing starts are clean in 10 of 10 iterations with two starts and in 5 of 5 with three: one start succeeds, the others fail with the diagnostic, one OpenVMM runs and the runtime record names it, and exec and stop then reach it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Cross-platform locking, deletion, and handoff behavior is highly race-sensitive and warrants final human review.

0 open findings

2 resolved since last review

🧠 Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings October 10, 2026 15:10

Copilot started reviewing on behalf of Pedro Henrique Penna (ppenna) October 10, 2026 15:11 View session