api: share_memory fork mode + fork_count/mem_locked on Instance#220
Closed
sjmiller609 wants to merge 9 commits into
Closed
api: share_memory fork mode + fork_count/mem_locked on Instance#220sjmiller609 wants to merge 9 commits into
sjmiller609 wants to merge 9 commits into
Conversation
Introduces a templates package and Standby-instance promotion path so later PRs can wire fork-from-template into firecracker. The registry persists one JSON file per template under <data>/templates/<id>/, with fork refcount accounting and Delete-blocked-when-in-use. The instance manager owns the registry because template lifecycle is coupled to instance lifecycle: promotion guards on Standby+HasSnapshot, deletion clears IsTemplate on the source instance, and forks (PR 3) will increment the refcount. Hypervisor wire-up is deferred — this PR ships the registry, the StoredMetadata flags, and the manager helpers (templateGuard, templateForFork, validateForkResolvedFromTemplate) so PR 3 can plug in.
When ForkInstanceRequest.TemplateID is set, the fork resolves the source
instance from the templates registry, skips the per-fork mem-file copy,
and installs a symlink to the template's snapshot mem-file instead.
firecracker mmaps the symlinked mem-file MAP_PRIVATE during restore, so
many concurrent forks COW from the same backing file rather than each
holding a private copy.
Also wires:
- templateGuard on Start/Restore so a template parent is never resumed
while live forks share its mem-file.
- Refcount lifecycle: bump on fork creation, decrement on fork delete.
- Delete safety: deleting a template instance refuses while ForkCount>0
via templates.ErrInUse.
- forkvm.CopyOptions.SkipRelPaths so callers can opt out of specific
files in the source dir without breaking the existing copy semantics.
Adds lib/uffd, a userfaultfd page server that backs many concurrent fan-out forks against a single read-only template mem-file instead of letting each fork mmap it privately. Firecracker connects to a per-fork UDS, hands us its userfaultfd via SCM_RIGHTS along with a JSON mappings handshake, and the server then services UFFD_EVENT_PAGEFAULT events with UFFDIO_COPY reads from the template. The Linux hot path lives behind a build tag; non-Linux builds return ErrUnsupported so callers can fall back to MAP_PRIVATE. Cross-platform tests cover the handshake parser and the server lifecycle. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a hot-page recorder + prefetch primitive on top of the userfaultfd page server. During a template's first warmup fork the server can record every served page (Config.RecordHotPages); the resulting HotPageList is stable-sorted, deduplicated, and saved to disk in a small binary format alongside the template. Later forks call Server.Prefetch(forkID, list) to issue UFFDIO_COPY for every recorded page against their userfaultfd before the guest unpauses, eliminating the fault round-trips on those addresses. The prefetcher is installed by the platform-specific listener once the fork's uffd has been received and registered, so callers can race Prefetch and the fault loop safely. EEXIST/EAGAIN are tolerated the same way the fault handler does to absorb first-touch races with vCPUs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wires firecracker's snapshot restore through a userfaultfd-backed page server when the fork descends from a template. Each template gets one uffd.Server lazily started on the first fork and torn down once the last fork is released, so non-template forks (the symlink-only path) are unaffected. Adds an E2E test that boots a firecracker source, standbys it, promotes it to a template, forks against the template, and asserts: fork reaches Running, mem-file is a symlink to the template's, refcount bumps to 1, the per-template uffd tracker registers the fork, and on fork delete the refcount drops and the tracker detaches. Also adds unit coverage for the uffd tracker lifecycle (lazy start, multi-fork share, last-release teardown, closeAll, empty-acquire rejection).
Unix domain socket paths cap at 108 bytes (sun_path). Putting the per- fork socket under <DataDir>/templates/<25-char-cuid>/uffd/<25-char- cuid>.uffd blew that limit on CI runners where t.TempDir() returns long prefixes, surfacing as "bind: invalid argument" in TestFirecrackerForkFromTemplate. Sockets are ephemeral, so anchoring them at /tmp/h-uffd/<templateID>/ keeps the path well under the limit regardless of how deep DataDir is. The tracker now also rm -rfs the per-template socket dir on the last release so we don't leak stale entries.
The previous fork-mem-file symlink looped through withSnapshotSourceDirAlias during firecracker restore: fork/.../memory -> source/.../memory, but the alias dance temporarily symlinks source dir -> fork dir, so resolution ping-pongs back to fork/.../memory and trips ELOOP. Switching to a hardlink makes the fork's mem-file resolve by inode so the temporary directory alias no longer affects it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Firecracker enables snapshot base reuse, which renames the post-restore snapshot dir from snapshot-latest to snapshot-base. The hardlink survives the rename (same inode), so the test just needs the right path.
ForkInstanceRequest gains share_memory: when true, the source instance is auto-promoted (idempotently) into a fan-out parent and the fork attaches to its mem-file via the existing template path instead of copying. While any share_memory fork is alive, the source is mem-locked: start/restore/ delete return 409 (ErrInvalidState) until the forks are deleted. Instance gains read-only fork_count and mem_locked fields so clients can introspect lock state on the existing GetInstance response without needing a separate templates resource. The internal templates registry stays in place but is not exposed in OpenAPI. templateGuard now returns ErrInvalidState rather than ErrNotSupported so the lock surfaces as a transient 409, not a 501 capability gap.
✱ Stainless preview builds for hypemanThis PR will update the Edit this comment to update it. It will appear in the SDK's changelogs. ✅ hypeman-typescript studio · code · diff
✅ hypeman-go studio · code · diff
✅ hypeman-openapi studio · code · diff
This comment is auto-generated by GitHub Actions and is automatically kept up to date as you push. |
4 tasks
sjmiller609
force-pushed
the
hypeship/uffd-firecracker-wiring
branch
from
030b69e to
0c36cb3
Compare
Collaborator
Author
|
Closing as obsolete — superseded by the state-machine template design in #229. The features this PR added are now provided by the state machine intrinsically:
If exposing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exposes the uffd/template-fork machinery through the existing
POST /instances/{id}/forkendpoint, without introducing a new/templatesresource.ForkInstanceRequest.share_memory: bool— when true, the source instance is auto-promoted (idempotently) into a fan-out parent and the fork attaches to its mem-file instead of copying. Subsequent share_memory forks against the same source reuse the same registry entry.Instance.fork_count: intandInstance.mem_locked: bool— read-only fields so clients can introspect lock state on the existingGetInstanceresponse.mem_locked = true, start/restore/delete on the source return 409 (ErrInvalidState) until all share_memory forks are deleted.templateGuardnow returnsErrInvalidStaterather thanErrNotSupportedso the lock surfaces as a transient 409, not a 501 capability gap.share_memoryis incompatible withtemplate_idand withfrom_running=true(the latter would re-restore the source after locking).The
lib/templatespackage andTemplateIDfield stay as internal machinery — no public templates resource.share_memory=falseforks are unchanged and may still be created against a locked source.Test plan
go vet ./...cleanlib/instances/share_memory_test.go:TestValidateForkRequest_ShareMemoryConflicts— rejects share_memory + template_id and share_memory + from_runningTestEnsureShareMemoryTemplate_AutoPromoteAndReuse— first call promotes, second reusesTestEnsureShareMemoryTemplate_RejectsNonStandby— non-Standby source returns ErrInvalidStateTestTemplateGuard_ReturnsInvalidStateNotUnsupported— guard now maps to 409TestHydrateForkLockState— fork_count/mem_locked reflect refcount changes