Skip to content

fork: hardlink snapshot mem-file into snapshot forks#221

Draft
sjmiller609 wants to merge 8 commits into
hypeship/fork-shared-memfilefrom
hypeship/snapshot-fork-share-memfile
Draft

fork: hardlink snapshot mem-file into snapshot forks#221
sjmiller609 wants to merge 8 commits into
hypeship/fork-shared-memfilefrom
hypeship/snapshot-fork-share-memfile

Conversation

@sjmiller609
Copy link
Copy Markdown
Collaborator

@sjmiller609 sjmiller609 commented May 8, 2026

Summary

  • Snapshot forks now hardlink the source mem-file into the fork's data dir instead of sparse-copying it.
  • The mem-file is skipped from the directory walk via CopyOptions.SkipRelPaths, then os.Link'd into place after the copy returns. Dodges the multi-GB sparse copy and the directory-walk overhead in one step.
  • Falls back to the normal copy when no raw mem-file is present (e.g. compressed-only snapshot whose decompression failed, or a snapshot kind that doesn't carry guest memory).

Why this is safe

  • Snapshot mem-files are immutable.
  • The hypervisor mmaps them MAP_PRIVATE on restore, so fork writes never reach the underlying file — all forks of a snapshot can share the same inode.
  • Hardlinks survive snapshot deletion via inode refcount, so deleting a snapshot never strands a running fork.
  • Same-FS guarantee holds: snapshot dir and fork dir are both under hypeman's data dir.

Stack

Test plan

  • go test ./lib/instances/ -run TestForkSnapshotHardlinksRawMemoryFile
  • go test ./lib/instances/ -run TestForkSnapshotFromCompressedSourceCopiesRawMemory still passes (compressed source still gets a real raw file in the fork)
  • go test ./lib/forkvm/
  • Manual: fork a Standby snapshot, confirm stat -c %i matches between the source mem-file and the fork's mem-file

sjmiller609 and others added 6 commits May 13, 2026 00:43
@sjmiller609 @claude
Wire new promote/demote template endpoints into Stainless config
294c9ff 
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@sjmiller609
fork: share template mem-file via hardlink for firecracker fan-out
a45d471 
When a Firecracker fork descends from a Template source, skip copying the
snapshot mem-file and hardlink it to the source's instead. Firecracker
mmaps the mem-file MAP_PRIVATE on restore, so all forks COW from the same
backing inode — no per-fork copy required.

Hardlink rather than symlink: firecracker's restore path temporarily
aliases the source data dir to the fork data dir while loading the
snapshot (withSnapshotSourceDirAlias). A symlink whose target traverses
the source dir would resolve back into the fork dir during that window
and trip ELOOP; a hardlink resolves by inode so the alias has no effect
on it. Hardlinks require both paths on the same filesystem, which holds
for our standard data-dir layout.

Gated to Firecracker only because other hypervisors (cloud-hypervisor,
qemu, vz) don't share MAP_PRIVATE semantics on their snapshot layouts.
Restricted to Template sources because they are explicitly promoted as
fork-only and can never be restored — sharing the mem-file with a
non-Template source would let a later RestoreInstance mutate the file
out from under live forks.

Stacked on hypeship/template-as-state so the Template state both gates
"this snapshot is safe to fan out from" and lets fork counts be derived
at read time.
@sjmiller609
fork: hardlink snapshot mem-file into snapshot forks
7992660 
Snapshot forks copy the source guest dir into the fork instance dir;
the dominant cost is the multi-GB mem-file. Hardlink it instead and
skip the file from the directory walk via CopyOptions.SkipRelPaths
(introduced for template forks).

This is safe because:
- snapshot mem-files are immutable
- the hypervisor mmaps them MAP_PRIVATE on restore, so fork writes
  never reach the underlying file
- hardlinks survive snapshot deletion via inode refcount, so a deleted
  snapshot never strands a running fork

Falls back to the regular copy walk when no raw mem-file is present.
@sjmiller609
Model template as an instance state instead of a separate registry
8c23e9f 
Adds StateTemplate to the instance state machine. A Standby instance is
auto-promoted to Template the first time it's forked from a snapshot,
and ForkCount is bumped on each subsequent fork. Templates can't wake
while ForkCount > 0; un-promote (Template -> Standby) and delete
(Template -> Stopped) are both refused until forks drain.

Fork bookkeeping lives on StoredMetadata (IsTemplate, ForkCount,
ForkOfTemplate, plus a reserved HotPagesPath for the prefetch path).
Deleting a fork decrements the parent template's ForkCount under the
parent's lock; deletion of the fork's own data has already happened, so
worst case is refcount drift that a future reconciliation pass fixes.

The running-fork flow keeps skipping promotion: it restores the source
back to Running afterward, and a template can't wake.
@sjmiller609
Compute template fork count at read time instead of tracking a refcount
ed5802a 
Drops the persisted ForkCount field from StoredMetadata and the
decrement bookkeeping in DeleteInstance. Live forks of a template are
now counted by scanning metadata for ForkOfTemplate matches via a new
countTemplateForks helper. The fork-of-template field itself remains
the single source of truth, so there's no drift to reconcile.

Template promotion on fork only flips IsTemplate when not already set;
deletion of a template still refuses when forks exist, but the count
is computed from disk rather than read from a denormalized field.
@sjmiller609 @claude
Make template promotion an explicit lifecycle call
deb49d8 
Previously ForkInstance auto-promoted a Standby source to Template the
first time it was forked from a snapshot, and RestoreInstance auto-demoted
a Template before waking it. That implicit lifecycle blurred the rules: a
Standby and a "Standby that has been forked once" behaved differently,
and callers had to know that restoring a Template was a two-step
operation under the hood.

Replace it with explicit PromoteToTemplate / DemoteTemplate manager
methods (and matching POST /instances/{id}/promote-template and
/demote-template endpoints). Promotion is now Standby -> Template only;
demotion is Template -> Standby only and refuses while live forks
reference the template. ForkInstance only records the parent linkage if
the source is already a Template, and RestoreInstance no longer
auto-demotes — callers must demote first.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@sjmiller609 sjmiller609 force-pushed the hypeship/fork-shared-memfile branch from d46be7a to 7b799f7 Compare May 13, 2026 18:13
@sjmiller609 sjmiller609 force-pushed the hypeship/snapshot-fork-share-memfile branch from 6d875e1 to 06abd04 Compare May 13, 2026 18:25
sjmiller609 and others added 2 commits May 13, 2026 15:30
@sjmiller609
Merge branch 'main' into hypeship/template-as-state
0d0ab12
@sjmiller609 @claude
Fail countTemplateForks on metadata load error instead of skipping
2738278 
Silently continuing past an unreadable metadata file could undercount
forks of a template, allowing DemoteTemplate or DeleteInstance to free
a template whose pages are still mapped by a live fork.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@sjmiller609 sjmiller609 force-pushed the hypeship/fork-shared-memfile branch from 7b799f7 to 355ad7f Compare May 13, 2026 20:15
@sjmiller609 sjmiller609 force-pushed the hypeship/snapshot-fork-share-memfile branch from 06abd04 to 13f3003 Compare May 13, 2026 20:16
@sjmiller609 sjmiller609 force-pushed the hypeship/fork-shared-memfile branch from 355ad7f to a45d471 Compare May 13, 2026 20:39
@sjmiller609 sjmiller609 force-pushed the hypeship/snapshot-fork-share-memfile branch from 13f3003 to 7992660 Compare May 13, 2026 20:40
@sjmiller609 sjmiller609 force-pushed the hypeship/fork-shared-memfile branch from a45d471 to 8b0000c Compare May 14, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sjmiller609