Amortize arena free-run ownership checks

Author: Frityet

notes/packed-arena-range-preflight-2026-07-13.md

@@ -0,0 +1,132 @@
+# Packed arena range preflight
+
+## Motivation
+
+The pushed typed-sweep amortization checkpoint `54714887` reduced repeated
+quarantine visits and global certificates, but a fresh loop-only hardware
+profile still measured 405.96 ns per `closures_upval` operation. The remaining
+free-run reconstruction path was the clearest isolated target:
+
+```text
+arena_clear_extent_range   51.42 ns/op
+arena_set_free_run         32.02 ns/op
+lj_arena_scan_free_runs    13.95 ns/op
+total                      97.39 ns/op
+```
+
+The first two functions repeatedly loaded root, recovery, destructor, READY,
+and lifetime state for every cell in a large free span. The same metadata is
+already stored in packed atomic planes, so this was observation overhead rather
+than useful semantic work.
+
+## Exact packed proof
+
+`arena_range_ownership_preflight()` now walks only the bitmap words intersected
+by the requested range. Existing exact converters map every nonzero root,
+recovery, and destructor state and every non-FREE lifetime nibble back to one
+bit per arena cell. Each word is intersected with a partial-range mask, so cells
+before or after the requested span cannot veto it.
+
+This is only an amortized form of the previous acquire observations. It does
+not authorize mutation, replace the owner/open-or-closed generation protocol,
+or remove any later validation. `arena_set_free_run()` still completes its
+entire first preflight before changing metadata, scrubs READY/cdata/destructor
+coverage, and calls the stricter `arena_clear_extent_range()` preflight again
+before removing interior boundaries.
+
+The two historical policies remain distinct:
+
+- `arena_set_free_run()` treats recovery ownership as an ordinary fail-closed
+  veto and returns without mutation;
+- `arena_clear_extent_range()` retains its corruption assertion and abort when
+  recovery is the first cell-ordered blocker.
+
+The cold fatal path also preserves the old diagnostic order inside a packed
+word. An ordinary blocker in a lower cell returns first; recovery at the same
+or an earlier cell remains fatal. Zero-length free-run publication is now
+rejected explicitly before end arithmetic or structural mutation. Other
+invalid bounds are rejected before an array access.
+
+READY deliberately differs between the two callers. An old typed READY bit
+vetoes direct extent removal, while free-run publication is allowed to scrub
+READY after lifetime FREE and zero destructor identity. Its second interior
+preflight then observes the scrubbed plane exactly as before.
+
+## Race and ordering argument
+
+The packed loads use the same acquire order class as the former per-cell
+accessors. Release publication of `block[]` changes and the atomic relaxed AND
+on concurrently markable `mark[]` are unchanged. A racing allocation,
+recovery, root, or remote-free publication is still governed by its lifetime
+claim and the surrounding arena generation. Grouping reads by word does not
+make a stale snapshot commit authority; later validation and the exact
+open/commit generation continue to reject a conflicting publisher.
+
+For a valid nonzero range, the new predicate is logically identical to the old
+one:
+
+- every root/recovery pair and lifetime nibble is converted exactly;
+- every destructor plane is ORed into cell geometry;
+- READY is included only for the caller which historically required it; and
+- plain arenas continue to omit the traversable-only lifetime plane.
+
+An independent concurrency/source audit found no unsafe-positive race or
+memory-order weakening after the recovery-policy, zero-length, and cold-order
+hardening.
+
+## Deterministic coverage
+
+An `LJ_ARENA_TEST_HELPERS` wrapper calls the actual static free-run function;
+it is absent from production builds and changes no public ABI. The focused
+arena-sweep fixture uses one 20-cell range spanning both packed lifetime
+positions 15/16 and bitmap positions 31/32. Whole-arena snapshots prove that:
+
+- zero-length and invalid-bound calls fail without mutation;
+- root, recovery-PENDING, destructor, and non-FREE lifetime blockers each fail
+  without changing any arena byte;
+- recovery remains a nonfatal veto on the set-free path;
+- blockers immediately outside the partial range do not leak through its
+  masks; and
+- a valid run scrubs READY/cdata, removes all interior block/mark boundaries,
+  and publishes exactly one mark-only free-run start.
+
+The fixture compiles with `-Wall -Wextra -Werror` and passed.
+
+## Validation
+
+All requested forced-clean gates passed:
+
+- `m2_arena_sweep`, before and after the direct packed-boundary fixture;
+- `m2_arena_gcsweep`;
+- `m3_gc2_recovery` in normal and assertions/GC2-paranoia builds; and
+- `m3_gc2_paranoia`, including all C oracles, 509/509 JIT tests, and 387/387
+  no-JIT tests.
+
+Every configuration-changing run restored the default build. The only known
+compiler diagnostic was the pre-existing GCC inlining warning around
+`gc2_root_rescan_later`/`la_load32_acq`.
+
+## Performance
+
+Five fresh independent processes with the valid stopped-GC wrapper measured:
+
+```text
+current active:  319.30, 317.84, 318.53, 320.70, 323.47; median 319.30 ns/op
+current stopped:  94.83,  94.02,  95.16,  94.28,  95.99; median  94.83 ns/op
+stock active:     37.51,  38.45,  36.98,  39.65,  38.28; median  38.28 ns/op
+stock stopped:    18.35,  21.46,  21.04,  20.61,  21.70; median  21.04 ns/op
+```
+
+The final zero-length/diagnostic hardening was performance-neutral in a
+three-process confirmation: 319.44 ns/op active and 94.14 ns/op stopped.
+
+Against pushed `54714887` (389.19 active, 113.31 stopped), the packed preflight
+improves active time by 17.96%, stopped time by 16.31%, and active-minus-stopped
+cost by 18.63%. It is 34.68% faster than the earlier 488.80 ns/op typed-body
+checkpoint. The main five-process medians remain 8.34x stock active and 4.51x
+stock stopped, so b1.2.0 is still performance-blocked.
+
+The next profile-directed tranche is an independently revalidated terminal
+FREE/FREEING word fast path in quarantine bitmap readiness/application,
+followed by all-free live-cell counting/adoption if the new profile still
+justifies it.

src/lj_arena.c

@@ -3629,28 +3629,70 @@ static uint64_t arena_range_mask(uint32_t lo, uint32_t nbits)
     (((uint64_t)1 << nbits) - 1u) << lo;
 }
 
-static int arena_clear_extent_range(GCArena *a, uint32_t start, uint32_t len)
+/* Prove that every cell in a structural-reuse range is free of persistent
+** ownership metadata. The packed-to-block converters are exact: any nonzero
+** root/recovery/dtor state and every non-FREE lifetime nibble maps to its
+** allocation cell. The surrounding owner-open/closed generation protocol is
+** unchanged; this summary merely groups the former acquire observations and
+** neither authorizes mutation nor replaces any later validation.
+**
+** READY is authoritative for arena_clear_extent_range(), where an old typed
+** publication must veto boundary removal. arena_set_free_run() deliberately
+** omits it: that path has already reached lifetime FREE with no destructor
+** identity and defensively scrubs READY before its second, stricter interior
+** validation through arena_clear_extent_range(). */
+static LJ_AINLINE int arena_range_ownership_preflight(GCArena *a,
+	uint32_t start, uint32_t len, int require_ready_clear,
+	int abort_on_recovery)
 {
   uint32_t pos, end;
+  int lifetime_managed;
   if (start >= LJ_ARENA_CELLS || len > LJ_ARENA_CELLS - start)
     return 0;
   end = start + len;
-  /* Keep the exact per-cell ownership precondition of arena_set_extent().
-  ** Validate the complete range before changing either bitmap, so a failed
-  ** side-plane check leaves every old boundary intact. */
-  for (pos = start; pos < end; pos++) {
-    uint32_t recovery = lj_arena_recovery_state_acq(a, pos);
-    lj_assertX(recovery == LJ_ARENA_RECOVERY_IDLE,
-	       "arena extent reuse crossed recovery ownership");
-    if (recovery != LJ_ARENA_RECOVERY_IDLE)
-      abort();
-    if (lj_arena_root_state_acq(a, pos) != LJ_ARENA_ROOT_NONE ||
-	lj_arena_dtor_kind_acq(a, pos) != LJ_ARENA_DTOR_NONE ||
-	lj_arena_ready_get(a, pos) ||
-	(arena_lifetime_managed(a) &&
-	 lj_arena_lifetime_state_acq(a, pos) != LJ_ARENA_LIFETIME_FREE))
+  lifetime_managed = arena_lifetime_managed(a);
+  for (pos = start; pos < end; ) {
+    uint32_t wi = pos >> 6;
+    uint32_t lo = pos & 63u;
+    uint32_t take = end - pos;
+    uint32_t room = 64u - lo;
+    uint64_t mask, recovery, blockers;
+    if (take > room)
+      take = room;
+    mask = arena_range_mask(lo, take);
+    recovery = arena_recovery_block_bits(a, wi) & mask;
+    blockers = arena_root_block_bits(a, wi) |
+	       arena_dtor_block_bits(a, wi);
+    if (lifetime_managed)
+      blockers |= arena_lifetime_block_bits(a, wi);
+    if (require_ready_clear)
+      blockers |= la_load64_acq(&a->ready[wi]);
+    blockers &= mask;
+    if (recovery != 0 && abort_on_recovery) {
+      uint64_t first_recovery = recovery & (0u - recovery);
+      /* Preserve the old cell-order diagnostic policy on the cold corrupt
+      ** path: an ordinary blocker in an earlier cell returns fail-closed;
+      ** recovery at the same or an earlier cell remains a fatal invariant. */
+      if ((blockers & (first_recovery - 1u)) == 0) {
+	lj_assertX(0, "arena extent reuse crossed recovery ownership");
+	abort();
+      }
+    }
+    if (recovery != 0 || blockers != 0)
       return 0;
+    pos += take;
   }
+  return 1;
+}
+
+static int arena_clear_extent_range(GCArena *a, uint32_t start, uint32_t len)
+{
+  uint32_t pos, end;
+  /* Validate the complete range before changing either bitmap, so a failed
+  ** side-plane check leaves every old boundary intact. */
+  if (!arena_range_ownership_preflight(a, start, len, 1, 1))
+    return 0;
+  end = start + len;
   /* The complete preflight proves this is already opaque reusable storage:
   ** READY and dtor are zero, every lifetime lane is FREE, and no root/recovery
   ** owner exists. Remove old structural boundaries with release stores;
@@ -3751,19 +3793,15 @@ static int arena_set_alloc(GCArena *a, uint32_t cell, uint32_t ncells,
 
 static int arena_set_free_run(GCArena *a, uint32_t start, uint32_t len)
 {
-  uint32_t i, pos = start, end = start + len;
-  if (arena_lifetime_managed(a) &&
-      lj_arena_lifetime_state_acq(a, start) != LJ_ARENA_LIFETIME_FREE)
+  uint32_t pos, end;
+  /* This is the complete pre-mutation ownership proof. READY is intentionally
+  ** scrubbed below; arena_clear_extent_range() then repeats the stricter
+  ** ownership validation over every interior cell before boundary removal. */
+  if (len == 0 ||
+      !arena_range_ownership_preflight(a, start, len, 0, 0))
     return 0;
-  for (i = 0; i < len; i++)
-    if (lj_arena_root_state_acq(a, start + i) != LJ_ARENA_ROOT_NONE ||
-	lj_arena_recovery_state_acq(a, start + i) !=
-	  LJ_ARENA_RECOVERY_IDLE ||
-	lj_arena_dtor_kind_acq(a, start + i) != LJ_ARENA_DTOR_NONE ||
-	(i != 0 && arena_lifetime_managed(a) &&
-	 lj_arena_lifetime_state_acq(a, start + i) !=
-	   LJ_ARENA_LIFETIME_FREE))
-      return 0;
+  pos = start;
+  end = start + len;
   while (pos < end) {
     uint32_t wi = pos >> 6;
     uint32_t lo = pos & 63u;
@@ -3783,6 +3821,13 @@ static int arena_set_free_run(GCArena *a, uint32_t start, uint32_t len)
   return 1;
 }
 
+#if defined(LJ_ARENA_TEST_HELPERS)
+int lj_arena_test_set_free_run(GCArena *a, uint32_t start, uint32_t len)
+{
+  return arena_set_free_run(a, start, len);
+}
+#endif
+
 static int arena_link_run_head(TGAlloc *alloc, GCArena *a, uint32_t start,
 			       uint32_t len)
 {

src/lj_arena.h

@@ -729,6 +729,8 @@ LJ_FUNC uint32_t lj_arena_test_remote_drain_paused(void);
 LJ_FUNC void lj_arena_test_remote_stats_reset(void);
 LJ_FUNC uint64_t lj_arena_test_remote_fast_skips(void);
 LJ_FUNC uint64_t lj_arena_test_remote_arena_probes(void);
+LJ_FUNC int lj_arena_test_set_free_run(GCArena *a, uint32_t start,
+					uint32_t len);
 #endif
 #if defined(LJ_ARENA_TEST_HELPERS) || defined(LJ_GC2_TEST_HELPERS)
 LJ_FUNC void lj_arena_test_lifetime_pause(int enabled);

tests/suites/m2_arena.lua

@@ -119,8 +119,12 @@ return function(add)
     name = "m2_arena_sweep",
     description = "owner-local arena sweep scaffold C fixture",
     run = function(t)
-      run_standalone_fixture(t, t:tmp("lj_t_arena_sweep"),
-                             "t-arena-sweep.c")
+      compile_and_run_sources(t, t:tmp("lj_t_arena_sweep"),
+        arena_sources(t, "t-arena-sweep.c"), {
+        cflags = "-DLUAJIT_SECURITY_PRNG=0 -DLJ_ARENA_TEST_HELPERS",
+        link_luajit = false,
+        libs = {}
+      })
     end
   })