jit: accelerate active-MT table length

Author: Frityet

notes/b1.2.1-roadmap-and-progress-2026-07-18.md

@@ -299,3 +299,53 @@ Development sequencing remains Linux-first. The complete macOS and Windows
 test/fix matrix is deferred until b1.2.1 is otherwise release-ready; incidental
 cross-build or compatibility smokes before then do not expand the active
 platform scope.
+
+## 2026-07-25 active-MT length performance checkpoint
+
+The bounded generated table-length path now has a separate x64 JIT-only ABI.
+It requires the exact `TMPREF IN1` root in the universe-aware current TG,
+published JIT entry, exact physical actor/state ownership, and a caller-owned
+table-vector epoch. JLOOP's live `jit_base` keeps GC bodies retained across the
+call; the epoch retains any paired array/hash generation captured by the
+helper. Root, generation, carrier and ownership are revalidated before the
+integer result escapes. Direct/interpreter calls fail closed, and the general
+rooted helper remains available for adversarial roots and non-x64 fallback.
+
+This removes the counted table lease and global SMR reader from the x64
+generated hot path without weakening the lockless contract. On a core-pinned
+single-core nine-sample run of 20 million traced `#table` operations, active-MT
+length fell from 64.125 ns to 9.332 ns. The stock pre-MT `IR_ALEN` path remains
+unchanged at 2.461 ns. Thus the new path is about 6.87x faster than the first
+bounded implementation and 3.79x the stock lowering; the remaining
+absolute/ratio gap stays explicit performance debt.
+
+Focused coverage includes direct misuse rejection, actual generated admission,
+generation-change retry and side-exit replay, exact nested epoch restoration,
+unchanged wait/allocation/lease accounting, `TMPREF IN1`, and the absence of
+active-MT `IR_ALEN`. This is a Linux-first checkpoint; macOS and Windows remain
+deferred until the rest of b1.2.1 is release-ready.
+
+The final Linux source passed clean focused M5/M6 gates, 20 repeated C-fixture
+runs, five repeated IR runs, 30 heavy six-writer/two-GC-worker resize runs, and
+the broader resize/JIT-read/weak-finalizer/remote-stack/worker-activation
+matrix. Strict GCC and Clang `-Wall -Werror` builds passed. Clang ASan and
+UBSan each passed their strict build, 20 focused repetitions, ten heavy stress
+repetitions, and the mixed concurrency matrix before the default build was
+restored.
+
+The interpreter wait is not closed by this optimization. Audit proved that a
+paused resize owner can replace a source with `FORWARD` before installing the
+value in its successor, with the only surviving value held in an owner-local C
+variable. Re-yielding or redispatching cannot manufacture that fact. Exact
+paused-owner completion needs a persistent resize descriptor published before
+retirement/forwarding, with durable per-slot moving state and helper-safe
+successor accounting.
+
+The safest structural precursor is a versioned help lane for ordinary shared
+new-key insertion. Its descriptor must publish immutable table/key/generation
+facts before the lane becomes visible and use intent-before-side-effect phases
+for freecount reservation, `KEYLOCK`, collision linking, key publication and
+rescan handoff. Pre-MT/private insertion stays stock. FINREG remains a separate
+follow-on because its indistinguishable `FINCLAIM` also transfers into the
+order-list/finalizer transaction and cannot be closed honestly by the ordinary
+new-key descriptor alone.

notes/bounded-rooted-table-length-jit-2026-07-25.md

@@ -50,9 +50,32 @@ the ordinary single-threaded hot trace unchanged.
 After MT activation, supported table-length recording emits:
 
 - a generated-frame `TMPREF IN1` `TValue` root for the table;
-- a nonthrowing `CALLS` to `lj_tab_len_rooted_try()`; and
+- on x64, a nonthrowing `CALLS` to `lj_tab_len_forjit_try()`; and
 - a guard that side exits when the result is `LJ_TAB_LEN_RETRY`.
 
+The generated-only helper cross-checks the dispatch `tg_hint` against the
+universe-aware current TG, requires the exact `tg->tmptv` root, live JIT entry,
+physical actor and state ownership, and enters an owner-written table-vector
+epoch before acquiring either structural root. JLOOP keeps `jit_base`
+published across ordinary generated calls, so destructive GC2 reclamation
+cannot begin and `tmptv` remains an enumerated exact root. The per-TG epoch
+independently prevents an old array or node generation from being reclaimed
+after it is captured. Root, paired generation, carrier, JIT and owner facts are
+all rechecked before returning the scalar result.
+
+This narrower ABI removes the global SMR reader and counted table-body lease
+from the generated hot path. The general `lj_tab_len_rooted_try()` remains
+unchanged for adversarial C roots and non-x64 generated fallback; direct or
+interpreter calls to the x64 ABI fail closed.
+
+The common dense/no-hash shape also has an exact bounded shortcut: after the
+paired snapshot it accepts `asize-1` only when the final positive array slot is
+a valid non-nil Lua value, then performs the same paired-current check. Empty
+tables are exact zero. A boundary hole or any hash part uses the full widening
+and binary search; internal/malformed observations request retry. FINCLAIM
+classification and the owner-written epoch operations are inline on this hot
+path, while their existing external entry points remain available elsewhere.
+
 The side exit replays the current bytecode in the interpreter. The bounded
 helper has made no Lua-visible mutation, so replay does not duplicate an
 effect. This lowering is shared by direct table `#`, Lua 5.2-compatible
@@ -72,8 +95,10 @@ The focused C fixture covers:
 - table-root replacement;
 - exact-owner loss and an invalid owner;
 - bounded generated-code retry, side exit, and interpreter replay; and
-- balanced SMR, lease, root-descriptor, and dynamic-root-anchor accounting,
-  with no table wait-counter movement and no helper allocation.
+- direct-call rejection by the generated-only ABI;
+- exact nested table-read depth/epoch restoration; and
+- balanced SMR, vector epoch, lease, root-descriptor, and dynamic-root-anchor
+  accounting, with no table wait-counter movement and no helper allocation.
 
 The JIT IR gate checks all live traces. It requires stock `IR_ALEN` and no
 rooted helper before MT, then requires `TMPREF IN1` plus the rooted `CALLS` and
@@ -104,11 +129,35 @@ correctness and hot-path performance.
 The important common-case property is exact: pre-MT traces retain stock
 `IR_ALEN`, so this change adds no generated-code instruction to that path.
 
-A preliminary single-run active-MT microbenchmark measured roughly 64 ns per
-`#table`, versus roughly 2.8 ns on the pre-MT stock lowering (about 23x). That
-is an absolute bounded cost, not performance parity, and it remains explicit
-optimization debt. The result is not a release benchmark or a substitute for
-the required reproducible three-sample performance gate.
+A nine-sample, core-pinned 20-million-iteration microbenchmark measured
+2.461 ns per pre-MT `#table` and 9.332 ns for the x64 active-MT generated
+path. The committed general rooted ABI measured 64.125 ns in the same run, so
+the generated-only epoch path is about 6.87x faster than the first bounded
+implementation and 3.79x the stock pre-MT lowering. A nearby active-MT rooted
+point read remained 64.361 ns, confirming that the length improvement is the
+removal of general lease/SMR machinery rather than a benchmark artifact.
+
+This is a reproducible microbenchmark result, not yet the complete b1.2.1
+three-sample suite gate. The remaining roughly 6.9 ns absolute delta is explicit
+parity debt and the 3.79x ratio is still above the planned 3x gate, but the
+former 25x cliff is gone and incremental optimization is now realistic.
+
+## Linux validation
+
+The final production-source diff fingerprint was
+`849fcdffb449db02d4803478a92c386f497a443ff33b4b0c19abe291b8814df7`.
+Clean focused M5 and M6 gates passed, followed by 20 repeated C-fixture runs
+and five repeated JIT-IR runs. A six-writer/two-GC-worker length-and-resize
+stress with 8,192 resize rounds and 4,096 observer rounds passed 30/30, and the
+broader resize, traced-read, weak-finalizer, remote-stack-GC and worker-
+activation matrix passed.
+
+Both GCC and Clang built the helper/assertion profile cleanly with
+`-Wall -Werror`. Clang ASan and UBSan each passed a strict helper build, the
+focused fixture 20/20, the heavy concurrent stress 10/10, and a mixed
+length/JIT-read/weak-finalizer/remote-GC run. The normal default build was then
+restored. This evidence is deliberately Linux/x64-only; the complete macOS and
+Windows pass remains deferred until b1.2.1 is otherwise release-ready.
 
 ## Remaining work
 
@@ -123,5 +172,12 @@ mechanical proof that no ordinary length caller reaches a peer-dependent wait.
 For structural mutation, the next larger table step remains a persistent,
 helpable new-key publication descriptor covering collision-chain `KEYLOCK`
 and `FINREG` states, followed by resize/range descriptors. Those designs should
-also provide the stable generation facts needed to replace the current
-active-MT helper cost with cheaper validated fast paths.
+also provide the stable old/successor facts needed for exact interpreter
+completion while a resize owner is paused.
+
+A yield-and-redispatch loop is not such a mechanism. During resize the current
+vectors remain `RETIRING` until owner-only migration completes, and a forwarded
+source value can exist only in the paused owner's C local before successor
+installation. Exact completion therefore requires a published resize/migration
+descriptor with value-preserving per-slot intent, helpable successor
+accounting, and idempotent root publication.

src/lj_cdata.c

@@ -210,17 +210,14 @@ void LJ_FASTCALL lj_cdata_free(global_State *g, GCcdata *cd)
   }
 }
 
-#define LJ_CDATA_FINCLAIM_U64 \
-  ((((uint64_t)LJ_TLIGHTUD) << 47) | (((uint64_t)1 << 47) - 1u))
-
 static void cdata_fin_setclaim(TValue *tv)
 {
   tv_rawstore(tv, LJ_CDATA_FINCLAIM_U64);
 }
 
 int lj_cdata_fin_isclaim(cTValue *tv)
 {
-  return tv_rawload(tv) == LJ_CDATA_FINCLAIM_U64;
+  return lj_cdata_fin_isclaim_inline(tv);
 }
 
 #if defined(LJ_CDATA_TEST_HELPERS)

src/lj_cdata.h

@@ -125,6 +125,15 @@ LJ_FUNC int lj_cdata_fin_claim_held(CTypeFinLease *lease, cTValue *key,
 LJ_FUNC int lj_cdata_fin_store_claim_held(CTypeFinLease *lease,
 					  cTValue *key, cTValue *src);
 LJ_FUNC int lj_cdata_fin_isclaim(cTValue *tv);
+/* FINCLAIM is checked by every shared table snapshot, including generated
+** numeric reads. Keep the raw-tag test inline for those hot readers while the
+** external helper remains available to cold translation units. */
+#define LJ_CDATA_FINCLAIM_U64 \
+  ((((uint64_t)LJ_TLIGHTUD) << 47) | (((uint64_t)1 << 47) - 1u))
+static LJ_AINLINE int lj_cdata_fin_isclaim_inline(cTValue *tv)
+{
+  return tv_rawload(tv) == LJ_CDATA_FINCLAIM_U64;
+}
 
 #if defined(LJ_CDATA_TEST_HELPERS)
 enum {

src/lj_ircall.h

@@ -210,6 +210,7 @@ typedef struct CCallInfo {
   _(ANY,	lj_tab_storetv_forjit_newref, 5, S, PGC, CCI_L|CCI_T) \
   _(ANY,	lj_tab_storetv_existing_forjit, 4, S, INT, CCI_L|CCI_T) \
   _(ANY,	lj_tab_len_rooted_try, 2, S, INT, CCI_L) \
+  _(ANY,	lj_tab_len_forjit_try, 2, S, INT, CCI_L) \
   _(ANY,	lj_tab_keyindex,	2,  FL, INT, 0) \
   _(ANY,	lj_vm_next,		2,  FL, PTR, 0) \
   _(ANY,	lj_tab_len,		1,  FL, INT, 0) \

src/lj_record.c

@@ -2950,7 +2950,11 @@ TRef lj_record_tab_len(jit_State *J, TRef tab)
 {
   if (lj_record_mt_shared_tab(J, tab)) {
     TRef tabroot = rec_tmpref_mode(J, tab, IRTMPREF_IN1);
+#if LJ_HAS_X64_MT_JIT_HELPERS
+    TRef len = lj_ir_call(J, IRCALL_lj_tab_len_forjit_try, tabroot);
+#else
     TRef len = lj_ir_call(J, IRCALL_lj_tab_len_rooted_try, tabroot);
+#endif
     emitir(IRTGI(IR_NE), len, lj_ir_kint(J, LJ_TAB_LEN_RETRY));
     return len;
   }