Repository navigation
Add StripedArrayPool and virtual-thread-aware HybridPool recycler pools - #1687
Conversation
StripedArrayPoolBase recycles through a fixed array of atomic slots scanned from a thread-id home index: acquire is getAndSet(null), release is a CAS into the first empty slot, and a release with every slot occupied drops the instance, which bounds retention. No queue node is allocated on either path. HybridPoolBase extends it: platform threads get the exact ThreadLocal leave-in behavior (SoftReference slot, unlinked, no-op release) and virtual threads share the striped slots, so their instances recycle across threads instead of dying with each thread. The virtual-thread check is a MethodHandle resolved once against Thread#isVirtual with an always-false fallback, so the code runs on the JDK 17 floor and the Android SDK 26 API baseline. Concrete BufferRecycler pools and factory methods follow the existing JsonRecyclerPools patterns (GLOBAL shared instances, construct(), readResolve). defaultPool() is unchanged. Tests cover lifecycle, the retention bound, capacity rounding, clear()/pooledCount() conformance, double-lease stress with platform and virtual threads, JDK serialization round-trips, and both paths of the hybrid pool; virtual-thread tests skip below JDK 21.
|
Fails on an Animal Sniffer check for Android support. You can use MethodHandles but we need to have a failover that kicks in MethodHandles are unsupported in the JVM (older Android releases). |
Move every java.lang.invoke reference into a nested holder and catch Throwable at the one call site, so on runtimes without method handles (Android before API 26) HybridPool degrades to exact ThreadLocalPool behavior instead of failing with LinkageError. The AnimalSniffer flag on MethodHandle.invokeExact is a signature-polymorphic false positive (java.lang.invoke is in the checked Android profile); ignore that one class in the pom with the rationale.
…ents The long-lived virtual pass keeps per-thread pool state alive across cycles, so it never exercises the thread-per-task lifecycle that motivates a shared structure for virtual threads. Add a third pass that starts a fresh virtual thread per cycle, and accept threads/warmup/measure as command-line arguments.
|
Updated to have a fallback for Android. |
|
@stevenschlansker this is very similar to the approach of Vertx we implemented with @mariofusco Would be worth referencing it, benchmarking and comparing against it too?🙏 In this current pr, I see no protection from false sharing and other diff which could be worth to be aligned under heavy contention. Another possible approach could be:
This should keep the acquired slot index stable and allow a release to hit the same stripe. This should help to keep the slots balanced, unless only a specific VT slot id is mostly utilized and then never again. |
|
Thank you for the suggestions! I was not aware of the vertx implementation - if you feel your implementation already covers these additional points well and would like to contribute it to Jackson, please feel free to open a PR and I will gladly close this one. Otherwise, I'll see which of the suggestions I can fit in and add a "inspired by" line. |
|
I see the vert.x implementation is Eclipse public license - I think that's okay for me to "read and be inspired by" and then contribute under Apache 2 license? Just want to clarify before I do anything "bad". Edit: Looks like it's "or Apache 2", so no problem. |
|
@stevenschlansker |
|
Whatever we can allow us (Vertx/Quarkus) to remove code is welcome 😁 |
The 16 default slots were 64 bytes with compressed references, one cache line, so every acquire or release from any thread bounced the same line. In the in-repo throughput harness on an 8-core aarch64 host the unpadded array was slower than no pooling under 8-way and 16-way contention (about 4.3M ops/s against 5.7M); with slots 16 references apart it is at threadLocal parity (about 14.5M against 15.7M), 3.2x to 3.7x on the striped pool and 2.8x to 3.6x on the hybrid pool's virtual-thread path. The hybrid platform path does not touch the array and is unchanged. Layout follows the vert.x HybridJacksonPool stripe spacing, pointed out by Francesco Nigro on the PR.
|
Thanks for pointing out the false sharing, as you say the effect is significant. I updated this PR to pad the entries and now the striped pool performs approximately at parity with the ThreadLocal implementation. |
cowtowncoder
added
the
performance
9f7b3a2 to
f681183
Compare
bb8279e to
abe4e17
Compare
|
I rebased the branch on latest master - I think something went wrong with a previous merge conflict resolution. |
| * may touch every slot, so very large pools are both memory-heavy | ||
| * and slow; this also keeps the backing array size from overflowing. | ||
| */ | ||
| public final static int MAX_CAPACITY = 1024; |
There was a problem hiding this comment.
@stevenschlansker Added maximum to bound as per Claude CR suggestion -- but value chosen is arbitrary; feel free to change if you think of a better limit.
|
@stevenschlansker Played with this a bit; planning to merge tomorrow, so we can get this in 3.3.0 release. |
cowtowncoder
changed the title
StripedArrayPool and virtual-thread-aware HybridPool recycler pools
The VirtualProbe holder and its LinkageError fallback were for Android runtimes without java.lang.invoke (before API 26). The 3.x line checks against the Android API 34 signature, and XJBWriter already needs VarHandle (API 33), so that fallback could not run on a supported runtime. A failed holder also threw and caught a new NoClassDefFoundError on every call. The method handle is now a static field, and _isVirtual() is private. The animal-sniffer exemption for all of java.lang.invoke.MethodHandle is replaced by the plugin's annotations option and a package-private IgnoreJRERequirement annotation on _isVirtual(), the one method with a signature-polymorphic invokeExact call. Without the annotation the check fails on that call only.
HybridPoolBase extended StripedArrayPoolBase, so a hybrid pool was an instanceof StripedArrayPoolBase whose pooledCount() and clear() covered only part of its state, and it allocated the slot array even where no virtual thread could use it. It also copied the SoftReference leave-in logic of ThreadLocalPool into a ThreadLocal per pool instance, so a platform thread kept one BufferRecycler for each hybrid pool it used. HybridPoolBase now extends StatefulImplBase and holds: - a ThreadLocalPoolBase, which gives each platform thread its leave-in. JsonRecyclerPools.HybridPool passes ThreadLocalPool.GLOBAL, so a platform thread uses the same instance as with threadLocalPool(). - private striped slots, allocated by the first release into them. A pool that only platform threads use through acquireAndLinkPooled(), which is how Jackson uses it, never allocates them. capacity(), pooledCount() and clear() now describe the shared slots only, and the class Javadoc no longer claims that platform threads behave exactly as with ThreadLocalPoolBase.
Each duplicated the sized factory with DEFAULT_CAPACITY. Like newBoundedPool(int), the new pools now have one factory for a private instance; sharedStripedArrayPool() and sharedHybridPool() keep the default size.
- An acquire on an empty pool or a release on a full one scanned every slot, up to 1024 cache lines at MAX_CAPACITY. Each scan now stops after 8 slots. With 4 retained instances in the 16 default slots, a scan of 8 from a random home slot (a new virtual thread) misses all of them about 4% of the time, against 27% for a scan of 4. - Slot 0 shared a cache line with the AtomicReferenceArray header and length, which every access reads. Slots now start one stride in. - The home slot came from the deprecated Thread.getId() and took the low bits of the Fibonacci product. It now uses the identity hash of the thread, which JDK 17 has, and the top bits of the product. - HybridPoolBase passes the thread it already has into the slot scan instead of calling Thread.currentThread() again.
ManualRecyclerPoolThroughput looked up Thread.startVirtualThread on every call, inside the measured per-task virtual-thread loop. The harness and StripedAndHybridPoolTest now resolve it once.
Replace the synchronized _allocateShared(), which locked on a pool that callers can reach through sharedHybridPool(), with a compareAndSet on _shared that discards the losing allocation. Derive MAX_PROBE from DEFAULT_CAPACITY and refer to the constant in the Javadoc. State that the SLOT_SHIFT spacing is 64 bytes only with compressed oops, and document the capacity-1 edge of _hashShift.
Raise SLOT_SHIFT from 4 to 5, so slots are 32 references (128 bytes with compressed oops) apart. This matches the JDK @contended padding that Striped64 uses, and covers 128-byte cache lines on Apple silicon and the x86 adjacent-line prefetcher. The cost is about 1 KB more for the default 16 slots and 66 KB more at 1024 slots.
The Javadoc of acquireAndLinkPooled() told callers to call releasePooled(). That leaves the instance linked, so the next acquireAndLinkPooled() of it fails with "already linked". A linked instance is released with WithPool.releaseToPool(), as IOContext does. releasePooled() pairs with acquirePooled(). Add a test of both pairings on a BoundedPool.
List each pool in the JsonRecyclerPools class Javadoc with its retention and sharing behavior, and state when to choose StripedArrayPool over HybridPool and the other way around.
A scan stopped after 8 slots from a home slot that is fixed per thread. Above 16 slots, a new thread (such as a per-task virtual thread) often missed every retained instance and created a new one. Its release then stored that instance in its own window, so retention rose toward the slot count. Instances that one thread released filled only the window of that thread, and other threads rarely scanned it. Acquire and release now scan every slot from the home slot and wrap around, so a release drops an instance only when all slots are full. MAX_CAPACITY drops from 1024 to 64, which limits a scan that finds nothing to 64 slot reads. The JsonRecyclerPools Javadoc now says how to size a striped or hybrid pool.
A native image without reflection metadata for Thread.isVirtual() can throw MissingReflectionRegistrationError, which is a LinkageError, from the method handle lookup. The lookup runs in a static initializer, so the error made HybridPoolBase fail with ExceptionInInitializerError and left the pool unusable. The lookup now returns null for a LinkageError, and every thread is then a platform thread.
On a platform thread, HybridPool returns the leave-in of ThreadLocalPool.GLOBAL, which calls new BufferRecycler() itself. A subclass that overrides createPooled() thus changes only the instances of virtual threads and of acquirePooled(). The Javadoc did not say so.
- StripedArrayPoolBase is not allocation-free: an acquire that finds no instance calls createPooled(). It has no lock and no queue node. - The isVirtual handle is resolved once per class initialization, not once per JVM. - BoundedPool takes the lock of its ArrayBlockingQueue on each call. It never waits for space or for an instance, but it does lock. - "Shared across threads" read as the global instance for defaultPool(), which returns a new pool on each call. The list now says "no per-thread state" and names the shared accessors where it means them.
Capacity 1 shifts the hash by 32, which Java reduces to 0, so only the mask keeps the home slot in range. Concurrent first releases on a new HybridPool race to allocate the shared slots, and a lost race would drop releases. The serialization test also did not use the shared slots of the deserialized pool.
HybridPool keeps the platform-thread instances per thread, so only the instances in the slots are open to any thread.
|
Thanks for the thorough review and suggestions regarding false sharing. The points are taken as valid and the latest version should fix them, with tests where feasible. Stack instead of slots: declined, this code is already complex and introducing a stack requires either Node allocation, or complicated atomic write patterns. Could consider an opportunity to followup. Thread routing: double lease and lost reuse fixed, with tests. VirtualProbe fallback: resolve once, don't throw, remove dead pre-API-26 code. Perf: limit pool size, shift the array to skip the first stride, 128 byte stride (same as jdk Docs clarified, class hierarchy improved, remove unnecessary factories from public API, reduce scope of Added a "when to use each pool" and fix acquireAndLinkedPool release in the Javadoc. I ran the first round of benchmarks and the performance is overall good, but there's an unexplained finding where a 4-slot array outperforms a 16-slot array with 8 threads. I don't think I'll finish this investigation tonight, but there might be one more small change depending on what I find tomorrow. I think what's here is good for another round of review though. |
|
@stevenschlansker Thank you! I think I'll give this another round of reviews and we can get it merged soon. |
…op IgnoreJRERequirement Also dedupe pool serialization tests and fix striped pool Javadoc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nces Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
📈 Overall Code Coverage
|
|
@stevenschlansker Merged after minor fixes Claude Opus 5.5 suggested. Did not do anything about concern wrt 64-slot cap -- this can be changed separately if that is seemed as too low but for now 64 sounds reasonable. EDIT: I asked Claude to test 128/256 caps, and it was unable to see any benefits: despite increased miss rates, actual performance differences were within measurement error range (i.e. no measurable difference). So 64 seems fine. |
|
Thanks! |
Context: While upgrading Blackbird, we identified contention in the buffer pool recycling algorithm. We found that pollFirst / offerLast on a ConcurrentLinkedDeque has less than optimal performance:
So, I asked Claude for options on a better design. Through a number of iteration rounds, we settled on a hybrid approach: platform threads continue to use soft thread locals, while virtual threads probe the slots of a striped atomic reference array, avoiding contending on a single head or tail. The array is padded to avoid false sharing effects.
The hope is that this hybrid pool ends up performing equal or better than any current implementation, and eventually could become default.
PR designed with assistance from Claude Fable 5 and Claude Opus 5.5.
Claude's report follows:
StripedArrayPoolBaserecycles through a fixed array of atomic slots: acquire isgetAndSet(null), release is a CAS into an empty slot. Each acquire and each release scans all slots. The scan starts at a home slot fromSystem.identityHashCodeof the thread (Fibonacci hash, top bits) and wraps around. A release drops the instance only when every slot is full, which bounds retention. Slots are 128 bytes apart with compressed oops (the@Contendeddefault), slot 0 is one stride off the array header, and no path allocates a queue node. Thus an acquire on an empty pool reads one cache line per slot before it creates an instance.DEFAULT_CAPACITYis 16 slots, a requested count rounds up to a power of two, andMAX_CAPACITYis 64. The Javadoc credits the Vert.x project for the approach (see eclipse-vertx/vert.x#4920).HybridPoolBaseholds aThreadLocalPoolBaseand its own striped slots. On a platform thread,acquireAndLinkPooled()returns the unlinked leave-in, andHybridPoolusesThreadLocalPool.GLOBAL, so it is the same instance thatthreadLocalPool()gives. On a virtual thread, it returns a linked instance from the slots. Every release, and directacquirePooled()/releasePooled()on any thread, uses the slots, so an instance can move between thread types with no double lease and no lost reuse.createPooled()supplies only the slot path, because the leave-ins come from theThreadLocalPoolBase. The first release allocates the slots, so a pool that Jackson uses only on platform threads never allocates them. The virtual-thread check is aThread#isVirtualMethodHandleresolved once. The handle is null where the method does not exist (the JDK 17 floor, the 3.x Android baseline of API 34), and also when the lookup throws aLinkageError, as in a native image without reflection metadata for the method.sharedStripedArrayPool()andsharedHybridPool()return the shared global instances, andnewStripedArrayPool(int)andnewHybridPool(int)construct new instances, with the existingGLOBAL/construct()/readResolve()patterns.defaultPool()is unchanged. UseHybridPoolwhen one instance per platform thread is acceptable. UseStripedArrayPoolwhen it is not: it is public because it is the bounded pool with no per-thread state, the like-for-like replacement for the defaultConcurrentDequePool. TheJsonRecyclerPoolsJavadoc lists when to use each pool. It also says to size a striped or hybrid pool near the peak number of threads that hold an instance at the same time, up to 64.Tests cover the lifecycle, the retention bound, capacity rounding and clamping, capacities 1 and 2,
clear()/pooledCount(), task-per-thread retention at 64 slots, reuse across threads, both hybrid paths, the hybrid leave-in fromThreadLocalPool.GLOBAL, the race of the first releases that allocate the hybrid slots, releases across thread types in both directions, double-lease stress that mixes direct and linked use, and serialization, with the slot path of a deserialized pool. Virtual-thread tests skip on JDK 17. The PR also fixes theacquireAndLinkPooled()Javadoc, which namedreleasePooled()instead ofWithPool.releaseToPool(), with a test.Benchmarks
I measured only linux-aarch64 at the final head: an Apple silicon VM with 8 CPUs, Fedora kernel 7.1.10, and OpenJDK 25.0.4.1. The benchmark is JMH on a 485-byte POJO document through databind 3.3, as paired rounds that rotate the pool order. Each round has one fork per pool, with 4 x 1 s warmup and 5 x 1 s measurement. Each cell gives the median change against the current default (
ConcurrentDequePool), the range, and the number of positive rounds.Platform threads, 10 rounds:
No platform-thread cell regressed. Deque writes do not scale above 4 threads, but the slots and the hybrid leave-ins scale to the 8 CPUs.
Virtual thread per task (4 drivers, each operation on a new virtual thread), 20 rounds, with GC allocation from
-prof gc:threadLocalPool()The slots recycle across virtual threads: hybrid and striped allocate 25 B/op less than the deque, and 18 KB/op less than no pooling on writes.
threadLocalPool()allocates more than no pooling, because each short-lived thread abandons its recycler,SoftReference, andThreadLocalmap entry. Throughput is near parity on writes. On reads, no pooling andthreadLocalPool()are 4-5% faster than the deque and about 2% faster than the slots (18 to 19 of 20 rounds), at the cost of 500-700 B/op more allocation. Hybrid against striped on virtual threads is -0.9% on writes and -0.3% on reads, inside the spread.A capacity sweep of the striped pool (4, 8, 16, 32 and 64 slots, 6 rounds, against 16 slots) checked the default. On virtual-thread writes, allocation is flat at 1,597-1,598 B/op and throughput shows no trend. On 8-thread platform writes, 8, 32 and 64 slots are within the spread of 16 slots. But 4 slots are faster by +36% (+26..+55%, 6/6), with 20-50 B/op more allocation, and I do not have an explanation. A separate count of pool misses (one run for each case) shows that drop-on-full engages only when the slots are fewer than the threads that hold an instance at the same time. On this 8-CPU host, that is 4 slots: 0.9% of writes miss at 8 platform threads, 1.2% at 16, and 0.15-0.30% with virtual threads. With 8 or more slots, a run of 0.8 to 1.6 million writes had at most 8 misses, for 4 to 64 threads.