Older version (2020.6.4) of PSPDFKIT served by the benchmark?

[Liedtke]

I'm investigating a potential improvement to the WebAssembly performance in V8 / Chrome with inlining support in V8's optimizing JIT compiler for wasm.

I had run https://pspdfkit.com/webassembly-benchmark/ and the wasm binary served seems to have changed in the recent days:
Previously (10 day sago) I was getting pspdfkit-ffc054f2e674b6fc.wasm (13MiB), now it uses pspdfkit-85b702f2ed274c68.wasm (9.7MiB).
Looking at the current accompanying .wasm.js file, this is version 2020.6.4 and I can find the same wasm binary in that npm package version.
Looking at more recent binaries' sizes, it seems that my the ffc0[...].wasm binary should be a fairly recent version.

Did something change in the benchmark that causes it to use an old(er) version now?
Is the expectation that the benchmark would serve new versions to benefit from improvements on the PSPDFKIT code or is the version meant to be fixed?

To provide more context:
We have seen a significant improvement on the PSPDFKIT benchmark (~10%) when enabling inlining (a feature that is currently only enabled for wasm-gc modules).
We plan on enabling it for linear-memory wasm (like pspdfkit) but have seen some significant compile time regressions.
I'm preparing a change to use better inlining heuristics to get the performance improvements while not regressing too much on compile time.
The characteristics for PSPDFKIT now changed "overnight", the benchmark seems to be slightly slower overall and the performance benefits of enabling inlining are significantly reduced (to ~5%).

[Liedtke]

Note: I have also tried to run the benchmark locally but even with older versions than the most recent one it fails at the 3rd step (Export ) with:

PSPDFKitError: Annotations: `text` is not an object with properties `format` and `value`

[matej]

Hello Matthias. It's great to see that the benchmark is proving useful for validating performance improvements in V8.

What you're observing here is surprising to me as we intentionally left a the benchmark on 2020.6.4 to have results be as comparable as possible between test runs. We mistakenly updated the version once during a release in 2021, but that change was quickly reverted and according to our git history there were no other explicit updates done from our side. I'll take a look internally to see if I can find out more.

That being said, while we do believe that constantly updating our WASM version would result in a less useful benchmark overall, I do acknowledge that the used version is now very outdated. We're considering doing a one time update and releasing what would essentially be a v2 of the benchmark with our latest release and WASM binary. With that we can also check out and see what's going on with that error you encountered.

Would this work for you and if so, what kind of timeline are you operating under for your performance improvements?

[Liedtke]

I'll take a look internally to see if I can find out more.

Thanks, I'd be certainly interested in case you find something out.
We experimented with Inlining about half a year ago and saw similar improvements as I did with the supposedly more recent version.

We're considering doing a one time update and releasing what would essentially be a v2 of the benchmark with our latest release and WASM binary. [...]
Would this work for you and if so, what kind of timeline are you operating under for your performance improvements?

That would be amazing.
For my current project I plan on finching (meaning releasing for parts of the user base) the Inlining feature as part of Chrome 123 which will hit production in about 4 weeks.
The main disadvantage is that we will mainly see the (background) compilation time metric (regressing). We might have less data points on the performance gains which is why there we have to rely at least partially on benchmarks where we have seen very different results (though no regressions on peak performance).

If I could fully run the benchmark locally, that would already be great.
Having an official PSPDFKit wasm benchmark v2 would still be very nice as the old version is great for long term comparisons but arguably the performance of a 4 year old application is less relevant than the performance of a current one.

[matej]

We went ahead and updated the benchmark. We already pushed the changes to this repository and https://pspdfkit.com/webassembly-benchmark/ should reflect the changes soon as well. You'll see a new footer that will indicate that PSPDFKit for Web 2024.1.0 is being used.

I hope this helps. I'd love it if you could tell us what kind of improvements you're seeing with this new deployment.

[Liedtke]

Thanks a lot! (I realized, I didn't reply here, yet.)
It's nice to have one more data point with this.
We currently test the inlining for wasm with linear memory on 50% chrome beta but I don't have any results yet.
Locally, the compile time regression for inlining with the new benchmark is lower but the benefits are lower as well.

For the old module we see a 5-10% performance improvement depending on architecture.
On the extreme side we have seen a 40% background compile time regression on Mac m1 mini, it mostly stays around 20%.
On M1 mini we see the following runtime changes:

• Searching: -33%
• Exporting: -32%
• Annotations: -25%
• Rendering: -4%
• Total: -8%

So there are huge improvements on the "smaller" parts of the benchmarks and only a small improvement on rendering.
I'll push for integrating the new benchmark into our benchmark suite as well, so I should get numbers on every platform for it as well, as said, locally I have seen smaller runtime improvements but also smaller compile time regressions (probably indicating that the newer version has better inlining decisions already happening during build time of the PSPDFKIT wasm module than the old module).

[matej]

Thank you for the update. I'm going to close the ticket, but please don't hesitate to chime back in if you need anything else of if you might have other results to share in the future.