New benchmark runner: Afterframe

[krausest]

So far the CPU benchmark measured duration by looking at the chrome timeline traces (the events one can see in the performance tab). This restricted testing to chrome only with the drivers: webdriver (selenium), puppeteer, playwright and webdriver-cdp.

If we want to measure across browsers we have to measure on the client side, since safari and firefox seem to have no api to access something similar like timeline tracing. We had some functionality in the benchmarks that tried to do client side measuring for the duration in the past: (the commit for its removal was 89b546f).
https://github.com/andrewiggins/afterframe takes a similar approach to get a callback after the rendering finished by using aMessageChannel and RAF.

It would be interesting to add such a benchmark driver and see if we can achieve cross browser testing.
The two classes to implement that are https://github.com/krausest/js-framework-benchmark/blob/master/webdriver-ts/src/benchmarksWebdriverAfterframe.ts and https://github.com/krausest/js-framework-benchmark/blob/master/webdriver-ts/src/forkedBenchmarkRunnerWebdriverAfterframe.ts. The test driver injects the afterframe code into the loaded page to compute the duration for the operation. This is similar to the other benchmark drivers where the whole duration from the click to finishing the repainting is measured.

[krausest]

The first question is: How close are the CPU results for chrome and webdriver and webdriver-afterframe?

When measuring with afterframe I haven't found any cross browser way to add a CPU slowdown, so all benchmarks were ran with full CPU speed. I took three frameworks and a closer look at their results:

As I hoped results are quite close for longer running benchmarks. Benchmarks below 1 FPS can vary more which was what I expected or feared. For almost all results the result from the chrome timeline was smaller that that from afterframe wich makes sense (afterframe should add some latency until the raf is executed).

Here's a chart showing the ratio of the duration reported by afterframe to the values reported by webdriver:

The three large outliers are the select row benchmarks.
Thus I suggest to skip select row for benchmark testing with afterframe.

Without select row the chart looks smoother:

What's interesting is that variance for afterframe isn't much higher that for webdriver.

[GermanJablo]

Hi @krausest, thank you for opening this thread and for the incredible work you do with this benchmark.

Some questions:

1. Are the results of your image from the same iteration? If the answer is yes, how in the case of 14 and 24 afterframe could it be faster than timeline? (using the last image as reference)
2. You said:

Chrome automation tools like puppeteer, playwright and webdriver can log the same events that the performance tab uses for its display

Could you explain to me how to access that information? (preferably with Playwright which is what I have the most experience with).

---

I would like to experiment to find a solution to this. Several options are discussed in this very interesting blog article.

As you noticed, afterframe uses rAF + MessageChannel. I think there are other options to explore such as:

• PerformanceObserver
• transitionEnd and animationEnd events
• MutationObserver
• Web workers