Elaborate on "deprecating" requestAnimationFrame()

[annevk]

I'd love to understand better as to why extensions to requestAnimationFrame() are not the way forward.

Re: https://github.com/WebKit/explainers/tree/main/animation-frame-rate.

[graouts]

First, I think it's important to consider that we're not suggesting the deprecation of requestAnimationFrame(). That function is part of the Web platform and here to stay.

This proposal is about controlling the frame rate of animations. While requestAnimationFrame() may be used to yield an animation through repeated calls, it does not in itself provide any animation feature. It has no notion of timing, duration, iteration, easing, progress, etc. This is the province of technologies based on the Web Animations model.

A frame rate makes sense for something that is, by nature, progressing, whereas requestAnimationFrame() registers a one-shot callback.

Separately to this explainer, we're advocating the support for custom effects to offer support for callback-based animations, which would tie in nicely with this proposal since this would allow authors to write such animations while specifying a frame rate.

[annevk]

All true (though I don't personally equate deprecating with removal), but assuming custom effects come to pass, what use cases would remain for requestAnimationFrame()?

[graouts]

I'm not sure I can fully answer this question, because I certainly don't know of all the uses for requestAnimationFrame(). But I believe that "virtual DOM" JS libraries rely on requestAnimationFrame() as a hook to commit their changes to the DOM, which is typically not an animation task but really a "do coordinated work at a future time where it might make sense" task. I expect there are (many) other uses cases for requestAnimationFrame() that aren't animation-related that I don't know about.

The naming of that function is really unfortunate :)

[annevk]

Right, so the idea is that you'd continue to use requestAnimationFrame() if you need something to happen at that point in time, but it's not expected you'd use it at 60Hz to move something around on the screen. I guess that's reasonable.

[mdrejhon]

Related: #89 -- This applies to both requestAnimationFrame() as well as Web Animations API.

Permission Prompt API for power-hungry animations including TestUFO

[mdrejhon]

We did more work on trying to port TestUFO to Web Animation API, and we've hit a blocker.

While TestUFO is also used as a "120Hz advertising" tool, different subpages / subtests within it are used by peer reviewed researchers, as well as many content creators (including LinucTechTips and RTINGS), and this is where we run into a science-quality blocker issue:

To Replace rAF() We Need An Equally Accurate Frame Presentation Callback Event added to Web Animation API

Purpose: Measure frame presentation times accurately, and detect frame drops (e.g. missed refresh cycles, e.g. 59fps at 60Hz, or 119fps at 120Hz, etc).

I have done some research why we continue to be able to port the 30+ motion tests at TestUFO.com (>2M uniq's per month in CloudFlare analytics, ~700K/mo recorded in Analytics, difference is from tracker/country blocking) away from requestAnimationFrame(), towards the new Web Animation API (still lower scientific quality for our needs, explained at #89 (comment)

Over 500 content creators (with a combined subscribership/viewership of >100M) use TestUFO for testing the motion quality of displays, and TestUFO is "motion education in a single link" as well as "120Hz advertising in a single link" on top of research -- so we're trying to find ways to support 120Hz apple devices better.

Web Animation API (that Apple prefers) is an adequate solution but there's one major scientific-mandatory feature missing:
I am pleased to say there's only one change to Web Animation API that we will need -- a callback API for all successfully presented web animation frames.

TestUFO is in several peer-reviewed papers (including one co-authored by NIST.gov) because I've successfully used requestAnimationFrame() as a single-framedrop detector.

Via analyzing the now() timing of requestAnimationFrame callbacks, I can detect single missed refresh cycles (tiny microstutters, sometimes only visible to a camera). I know when I'm only running 59fps at 60Hz, or 239fps at 240Hz, and can instantly invalidate a scientific test on time.

To replace requestAnimationFrame() I need a modification to the Web Animation API to add a callback purely for framedrop analysis purpose. I also will settle for a callback upon missed frames (at a speed of within 1/Hz, so I can render a color-coded error message as quickly as possible at the bottom of TestUFO, to help all researchers and content creators accept/invalidate photographic tests of TestUFO animations).

More info why I need a frame presentation time callback:
#89 (comment)

I'm not sure why you qualify Web Animations as a "low-scientific-quality fallback". They should provide identical callback reliability to requestAnimationFrame.

@smfr, we need reliable single-framedrop detection logic (e.g. suddenly know when we see 2 refresh intervals pass instead of 1 refresh interval), to automatically invalidate tests --

Other animation APIs have more callback-timing-jitter than rAF() which is the problem here...

Some tests require instant notification (e.g. change color away from green) immediately upon a framedrop (any divergence away from framerate=Hz), because the scientific researcher is pointing a camera at the screen, with a mandatory requirement of framerate=Hz. Sometimes it's a pursuit camera, like this photo:

TestUFO Ghosting: www.testufo.com/ghosting

This is all HTML5 driven -- one of the invention implementations are at www.testufo.com/ghosting thanks to my super-accurate framedrop detector in requestAnimationFrame()
Research DOI of my paper: http://dx.doi.org/10.13140/2.1.4950.8164

This inexpensive DIY moving-camera rig outperformed a $30,000 laboratory piece of equipment, thanks to the temporal test pattern invented by me (but requires mandatory perfect framerate=Hz during the photograph, and instant invalidation upon framedrop)

This DIY rig is now peer reviewed by National Institute of Standards and Technology (see my research paper that is co-authored by a NIST.gov researcher), and consequently is used by many researchers worldwide.

If the photograph contains anything that's not a green shade ("READY") it means something happened (e.g. a frame drop shows as an orange or red), and the photograph can be thrown away, and the scientific test restarted.

Now over 500 content creators (combined audience: 100 million -- RTINGS 9M, LinusTechTips 14M) use this test, because I've made it so inexpensive for them.

Another example more applicable to consumers is TestUFO is www.testufo.com/frameskipping Which is used at a lesser scale by display testers to see if there are bugs in a display that is frameskipping (e.g. early ViewSonic XG2530s), or when overclocking a display. This is a popular test at the Overclock-NET forums and also requires a camera.

There are other tests that require scientific-reliability framedrop detection.

So different use cases are covered by different audiences (end users thru researchers)

The problem is at first impression, the callback timing data is more opaque.

Although single-framedrop detection is undocumented (it is not part of any web specification), it is relatively reliable in rAF() but not reliable in other animation APIs.

TestUFO works in all kinds of browsers (smart TV browsers, streaming-device browsers, mobile browsers, and PC browsers), many of which do not currently support other animation APIs but reliably supports rAF(). It will take a long time to discontinue rAF(). And TestUFO is a "educate yourself in a single link" (e.g. demos like www.testufo.com/persistence are hugely self-educational) without needing to install an executable application. So we're very wedded to using browser architectures for mass appeal and widespread flexibility in all kinds of odd-equipment odd-testing situations --

Scientifically-reliable framedrop detection is a mandatory part of TestUFO; and increased timer jitter (Meltdown/Spectre or GPU rendering or different precision than rAF) can make this harder, especially during ultratiny refresh intervals;

You can see some hints of the algorithm in action at https://www.testufo.com/animation-time-graph -- disrupt it by doing things like window resizes, simultaneous tabs, etc, and see the yellow/red spikes.

Perhaps in time the APIs will improve; but right now it's still a low-scientific-quality, because of various factors; many other animation APIs don't provide sufficiently-reliable callback-interval timing stream to correctly detect refresh rate & use the data to detect framedrops (e.g. missed refresh cycles). Either there is more timing jitter, or it is algorithmically unsynchronized (e.g. half frame rate, third frame rate) with less consistency between web browsers.

As refresh rates goes up, the timing precision required goes up, and rAF() stayed precise while others animation APIs (e.g. WebGL) jitters too much (e.g. 3D rendering pipelining effects, and 3D graphics is slower than 2D, etc), given the emergence of 120Hz-500Hz displays. This is not the only issue though;

Once new animation APIs become available, I will test them more fully for low-jitter suitable for realtime data analysis to detect framedrops (e.g. detecting refresh rate, detecting whether animation is sync'd to refresh rate, and also detecting double-interval and triple-interval events)

Also in our old 120Hz browser tests from almost 10 years ago, we discovered we could still accurately heuristically detect framedrops even if the animation rendering jittered massively from 0% to 80% of a refresh cycle interval (e.g. frame took 0.8/60sec to render on a 60Hz display).

The margins are really tight at 240Hz-500Hz+ (and 1000Hz displays in the lab already -- I helped Microsoft add a registry tweak to Windows Insider build to unlock 1000Hz display support) and rAF() succeeds where other animation APIs fail. TestUFO runs with correct refresh rate detection at 500fps 500Hz (on the new 500Hz display) for all tests on modern AMD and NVIDIA GPUs.

If animation frames slew all over the place without clear interval-doublings, then framedrop detection logic fails. 500Hz means 2ms per refresh cycle, and detecting difference between 2ms (normal) and 4ms callback interval (framedropped) in the midst of timer jitter, is much harder. Or say, lower-end GPUs (streaming sticks, Fire TV, previous model Apple TV, etc) that takes a lot of CPU to run certain TestUFO tests -- so this problem is still applicable at 60Hz too;

I will need to retest Animation API to see if least some browsers now has sufficient callback-interval precision for non-rAF() animation APIs accurate enough for needs. Running on underpowered Intel GPUs on a 4K display, is a lot less reliable in other APIs than rAF() -- e.g. budget laptops or chromebooks connected to a 4K TV for testing. Or even running in the built-in SmartTV browser or on a streaming box (Fire TV, Apple TV, etc), or even Android phone connected by USB-C to HDMI in a "use limited equipment at hand' situation worldwide. I already prevent the green READY from showing up on devices that have insufficient callback-timing precisions -- forcing scientific-quality users to use a different higher-performing device;

I will likely run a new pass of timer-jitter accuracy tests on the various callback accuracies of other non-rAF() APIs, but right now, rAF() is the gold standard for low-timing-jitter that is easy to heuristically detect real framedrops (e.g. very clear sudden double-intervals, and such effects).

Questions

1. Is there any data or any new non-rAF animation timing API suitable for single framedrop detection that is as reliable as the logic I've developed for rAF()?
2. If not, is there still an opportunity to standardize framedrop-detection capability more officially?

Perhaps this is an opportunity to standardize callback timing data, to provide sufficient enough information to distinguish performance-throttling versus real-refresh-rates, plus also the ability to detect framedrops (e.g. 59fps at 60Hz, or 239fps at 240Hz) during fixed-Hz fixed-framerate scientific tests.

Also relevant (for those wondering why TestUFO has been accepted by high-Hz gaming display testers as being superior to other motion tests):
Making Of: Why Are TestUFO Display Motion Tests 960 Pixels Per Second?

Apple wants to advertise 120Hz more and I want to help Apple sell 120Hz displays -- so working in finding ways to help me gain access to newer web standards (& framerate=Hz animations in browsers). So it's in Apple's economic interests to find ways to improve Web Animation API to be a superior replacement to requestAnimationFrame() in all the necessary checkboxes for TestUFO needs.... no?

[mdrejhon]

Also:

Another reason is we automatically use multiple divisors of Hz for many educational tests. So on a 165 Hz monitor, some animations will play at exactly half 165Hz (87.5fps) or exactly third of 165 Hz (55fps), to compare against full frame rate, e.g. https://www.testufo.com/blackframes#count=3&bonusufo=1&equalizer=1&background=000000&multistrobe=1&pps=960
which demonstrates that pixel visibility time is directly proportional to display motion blur (click it -- it's REALLY educational, especially on a 120Hz Chrome browser).

Remember, TestUFO has over 30 tests (selectable at upper right corner) with millions of customizable parameters for each (totalled).

So to cherrypick one example (more education-related rather than researcher-related):

Web Animation API has no easy method to port this specific test -- www.testufo.com/blackframes (or a fancy configured version like testufo.com/blackframes#count=3&bonusufo=1&equalizer=1&background=000000&multistrobe=1&pps=960 -- every combination is fully bookmarkable and shareable, which is a neat feature of TestUFO for education's sake) -- and accurate framedrop-free detection, since the BFI animation erratically flickers if there's even just 1 framedrop, so some motion algorithms need perfect framerate=Hz to be properly educational.

By utilizing my framedrop detector I use for requestAnimationFrame() I can tell whether the frame rate deviates from perfect framerate=Hz, I can instantly tell the user to close apps/tabs until the animation runs again at full frame rate -- try it and try forcing stutters. You will see the bottom message change into orange or red, telling users.

Also, we have held off the mobile-compatible version of TestUFO because of Apple's lack of ability to provide an acceptable route for TestUFO.

The lack of a frame-presented callback event in Web Animation API is highly problematic for a lot of needs of TestUFO, and eventually popup messages at TestUFO will soon display a globally blacklisted-warning for all Safari browsers (never shows a green "READY"), once 120Hz Apple devices are more popular than 60Hz Apple devices (in the next 5 years or so).

We urgently need to fix the Web Animation API limitation of a lack of a reliable at-least-millisecond-accurate frame-presentation callback event (microsecond like Chrome is preferred, but we will settle for millisecond accuracy).

Of some of the 30 TestUFO tests, has some interesting requirements (for science, for research, etc):

• Some animations need exact integer pixel steps
• Some animations need to automatically slow down (for scientific reasons) for every dropped frame
• Some animations need to stay the same physical speed independent of Hz
• Some animations need to add intentional single-frame stutter (e.g. the selectable "Stutter" setting of the TestUFO DOTA demo)

Since TestUFO educates people on display motion physics (including VRR simulation, stutter simulation, software-simulation of a CRT tube, etc), requires perfect single-frame control and self-monitoring.

Auto-self-animating APIs (e.g. CSS animations) are unsuitable for more than half of the selectable TestUFO tests because of their scientific/education opaqueness.

If Web Animation API adds a frame callback, we are able to do the rest.

Because of the above, it is still not yet possible to deprecate requestAnimationFrame() for our needs.

Minor revision to Web Animations API is needed; what would be the best Apple-sanctioned channel to submit the necessary changes to Web Animations API in order to be able to deprecate requestAnimationFrame()? (120fps 120Hz Apple device support + realtime on-the-fly per-frame timestamp callbacks for use for framedrop detection algorithms)