Perceptual speed index not working well for animated content

[wibblymat]

I've written an app that, on first load, shows a full screen video that is being fed by getUserMedia.

If I stick something over my camera I get a good score for perceptual speed index (1.3s), whereas if I run LH while the camera is uncovered and I'm waving my hands about, I get bad scores (4.7s is my worst)

Clearly the test is not measuring anything useful in this case and I wonder if it can be improved.

[ebidel]

Is this a flaw of the SI metric? I've seen similar issues on sites that have a continuous animation in the background (or a video). The I/O 2016 is an example where SI gets pushed out b/c the animation keeps changing. WPT never sees the page as "visually complete".

[wibblymat]

Yes, I think it's a flaw in the module that is being used. I think it screenshots the page at intervals and waits for the difference between consecutive screenshots to be small.

[patrickhulce]

Yes this is a critical flaw of all metrics (SI and PSI) using a single target frame as the "ideal" or "idle" state and judging progress toward that frame. There are some other critical problems with PSI discussed over in speedline like paulirish/speedline#52 and paulirish/speedline#50. Plenty of work to do here :)

[ebidel]

Should we just close this issue? The problem lies in the metric and not LH :)

[brendankenny]

I know @patrickhulce wants to keep investigating improvements, though he can comment on timescale for that. It would be nice to have a tracking bug over here, though it may make sense to open a new one with clearer title and summary of current state and what could be improved.

[patrickhulce]

Yeah I'm not sure if any immediate progress will be made on this specific problem though since speed index is so well established as a thing at this point, at best we'll come up with a variant in a new metric. It might be fine to leave this open to coalesce anyone filing issues like #2607 :)

[brendankenny]

Should we just close this issue? The problem lies in the metric and not LH :)

SG :)

It would be nice to have a tracking bug over here

Sounds terrible, who writes things like this.