Core performance tests improvements (quick wins)

[swissspidy]

We talked about potential improvements to the core performance test suite in the past, but haven't worked on it much so far.

After we saw some slight performance regressions after 6.7 not only in lab data but also in the field, now is a good time to at least work on some quick wins.

Note

This is split out from #1093, which is a larger ticket with long discussions about multiple aspects of performance testing.

The following items have been discussed with @felixarntz:

Short term:

• Update baseline in tests to 6.7
• Test TT5 theme
• Raise awareness of the dashboard among the team by adding to the handbook and any checklists, so people check it more regularly

Mid term:

• Contribute improvements to the codevitals repo to make it less overwhelming when there are more stats (e.g. a checkbox to hide/show certain fields)
• With such improvements in place, ensure that all possible data is shown in the dashboard (e.g. data for all themes can be viewed)
• Improve dummy content to be more realistic and document it
• Test single post URLs as well.

Bonus:

• We didn't send data to the dashboard for a while (May - October). During this time, and probably after the 6.6 release (August), there was a TTFB regression in block themes. Find commit in trunk between August and October to find potential culprit.
• Build some alerting if nothing gets reported or if there is a drastic change in metrics

[joemcgill]

Test TT5 theme

There is https://core.trac.wordpress.org/ticket/62148 open to track this, along with this PR that needs to be reviewed.

[felixarntz]

Thanks @swissspidy for the great summary!

A few additional thoughts:

• Ideas for enhancing the Code Vitals dashboard UI to make it easier to overview and use:
  • Select <=4 key metrics to highlight, with all other ones being shown in a less prominent way (e.g. "advanced" UI, or simply in a different place, or smaller). There are way too many metrics right now all presented equally, but some of them are far less important than others.
    • My first intuition would be to highlight "LCP - TTFB" (client-side performance) and either "TTFB" or "Server-Timing total" (server-side performance).
    • Both of these metrics could be highlighted for the most recent block theme and the most recent classic theme, which would give us a total of 4 key metrics.
  • Instead of the "block theme" and "classic theme" labeling, use the actual theme names (or slugs).
    • Related: Consider defining a "policy" on which themes get included, e.g. the 3 most recent default block themes and default classic themes (the latter would probably forever be 2021, 2020, and 2019).
  • Separate metrics for different "products" (e.g. the different themes) in different tabs.
• I think we should remove the point about the TTFB regression, as that's an actual performance issue possibly related to 6.7 - I think that would be best in its own issue decoupled from this, as it has nothing to do really with improving the metric collection and dashboard.

[swissspidy]

Instead of the "block theme" and "classic theme" labeling, use the actual theme names (or slugs)

As discussed, we can do this while preserving "backward compatibility" with the data, i.e. we can keep the existing home-classic-theme and home-block-theme tests for the current themes we're testing, and then use the actual theme slugs for the new ones.

[swissspidy]

Improve dummy content to be more realistic and document it

The homepages are currently suboptimal with the theme unit test data. Either it's a full list of the 10 latest posts (which is gigantic) or it's the "Front page" page, which just contains 1 paragraph and nothing else.

Test single post URLs as well.

In the theme unit test data there is a "Block: Image" post with multiple images in them that I think is a reasonable start:

[felixarntz]

@swissspidy

Test single post URLs as well.

In the theme unit test data there is a "Block: Image" post with multiple images in them that I think is a reasonable start:

Agreed. Though I'd love for this post to also get a featured image, as that makes a big impact on performance for single post URLs in most themes - or alternatively, use that post but with the image as the very first block. Currently, I think the image is too far down the page to be in the initial viewport for most mobile devices.

[swissspidy]

Currently, I think the image is too far down the page to be in the initial viewport for most mobile devices.

FWIW we don't actually test mobile at all. All tests run using Desktop Chrome with a 1280x720 viewport. (Though there the image is not in the initial viewport either)