-
Notifications
You must be signed in to change notification settings - Fork 101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Performance Benchmarks for WordPress 6.7 #1572
Comments
WP 6.7 Beta 1 Performance Benchmark Report: I ran benchmarks comparing WordPress 6.7 Beta 1 and the latest released 6.6 branch (6.6.2). I took measurements with the twentytwentyone (TT1) theme using en_US locale, both with object caching disabled and enabled, then again with twentytwentyfour (TT4), This version of WordPress shows a regression in server rendering time from the previous release for TT4. With object caching disabled, Classic themes (TT1) show a ~3% regression (0.35ms slower) in Here are the results of the test runs: Caching Disabled: Caching Enabled: cc. @felixarntz @joemcgill @adamsilverstein @westonruter @swissspidy |
Thanks @mukeshpanchal27. Repeating what I said in our weekly performance team chat earlier today:
I started trying to investigate the likely cause by looking at the WordPress Code Vitals dashboard and realized that the last commit recorded to that dashboard seems to be this one from five months ago 😱. It's likely that the following commit, r58165, likely broke these. Fortunately, the Gutenberg tests are still being recorded, and it looks like this commit is worth investigating further. It was synced to WordPress in r58936. Here's the performance run from that commit in WP-dev. |
I've opened https://core.trac.wordpress.org/ticket/62153 to fix the dashboard issue. |
@joemcgill The code https://github.com/WordPress/wordpress-develop/blob/trunk/src/wp-includes/class-wp-theme-json.php#L3239-L3251 is too expensive see the profiling result. I removed the code and did the comparison. Server-timing
🔴 It clear shows that the code introduce ~3% regression |
JFYI, I re-open https://core.trac.wordpress.org/ticket/61858 so editor team also can check it. |
Opened PR WordPress/wordpress-develop#7486 that solve the regression. Waiting for the feedback. |
Since we now reinstated the Code Vitals dashboard, I just took a look at it and noticed https://www.codevitals.run/project/wordpress/home-block-theme-lcpMinusTtfb shows a notable regression that must have happened at some point while it was not working. It's interesting that our Web Vitals benchmarks for "LCP - TTFB" didn't show a similar regression. Since the Code Vitals dashboard is still using the TT3 theme (which we should probably update at some point), it would be helpful to run another Web Vitals benchmark comparison using TT3, to check whether we see the regression there too - in which case the problem might be something that mostly affects TT3 for some reason. |
I ran performance benchmarks for Beta 1 with TT3 theme: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11228728635 Web Vitals benchmarks
The benchmarks didn't show the regression that we see in dashboard. |
WP 6.7 Beta 2 Performance Benchmark Report: With object caching disabled,
Here are the results of the test runs: TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11228940062 TT1 (twentytwentyone) benchmarksWeb Vitals
Server-Timing
TT3 (twentytwentythree) benchmarksWeb Vitals
Server-Timing
TT4 (twentytwentyfour) benchmarksWeb Vitals
Server-Timing
🔴 alert for TT4, In WordPress/wordpress-develop#7486 we are checking one approach to solve the background image regression that account for 3-4% regression in block theme. |
Following up on #1572 (comment), I noticed while testing a fix to the Performance Workflow that the LCP-TTFB metric has returned to a more "normal" range but doesn't seem related to any specific code change, so I'm thinking there was a temporary problem with the workflow that has cleared itself up. |
It means we should debug as the difference it ~6ms for normal code commit. |
WP 6.7 Beta 3 Performance Benchmark Report: With object caching disabled,
Here are the results of the test runs: TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11339147727 TT1 (twentytwentyone) benchmarksWeb Vitals
Server-Timing
TT3 (twentytwentythree) benchmarksWeb Vitals
Server-Timing
TT4 (twentytwentyfour) benchmarksWeb Vitals
Server-Timing
Beta 3 shows improvement compared to Beta 2, but there is still a regression in Web Vitals and Server-Timing. |
Just for visibility, I found another background image related PR WordPress/wordpress-develop#6836 (comment) that is responsible for regression. |
Following the set up in the docs, I'm having trouble replicating the figures you quoted here locally. Comparing with and then without those changes, my benchmark stats are unmoved or are neglible in TT5, TT4 and TT3. I know I'm probably doing something wrong though. E.g., TT3,
That the "without change" measurement is higher for me I cannot explain, but I ran it multiple times and it swings +- by 0.5 on each side. |
Again ran the benchmark for TT4 Server-Timing benchmarks
|
Thanks for running it again. I think I still need some clarification on how to interpret these results. Based on the tables, the server load time increases when the changeset in question is reverted. Unless a higher number is better 😅 ? My assumption is that a lower server load time number is preferable. Anyway, I'll keep testing and report back if I find something significant. It's probably worth looking into caching the output of the method as well. Thank you for your help! |
I've run tests both locally and on CI and cannot find any conclusive evidence that WordPress/wordpress-develop#6836 causes a statistically significant regression in performance. In fact, I'm more confounded than anything. Some reports suggest that trunk is faster than the reverted branch, other times the needle falls in the other direction by between 1-4ms. Example from a complete revert on CI ("After" is the revert): Locally the benchmark tests reveal nothing significant for me as mentioned above. I've been running all variations for a couple of hours and am yet to find any reliable results. Furthermore I added some relative paths to TT4 and tried caching the results of All that said, I do believe we need a caching strategy for these resolved URIs however. I think perhaps caching them in Edit: This will improve things in general perhaps: |
I've also followed the steps outlined in the docs, using a completely fresh site locally for testing. Every test run appears to yield inconsistent results for me. This was the latest set of results for me, running a batch of 10000 requests, with
💯 in the same boat.
+1 |
Have you both checked for TT4? |
Yes. |
Just committed r59256, which shows pretty good improvements for TT4 |
@joemcgill does r59256 require backporting to Gutenberg? Happy to take care of it if you don't have time. |
I believe it does. I was also planning to help out there if needed 👍 Core's action calling |
Took a naive stab at this: WordPress/gutenberg#66155 |
WP 6.7 RC 1 Performance Benchmark Report: With object caching disabled,
Here are the results of the test runs: TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11472667306 TT1 (twentytwentyone) benchmarksWeb Vitals
Server-Timing
TT3 (twentytwentythree) benchmarksWeb Vitals
Server-Timing
TT4 (twentytwentyfour) benchmarksWeb Vitals
Server-Timing
|
WP 6.7 RC 2 Performance Benchmark Report: With object caching disabled,
Here are the results of the test runs: TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11579114106 TT1 (twentytwentyone) benchmarksWeb Vitals
Server-Timing
TT3 (twentytwentythree) benchmarksWeb Vitals
Server-Timing
TT4 (twentytwentyfour) benchmarksWeb Vitals
Server-Timing
RC 2 shows improvement compared to RC 1 in block theme. |
I ran local benchmarks for TT4 on versions 6.6.2 and 6.7 RC2 and saw similar results. Core didn’t introduce any performance-related changes between RC1 and RC2, though the Editor packages were updated in the meantime. Interestingly, the Code Vitals analysis didn’t show any improvement 🤔 |
WP 6.7 RC3 Performance Benchmark Report: With object caching disabled,
Here are the results of the test runs: TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11696995231 TT1 (twentytwentyone) benchmarksWeb Vitals
Server-Timing
TT3 (twentytwentythree) benchmarksWeb Vitals
Server-Timing
TT4 (twentytwentyfour) benchmarksWeb Vitals
Server-Timing
RC3 shows slight regression compared to RC2 in block theme. |
WP 6.7 Performance Benchmark Report: With object caching disabled,
Here are the results of the test runs: TT1: https://github.com/mukeshpanchal27/compare-wp-performance/actions/runs/11931170428 TT1 (twentytwentyone) benchmarksWeb Vitals
Server-Timing
TT3 (twentytwentythree) benchmarksWeb Vitals
Server-Timing
TT4 (twentytwentyfour) benchmarksWeb Vitals
Server-Timing
|
As we enter the beta phase for WordPress 6.7, I am opening this issue to gather and track all performance benchmarks. This will help us collaborate more effectively, identify potential regressions, and ensure all performance-related data is centralized in one place.
Feel free to contribute with your benchmarks, results, or any findings related to performance in WordPress 6.7.
The text was updated successfully, but these errors were encountered: