Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove expensive array ops #8838

Merged
merged 6 commits into from
Jul 20, 2021

Conversation

carnage
Copy link
Contributor

@carnage carnage commented Jul 11, 2021

Rebuild/Retarget #6619

@carnage
Copy link
Contributor Author

carnage commented Jul 11, 2021

not sure what's causing the test failures and have run out of time for today any ideas @Ocramius ?

@greg0ire
Copy link
Member

e8119c6 is the first commit that fails the build.

@greg0ire
Copy link
Member

That commit looks slightly different than b69ddf8 though

@greg0ire
Copy link
Member

I think the conflict resolution went wrong, and I have another resolution I am going to push :)

@greg0ire
Copy link
Member

Great! Now let's run git rebase --exec "vendor/bin/phpcbf" origin/2.10.x, hopefully we can get rid of the CS commit :)

@greg0ire
Copy link
Member

Done! Now, let's use https://phpbench.readthedocs.io/en/latest/guides/regression-testing.html to figure out what the improvements are.

@greg0ire
Copy link
Member

greg0ire commented Jul 11, 2021

9c5d676

Noticeable improvement of ProxyInitializationTimeBench::benchInitializationOfAlreadyInitialized*

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak        | mode                 | rstdev          |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 1    | 10  | 6.739mb -0.01%  | 5,115.822μs -0.03%   | ±0.67% -18.65%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 1    | 10  | 6.752mb -0.01%  | 5,194.591μs +0.22%   | ±0.88% -5.96%   |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 1    | 10  | 6.797mb -0.01%  | 5,340.190μs +1.28%   | ±0.98% -16.21%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 1    | 10  | 7.126mb -0.01%  | 7,413.008μs -0.34%   | ±0.28% -17.83%  |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 1    | 10  | 18.880mb -0.00% | 80,082.712μs +0.12%  | ±0.49% +139.57% |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 26.541mb 0.00%  | 81,968.178μs +0.91%  | ±0.74% +21.70%  |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 1    | 10  | 68.928mb -0.00% | 199,683.830μs +1.28% | ±0.56% +7.75%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 1    | 10  | 7.184mb -0.01%  | 2,552.916μs +0.28%   | ±1.31% +566.11% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 1    | 10  | 7.328mb -0.01%  | 3,134.973μs -0.12%   | ±0.65% +0.72%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 1    | 10  | 7.205mb -0.01%  | 2,638.411μs -0.22%   | ±0.96% +102.90% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 1    | 10  | 7.404mb -0.01%  | 3,648.041μs -0.03%   | ±0.33% -18.88%  |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 32.582mb 0.00%  | 77,107.706μs +0.01%  | ±0.37% -46.15%  |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 1    | 10  | 15.232mb 0.00%  | 24,655.822μs -0.49%  | ±0.39% -18.99%  |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 1    | 10  | 18.733mb 0.00%  | 55,626.168μs -0.14%  | ±0.62% -57.14%  |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 1    | 10  | 13.005mb -0.48% | 47,061.945μs -0.99%  | ±1.24% -19.01%  |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 1    | 10  | 15.113mb 0.00%  | 38,293.796μs -1.12%  | ±1.40% -32.27%  |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 1    | 10  | 63.384mb -0.00% | 234,303.092μs -0.34% | ±0.43% -70.02%  |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 1    | 10  | 5.767mb -0.01%  | 1,023.301μs -1.06%   | ±0.57% -55.46%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 1    | 10  | 6.044mb 0.00%   | 2,068.380μs -2.54%   | ±0.72% -50.30%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 1    | 10  | 6.014mb 0.00%   | 1,414.781μs -1.41%   | ±0.47% -81.60%  |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 1    | 10  | 17.812mb 0.00%  | 35,566.065μs -0.95%  | ±0.36% -71.33%  |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 1    | 10  | 17.812mb 0.00%  | 6,391.380μs -0.21%   | ±0.27% -65.24%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 1    | 10  | 17.812mb 0.00%  | 425.978μs -12.16%    | ±0.85% -35.99%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 1    | 10  | 17.812mb 0.00%  | 366.155μs -4.76%     | ±1.85% -23.81%  |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 1    | 10  | 5.758mb 0.00%   | 84,365.487μs +0.02%  | ±0.33% -27.68%  |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 1    | 10  | 5.604mb 0.00%   | 49,684.746μs -0.15%  | ±0.42% -5.86%   |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+

8b3d584

Noticeable degradation of ProxyInitializationTimeBench::benchInitializationOfAlreadyInitialized*

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak        | mode                 | rstdev          |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 1    | 10  | 6.738mb -0.00%  | 5,106.507μs -0.18%   | ±1.19% +77.78%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 1    | 10  | 6.751mb -0.00%  | 5,207.022μs +0.24%   | ±0.36% -59.40%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 1    | 10  | 6.797mb -0.00%  | 5,283.178μs -1.07%   | ±0.50% -48.94%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 1    | 10  | 7.125mb -0.00%  | 7,462.454μs +0.67%   | ±0.31% +11.05%  |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 1    | 10  | 18.880mb -0.00% | 80,232.828μs +0.19%  | ±0.42% -14.46%  |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 26.541mb -0.00% | 81,757.638μs -0.26%  | ±0.47% -36.43%  |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 1    | 10  | 68.927mb -0.00% | 199,056.382μs -0.31% | ±0.81% +44.02%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 1    | 10  | 7.183mb -0.00%  | 2,565.497μs +0.49%   | ±1.23% -6.01%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 1    | 10  | 7.328mb -0.00%  | 3,141.646μs +0.21%   | ±0.66% +1.37%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 1    | 10  | 7.205mb -0.00%  | 2,655.295μs +0.64%   | ±0.63% -34.65%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 1    | 10  | 7.403mb -0.00%  | 3,672.992μs +0.68%   | ±0.72% +116.48% |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 32.582mb -0.00% | 76,558.055μs -0.71%  | ±0.44% +20.29%  |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 1    | 10  | 15.232mb -0.00% | 24,560.401μs -0.39%  | ±0.22% -42.64%  |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 1    | 10  | 18.733mb -0.00% | 55,186.728μs -0.79%  | ±0.58% -7.13%   |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 1    | 10  | 12.943mb -0.00% | 46,791.178μs -0.58%  | ±0.37% -69.88%  |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 1    | 10  | 15.113mb -0.00% | 38,167.155μs -0.33%  | ±0.23% -83.29%  |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 1    | 10  | 63.383mb -0.00% | 232,607.773μs -0.72% | ±0.46% +8.84%   |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 1    | 10  | 5.767mb -0.00%  | 1,021.010μs -0.22%   | ±0.54% -5.30%   |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 1    | 10  | 6.044mb -0.00%  | 2,064.454μs -0.19%   | ±0.43% -39.88%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 1    | 10  | 6.014mb -0.00%  | 1,410.652μs -0.29%   | ±0.35% -26.38%  |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 1    | 10  | 17.812mb -0.00% | 35,850.886μs +0.80%  | ±0.55% +55.02%  |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 1    | 10  | 17.812mb -0.00% | 6,431.714μs +0.63%   | ±0.62% +132.93% |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 1    | 10  | 17.812mb -0.00% | 481.949μs +13.14%    | ±1.27% +48.71%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 1    | 10  | 17.812mb -0.00% | 390.591μs +6.67%     | ±1.62% -12.51%  |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 1    | 10  | 5.758mb -0.00%  | 84,596.722μs +0.27%  | ±0.37% +10.32%  |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 1    | 10  | 5.604mb -0.00%  | 49,692.123μs +0.01%  | ±0.69% +64.01%  |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+

f9b9a14

No noticeable improvement or degradation

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak        | mode                 | rstdev          |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 1    | 10  | 6.738mb -0.00%  | 5,110.744μs +0.08%   | ±0.46% -61.48%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 1    | 10  | 6.751mb -0.00%  | 5,188.926μs -0.35%   | ±0.50% +40.01%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 1    | 10  | 6.797mb -0.00%  | 5,262.683μs -0.39%   | ±0.22% -56.14%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 1    | 10  | 7.125mb -0.00%  | 7,431.362μs -0.42%   | ±0.31% +0.84%   |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 1    | 10  | 18.880mb -0.00% | 79,529.458μs -0.88%  | ±0.55% +30.87%  |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 26.541mb -0.00% | 81,405.018μs -0.43%  | ±0.23% -50.12%  |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 1    | 10  | 68.927mb -0.00% | 196,991.591μs -1.04% | ±0.82% +1.62%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 1    | 10  | 7.183mb -0.00%  | 2,541.078μs -0.95%   | ±0.49% -60.08%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 1    | 10  | 7.328mb -0.00%  | 3,125.789μs -0.50%   | ±1.03% +54.99%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 1    | 10  | 7.205mb -0.00%  | 2,642.890μs -0.47%   | ±0.65% +4.25%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 1    | 10  | 7.403mb -0.00%  | 3,629.010μs -1.20%   | ±0.85% +16.76%  |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 32.582mb -0.00% | 76,688.736μs +0.17%  | ±0.14% -67.22%  |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 1    | 10  | 15.232mb -0.00% | 24,518.926μs -0.17%  | ±0.31% +39.83%  |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 1    | 10  | 18.733mb -0.00% | 55,143.658μs -0.08%  | ±0.48% -16.64%  |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 1    | 10  | 12.943mb -0.00% | 46,647.078μs -0.31%  | ±0.36% -3.77%   |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 1    | 10  | 15.113mb -0.00% | 38,204.609μs +0.10%  | ±0.27% +13.54%  |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 1    | 10  | 63.383mb -0.00% | 233,822.080μs +0.52% | ±1.32% +184.03% |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 1    | 10  | 5.767mb -0.00%  | 1,026.667μs +0.55%   | ±1.40% +158.77% |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 1    | 10  | 6.044mb -0.00%  | 2,097.528μs +1.60%   | ±1.29% +197.78% |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 1    | 10  | 6.014mb -0.00%  | 1,413.159μs +0.18%   | ±0.74% +114.11% |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 1    | 10  | 17.812mb -0.00% | 35,721.233μs -0.36%  | ±0.31% -43.83%  |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 1    | 10  | 17.812mb -0.00% | 6,423.515μs -0.13%   | ±0.35% -43.57%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 1    | 10  | 17.812mb -0.00% | 488.350μs +1.33%     | ±1.34% +5.48%   |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 1    | 10  | 17.812mb -0.00% | 385.176μs -1.39%     | ±0.82% -49.24%  |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 1    | 10  | 5.758mb -0.00%  | 84,103.632μs -0.58%  | ±0.57% +54.27%  |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 1    | 10  | 5.604mb -0.00%  | 49,626.695μs -0.13%  | ±0.37% -45.64%  |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+

102484d

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+----------------+----------------------+-----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak       | mode                 | rstdev          |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+----------------+----------------------+-----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 1    | 10  | 6.738mb 0.00%  | 5,096.881μs -0.27%   | ±0.33% -27.67%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 1    | 10  | 6.751mb 0.00%  | 5,190.446μs +0.03%   | ±1.11% +121.71% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 1    | 10  | 6.797mb 0.00%  | 5,254.528μs -0.15%   | ±0.65% +197.00% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 1    | 10  | 7.125mb 0.00%  | 7,418.078μs -0.18%   | ±0.39% +23.54%  |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 1    | 10  | 18.880mb 0.00% | 79,539.616μs +0.01%  | ±0.23% -58.22%  |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 26.541mb 0.00% | 81,162.505μs -0.30%  | ±0.49% +110.59% |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 1    | 10  | 68.927mb 0.00% | 196,880.886μs -0.06% | ±0.76% -7.37%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 1    | 10  | 7.183mb 0.00%  | 2,537.479μs -0.14%   | ±1.31% +166.74% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 1    | 10  | 7.327mb 0.00%  | 3,134.648μs +0.28%   | ±1.08% +4.85%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 1    | 10  | 7.205mb 0.00%  | 2,643.998μs +0.04%   | ±0.87% +32.83%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 1    | 10  | 7.403mb 0.00%  | 3,632.252μs +0.09%   | ±0.30% -64.51%  |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 32.582mb 0.00% | 76,506.018μs -0.24%  | ±0.46% +215.90% |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 1    | 10  | 15.232mb 0.00% | 24,620.178μs +0.41%  | ±0.60% +94.15%  |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 1    | 10  | 18.733mb 0.00% | 55,379.004μs +0.43%  | ±0.47% -2.55%   |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 1    | 10  | 12.943mb 0.00% | 46,485.221μs -0.35%  | ±0.33% -8.98%   |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 1    | 10  | 15.113mb 0.00% | 38,010.579μs -0.51%  | ±0.55% +107.19% |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 1    | 10  | 63.383mb 0.00% | 232,184.022μs -0.70% | ±0.52% -60.89%  |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 1    | 10  | 5.766mb 0.00%  | 1,015.771μs -1.06%   | ±0.78% -43.94%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 1    | 10  | 6.043mb 0.00%  | 2,057.329μs -1.92%   | ±0.29% -77.58%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 1    | 10  | 6.014mb 0.00%  | 1,406.984μs -0.44%   | ±0.46% -38.49%  |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 1    | 10  | 17.812mb 0.00% | 35,632.579μs -0.25%  | ±0.36% +15.71%  |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 1    | 10  | 17.812mb 0.00% | 6,388.849μs -0.54%   | ±0.43% +21.44%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 1    | 10  | 17.812mb 0.00% | 484.215μs -0.85%     | ±1.27% -4.83%   |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 1    | 10  | 17.812mb 0.00% | 384.697μs -0.12%     | ±1.30% +58.28%  |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 1    | 10  | 5.758mb 0.00%  | 84,479.346μs +0.45%  | ±0.30% -48.00%  |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 1    | 10  | 5.604mb 0.00%  | 49,450.364μs -0.36%  | ±0.09% -75.28%  |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+----------------+----------------------+-----------------+

91de49e

No noticeable imporvement or degradation

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+----------------+----------------------+-----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak       | mode                 | rstdev          |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+----------------+----------------------+-----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 1    | 10  | 6.738mb 0.00%  | 5,107.008μs +0.20%   | ±0.75% +126.35% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 1    | 10  | 6.751mb 0.00%  | 5,184.382μs -0.12%   | ±0.48% -56.97%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 1    | 10  | 6.797mb 0.00%  | 5,265.767μs +0.21%   | ±0.78% +19.38%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 1    | 10  | 7.125mb 0.00%  | 7,468.016μs +0.67%   | ±0.49% +26.87%  |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 1    | 10  | 18.880mb 0.00% | 79,451.603μs -0.11%  | ±0.31% +34.11%  |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 26.541mb 0.00% | 81,549.517μs +0.48%  | ±0.30% -38.39%  |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 1    | 10  | 68.927mb 0.00% | 198,607.294μs +0.88% | ±0.82% +7.69%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 1    | 10  | 7.183mb 0.00%  | 2,543.902μs +0.25%   | ±0.54% -59.02%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 1    | 10  | 7.327mb 0.00%  | 3,135.335μs +0.02%   | ±0.58% -46.38%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 1    | 10  | 7.205mb 0.00%  | 2,643.714μs -0.01%   | ±0.38% -56.51%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 1    | 10  | 7.403mb 0.00%  | 3,648.654μs +0.45%   | ±0.65% +115.40% |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 32.582mb 0.00% | 76,649.329μs +0.19%  | ±0.23% -48.88%  |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 1    | 10  | 15.232mb 0.00% | 24,490.155μs -0.53%  | ±0.58% -4.33%   |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 1    | 10  | 18.733mb 0.00% | 55,326.568μs -0.09%  | ±1.04% +120.16% |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 1    | 10  | 12.943mb 0.00% | 46,621.123μs +0.29%  | ±0.85% +160.35% |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 1    | 10  | 15.113mb 0.00% | 38,021.270μs +0.03%  | ±0.51% -6.59%   |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 1    | 10  | 63.383mb 0.00% | 231,592.219μs -0.25% | ±0.34% -34.62%  |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 1    | 10  | 5.766mb 0.00%  | 1,020.225μs +0.44%   | ±1.39% +77.72%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 1    | 10  | 6.043mb 0.00%  | 2,072.808μs +0.75%   | ±0.76% +163.56% |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 1    | 10  | 6.014mb 0.00%  | 1,410.703μs +0.26%   | ±1.41% +209.06% |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 1    | 10  | 17.812mb 0.00% | 35,729.348μs +0.27%  | ±0.51% +42.14%  |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 1    | 10  | 17.812mb 0.00% | 6,390.973μs +0.03%   | ±0.12% -71.68%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 1    | 10  | 17.812mb 0.00% | 488.720μs +0.93%     | ±1.78% +39.48%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 1    | 10  | 17.812mb 0.00% | 390.223μs +1.44%     | ±0.83% -36.40%  |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 1    | 10  | 5.758mb 0.00%  | 83,853.607μs -0.74%  | ±0.60% +101.16% |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 1    | 10  | 5.604mb 0.00%  | 49,515.630μs +0.13%  | ±0.16% +69.27%  |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+----------------+----------------------+-----------------+

74ffc25

Small performance improvements on SimpleQueryPartialObjectHydrationPerformanceBench::benchHydration, and ProxyInitializationTimeBench::benchInitializationOfAlreadyInitialized*

Overall

Small performance improvement on ProxyInitializationTimeBench::benchInitializationOfAlreadyInitializedCmsUsers and only there.

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak        | mode                 | rstdev          |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 1    | 10  | 6.739mb -0.01%  | 5,110.376μs +0.20%   | ±0.31% +23.50%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 1    | 10  | 6.752mb -0.01%  | 5,187.644μs +0.37%   | ±0.29% -40.97%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 1    | 10  | 6.797mb -0.01%  | 5,259.317μs +0.23%   | ±0.73% +137.77% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 1    | 10  | 7.126mb -0.01%  | 7,415.384μs -0.47%   | ±1.13% +70.42%  |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 1    | 10  | 18.880mb -0.00% | 80,009.622μs +0.06%  | ±0.25% -25.04%  |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 26.541mb -0.00% | 80,639.955μs -0.13%  | ±0.57% +10.60%  |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 1    | 10  | 68.928mb -0.00% | 198,168.276μs +0.49% | ±1.29% +76.94%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 1    | 10  | 7.184mb -0.01%  | 2,548.196μs +0.30%   | ±0.52% +38.65%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 1    | 10  | 7.328mb -0.01%  | 3,130.728μs +0.23%   | ±0.61% +61.94%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 1    | 10  | 7.205mb -0.01%  | 2,641.360μs +0.42%   | ±0.43% +36.35%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 1    | 10  | 7.404mb -0.01%  | 3,625.818μs -0.13%   | ±0.57% +349.67% |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 32.582mb -0.00% | 76,519.650μs +0.05%  | ±0.27% -31.96%  |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 1    | 10  | 15.232mb -0.00% | 24,476.493μs -0.09%  | ±0.51% +49.15%  |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 1    | 10  | 18.733mb -0.00% | 55,231.389μs +0.56%  | ±1.10% +23.55%  |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 1    | 10  | 13.005mb -0.48% | 46,687.313μs +0.36%  | ±0.34% -22.79%  |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 1    | 10  | 15.113mb -0.00% | 38,083.286μs +0.17%  | ±0.83% +86.97%  |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 1    | 10  | 63.384mb -0.00% | 233,439.331μs +0.86% | ±0.38% -18.15%  |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 1    | 10  | 5.767mb -0.01%  | 1,011.593μs -0.83%   | ±0.55% -51.10%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 1    | 10  | 6.044mb -0.00%  | 2,063.795μs +0.11%   | ±0.33% -76.18%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 1    | 10  | 6.014mb -0.00%  | 1,411.781μs -0.06%   | ±0.59% -68.14%  |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 1    | 10  | 17.812mb -0.00% | 35,666.444μs +0.05%  | ±0.36% +24.77%  |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 1    | 10  | 17.812mb -0.00% | 6,408.143μs +0.32%   | ±0.38% -4.28%   |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 1    | 10  | 17.812mb -0.00% | 474.370μs -1.77%     | ±1.00% +5.39%   |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 1    | 10  | 17.812mb -0.00% | 385.129μs +0.21%     | ±1.32% +53.35%  |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 1    | 10  | 5.758mb -0.00%  | 84,325.149μs -0.31%  | ±0.35% -29.02%  |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 1    | 10  | 5.604mb -0.00%  | 49,452.609μs +0.01%  | ±0.06% -54.32%  |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+

It's possible that the code changed in this PR is not run during the benchmarks, all we know is that it does not make the scenarios tested here worse.

After dropping 8b3d584

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak        | mode                 | rstdev          |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 1    | 10  | 6.739mb -0.01%  | 5,131.260μs +0.61%   | ±0.83% +235.20% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 1    | 10  | 6.752mb -0.01%  | 5,183.509μs +0.29%   | ±0.44% -10.19%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 1    | 10  | 6.797mb -0.01%  | 5,279.636μs +0.61%   | ±0.57% +84.58%  |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 1    | 10  | 7.126mb -0.01%  | 7,435.722μs -0.20%   | ±0.49% -25.88%  |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 1    | 10  | 18.880mb -0.00% | 80,038.415μs +0.09%  | ±0.22% -33.97%  |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 26.541mb -0.00% | 80,933.386μs +0.24%  | ±0.69% +33.82%  |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 1    | 10  | 68.928mb -0.00% | 198,066.558μs +0.44% | ±0.78% +6.61%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 1    | 10  | 7.184mb -0.01%  | 2,546.945μs +0.25%   | ±1.28% +243.69% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 1    | 10  | 7.328mb -0.01%  | 3,118.687μs -0.16%   | ±0.41% +9.01%   |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 1    | 10  | 7.205mb -0.01%  | 2,638.671μs +0.32%   | ±0.43% +34.20%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 1    | 10  | 7.404mb -0.01%  | 3,682.282μs +1.43%   | ±0.69% +439.84% |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 1    | 10  | 32.582mb -0.00% | 77,474.481μs +1.30%  | ±0.18% -55.23%  |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 1    | 10  | 15.232mb -0.00% | 24,752.599μs +1.04%  | ±0.48% +39.36%  |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 1    | 10  | 18.733mb -0.00% | 55,230.155μs +0.55%  | ±0.85% -3.96%   |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 1    | 10  | 13.005mb -0.48% | 46,738.521μs +0.47%  | ±0.55% +24.58%  |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 1    | 10  | 15.113mb -0.00% | 38,147.221μs +0.34%  | ±0.77% +73.56%  |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 1    | 10  | 63.384mb -0.00% | 233,703.370μs +0.97% | ±0.59% +27.76%  |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 1    | 10  | 5.767mb -0.01%  | 1,022.008μs +0.19%   | ±0.69% -38.81%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 1    | 10  | 6.044mb -0.00%  | 2,062.890μs +0.06%   | ±0.84% -40.12%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 1    | 10  | 6.014mb -0.00%  | 1,404.321μs -0.59%   | ±0.72% -60.92%  |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 1    | 10  | 17.812mb -0.00% | 35,586.364μs -0.18%  | ±0.45% +57.65%  |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 1    | 10  | 17.812mb -0.00% | 6,381.975μs -0.08%   | ±0.25% -37.64%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 1    | 10  | 17.812mb -0.00% | 424.090μs -12.18%    | ±1.74% +84.40%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 1    | 10  | 17.812mb -0.00% | 385.309μs +0.26%     | ±1.37% +59.37%  |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 1    | 10  | 5.758mb -0.00%  | 83,812.986μs -0.92%  | ±0.56% +12.57%  |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 1    | 10  | 5.604mb -0.00%  | 49,622.370μs +0.35%  | ±0.21% +73.67%  |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+-----------------+

@Ocramius
Copy link
Member

Ocramius commented Jul 11, 2021 via email

@greg0ire
Copy link
Member

Good to know! I'm trying also to figure out if there is a reliable way to know if a given piece of code is executed or not during the benchmark.

@greg0ire
Copy link
Member

greg0ire commented Jul 11, 2021

with an error threshold

Not sure what you mean by that, but I have reduced the retry threshold.

Here are the results with --retry-threshold=2 --iterations=10 --revs=100 --warmup=2:

Overall

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak        | mode                 | rstdev         |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 100  | 10  | 8.207mb -0.01%  | 285.208μs +0.10%     | ±0.72% +42.69% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 100  | 10  | 8.207mb -0.01%  | 317.884μs -0.02%     | ±0.40% -36.01% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 100  | 10  | 8.241mb -0.01%  | 330.984μs +0.01%     | ±0.73% +16.56% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 100  | 10  | 11.853mb -0.00% | 913.761μs -0.94%     | ±0.43% +10.67% |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 100  | 10  | 18.880mb -0.00% | 3.066μs +1.26%       | ±0.78% -13.68% |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 100  | 10  | 26.541mb -0.00% | 1.684μs +1.38%       | ±0.88% -8.47%  |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 100  | 10  | 68.928mb -0.00% | 107,754.712μs +1.20% | ±0.87% +44.97% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 100  | 10  | 7.184mb -0.01%  | 240.362μs +0.35%     | ±1.03% +33.28% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 100  | 10  | 7.328mb -0.01%  | 511.250μs +1.86%     | ±1.20% +39.40% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 100  | 10  | 7.205mb -0.01%  | 275.859μs +0.58%     | ±0.63% -25.69% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 100  | 10  | 7.404mb -0.01%  | 752.064μs -0.11%     | ±0.82% -22.97% |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 100  | 10  | 32.582mb -0.00% | 1.080μs -0.41%       | ±0.62% -32.80% |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 100  | 10  | 15.232mb -0.00% | 0.870μs -0.06%       | ±0.92% -10.81% |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 100  | 10  | 18.733mb -0.00% | 41,758.441μs -0.43%  | ±0.61% +54.80% |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 100  | 10  | 13.005mb -0.48% | 2.627μs +0.51%       | ±0.91% -9.22%  |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 100  | 10  | 15.113mb -0.00% | 1.019μs +0.74%       | ±1.09% -8.76%  |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 100  | 10  | 63.384mb -0.00% | 2.558μs -0.25%       | ±0.95% -5.06%  |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 100  | 10  | 5.767mb -0.01%  | 913.484μs +0.85%     | ±0.46% +2.41%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 100  | 10  | 6.044mb -0.00%  | 90.669μs +0.63%      | ±0.64% -16.16% |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 100  | 10  | 6.014mb -0.00%  | 13.731μs +0.11%      | ±0.68% +17.39% |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 100  | 10  | 17.812mb -0.00% | 276.183μs -2.97%     | ±0.67% -46.20% |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 100  | 10  | 17.812mb -0.00% | 263.353μs -0.59%     | ±0.39% -43.33% |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 100  | 10  | 17.812mb -0.00% | 269.409μs -0.58%     | ±1.02% +27.80% |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 100  | 10  | 17.812mb -0.00% | 263.476μs -0.20%     | ±0.77% +12.91% |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 100  | 10  | 5.758mb -0.00%  | 69,944.310μs +0.24%  | ±0.43% +41.24% |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 100  | 10  | 5.604mb -0.00%  | 42,080.402μs +0.07%  | ±0.13% -78.74% |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+----------------+

After dropping 8b3d584

There no longer seems to be a big benefit in dropping this commit

+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+----------------+
| benchmark                                                 | subject                                             | set | revs | its | mem_peak        | mode                 | rstdev         |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+----------------+
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFixContracts                             | 0   | 100  | 10  | 8.207mb -0.01%  | 287.779μs +1.00%     | ±0.55% +10.60% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertFlexContracts                            | 0   | 100  | 10  | 8.207mb -0.01%  | 319.719μs +0.56%     | ±0.54% -14.20% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertUltraContracts                           | 0   | 100  | 10  | 8.241mb -0.01%  | 331.088μs +0.04%     | ±0.70% +11.96% |
| SingleTableInheritanceInsertPerformanceBench              | benchInsertAllContracts                             | 0   | 100  | 10  | 11.853mb -0.00% | 919.118μs -0.36%     | ±0.65% +66.95% |
| MixedQueryFetchJoinFullObjectHydrationPerformanceBench    | benchHydration                                      | 0   | 100  | 10  | 18.880mb -0.00% | 3.074μs +1.49%       | ±1.03% +13.51% |
| SimpleQueryPartialObjectHydrationPerformanceBench         | benchHydration                                      | 0   | 100  | 10  | 26.541mb -0.00% | 1.638μs -1.41%       | ±0.73% -24.20% |
| SimpleHydrationBench                                      | benchHydration                                      | 0   | 100  | 10  | 68.928mb -0.00% | 107,689.096μs +1.14% | ±0.86% +43.58% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFixContracts                            | 0   | 100  | 10  | 7.184mb -0.01%  | 242.237μs +1.13%     | ±1.15% +49.44% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateFlexContracts                           | 0   | 100  | 10  | 7.328mb -0.01%  | 501.550μs -0.07%     | ±0.82% -4.51%  |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateUltraContracts                          | 0   | 100  | 10  | 7.205mb -0.01%  | 274.045μs -0.08%     | ±0.96% +13.56% |
| SingleTableInheritanceHydrationPerformanceBench           | benchHydrateAllContracts                            | 0   | 100  | 10  | 7.404mb -0.01%  | 756.115μs +0.42%     | ±0.86% -19.03% |
| MixedQueryFetchJoinArrayHydrationPerformanceBench         | benchHydration                                      | 0   | 100  | 10  | 32.582mb -0.00% | 1.071μs -1.21%       | ±0.74% -18.81% |
| SimpleQueryScalarHydrationPerformanceBench                | benchHydration                                      | 0   | 100  | 10  | 15.232mb -0.00% | 0.871μs +0.02%       | ±0.91% -11.01% |
| SimpleInsertPerformanceBench                              | benchHydration                                      | 0   | 100  | 10  | 18.733mb -0.00% | 41,544.122μs -0.94%  | ±0.73% +84.87% |
| MixedQueryFetchJoinPartialObjectHydrationPerformanceBench | benchHydration                                      | 0   | 100  | 10  | 13.005mb -0.48% | 2.668μs +2.08%       | ±0.99% -0.77%  |
| SimpleQueryArrayHydrationPerformanceBench                 | benchHydration                                      | 0   | 100  | 10  | 15.113mb -0.00% | 1.020μs +0.85%       | ±0.59% -50.89% |
| SimpleQueryFullObjectHydrationPerformanceBench            | benchHydration                                      | 0   | 100  | 10  | 63.384mb -0.00% | 2.571μs +0.24%       | ±1.13% +11.93% |
| UnitOfWorkComputeChangesBench                             | benchChangeSetComputation                           | 0   | 100  | 10  | 5.767mb -0.01%  | 913.421μs +0.85%     | ±0.41% -9.72%  |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithInferredParameterType    | 0   | 100  | 10  | 6.044mb -0.00%  | 91.101μs +1.11%      | ±0.51% -32.81% |
| QueryBoundParameterProcessingBench                        | benchExecuteParsedQueryWithDeclaredParameterType    | 0   | 100  | 10  | 6.014mb -0.00%  | 13.797μs +0.59%      | ±0.97% +66.24% |
| ProxyInitializationTimeBench                              | benchCmsUserInitialization                          | 0   | 100  | 10  | 17.812mb -0.00% | 275.785μs -3.11%     | ±0.38% -69.10% |
| ProxyInitializationTimeBench                              | benchCmsEmployeeInitialization                      | 0   | 100  | 10  | 17.812mb -0.00% | 263.367μs -0.59%     | ±0.45% -33.86% |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsUsers     | 0   | 100  | 10  | 17.812mb -0.00% | 270.233μs -0.28%     | ±0.86% +7.67%  |
| ProxyInitializationTimeBench                              | benchInitializationOfAlreadyInitializedCmsEmployees | 0   | 100  | 10  | 17.812mb -0.00% | 264.554μs +0.21%     | ±0.46% -32.42% |
| ProxyInstantiationTimeBench                               | benchCmsUserInstantiation                           | 0   | 100  | 10  | 5.758mb -0.00%  | 69,942.807μs +0.24%  | ±0.38% +23.50% |
| ProxyInstantiationTimeBench                               | benchCmsEmployeeInstantiation                       | 0   | 100  | 10  | 5.604mb -0.00%  | 42,061.578μs +0.02%  | ±0.16% -74.64% |
+-----------------------------------------------------------+-----------------------------------------------------+-----+------+-----+-----------------+----------------------+----------------+

@Ocramius
Copy link
Member

Ocramius commented Jul 11, 2021 via email

@greg0ire
Copy link
Member

Did you try using --tag and --ref to get a comparative result?

Yes, that's exactly what you are looking at :)

  1. there are none
  2. Before I dive into this, it would be great to know of a reliable way to figure out if the benchmark executes the changed pieces of code or not. That being said, doesn't the rstdev demonstrate the benchmark is pretty stable already?

@Ocramius
Copy link
Member

Before I dive into this, it would be great to know of a reliable way to figure out if the benchmark executes the changed pieces of code or not.

Difficult to say: would require running these benchmarks with coverage, which I don't know how to do, especially since they run in forked processes.

That being said, doesn't the rstdev demonstrate the benchmark is pretty stable already?

The STDDEV is higher than the reported increases/losses in performance, hence a bit too imprecise.

If you are in an ubuntu/linux environment, I can endorse http://konkor.github.io/cpufreq/ to have a quick switch to enable/disable turbo boost.

@greg0ire
Copy link
Member

I use Fedora and not Gnome, so I'll pass… had a look at my BIOS settings, and didn't find turbo boost either.

As for the code, I have a way to know if a piece of code is executed. If I use die('hello') in getCommitOrder, it will cause errors. I don't know if the error is guaranteed to happen though, but now we know at least getCommitOrder is covered :P

@greg0ire
Copy link
Member

Ah I know, I'll use sleep(60)

@@ -68,12 +68,16 @@ public function loadCacheEntry(ClassMetadata $metadata, CollectionCacheKey $key,
}

foreach ($entityEntries as $index => $entityEntry) {
$list[$index] = $this->uow->createEntity($entityEntry->class, $entityEntry->resolveAssociationEntries($this->em), self::$hints);
}
$entity = $this->uow->createEntity(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not covered

@@ -170,7 +170,7 @@ public function get(QueryCacheKey $key, ResultSetMapping $rsm, array $hints = []
$assocEntries = $assocRegion->getMultiple($assocKeys);

foreach ($assoc['list'] as $assocIndex => $assocId) {
$assocEntry = is_array($assocEntries) && array_key_exists($assocIndex, $assocEntries) ? $assocEntries[$assocIndex] : null;
$assocEntry = is_array($assocEntries) ? ($assocEntries[$assocIndex] ?? null) : null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not covered

@@ -133,7 +133,7 @@ public function storeCollectionCache(CollectionCacheKey $key, $elements)
// Only preserve ordering if association configured it
if (! (isset($associationMapping['indexBy']) && $associationMapping['indexBy'])) {
// Elements may be an array or a Collection
$elements = array_values(is_array($elements) ? $elements : $elements->getValues());
$elements = array_values($elements instanceof Collection ? $elements->getValues() : $elements);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not covered

Copy link
Member

@greg0ire greg0ire Jul 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered by testFindEntityOneToManyWithCache

return array_values(array_diff_key(
array_combine(array_map('spl_object_id', $this->snapshot), $this->snapshot),
array_combine(array_map('spl_object_id', $collectionItems), $collectionItems)
));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not covered

return array_values(array_diff_key(
array_combine(array_map('spl_object_id', $collectionItems), $collectionItems),
array_combine(array_map('spl_object_id', $this->snapshot), $this->snapshot)
));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not covered

@@ -1249,8 +1249,7 @@ private function executeDeletions(ClassMetadata $class): void
*/
private function getCommitOrder(): array
{
$entityChangeSet = array_merge($this->entityInsertions, $this->entityUpdates, $this->entityDeletions);
$calc = $this->getCommitOrderCalculator();
$calc = $this->getCommitOrderCalculator();
Copy link
Member

@greg0ire greg0ire Jul 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered by

\Doctrine\Performance\Hydration\SingleTableInheritanceInsertPerformanceBench, \Doctrine\Performance\Hydration\SimpleHydrationBench

and

\Doctrine\Performance\Hydration\SingleTableInheritanceHydrationPerformanceBench

$countArray = array_map('count', $this->identityMap);

return array_sum($countArray);
return array_sum(array_map('count', $this->identityMap));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not covered

@greg0ire
Copy link
Member

With all that in mind, I can create a benchmark with the relevant methods, on the relevant commit :)

@greg0ire
Copy link
Member

Here are the results:

+-------------------------------------------------+----------------------------+-----+------+-----+-----------------+----------------------+----------------+
| benchmark                                       | subject                    | set | revs | its | mem_peak        | mode                 | rstdev         |
+-------------------------------------------------+----------------------------+-----+------+-----+-----------------+----------------------+----------------+
| SingleTableInheritanceInsertPerformanceBench    | benchInsertFixContracts    | 0   | 100  | 10  | 8.207mb -0.00%  | 286.974μs +0.92%     | ±0.74% +55.89% |
| SingleTableInheritanceInsertPerformanceBench    | benchInsertFlexContracts   | 0   | 100  | 10  | 8.208mb -0.00%  | 319.154μs -0.27%     | ±0.42% -41.78% |
| SingleTableInheritanceInsertPerformanceBench    | benchInsertUltraContracts  | 0   | 100  | 10  | 8.241mb -0.00%  | 331.079μs -0.37%     | ±0.43% -32.73% |
| SingleTableInheritanceInsertPerformanceBench    | benchInsertAllContracts    | 0   | 100  | 10  | 11.853mb -0.00% | 922.612μs +0.15%     | ±0.63% -49.77% |
| SimpleHydrationBench                            | benchHydration             | 0   | 100  | 10  | 68.928mb -0.00% | 107,287.688μs +0.62% | ±0.98% +51.64% |
| SingleTableInheritanceHydrationPerformanceBench | benchHydrateFixContracts   | 0   | 100  | 10  | 7.184mb -0.00%  | 239.466μs -1.56%     | ±0.69% -23.33% |
| SingleTableInheritanceHydrationPerformanceBench | benchHydrateFlexContracts  | 0   | 100  | 10  | 7.328mb -0.00%  | 508.997μs +0.93%     | ±0.69% -36.71% |
| SingleTableInheritanceHydrationPerformanceBench | benchHydrateUltraContracts | 0   | 100  | 10  | 7.205mb -0.00%  | 275.255μs -0.22%     | ±0.91% +2.21%  |
| SingleTableInheritanceHydrationPerformanceBench | benchHydrateAllContracts   | 0   | 100  | 10  | 7.404mb -0.00%  | 747.720μs +0.14%     | ±1.05% +27.18% |
+-------------------------------------------------+----------------------------+-----+------+-----+-----------------+----------------------+----------------+

I think that improvement is negligible when compared to the fluctuations you can have while executing the rest of the code. Since there is no visible harm done, LGTM


return array_values(array_diff_key(
array_combine(array_map('spl_object_id', $this->snapshot), $this->snapshot),
array_combine(array_map('spl_object_id', $collectionItems), $collectionItems)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spl_object_id is not available on 7.1, but #8837 will introduce a polyfill for it, so let's merge that PR first.

@Ocramius
Copy link
Member

Ocramius commented Jul 11, 2021 via email

@greg0ire
Copy link
Member

You can read more about this here: #8613 (comment)

@beberlei
Copy link
Member

Not a single benchmark executes second level cache code or am i mistaken?

@greg0ire
Copy link
Member

greg0ire commented Jul 20, 2021

@beberlei I think you are correct! I don't know that area of the code well, but adding a die() statement in the constructor of DefaultRegion.php does not cause any issues 😬

That being said, I notice that we have a file called tests/Doctrine/Tests/ORM/Performance/SecondLevelCacheTest.php in the project… apparently it's a phpunit test, not a phpbench test, so maybe we should convert it.

Before this PR

vendor/bin/phpunit --group=performance
PHPUnit 9.5.6 by Sebastian Bergmann and contributors.


Using DB driver Doctrine\DBAL\Driver\PDO\SQLite\Driver
.
testFindEntityWithoutCache
[0.010225] persist 500 countries
[0.244405] find 500 countries (10 times)
--------------------------------------------------
.
testFindEntityWithCache
[0.013658] persist 500 countries
[0.057642] find 500 countries (10 times)
--------------------------------------------------
.
testFindAllEntityWithoutCache
[0.001249] persist 50 countries
[0.029624] find 50 countries (100 times)
--------------------------------------------------
.
testFindAllEntityWithCache
[0.001411] persist 50 countries
[0.035871] find 50 countries (100 times)
--------------------------------------------------
.
testFindEntityOneToManyWithoutCache
[0.016983] persist 15 states and 450 cities
[0.012043] find 15 states and 450 cities (50 times)
--------------------------------------------------
.
testFindEntityOneToManyWithCache
[0.023742] persist 15 states and 450 cities
[0.008751] find 15 states and 450 cities (50 times)
--------------------------------------------------
.
testQueryEntityWithoutCache
[0.016269] persist 500 countries
[0.301352] select 500 countries (100 times)
--------------------------------------------------
.                                                            8 / 8 (100%)
testQueryEntityWithCache
[0.012689] persist 500 countries
[0.240104] select 500 countries (100 times)
--------------------------------------------------


Time: 00:01.124, Memory: 92.00 MB

After this PR

vendor/bin/phpunit --group=performance
PHPUnit 9.5.6 by Sebastian Bergmann and contributors.


Using DB driver Doctrine\DBAL\Driver\PDO\SQLite\Driver
.
testFindEntityWithoutCache
[0.010235] persist 500 countries
[0.247301] find 500 countries (10 times)
--------------------------------------------------
.
testFindEntityWithCache
[0.013622] persist 500 countries
[0.057084] find 500 countries (10 times)
--------------------------------------------------
.
testFindAllEntityWithoutCache
[0.001134] persist 50 countries
[0.029797] find 50 countries (100 times)
--------------------------------------------------
.
testFindAllEntityWithCache
[0.001295] persist 50 countries
[0.035365] find 50 countries (100 times)
--------------------------------------------------
.
testFindEntityOneToManyWithoutCache
[0.016294] persist 15 states and 450 cities
[0.012077] find 15 states and 450 cities (50 times)
--------------------------------------------------
.
testFindEntityOneToManyWithCache
[0.023316] persist 15 states and 450 cities
[0.008795] find 15 states and 450 cities (50 times)
--------------------------------------------------
.
testQueryEntityWithoutCache
[0.015849] persist 500 countries
[0.296401] select 500 countries (100 times)
--------------------------------------------------
.                                                            8 / 8 (100%)
testQueryEntityWithCache
[0.012073] persist 500 countries
[0.238900] select 500 countries (100 times)
--------------------------------------------------


Time: 00:01.116, Memory: 92.00 MB

It seems overall better, but I don't know if this test can be trusted.

@@ -170,7 +170,7 @@ public function get(QueryCacheKey $key, ResultSetMapping $rsm, array $hints = []
$assocEntries = $assocRegion->getMultiple($assocKeys);

foreach ($assoc['list'] as $assocIndex => $assocId) {
$assocEntry = is_array($assocEntries) && array_key_exists($assocIndex, $assocEntries) ? $assocEntries[$assocIndex] : null;
Copy link
Member

@greg0ire greg0ire Jul 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Covered by testFindEntityOneToManyWithCache

@greg0ire
Copy link
Member

in that phpunit test, testFindEntityOneToManyWithCache seems to be the only method relevant to this PR. Here are the corresponding excerpts from the output above:

Before

testFindEntityOneToManyWithCache
[0.023742] persist 15 states and 450 cities
[0.008751] find 15 states and 450 cities (50 times)

After

testFindEntityOneToManyWithCache
[0.023316] persist 15 states and 450 cities
[0.008795] find 15 states and 450 cities (50 times)

Pretty underwhelming 😅

@greg0ire
Copy link
Member

Merging since this seems harmless. I think there is an improvement that could be demonstrated, but we would have to write a benchmark targeting a much smaller part of the code for this to happen. Thanks for porting this back @carnage !

@greg0ire greg0ire merged commit 52f5528 into doctrine:2.10.x Jul 20, 2021
@carnage carnage deleted the remove-expensive-array-ops branch July 24, 2021 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants