Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimization: Small changes on many functions V3 (wip) #217

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
  •  
  •  
  •  
59 changes: 59 additions & 0 deletions bench-results/small-changes-v3/test1/bench-base.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
lambda/decode_shl_sub time: [2.5098 ms 2.5255 ms 2.5368 ms]
lambda/deploy_simple_contract
time: [3.3569 ms 3.3583 ms 3.3601 ms]
lambda/heap_read_write time: [515.53 ms 515.78 ms 516.03 ms]
lambda/slot_hash_collision
time: [2.2614 ms 2.2691 ms 2.2760 ms]
lambda/write_and_decode time: [2.2485 ms 2.2496 ms 2.2509 ms]
lambda/event_spam time: [377.65 ms 377.91 ms 378.15 ms]
Found 2 outliers among 30 measurements (6.67%)
1 (3.33%) low mild
1 (3.33%) high mild
lambda/access_memory time: [2.2379 ms 2.2478 ms 2.2553 ms]
lambda/finish_eventful_frames
time: [2.7640 ms 2.7665 ms 2.7694 ms]
lambda/call_far time: [513.77 ms 514.48 ms 515.23 ms]

lambda/decode_shl_sub/full
time: [11.469 ms 11.471 ms 11.473 ms]
lambda/deploy_simple_contract/full
time: [12.679 ms 12.683 ms 12.687 ms]
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) high mild
lambda/heap_read_write/full
time: [523.69 ms 523.91 ms 524.16 ms]
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) high mild
lambda/slot_hash_collision/full
time: [11.576 ms 11.578 ms 11.581 ms]
Found 2 outliers among 30 measurements (6.67%)
2 (6.67%) low mild
lambda/write_and_decode/full
time: [11.579 ms 11.584 ms 11.588 ms]
lambda/event_spam/full time: [403.94 ms 405.29 ms 406.52 ms]
lambda/access_memory/full
time: [11.624 ms 11.626 ms 11.629 ms]
Found 2 outliers among 30 measurements (6.67%)
1 (3.33%) low mild
1 (3.33%) high mild
lambda/finish_eventful_frames/full
time: [12.303 ms 12.307 ms 12.312 ms]
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) low mild
lambda/call_far/full time: [596.91 ms 597.82 ms 598.69 ms]
Found 6 outliers among 30 measurements (20.00%)
2 (6.67%) low severe
1 (3.33%) low mild
2 (6.67%) high mild
1 (3.33%) high severe

lambda/load_test time: [1.9141 ms 1.9152 ms 1.9168 ms]
lambda/load_test_realistic
time: [1.5915 ms 1.5920 ms 1.5926 ms]
Found 4 outliers among 30 measurements (13.33%)
4 (13.33%) high mild
lambda/load_test_heavy time: [4.2946 ms 4.2976 ms 4.3004 ms]
Found 2 outliers among 30 measurements (6.67%)
1 (3.33%) high mild
1 (3.33%) high severe

108 changes: 108 additions & 0 deletions bench-results/small-changes-v3/test1/bench-compare.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
lambda/decode_shl_sub time: [2.1937 ms 2.1948 ms 2.1961 ms]
change: [-12.663% -12.294% -11.938%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 30 measurements (6.67%)
2 (6.67%) high mild
lambda/deploy_simple_contract
time: [3.2688 ms 3.2698 ms 3.2707 ms]
change: [-2.7906% -2.7103% -2.6318%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 30 measurements (6.67%)
2 (6.67%) high mild
lambda/heap_read_write time: [504.64 ms 505.01 ms 505.38 ms]
change: [-2.1718% -2.0893% -2.0013%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) high mild
lambda/slot_hash_collision
time: [2.1237 ms 2.1290 ms 2.1366 ms]
change: [-6.0710% -5.6377% -5.1654%] (p = 0.00 < 0.05)
Performance has improved.
lambda/write_and_decode time: [2.1724 ms 2.1744 ms 2.1764 ms]
change: [-3.4328% -3.3345% -3.2501%] (p = 0.00 < 0.05)
Performance has improved.
lambda/event_spam time: [355.99 ms 356.51 ms 357.03 ms]
change: [-5.8181% -5.6616% -5.5130%] (p = 0.00 < 0.05)
Performance has improved.
lambda/access_memory time: [2.1615 ms 2.1625 ms 2.1634 ms]
change: [-3.5803% -3.2947% -3.0170%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 30 measurements (16.67%)
1 (3.33%) low mild
1 (3.33%) high mild
3 (10.00%) high severe
lambda/finish_eventful_frames
time: [2.7032 ms 2.7044 ms 2.7057 ms]
change: [-2.6575% -2.4906% -2.3263%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 30 measurements (10.00%)
3 (10.00%) high mild
lambda/call_far time: [511.11 ms 511.61 ms 512.10 ms]
change: [-0.7285% -0.5574% -0.3873%] (p = 0.00 < 0.05)
Change within noise threshold.

lambda/decode_shl_sub/full
time: [11.334 ms 11.336 ms 11.338 ms]
change: [-1.2120% -1.1844% -1.1552%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) high mild
lambda/deploy_simple_contract/full
time: [12.389 ms 12.391 ms 12.393 ms]
change: [-2.4166% -2.3632% -2.3091%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) high mild
lambda/heap_read_write/full
time: [517.27 ms 517.63 ms 518.00 ms]
change: [-1.2808% -1.1994% -1.1128%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 30 measurements (6.67%)
1 (3.33%) low mild
1 (3.33%) high mild
lambda/slot_hash_collision/full
time: [11.464 ms 11.468 ms 11.472 ms]
change: [-0.9749% -0.9277% -0.8788%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) high mild
lambda/write_and_decode/full
time: [11.357 ms 11.410 ms 11.462 ms]
change: [-1.4043% -0.9058% -0.4741%] (p = 0.00 < 0.05)
Change within noise threshold.
lambda/event_spam/full time: [387.89 ms 388.49 ms 389.12 ms]
change: [-4.4694% -4.1456% -3.7969%] (p = 0.00 < 0.05)
Performance has improved.
lambda/access_memory/full
time: [11.344 ms 11.348 ms 11.351 ms]
change: [-2.4006% -2.3458% -2.2802%] (p = 0.00 < 0.05)
Performance has improved.
lambda/finish_eventful_frames/full
time: [11.898 ms 11.902 ms 11.906 ms]
change: [-3.3065% -3.2632% -3.2193%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) high mild
lambda/call_far/full time: [588.73 ms 590.17 ms 591.80 ms]
change: [-1.5546% -1.2799% -0.9632%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 1 outliers among 30 measurements (3.33%)
1 (3.33%) high mild

lambda/load_test time: [1.8695 ms 1.8702 ms 1.8710 ms]
change: [-2.6441% -2.5417% -2.4427%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 30 measurements (16.67%)
5 (16.67%) low mild
lambda/load_test_realistic
time: [1.5628 ms 1.5637 ms 1.5647 ms]
change: [-1.7941% -1.7034% -1.6004%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 30 measurements (10.00%)
3 (10.00%) high mild
lambda/load_test_heavy time: [4.1022 ms 4.1040 ms 4.1063 ms]
change: [-4.7391% -4.5978% -4.4874%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 30 measurements (6.67%)
2 (6.67%) high mild

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"group_id":"lambda","function_id":"access_memory","value_str":null,"throughput":null,"full_id":"lambda/access_memory","directory_name":"lambda/access_memory","title":"lambda/access_memory"}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":2231867.278502892,"upper_bound":2243451.405133254},"point_estimate":2237352.9068279616,"standard_error":2968.2473008000466},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":2227672.743055556,"upper_bound":2231350.8273809524},"point_estimate":2229998.185087719,"standard_error":1814.715242101092},"median_abs_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":2523.2987509542136,"upper_bound":13145.386000597093},"point_estimate":5719.6406136749765,"standard_error":3005.6161206756237},"slope":{"confidence_interval":{"confidence_level":0.95,"lower_bound":2237882.8940691403,"upper_bound":2255341.2155146482},"point_estimate":2247805.308690287,"standard_error":4475.432557387813},"std_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":11956.515669871082,"upper_bound":18833.698929266186},"point_estimate":16477.789676399687,"standard_error":1770.2391070251344}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"sampling_mode":"Linear","iters":[3.0,6.0,9.0,12.0,15.0,18.0,21.0,24.0,27.0,30.0,33.0,36.0,39.0,42.0,45.0,48.0,51.0,54.0,57.0,60.0,63.0,66.0,69.0,72.0,75.0,78.0,81.0,84.0,87.0,90.0],"times":[6766130.0,13341129.0,19974870.0,26776869.0,33424727.0,40061056.0,46857214.0,53463365.0,60213442.0,66904162.0,73565615.0,80197390.0,86739277.0,93913066.0,100228591.0,106879990.0,113763332.0,120271483.0,127110042.0,133799738.0,140191819.0,146616341.0,153828032.0,163315333.0,169449465.0,176913081.0,183339944.0,190186993.0,197077683.0,203769043.0]}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[2157434.5420634914,2192348.655753968,2285452.9589285715,2320367.072619048]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":-0.03580291359875877,"upper_bound":-0.030170094023357796},"point_estimate":-0.032947137642697744,"standard_error":0.0014380225479544528},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":-0.03132763214334632,"upper_bound":-0.029208966757853207},"point_estimate":-0.030479392807488237,"standard_error":0.000835220924072084}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"group_id":"lambda","function_id":"access_memory","value_str":null,"throughput":null,"full_id":"lambda/access_memory","directory_name":"lambda/access_memory","title":"lambda/access_memory"}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":2160969.9060336733,"upper_bound":2166594.5163442045},"point_estimate":2163638.532651411,"standard_error":1445.5674832678196},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":2160934.032212885,"upper_bound":2164036.9283950618},"point_estimate":2162029.1944444445,"standard_error":856.2046557168637},"median_abs_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":1975.3278201423407,"upper_bound":8048.334225262091},"point_estimate":3402.61036114945,"standard_error":1646.1442270448822},"slope":{"confidence_interval":{"confidence_level":0.95,"lower_bound":2161523.047136157,"upper_bound":2163373.4656631765},"point_estimate":2162489.1480698045,"standard_error":470.5899289245763},"std_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":4851.520910711272,"upper_bound":10159.950238971947},"point_estimate":8010.452317526981,"standard_error":1352.0779999402953}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"sampling_mode":"Linear","iters":[3.0,6.0,9.0,12.0,15.0,18.0,21.0,24.0,27.0,30.0,33.0,36.0,39.0,42.0,45.0,48.0,51.0,54.0,57.0,60.0,63.0,66.0,69.0,72.0,75.0,78.0,81.0,84.0,87.0,90.0],"times":[6548267.0,13058281.0,19614503.0,26224106.0,32485160.0,38781128.0,45197295.0,51716575.0,58242960.0,64626808.0,71342535.0,77652380.0,84246330.0,90606939.0,97624654.0,103877186.0,110213002.0,116771637.0,123186776.0,129628454.0,136157503.0,143000435.0,149353288.0,155675764.0,162304030.0,168618021.0,175285630.0,181509620.0,188120280.0,194759670.0]}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[2147664.2121167784,2153952.250609671,2170720.3532573855,2177008.3917502784]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"group_id":"lambda","function_id":"access_memory/full","value_str":null,"throughput":null,"full_id":"lambda/access_memory/full","directory_name":"lambda/access_memory_full","title":"lambda/access_memory/full"}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":11623399.337808622,"upper_bound":11630953.491012163},"point_estimate":11627099.353752574,"standard_error":1923.3946738929228},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":11624038.777777778,"upper_bound":11629096.663888888},"point_estimate":11626235.97985348,"standard_error":1267.5262028688767},"median_abs_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":3227.715613151222,"upper_bound":12145.558236361378},"point_estimate":6820.721770574803,"standard_error":2340.174596400625},"slope":{"confidence_interval":{"confidence_level":0.95,"lower_bound":11624057.19426448,"upper_bound":11628732.701081954},"point_estimate":11626287.984699454,"standard_error":1189.9452267103948},"std_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":6619.588260378467,"upper_bound":14136.709886128},"point_estimate":10741.784596308938,"standard_error":1933.9100034002674}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"sampling_mode":"Linear","iters":[3.0,6.0,9.0,12.0,15.0,18.0,21.0,24.0,27.0,30.0,33.0,36.0,39.0,42.0,45.0,48.0,51.0,54.0,57.0,60.0,63.0,66.0,69.0,72.0,75.0,78.0,81.0,84.0,87.0,90.0],"times":[34975708.0,69744981.0,104399447.0,139557188.0,174332280.0,209230453.0,243929192.0,279002045.0,314266035.0,349123863.0,383526154.0,418826694.0,452837419.0,488616769.0,524083242.0,557818559.0,593294359.0,627891676.0,663297584.0,697499229.0,731756979.0,767399310.0,802244165.0,837446810.0,871391801.0,906829490.0,941392703.0,976622040.0,1011380390.0,1046428887.0]}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[11590393.004221128,11606265.053036489,11648590.516544119,11664462.56535948]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":-0.02400561844935276,"upper_bound":-0.02280160780067924},"point_estimate":-0.023457825339675442,"standard_error":0.0003096612870193098},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":-0.024405519211268767,"upper_bound":-0.02320131261642433},"point_estimate":-0.023845637069762726,"standard_error":0.00035244524789541974}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"group_id":"lambda","function_id":"access_memory/full","value_str":null,"throughput":null,"full_id":"lambda/access_memory/full","directory_name":"lambda/access_memory_full","title":"lambda/access_memory/full"}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":11348739.103423418,"upper_bound":11360360.582873467},"point_estimate":11354352.887905193,"standard_error":2982.9572285140966},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":11342332.767948717,"upper_bound":11355367.972222222},"point_estimate":11349000.976190476,"standard_error":3694.8799756756694},"median_abs_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":6323.548074147964,"upper_bound":22885.172271206146},"point_estimate":12187.180542403785,"standard_error":4641.559184707931},"slope":{"confidence_interval":{"confidence_level":0.95,"lower_bound":11344151.408299914,"upper_bound":11350819.338333333},"point_estimate":11347619.927163757,"standard_error":1712.2109560863178},"std_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":12470.942631310802,"upper_bound":19352.26044298274},"point_estimate":16644.872798611836,"standard_error":1754.8912913477652}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"sampling_mode":"Linear","iters":[3.0,6.0,9.0,12.0,15.0,18.0,21.0,24.0,27.0,30.0,33.0,36.0,39.0,42.0,45.0,48.0,51.0,54.0,57.0,60.0,63.0,66.0,69.0,72.0,75.0,78.0,81.0,84.0,87.0,90.0],"times":[34171385.0,68272189.0,102318279.0,136585414.0,170328796.0,204869636.0,238924272.0,273056584.0,307155093.0,340642442.0,373874422.0,408207593.0,442301232.0,476680258.0,510441072.0,544306242.0,578772072.0,612687035.0,646396754.0,680472440.0,714193509.0,748772502.0,783028817.0,816474318.0,850537833.0,884639387.0,919605938.0,953966811.0,987712696.0,1021993459.0]}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[11267248.463141024,11304171.60977564,11402633.334134616,11439556.480769232]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"group_id":"lambda","function_id":"call_far","value_str":null,"throughput":null,"full_id":"lambda/call_far","directory_name":"lambda/call_far","title":"lambda/call_far"}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":513774637.0,"upper_bound":515226142.1483333},"point_estimate":514479473.6333333,"standard_error":369009.8893551406},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":513047060.0,"upper_bound":515362083.0},"point_estimate":514518105.5,"standard_error":503286.2490562801},"median_abs_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":1350706.1564976734,"upper_bound":2952669.0123796463},"point_estimate":2295837.935140729,"standard_error":408025.964694981},"slope":null,"std_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":1514779.4968823015,"upper_bound":2488867.536508034},"point_estimate":2053594.881255836,"standard_error":250333.51647087611}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"sampling_mode":"Flat","iters":[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0],"times":[513755167.0,513142114.0,519528360.0,514451035.0,512952006.0,518443670.0,512020846.0,515673492.0,516049048.0,516442949.0,512012881.0,510754114.0,512030034.0,515640281.0,516049049.0,514371164.0,512411450.0,515883317.0,514585176.0,514000377.0,517133888.0,514597169.0,512903044.0,516955913.0,514714550.0,515083885.0,514862929.0,512775806.0,512473537.0,512686958.0]}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[503737879.75,508272747.625,520365728.625,524900596.5]
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":-0.007284596246690344,"upper_bound":-0.0038728237673568713},"point_estimate":-0.00557364717860509,"standard_error":0.000868590745833906},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":-0.007486127717783897,"upper_bound":-0.0025099833608994704},"point_estimate":-0.005441067029661628,"standard_error":0.001114119023761169}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"group_id":"lambda","function_id":"call_far","value_str":null,"throughput":null,"full_id":"lambda/call_far","directory_name":"lambda/call_far","title":"lambda/call_far"}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"mean":{"confidence_interval":{"confidence_level":0.95,"lower_bound":511112849.34000003,"upper_bound":512100197.1725},"point_estimate":511611946.56666666,"standard_error":252295.3727567797},"median":{"confidence_interval":{"confidence_level":0.95,"lower_bound":511224604.0,"upper_bound":512206491.0},"point_estimate":511718578.0,"standard_error":273804.9479111366},"median_abs_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":713504.9438327551,"upper_bound":2251889.2241209745},"point_estimate":1604216.1669194698,"standard_error":398567.2864210508},"slope":null,"std_dev":{"confidence_interval":{"confidence_level":0.95,"lower_bound":1089545.6133154533,"upper_bound":1638034.7144549561},"point_estimate":1406994.181063101,"standard_error":139631.2841212152}}
Loading
Loading