Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix comment posting on failed CI run and cleanup benchmarks #46

Merged
merged 3 commits into from
Aug 20, 2024

Conversation

joamaki
Copy link
Contributor

@joamaki joamaki commented Aug 20, 2024

If a PR fails the failed "make test" output wasn't posted. Add a "if: always()" to the comment posting to always post it.
This was manually tested.

Fix the secondary index benchmark not to recreate the tags set on each insert as that's not what is important for the benchmark.
Additionally drop the very large batch sizes from benchmarks as those use up fair bit of memory and don't provide a useful signal.

Post the comment showing the build & tests regardless of failure.

Signed-off-by: Jussi Maki <[email protected]>
Drop the benchmarks for the 10k and 100k batch sizes. They're
not providing further insights and use up fair bit of memory to execute.

Signed-off-by: Jussi Maki <[email protected]>
BenchmarkDB_WriteTxn_100_SecondaryIndex was reconstructing the tags set
on every insert which cost a bit of time and wasn't relevant for the benchmark.

While at it simplify the batch benchmarks to not have two loops.

Signed-off-by: Jussi Maki <[email protected]>
@joamaki joamaki requested review from a team as code owners August 20, 2024 10:00
@joamaki joamaki requested review from viktor-kurchenko and dylandreimerink and removed request for a team August 20, 2024 10:00
Copy link

$ make test
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading golang.org/x/term v0.16.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading golang.org/x/sys v0.17.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading golang.org/x/text v0.14.0
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
ok  	github.com/cilium/statedb	5.668s	coverage: 89.1% of statements
ok  	github.com/cilium/statedb/index	0.005s	coverage: 25.5% of statements
ok  	github.com/cilium/statedb/internal	0.017s	coverage: 93.3% of statements
ok  	github.com/cilium/statedb/part	1.759s	coverage: 82.8% of statements
ok  	github.com/cilium/statedb/reconciler	2.304s	coverage: 92.0% of statements
-----
$ make bench
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkDB_WriteTxn_1-4                    	  451348	      2613 ns/op	    382737 objects/sec	    2720 B/op	      32 allocs/op
BenchmarkDB_WriteTxn_10-4                   	 1000000	      1002 ns/op	    997868 objects/sec	     716 B/op	      10 allocs/op
BenchmarkDB_WriteTxn_100-4                  	 1407030	       780.7 ns/op	   1280839 objects/sec	     580 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_1000-4                 	 1438066	       825.0 ns/op	   1212172 objects/sec	     530 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4   	  669882	      1685 ns/op	    593422 objects/sec	    1023 B/op	      19 allocs/op
BenchmarkDB_Modify-4                        	    1330	    934326 ns/op	   1070292 objects/sec	  755805 B/op	    8463 allocs/op
BenchmarkDB_GetInsert-4                     	    1203	   1027869 ns/op	    972897 objects/sec	  744426 B/op	    8469 allocs/op
BenchmarkDB_RandomInsert-4                  	    2284	    508791 ns/op	   1965443 objects/sec	  385747 B/op	    7096 allocs/op
BenchmarkDB_RandomReplace-4                 	     402	   2889796 ns/op	    346045 objects/sec	 1852544 B/op	   30567 allocs/op
BenchmarkDB_SequentialInsert-4              	    1528	    818475 ns/op	   1221784 objects/sec	  534015 B/op	    7288 allocs/op
BenchmarkDB_Changes_Baseline-4              	    1197	    991937 ns/op	   1008130 objects/sec	  536254 B/op	   10246 allocs/op
BenchmarkDB_Changes-4                       	     664	   1786425 ns/op	    559777 objects/sec	  963933 B/op	   14541 allocs/op
BenchmarkDB_RandomLookup-4                  	   22353	     55150 ns/op	  18132251 objects/sec	     144 B/op	       1 allocs/op
BenchmarkDB_SequentialLookup-4              	   26636	     46389 ns/op	  21557092 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_All-4             	     561	   2144624 ns/op	  46628327 objects/sec	     360 B/op	       7 allocs/op
BenchmarkDB_FullIteration_Get-4             	     206	   5846207 ns/op	  17105145 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4              	  468739	      2352 ns/op	        20.00 50th_µs	        24.00 90th_µs	        64.00 99th_µs	    1514 B/op	      24 allocs/op
PASS
ok  	github.com/cilium/statedb	25.522s
PASS
ok  	github.com/cilium/statedb/index	0.005s
PASS
ok  	github.com/cilium/statedb/internal	0.004s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_Insert_RootOnlyWatch-4    	    9009	    135763 ns/op	   7365785 objects/sec	  104118 B/op	    2041 allocs/op
Benchmark_Insert-4                  	    5923	    183470 ns/op	   5450492 objects/sec	  202684 B/op	    3065 allocs/op
Benchmark_Modify-4                  	    7294	    155856 ns/op	   6416197 objects/sec	  211998 B/op	    1205 allocs/op
Benchmark_GetInsert-4               	    6309	    188564 ns/op	   5303243 objects/sec	  211797 B/op	    1204 allocs/op
Benchmark_Replace-4                 	27606392	        43.84 ns/op	  22809317 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Replace_RootOnlyWatch-4   	27376837	        44.83 ns/op	  22305043 objects/sec	       0 B/op	       0 allocs/op
Benchmark_txn_1-4                   	 2831557	       423.5 ns/op	   2361226 objects/sec	     432 B/op	       7 allocs/op
Benchmark_txn_10-4                  	 7024522	       167.7 ns/op	   5964334 objects/sec	     152 B/op	       2 allocs/op
Benchmark_txn_100-4                 	 7344321	       169.8 ns/op	   5888868 objects/sec	     223 B/op	       2 allocs/op
Benchmark_txn_1000-4                	 7008793	       169.5 ns/op	   5901373 objects/sec	     215 B/op	       2 allocs/op
Benchmark_txn_delete_1-4            	 3290622	       361.7 ns/op	   2765104 objects/sec	     840 B/op	       6 allocs/op
Benchmark_txn_delete_10-4           	 8917460	       135.4 ns/op	   7387764 objects/sec	     131 B/op	       1 allocs/op
Benchmark_txn_delete_100-4          	10483654	       113.2 ns/op	   8836474 objects/sec	      60 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4         	11645161	       103.4 ns/op	   9674594 objects/sec	      26 B/op	       1 allocs/op
Benchmark_Get-4                     	   40327	     29892 ns/op	  33454154 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterate-4                 	  163784	      7336 ns/op	 136309534 objects/sec	      80 B/op	       3 allocs/op
Benchmark_Hashmap_Insert-4          	   14990	     79629 ns/op	  12558219 objects/sec	   86549 B/op	      64 allocs/op
Benchmark_Hashmap_Get_Uint64-4      	  159499	      7535 ns/op	 132705565 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4       	  157443	      7483 ns/op	 133642256 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Uint64Map_Random-4        	    1226	    969004 ns/op	   1031989 items/sec	 2688311 B/op	    9028 allocs/op
Benchmark_Uint64Map_Sequential-4    	    1258	    919292 ns/op	   1087794 items/sec	 2476395 B/op	    9749 allocs/op
PASS
ok  	github.com/cilium/statedb/part	28.422s
PASS
ok  	github.com/cilium/statedb/reconciler	0.005s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 3.07 seconds (batch size 1000)
Throughput 326246.84 objects per second
Allocated 6011420 objects, 409092kB bytes, 538816kB bytes still in use

Copy link
Member

@dylandreimerink dylandreimerink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@joamaki joamaki merged commit f97a090 into main Aug 20, 2024
1 check passed
@joamaki joamaki deleted the pr/joamaki/workflow-bench-fixes branch August 20, 2024 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants