Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify DB and Handle types into just DB #51

Merged
merged 2 commits into from
Sep 6, 2024
Merged

Conversation

joamaki
Copy link
Contributor

@joamaki joamaki commented Sep 6, 2024

This removes the separate Handle type to allow decorating DB with the handle name in Hive:

       hive.Options{
               // Create a named DB handle for each module.
               ModuleDecorator: func(db *statedb.DB, id cell.ModuleID) *statedb.DB {
                       return db.NewHandle(string(id))
               },
       },

With this we can get StateDB metrics accounted by module, allowing us to for example seeing which module
has long-running WriteTxns etc.

Instead of having separate DB and Handle types, merge them into
just DB. This allows Hive to decorate the "*DB" type for each module
and use the module's ID as the handle name.

Signed-off-by: Jussi Maki <[email protected]>
The example expvar metrics implementation incremented the
total errors variable even when there wasn't an error.

Signed-off-by: Jussi Maki <[email protected]>
@joamaki joamaki requested a review from a team as a code owner September 6, 2024 07:38
@joamaki joamaki requested review from pippolo84 and removed request for a team September 6, 2024 07:38
Copy link

github-actions bot commented Sep 6, 2024

$ make test
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading golang.org/x/term v0.16.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading golang.org/x/sys v0.17.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading golang.org/x/text v0.14.0
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
ok  	github.com/cilium/statedb	5.584s	coverage: 88.3% of statements
ok  	github.com/cilium/statedb/index	0.004s	coverage: 44.6% of statements
ok  	github.com/cilium/statedb/internal	0.007s	coverage: 93.3% of statements
ok  	github.com/cilium/statedb/part	2.549s	coverage: 82.8% of statements
ok  	github.com/cilium/statedb/reconciler	2.506s	coverage: 92.9% of statements
-----
$ make bench
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkDB_WriteTxn_1-4                    	  444542	      2578 ns/op	    387890 objects/sec	    2800 B/op	      32 allocs/op
BenchmarkDB_WriteTxn_10-4                   	 1244499	       983.8 ns/op	   1016502 objects/sec	     742 B/op	      10 allocs/op
BenchmarkDB_WriteTxn_100-4                  	 1585818	       757.2 ns/op	   1320717 objects/sec	     595 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_1000-4                 	 1520791	       788.2 ns/op	   1268778 objects/sec	     550 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4   	  504988	      2260 ns/op	    442385 objects/sec	    1489 B/op	      37 allocs/op
BenchmarkDB_Modify-4                        	    1344	    914203 ns/op	   1093850 objects/sec	  768756 B/op	    8457 allocs/op
BenchmarkDB_GetInsert-4                     	    1246	   1023935 ns/op	    976625 objects/sec	  752813 B/op	    8458 allocs/op
BenchmarkDB_RandomInsert-4                  	    2368	    496158 ns/op	   2015485 objects/sec	  402239 B/op	    7098 allocs/op
BenchmarkDB_RandomReplace-4                 	     330	   3670694 ns/op	    272428 objects/sec	 2344571 B/op	   48567 allocs/op
BenchmarkDB_SequentialInsert-4              	    1569	    792173 ns/op	   1262350 objects/sec	  551510 B/op	    7287 allocs/op
BenchmarkDB_Changes_Baseline-4              	    1266	    947001 ns/op	   1055966 objects/sec	  553176 B/op	   10253 allocs/op
BenchmarkDB_Changes-4                       	     691	   1685090 ns/op	    593440 objects/sec	  993102 B/op	   14493 allocs/op
BenchmarkDB_RandomLookup-4                  	   21475	     58606 ns/op	  17063066 objects/sec	     144 B/op	       1 allocs/op
BenchmarkDB_SequentialLookup-4              	   27534	     44645 ns/op	  22399064 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_All-4             	     817	   1253307 ns/op	  79789072 objects/sec	     464 B/op	      12 allocs/op
BenchmarkDB_FullIteration_Get-4             	     224	   5291197 ns/op	  18899351 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4              	  507836	      2298 ns/op	        20.00 50th_µs	        23.00 90th_µs	        85.00 99th_µs	    1585 B/op	      24 allocs/op
PASS
ok  	github.com/cilium/statedb	26.559s
PASS
ok  	github.com/cilium/statedb/index	0.004s
PASS
ok  	github.com/cilium/statedb/internal	0.005s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_Insert_RootOnlyWatch-4    	    8811	    131821 ns/op	   7586032 objects/sec	  104162 B/op	    2041 allocs/op
Benchmark_Insert-4                  	    6021	    185995 ns/op	   5376499 objects/sec	  219063 B/op	    3064 allocs/op
Benchmark_Modify-4                  	    8293	    141783 ns/op	   7053020 objects/sec	  212736 B/op	    1205 allocs/op
Benchmark_GetInsert-4               	    5884	    184698 ns/op	   5414263 objects/sec	  212267 B/op	    1204 allocs/op
Benchmark_Replace-4                 	27058000	        44.47 ns/op	  22486298 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Replace_RootOnlyWatch-4   	27015212	        44.59 ns/op	  22426503 objects/sec	       0 B/op	       0 allocs/op
Benchmark_txn_1-4                   	 3012993	       399.7 ns/op	   2501786 objects/sec	     448 B/op	       7 allocs/op
Benchmark_txn_10-4                  	 7332902	       164.8 ns/op	   6066594 objects/sec	     154 B/op	       2 allocs/op
Benchmark_txn_100-4                 	 7945231	       150.3 ns/op	   6652305 objects/sec	     224 B/op	       2 allocs/op
Benchmark_txn_1000-4                	 7122537	       174.2 ns/op	   5741760 objects/sec	     216 B/op	       2 allocs/op
Benchmark_txn_delete_1-4            	 3108771	       382.3 ns/op	   2615625 objects/sec	     856 B/op	       6 allocs/op
Benchmark_txn_delete_10-4           	 8484058	       138.9 ns/op	   7200028 objects/sec	     132 B/op	       1 allocs/op
Benchmark_txn_delete_100-4          	10795306	       110.2 ns/op	   9072311 objects/sec	      60 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4         	11536909	       106.4 ns/op	   9399148 objects/sec	      26 B/op	       1 allocs/op
Benchmark_Get-4                     	   38018	     30721 ns/op	  32551548 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterate-4                 	  172851	      6938 ns/op	 144134974 objects/sec	      80 B/op	       3 allocs/op
Benchmark_Hashmap_Insert-4          	   15686	     76051 ns/op	  13149062 objects/sec	   86549 B/op	      64 allocs/op
Benchmark_Hashmap_Get_Uint64-4      	  164020	      7450 ns/op	 134233166 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4       	  155763	      7682 ns/op	 130175919 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Uint64Map_Random-4        	    1399	    848544 ns/op	   1178491 items/sec	 2701287 B/op	    9028 allocs/op
Benchmark_Uint64Map_Sequential-4    	    1508	    791196 ns/op	   1263910 items/sec	 2492405 B/op	    9749 allocs/op
PASS
ok  	github.com/cilium/statedb/part	28.284s
PASS
ok  	github.com/cilium/statedb/reconciler	0.004s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 2.90 seconds (batch size 1000)
Throughput 344365.87 objects per second
Allocated 6011442 objects, 424776kB bytes, 516056kB bytes still in use

Copy link
Member

@pippolo84 pippolo84 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

@joamaki joamaki merged commit 574b83f into main Sep 6, 2024
1 check passed
@joamaki joamaki deleted the pr/joamaki/unify-db-handle branch September 6, 2024 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants