[WIP] benchmark createClusterCatalog func #1651
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
openshift-ci
bot
added
the
do-not-merge/work-in-progress
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1651 +/- ##
==========================================
- Coverage 67.42% 67.40% -0.03%
==========================================
Files 55 55
Lines 4632 4632
==========================================
- Hits 3123 3122 -1
- Misses 1284 1285 +1
Partials 225 225
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
jiazha-mac:e2e jiazha$ go test -run=^$ -bench=. -count=10 -memprofile=mem.out -cpuprofile=cpu.out
goos: darwin
goarch: arm64
pkg: github.com/operator-framework/operator-controller/test/e2e
cpu: Apple M1 Pro
BenchmarkCreateClusterCatalog-10 1 2093042334 ns/op
BenchmarkCreateClusterCatalog-10 4 611432146 ns/op
BenchmarkCreateClusterCatalog-10 10 224809304 ns/op
BenchmarkCreateClusterCatalog-10 13 92630189 ns/op
BenchmarkCreateClusterCatalog-10 6 174559444 ns/op
BenchmarkCreateClusterCatalog-10 12 107088038 ns/op
BenchmarkCreateClusterCatalog-10 1 1003581583 ns/op
BenchmarkCreateClusterCatalog-10 3 379469264 ns/op
BenchmarkCreateClusterCatalog-10 2 606936271 ns/op
BenchmarkCreateClusterCatalog-10 1 2773485917 ns/op
PASS
ok github.com/operator-framework/operator-controller/test/e2e 34.115s
jiazha-mac:e2e jiazha$ go tool pprof mem.out
File: e2e.test
Type: alloc_space
Time: Jan 27, 2025 at 5:22pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 8433.33kB, 62.21% of 13555.35kB total
Showing top 10 nodes out of 126
flat flat% sum% cum cum%
1536.16kB 11.33% 11.33% 1536.16kB 11.33% golang.org/x/net/http2.(*ClientConn).roundTrip
1536.09kB 11.33% 22.66% 1536.09kB 11.33% path.Join
1184.27kB 8.74% 31.40% 1184.27kB 8.74% runtime/pprof.StartCPUProfile
902.59kB 6.66% 38.06% 1553.21kB 11.46% compress/flate.NewWriter
650.62kB 4.80% 42.86% 650.62kB 4.80% compress/flate.(*compressor).init
553.04kB 4.08% 46.94% 553.04kB 4.08% github.com/gogo/protobuf/proto.RegisterType
528.17kB 3.90% 50.84% 528.17kB 3.90% regexp.(*bitState).reset
516.01kB 3.81% 54.64% 516.01kB 3.81% google.golang.org/protobuf/internal/filedesc.(*File).initDecls
513.69kB 3.79% 58.43% 513.69kB 3.79% regexp.mergeRuneSets.func2
512.69kB 3.78% 62.21% 512.69kB 3.78% regexp/syntax.(*compiler).inst
(pprof) exit
jiazha-mac:e2e jiazha$ go tool pprof cpu.out
File: e2e.test
Type: cpu
Time: Jan 27, 2025 at 5:22pm (CST)
Duration: 33.19s, Total samples = 380ms ( 1.14%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 310ms, 81.58% of 380ms total
Showing top 10 nodes out of 154
flat flat% sum% cum cum%
90ms 23.68% 23.68% 90ms 23.68% runtime.kevent
50ms 13.16% 36.84% 50ms 13.16% runtime.pthread_cond_signal
50ms 13.16% 50.00% 50ms 13.16% syscall.syscall
40ms 10.53% 60.53% 40ms 10.53% runtime.pthread_cond_wait
30ms 7.89% 68.42% 60ms 15.79% runtime.scanobject
10ms 2.63% 71.05% 20ms 5.26% k8s.io/client-go/rest.(*Request).tryThrottleWithInfo
10ms 2.63% 73.68% 10ms 2.63% runtime.(*itabTableType).find
10ms 2.63% 76.32% 10ms 2.63% runtime.(*mheap).allocSpan
10ms 2.63% 78.95% 10ms 2.63% runtime.(*mspan).heapBitsSmallForAddr
10ms 2.63% 81.58% 10ms 2.63% runtime.(*unwinder).resolveInternal
(pprof) exit |
joelanford
reviewed
Jan 28, 2025
| } | ||
|
|
||
| // GetRandomString generates a random string of the given length | ||
| func getRandomString(length int) string { |
There was a problem hiding this comment.
k8s.io/apimachinery has a rand.String function that can be used instead of us maintaining a separate implementation. https://pkg.go.dev/k8s.io/apimachinery/pkg/util/rand#String
joelanford
reviewed
Jan 28, 2025
| "time" | ||
| ) | ||
|
|
||
| func BenchmarkCreateClusterCatalog(b *testing.B) { |
There was a problem hiding this comment.
I'm not sure this benchmark tells us a whole lot because it is mostly measuring:
- client CPU/memory used to submit a CREATE request to the apiserver
- client CPU/memory used to submit a DELETE request to the apiserver
But it notably isn't able to account for the resources or time spent by the controller to reconcile the ClusterCatalog, which I think is what we actually want to measure.
But regardless, in my mind the biggest open question in is NOT "what machinery do we use to take measurements?". The real question is "where/how do we store/retrieve historical measurements that we can use as a baseline for comparison?"
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
See more: #920
Reviewer Checklist