chore: basic isEnabled benchmarks #73
Conversation
| "..", | ||
| "..", | ||
| "..", | ||
| "client-specification", |
There was a problem hiding this comment.
@daveleek I feel like I've downed a bottle of crazy pills here. Where the hell is this assembly actually executing!?
Surely there's something better than this crazy down path?
There was a problem hiding this comment.
Long time since I looked at that, but back in the day mstest would move things to run in a subfolder deep down. you could output TestContext.CurrentContext.TestDirectory to the console and have a look 😅
There was a problem hiding this comment.
Don't think that will work, this isn't running in a test context, its a standalone executable so I can remove test harness fluff from the benchmarks
But if nothing stands out to you, it can sit, it works, it's just ugly
| { | ||
| public Config() | ||
| { | ||
| AddColumn(BenchmarkDotNet.Columns.StatisticColumn.OperationsPerSecond); |
There was a problem hiding this comment.
Not strictly necessary but gives us the output we want to be able to compare against the others
| @@ -6,6 +6,12 @@ | |||
| * Detailed information about configuring a multi-project build in Gradle can be found | |||
| * in the user manual at https://docs.gradle.org/8.1/userguide/multi_project_builds.html | |||
| */ | |||
| pluginManagement { | |||
There was a problem hiding this comment.
No idea why this needed to happen but Gradle flat out refused to resolve jmh without it for some reason
| @@ -35,3 +40,14 @@ tasks.named('test') { | |||
| // Use JUnit Platform for unit tests. | |||
| useJUnitPlatform() | |||
| } | |||
|
|
|||
| jmh { | |||
There was a problem hiding this comment.
The default settings for jmh spin off a ~30 minute benchmark run. I do not have the patience for that so I've fiddled so that they execute in a similar 30-60 off seconds like the others but this config could use more eyes.
@gastonfournier sanity check?
| There's also a `mem_check.rb` in the scripts folder. This is not a bullet proof test, but it can be helpful for detecting large leaks. This requires human interaction - you need to read the output and understand what it's telling you, so it's not run as part of the test suite. | ||
|
|
||
| ```bash | ||
| ruby scripts/mem_check.rb | ||
| rspec scripts/mem_check.rb |
There was a problem hiding this comment.
Just boyscouting, the previous command never worked
| } | ||
|
|
||
| @Setup | ||
| public void setUp() { |
There was a problem hiding this comment.
try setting the context here and reuse it in the benchmarkFeatureToggle() method
There was a problem hiding this comment.
Yep! Gave it a spin, no meaningful change to output
| @@ -77,7 +77,6 @@ private <T> T read(Pointer pointer, TypeReference<T> typeReference) { | |||
| String str = pointer.getString(0, UTF_8); | |||
| yggdrasil.freeResponse(pointer); | |||
| try { | |||
| System.out.println(str); // TODO use a logging library. SLF4J? | |||
There was a problem hiding this comment.
Yeeted this for now since std in a loop generally hurts benchmark reliability
| "..", | ||
| "..", | ||
| "..", | ||
| "client-specification", |
There was a problem hiding this comment.
Long time since I looked at that, but back in the day mstest would move things to run in a subfolder deep down. you could output TestContext.CurrentContext.TestDirectory to the console and have a look 😅
Adds some benchmarks purely for the equivalent
isEnabledcall for Ruby, Java and .NET. These are not intended to be extensive benchmarks, just enough to check that each SDK is correctly managing the FFI layer reasonably and get a loose sense of the computational differences between the implementations.Each implementation has instructions in the relevant README for running the benchmarks.
Output converted to a human readable, comparable form:
Java - 123 k operations/s
Ruby - 283 k operations/s
.NET - 493 k operations/s
Ruby is roughly twice the speed of the equivalent code in the SDK, which is within a reasonable target. Haven't checked how .net compares to the baseline implementation. Something appears to be off with Java, but that can be discussed elsewhere.