Significant Performance Degradation in Unleash Client v5 Compared to v4

[xiaoyangwang-geotab]

Hello Unleash Client Team,

We have observed a significant performance and memory allocation degradation in the Unleash.Client library after upgrading from version 4.1.11 to 5.0.4. Using benchmark tests, we compared the performance of key methods (IsEnabled and GetVariant) across the two versions. Below are the details of our tests:

Steps to Reproduce

Clone the following repository for the benchmark test: [UnleashClientBenchmarkTest Github Repo]
Follow the steps in the README.md to upgrade Unleash.Client to the desired version.
Run the benchmark with:
dotnet run --configuration Release

Benchmark Results

V4.1.11

| Method         | Mean      | Error    | StdDev   | Gen0   | Allocated |
|--------------- |----------:|---------:|---------:|-------:|----------:|
| TestIsEnabled  |  87.88 ns | 1.788 ns | 4.036 ns | 0.0185 |     232 B |
| TestGetVariant | 184.79 ns | 3.554 ns | 3.490 ns | 0.0253 |     320 B |

V5.0.4

| Method         | Mean     | Error     | StdDev    | Gen0   | Allocated |
|--------------- |---------:|----------:|----------:|-------:|----------:|
| TestIsEnabled  | 2.931 us | 0.0585 us | 0.0928 us | 0.1678 |   2.09 KB |
| TestGetVariant | 4.950 us | 0.0976 us | 0.0913 us | 0.2899 |   3.67 KB |

• Execution Time:
  TestIsEnabled is ~33x slower in v5 (87.88 ns → 2.931 us).
  TestGetVariant is ~27x slower in v5 (184.79 ns → 4.950 us).

• Memory Allocations:
  TestIsEnabled allocates ~9x more memory in v5 (232 B → 2.09 KB).
  TestGetVariant allocates ~11x more memory in v5 (320 B → 3.67 KB).

• Garbage Collection:
  Gen0 allocations increased significantly for both methods in v5, indicating higher memory churn.

The degradation affects scenarios where IsEnabled and GetVariant are called frequently, leading to increased latency and higher memory consumption, which could significantly impact applications under high load. Therefore, we expect that the v5 implementation maintains similar performance and memory characteristics as v4, particularly for key methods like IsEnabled and GetVariant.

Thank you for investigating this issue!

[sighphyre]

Thanks for reporting this and providing the benchmarks!

Just eyeballing those benchmarks, is that a real usecase for you? Those are effectively a no-op for the SDK. Are you seeing a similar regression with complex, real life toggles?

I'll take a look either way

[xiaoyangwang-geotab]

Hi Simon, Thanks for taking a look at this issue. You are correct that we use both methods in our application, which is why we included them in our benchmarks. It looks like these two methods are the primary ways to access features using the Unleash client, so it's important for us to ensure they are both performing optimally. Please let me know if you have any further questions or require more information. Thanks, Xiaoyang Xiaoyang Wang Geotab Software Developer Direct Toll-free Visit +1 (226) 581-2886 +1 (877) 436-8221 www.geotab.com Twitter <https://twitter.com/geotab> | Facebook <https://www.facebook.com/Geotab> | YouTube <https://www.youtube.com/user/MyGeotab> | LinkedIn <https://www.linkedin.com/company/geotab/>

…

On Wed, Jan 29, 2025 at 2:13 AM Simon Hornby ***@***.***> wrote: Thanks for reporting this and providing the benchmarks! Just eyeballing those benchmarks, is that a real usecase for you? Those are effectively a no-op for the SDK. Are you seeing a similar regression with complex, real life toggles? I'll take a look either way — Reply to this email directly, view it on GitHub <#272 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A32Y7W5WZJR356L7L4GWYRL2NB5SRAVCNFSM6AAAAABWBBC33SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMRQHA3DSNBSGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[sighphyre]

Hey @xiaoyangwang-geotab, what I'm really asking is that from looking at the benchmarks you've provided, it's hard to tell what data the SDK is being hydrated with. If there's no matching toggle being provided from the Unleash API then the SDK is short circuiting to off because it can't find the toggle. I'm asking if that's a real usecase for you.

[xiaoyangwang-geotab]

@sighphyre I removed the real features and other properties that we used in production as the github repo is public. When I ran my test, I added all the properties that we used in production and used the features existing in our unleash project.

I just ran another pair of tests with different feature names and the result is similar:

v5

Method	Mean	Error	StdDev	Gen0	Allocated
TestIsEnabled	4.067 us	0.0809 us	0.1616 us	0.1755	2.23 KB
TestGetVariant	8.928 us	0.1760 us	0.2892 us	0.4272	5.6 KB

v4

Method	Mean	Error	StdDev	Gen0	Allocated
TestIsEnabled	338.5 ns	6.46 ns	9.26 ns	0.0598	752 B
TestGetVariant	216.0 ns	3.98 ns	3.72 ns	0.0203	256 B

[sighphyre]

@xiaoyangwang-geotab Gotcha, thanks!

Okay, looking into this, I'll have a release when we've got this in a better state. Thanks again for reporting!