Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate param benchmarks #15885

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

MiniaczQ
Copy link
Contributor

Objective

Benchmark overhead of validation for:

  • DynSystemParam,
  • ParamSet,
  • combinator systems.

Needed for #15606

Solution

As noted in objective, I've added 3 benchmarks, where each uses an excessive amount of the specific functionality.
I benchmark on the level of schedules, rather than individual validate_param calls, so we get a better idea how changes to the code impact memory-lookup, etc. related side effects.

Testing

param/combinator_system/8_piped_systems
                        time:   [1.7560 µs 1.7865 µs 1.8180 µs]
                        change: [+4.5244% +6.7955% +9.1413%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

param/combinator_system/8_dyn_params_system
                        time:   [89.354 ns 89.790 ns 90.300 ns]
                        change: [+0.6751% +1.6825% +2.6842%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
  6 (6.00%) high mild
  3 (3.00%) high severe

param/combinator_system/8_variant_param_set_system
                        time:   [88.295 ns 89.202 ns 90.208 ns]
                        change: [+0.1320% +1.0060% +1.8482%] (p = 0.02 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

2 back-to-back runs of the benchmarks, there is quire a lot of noise, can use feedback on fixing that

@alice-i-cecile alice-i-cecile added A-ECS Entities, components, systems, and events C-Benchmarks Stress tests and benchmarks used to measure how fast things are X-Uncontroversial This work is generally agreed upon S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Oct 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-ECS Entities, components, systems, and events C-Benchmarks Stress tests and benchmarks used to measure how fast things are S-Needs-Review Needs reviewer attention (from anyone!) to move forward X-Uncontroversial This work is generally agreed upon
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants