Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize param validation through get_param(...) -> Option<Out> #15606

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Conversation

MiniaczQ
Copy link
Contributor

@MiniaczQ MiniaczQ commented Oct 2, 2024

Objective

Fixes #15505 through a new solution which came up during development.
It changes some of the original assumptions about checking all system params before running the systems,
which isn't correct for combinator systems (they can invalidate their own sub-system params).
Alternative to #15571

Solution

get_param returns an Option, which makes run_unsafe also return an option.
validate_param now defaults to get_param().is_some() and we overwrite it for non-send and grouped params to behave correctly.

The original assumption about validating all params before running a system is no longer valid.
Combinator system can invalidate it's own params in a case of:
a.pipe(b).pipe(a) where b removes required params for a.
Because of that, we will no longer validate all params of a system, but only those of the first one.
This allows us to prevent spawning tasks if we cannot start running a system and reduced access overhead, because we only check a single subsystem's params immediately followed by getting them (albeit in another thread).

Testing

Ran parts of CI locally,
Ran fallible_params example.

@alice-i-cecile alice-i-cecile added A-ECS Entities, components, systems, and events C-Performance A change motivated by improving speed, memory usage or compile times S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged labels Oct 3, 2024
@MiniaczQ MiniaczQ added the D-Unsafe Touches with unsafe code in some way label Oct 3, 2024
@hymm hymm self-requested a review October 3, 2024 21:40
@alice-i-cecile alice-i-cecile added this to the 0.15 milestone Oct 3, 2024
@MiniaczQ
Copy link
Contributor Author

MiniaczQ commented Oct 3, 2024

The steps taken to get here look as following:

  1. We question whether 1.2 (from Optimize validate_param #15505) is possible to implement. While lifetimes and type erasure can be worked around, we can't fetch all system parameters ahead of running the system, because in case of compound systems (e.g. a.pipe(b)), the respective sub-systems can have mutually exclusive access. While this isn't technically UB, since the parameters will only be used in mutually exclusive contexts, this will lead to multiple im-/mutable references to the same data existing at once.
  2. To get to 1.2, we need proposition 1.1 anyways. It gives better internal ergonomics than current solution. Because it's only applied when running systems, we don't save on tasks in multithreaded executors nor do we retain the original behavior of running only entire compound systems in executors (e.g. a.pipe(b) could run a but stop at b since params were "checked" lazily).
  3. We can fix that by reintroducing validate_param, but default it's implementation as get_param().is_some(). We expect this implementation to be optimized by compiler. Because default implementation breaks for non-send resources, we overwrite it. We also overwrite it for compound resources (SystemParam tuples, ParamSet, DynSystemParam) to rely on their sub-params implementations.

To sum it up, without going the additional step into proposition 1.2 we simply cannot reduce double checks in executors.

An alternative to 1.2 idea could split "fetching" into 2 steps:

  • checking availability and creating an "access key" - validate_param() -> Option<Accessor>
  • using the "access key" to construct system params - get_param(Accessor)
  • to avoid additional boxing in executors, Acccessors can be stored inside the bottom-level Systems (exclusive/function) when fetch succeeds and removed after running the system.

@MiniaczQ MiniaczQ added S-Needs-Benchmarking This set of changes needs performance benchmarking to double-check that they help S-Needs-Review Needs reviewer attention (from anyone!) to move forward and removed S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged labels Oct 4, 2024
@MiniaczQ MiniaczQ marked this pull request as ready for review October 4, 2024 14:19
@MiniaczQ
Copy link
Contributor Author

MiniaczQ commented Oct 4, 2024

Marking it as ready for review since code is basically done, just needs benchmarks

@MiniaczQ MiniaczQ added the D-Modest A "normal" level of difficulty; suitable for simple features or challenging fixes label Oct 4, 2024
@MiniaczQ
Copy link
Contributor Author

MiniaczQ commented Oct 4, 2024

Note for @alice-i-cecile, since I've only made groundwork towards 1.2, but not implemented it, this doesn't have a significant impact on performance and we can delay it.
I've had trouble finding all the broken code, but now that I've done it once it should go much quicker

@alice-i-cecile alice-i-cecile removed this from the 0.15 milestone Oct 4, 2024
@MiniaczQ
Copy link
Contributor Author

MiniaczQ commented Oct 5, 2024

I found an edge case that asks for change in requirements:
https://discord.com/channels/691052431525675048/749335865876021248/1292140022375514145
I'm re-drafting this until I implement it

@MiniaczQ MiniaczQ marked this pull request as draft October 5, 2024 15:30
@MiniaczQ MiniaczQ added S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Oct 5, 2024
@MiniaczQ MiniaczQ marked this pull request as ready for review October 12, 2024 15:40
@MiniaczQ
Copy link
Contributor Author

Core changes are done, just need to update documentation

@MiniaczQ MiniaczQ added S-Needs-Review Needs reviewer attention (from anyone!) to move forward and removed S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged labels Oct 12, 2024
@MiniaczQ MiniaczQ added this to the 0.16 milestone Oct 14, 2024
github-merge-queue bot pushed a commit that referenced this pull request Oct 15, 2024
# Objective

Benchmark overhead of validation for:
- `DynSystemParam`,
- `ParamSet`,
- combinator systems.

Needed for #15606

## Solution

As noted in objective, I've added 3 benchmarks, where each uses an
excessive amount of the specific functionality.
I benchmark on the level of schedules, rather than individual
`validate_param` calls, so we get a better idea how changes to the code
impact memory-lookup, etc. related side effects.

## Testing

```
param/combinator_system/8_piped_systems
                        time:   [1.7560 µs 1.7865 µs 1.8180 µs]
                        change: [+4.5244% +6.7955% +9.1413%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

param/combinator_system/8_dyn_params_system
                        time:   [89.354 ns 89.790 ns 90.300 ns]
                        change: [+0.6751% +1.6825% +2.6842%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
  6 (6.00%) high mild
  3 (3.00%) high severe

param/combinator_system/8_variant_param_set_system
                        time:   [88.295 ns 89.202 ns 90.208 ns]
                        change: [+0.1320% +1.0060% +1.8482%] (p = 0.02 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild
```

2 back-to-back runs of the benchmarks, there is quire a lot of noise,
can use feedback on fixing that
@MiniaczQ
Copy link
Contributor Author

MiniaczQ commented Oct 15, 2024

Before (main):

param/combinator_system/8_piped_systems
                        time:   [1.6549 µs 1.6689 µs 1.6852 µs]
                        change: [-13.250% -10.268% -7.3630%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

param/combinator_system/8_dyn_params_system
                        time:   [95.545 ns 95.841 ns 96.172 ns]
                        change: [-2.9593% -1.3542% -0.0548%] (p = 0.06 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

param/combinator_system/8_variant_param_set_system
                        time:   [91.596 ns 91.962 ns 92.365 ns]
                        change: [+0.1790% +1.0297% +1.8957%] (p = 0.02 < 0.05)
                        Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
  6 (6.00%) high mild
  3 (3.00%) high severe

After:

param/combinator_system/8_piped_systems
                        time:   [1.6590 µs 1.6863 µs 1.7149 µs]
                        change: [-3.5415% -1.5778% +0.5378%] (p = 0.15 > 0.05)
                        No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

param/combinator_system/8_dyn_params_system
                        time:   [93.629 ns 94.065 ns 94.551 ns]
                        change: [-2.3199% -1.2717% -0.1379%] (p = 0.01 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

param/combinator_system/8_variant_param_set_system
                        time:   [91.913 ns 92.361 ns 92.842 ns]
                        change: [+0.0434% +0.8115% +1.6509%] (p = 0.05 < 0.05)
                        Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

There is a lot of noise in the measurements, but at the very least I don't think this is a regression

It'd be good if few more people measured, I use cargo bench -- param/
Here is the specific main commit I used acbed6040eeae99199657e44a9668772738ac344.

@MiniaczQ MiniaczQ removed the S-Needs-Benchmarking This set of changes needs performance benchmarking to double-check that they help label Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-ECS Entities, components, systems, and events C-Performance A change motivated by improving speed, memory usage or compile times D-Modest A "normal" level of difficulty; suitable for simple features or challenging fixes D-Unsafe Touches with unsafe code in some way S-Needs-Review Needs reviewer attention (from anyone!) to move forward
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimize validate_param
2 participants