Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support repeat for network and other custom tracking #29

Open
oruebel opened this issue Feb 21, 2024 · 1 comment
Open

Support repeat for network and other custom tracking #29

oruebel opened this issue Feb 21, 2024 · 1 comment

Comments

@oruebel
Copy link
Contributor

oruebel commented Feb 21, 2024

          > One remaining item may be how we want to handle repeats for network benchmarks, but I think we can deal with hat in a separate Issue/PR.

A follow-up sounds good

TBH I think we could just set a repeat attribute for consistency, and then have a basic for loop that goes around the context + operation, appending a samples: list from each iteration, and that's what it returns - this would produce identical structure in the results to what we see in the timing tests

The only problem might be for any tests that have any kind of caching (including in-memory LRU) the operations would be performed on the same process, and so the first run may have heterogeneous statistics from the rest of the samples 🤔for comparison, the reason the timing tests can repeat so easily is because that's a built-in feature of timeit, which runs on new processes each time

Originally posted by @CodyCBakerPhD in #21 (comment)

@oruebel
Copy link
Contributor Author

oruebel commented Feb 22, 2024

The only problem might be for any tests that have any kind of caching (including in-memory LRU) the operations would be performed on the same process, and so the first run may have heterogeneous statistics from the rest of the samples

I think that depends in part on what we put into the setup method. For the network benchmarks we are controlling what is being measured via the network tracking decorator, i.e., if necessary, we could put setup and clean-up code inside the benchmark function to make sure we have clean repeats

@CodyCBakerPhD CodyCBakerPhD changed the title Discuss support for repeat of network benchmarks Support repeat for network and other custom benchmarks Mar 11, 2024
@CodyCBakerPhD CodyCBakerPhD changed the title Support repeat for network and other custom benchmarks Support repeat for network and other custom tracking Mar 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant