Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[benchmark] notebook for interactively computing benchmark config #14757

Open
wants to merge 1 commit into
base: ehigham/benchmark-suite-improvements
Choose a base branch
from
Open

[benchmark] notebook for interactively computing benchmark config #14757

wants to merge 1 commit into from
+1,060 −230

Conversation

ehigham
Copy link
Member

@ehigham ehigham commented Nov 14, 2024 •
edited
Loading

Change Description

Adds tools for analyzing benchmark variability and detecting minimal slowdowns when running benchmarks on Hail Batch. The analysis is based on methods from Laaber et al.'s paper on software microbenchmarking in the cloud.

Key additions:

  • New Jupyter notebook for exploring benchmark variability
  • Statistical analysis tools for computing confidence intervals and detecting performance changes
  • Functions for determining minimal detectable slowdowns using different testing strategies
  • Improved benchmark data import/export capabilities
  • Enhanced visualization tools for benchmark results

Security Assessment

This change has no security impact

Impact Description

Analysis and visualization code that operates on benchmark data only, with no access to sensitive information or systems.

@ehigham Graphite App
Copy link
Member Author

ehigham commented Nov 14, 2024 •
edited
Loading

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@ehigham ehigham mentioned this pull request Nov 14, 2024
Open
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch 3 times, most recently from 8d65393 to 39eb23a Compare November 15, 2024 20:58
@ehigham ehigham marked this pull request as ready for review November 15, 2024 20:58
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 39eb23a to 20ea9e5 Compare November 19, 2024 04:58
@ehigham ehigham force-pushed the ehigham/tmpdir-per-query branch from d5575e9 to 2543350 Compare November 19, 2024 17:09
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch 2 times, most recently from fd48477 to 5041735 Compare November 19, 2024 18:11
@ehigham ehigham force-pushed the ehigham/tmpdir-per-query branch from 2543350 to 7965c8f Compare November 20, 2024 17:01
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 5041735 to 207fa9b Compare November 20, 2024 17:01
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch 2 times, most recently from 7ee594a to e47b948 Compare December 5, 2024 19:45
@ehigham ehigham mentioned this pull request Dec 5, 2024
Open
1 task
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from e47b948 to df2624c Compare December 5, 2024 19:58
@ehigham ehigham mentioned this pull request Dec 5, 2024
Open
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch 3 times, most recently from 5828ffd to d8e0986 Compare December 5, 2024 21:42
@ehigham ehigham changed the base branch from ehigham/tmpdir-per-query to ehigham/optimise-implicit-context December 5, 2024 21:42
This was referenced Dec 5, 2024
Merged
Merged
Merged
Merged
Closed
Open
Open
Open
Closed
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 78afb48 to 2214856 Compare January 31, 2025 19:28
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from cfc5dc4 to 67809f1 Compare January 31, 2025 21:22
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 2214856 to 573cf80 Compare January 31, 2025 21:22
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 67809f1 to 30610c5 Compare February 3, 2025 17:21
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 573cf80 to ff1c0f3 Compare February 3, 2025 17:21
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 30610c5 to 5fc3af5 Compare February 3, 2025 18:02
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from ff1c0f3 to 61d4c21 Compare February 3, 2025 18:02
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 5fc3af5 to c28303f Compare February 3, 2025 20:52
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 61d4c21 to b77e1fc Compare February 3, 2025 20:52
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from c28303f to 1359af6 Compare February 3, 2025 23:07
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from b77e1fc to 199105c Compare February 3, 2025 23:07
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 1359af6 to 01bf46e Compare February 4, 2025 22:45
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 199105c to da6a06d Compare February 4, 2025 22:45
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 01bf46e to d0c7f40 Compare February 6, 2025 18:44
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from da6a06d to 06cf763 Compare February 6, 2025 18:44
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from d0c7f40 to afc3754 Compare February 20, 2025 16:50
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch 2 times, most recently from 6b1cb9e to f13de5e Compare February 20, 2025 17:28
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from f13de5e to 9deea97 Compare February 27, 2025 22:16
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 74c4273 to b5cb918 Compare February 27, 2025 22:17
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 9deea97 to ff7f6f7 Compare March 3, 2025 17:55
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from b5cb918 to 04b8401 Compare March 3, 2025 17:55
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from ff7f6f7 to dd7255d Compare March 17, 2025 21:54
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 04b8401 to 5a2655b Compare March 17, 2025 21:54
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from dd7255d to 8c5bf90 Compare March 17, 2025 22:48
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 5a2655b to 849173b Compare March 17, 2025 22:48
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 8c5bf90 to 6043648 Compare March 18, 2025 03:43
@ehigham
[benchmark] notebook for interactively computing benchmark config

Verified

This commit was signed with the committer’s verified signature.
scala-steward Scala Steward
GPG key ID: 7FE176635A140508
Verified
Learn about vigilant mode
Loading
Loading status checks…
800a3f3
@ehigham ehigham force-pushed the ehigham/benchmark-in-batch-variability branch from 6043648 to 800a3f3 Compare March 19, 2025 19:13
@ehigham ehigham force-pushed the ehigham/benchmark-suite-improvements branch from 3595f2e to 7a6d053 Compare March 19, 2025 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@ehigham