Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support call tracking within Q# code #1791

Merged
merged 10 commits into from
Aug 15, 2024
Merged

Support call tracking within Q# code #1791

merged 10 commits into from
Aug 15, 2024

Conversation

swernli
Copy link
Collaborator

@swernli swernli commented Jul 25, 2024

This change adds callable tracking APIs to the Diagnostics namespace that allow Q# code to register for tracking of specific callables and getting the count of times that callable was invoked. This is useful in katas and auto-grading scenarios to verify that exercise restrictions are being appropriately followed.

Fixes #1154

@swernli
Support call tracking within Q# code
920b49d 
This change adds callable tracking APIs to the Diagnostics namespace that allow Q# code to register for tracking of specific callables and getting the count of times that callable was invoked. This is useful in katas and auto-grading scenarios to verify that exercise restrictions are being appropriately followed.

Fixes #1154
@github-actions GitHub Actions
Copy link

Change in memory usage detected by benchmark.

Memory Report for b9a5957

Test This Branch On Main Difference
compile core + standard lib 17980128 bytes 17962616 bytes 17512 bytes

@github-actions GitHub Actions
Copy link

Benchmark for b9a5957

Click to view benchmark
Test Base PR %
Array append evaluation 338.7±2.94µs 339.6±7.24µs +0.27%
Array literal evaluation 192.7±3.61µs 187.2±3.24µs -2.85%
Array update evaluation 417.3±4.37µs 416.9±3.58µs -0.10%
Core + Standard library compilation 23.1±1.21ms 23.4±1.31ms +1.30%
Deutsch-Jozsa evaluation 4.9±0.06ms 4.9±0.05ms 0.00%
Large file parity evaluation 34.3±1.77ms 34.7±0.59ms +1.17%
Large input file compilation 13.3±0.67ms 13.5±0.61ms +1.50%
Large input file compilation (interpreter) 51.7±3.83ms 54.9±2.67ms +6.19%
Large nested iteration 32.6±0.37ms 33.0±1.14ms +1.23%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample 1564.6±53.66µs 1587.0±88.18µs +1.43%
Perform Runtime Capabilities Analysis (RCA) on large file sample 7.7±0.10ms 7.9±0.14ms +2.60%
Perform Runtime Capabilities Analysis (RCA) on teleport sample 1420.4±40.01µs 1454.5±129.60µs +2.40%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries 27.9±1.04ms 28.2±0.84ms +1.08%
Teleport evaluation 92.0±7.32µs 94.0±4.01µs +2.17%

@github-actions GitHub Actions
Copy link

Change in memory usage detected by benchmark.

Memory Report for acda466

Test This Branch On Main Difference
compile core + standard lib 17980128 bytes 17962616 bytes 17512 bytes

@github-actions GitHub Actions
Copy link

Benchmark for acda466

Click to view benchmark
Test Base PR %
Array append evaluation 335.0±2.40µs 341.8±2.80µs +2.03%
Array literal evaluation 173.1±2.79µs 174.1±6.21µs +0.58%
Array update evaluation 412.5±1.56µs 422.2±3.04µs +2.35%
Core + Standard library compilation 22.9±1.29ms 21.7±0.51ms -5.24%
Deutsch-Jozsa evaluation 4.9±0.06ms 5.0±0.06ms +2.04%
Large file parity evaluation 34.0±0.13ms 34.7±0.55ms +2.06%
Large input file compilation 13.1±0.28ms 13.2±0.43ms +0.76%
Large input file compilation (interpreter) 52.0±2.13ms 51.8±1.64ms -0.38%
Large nested iteration 32.2±0.53ms 33.2±0.18ms +3.11%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample 1599.2±57.74µs 1561.7±35.87µs -2.34%
Perform Runtime Capabilities Analysis (RCA) on large file sample 8.0±0.21ms 7.7±0.09ms -3.75%
Perform Runtime Capabilities Analysis (RCA) on teleport sample 1455.6±94.46µs 1430.8±56.14µs -1.70%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries 28.4±0.26ms 27.6±0.25ms -2.82%
Teleport evaluation 91.0±3.20µs 93.8±3.98µs +3.08%

@swernli swernli requested a review from tcNickolas August 5, 2024 05:00
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 5, 2024

Change in memory usage detected by benchmark.

Memory Report for eb7cad7

Test This Branch On Main Difference
compile core + standard lib 17985040 bytes 17963208 bytes 21832 bytes

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 5, 2024

Benchmark for eb7cad7

Click to view benchmark
Test Base PR %
Array append evaluation 339.9±3.43µs 334.6±9.60µs -1.56%
Array literal evaluation 186.4±4.19µs 171.9±1.05µs -7.78%
Array update evaluation 417.7±2.19µs 414.8±8.02µs -0.69%
Core + Standard library compilation 21.7±0.61ms 22.0±0.72ms +1.38%
Deutsch-Jozsa evaluation 5.0±0.06ms 5.1±0.08ms +2.00%
Large file parity evaluation 34.5±0.22ms 34.6±1.94ms +0.29%
Large input file compilation 12.9±0.25ms 13.0±0.21ms +0.78%
Large input file compilation (interpreter) 50.0±1.68ms 52.2±1.98ms +4.40%
Large nested iteration 33.3±1.20ms 32.7±0.31ms -1.80%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample 1567.4±32.93µs 1580.6±65.34µs +0.84%
Perform Runtime Capabilities Analysis (RCA) on large file sample 7.8±0.07ms 7.9±0.11ms +1.28%
Perform Runtime Capabilities Analysis (RCA) on teleport sample 1429.7±44.36µs 1438.1±99.22µs +0.59%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries 27.7±0.19ms 27.7±0.20ms 0.00%
Teleport evaluation 93.2±3.10µs 94.9±3.82µs +1.82%

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 5, 2024

Change in memory usage detected by benchmark.

Memory Report for 7695dea

Test This Branch On Main Difference
compile core + standard lib 17984728 bytes 17962896 bytes 21832 bytes

@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 5, 2024

Benchmark for 7695dea

Click to view benchmark
Test Base PR %
Array append evaluation 333.4±5.37µs 334.3±1.88µs +0.27%
Array literal evaluation 186.4±1.06µs 190.5±2.77µs +2.20%
Array update evaluation 417.0±5.24µs 416.5±4.10µs -0.12%
Core + Standard library compilation 21.6±0.66ms 21.5±0.46ms -0.46%
Deutsch-Jozsa evaluation 5.0±0.06ms 5.2±0.08ms +4.00%
Large file parity evaluation 34.5±0.39ms 34.6±0.82ms +0.29%
Large input file compilation 13.0±0.33ms 13.1±0.33ms +0.77%
Large input file compilation (interpreter) 50.9±2.11ms 54.1±3.50ms +6.29%
Large nested iteration 33.1±0.29ms 33.0±0.32ms -0.30%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample 1562.5±42.47µs 1573.6±28.30µs +0.71%
Perform Runtime Capabilities Analysis (RCA) on large file sample 7.7±0.08ms 7.9±0.30ms +2.60%
Perform Runtime Capabilities Analysis (RCA) on teleport sample 1425.1±41.70µs 1441.1±41.28µs +1.12%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries 27.7±0.17ms 27.7±0.37ms 0.00%
Teleport evaluation 92.3±3.77µs 96.1±3.43µs +4.12%

tcNickolas
tcNickolas reviewed Aug 14, 2024
View reviewed changes
library/std/src/diagnostics.qs Outdated Show resolved Hide resolved
library/std/src/diagnostics.qs Outdated Show resolved Hide resolved
library/std/src/diagnostics.qs Outdated Show resolved Hide resolved
library/std/src/diagnostics.qs Show resolved Hide resolved
library/src/tests/diagnostics.rs Show resolved Hide resolved
library/src/tests/diagnostics.rs Show resolved Hide resolved
@github-actions GitHub Actions
Copy link

Change in memory usage detected by benchmark.

Memory Report for dfb8b6d

Test This Branch On Main Difference
compile core + standard lib 17991976 bytes 17966344 bytes 25632 bytes

@github-actions GitHub Actions
Copy link

Benchmark for dfb8b6d

Click to view benchmark
Test Base PR %
Array append evaluation 350.6±4.35µs 340.6±7.85µs -2.85%
Array literal evaluation 192.2±0.83µs 190.3±1.76µs -0.99%
Array update evaluation 429.0±2.57µs 413.0±18.23µs -3.73%
Core + Standard library compilation 22.1±0.30ms 23.0±0.79ms +4.07%
Deutsch-Jozsa evaluation 5.0±0.05ms 4.9±0.18ms -2.00%
Large file parity evaluation 34.9±0.27ms 34.6±0.14ms -0.86%
Large input file compilation 14.1±0.30ms 13.9±0.36ms -1.42%
Large input file compilation (interpreter) 53.1±1.89ms 52.8±2.12ms -0.56%
Large nested iteration 34.5±0.14ms 32.7±0.18ms -5.22%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample 1564.5±43.28µs 1560.1±31.55µs -0.28%
Perform Runtime Capabilities Analysis (RCA) on large file sample 7.8±0.11ms 8.0±0.31ms +2.56%
Perform Runtime Capabilities Analysis (RCA) on teleport sample 1430.4±46.64µs 1429.0±52.36µs -0.10%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries 27.5±0.11ms 27.8±0.42ms +1.09%
Teleport evaluation 93.8±3.22µs 95.3±3.55µs +1.60%

@github-actions GitHub Actions
Copy link

Change in memory usage detected by benchmark.

Memory Report for dd8fcf4

Test This Branch On Main Difference
compile core + standard lib 17991976 bytes 17966344 bytes 25632 bytes

@github-actions GitHub Actions
Copy link

Benchmark for dd8fcf4

Click to view benchmark
Test Base PR %
Array append evaluation 333.5±2.49µs 326.1±3.16µs -2.22%
Array literal evaluation 194.0±3.99µs 172.5±3.13µs -11.08%
Array update evaluation 415.8±1.91µs 408.0±1.69µs -1.88%
Core + Standard library compilation 22.8±0.44ms 22.8±0.34ms 0.00%
Deutsch-Jozsa evaluation 5.0±0.07ms 4.9±0.06ms -2.00%
Large file parity evaluation 34.9±0.47ms 34.5±0.65ms -1.15%
Large input file compilation 14.2±0.40ms 14.3±0.42ms +0.70%
Large input file compilation (interpreter) 54.5±1.70ms 54.7±2.06ms +0.37%
Large nested iteration 33.0±0.24ms 32.6±0.18ms -1.21%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample 1562.9±43.76µs 1570.4±60.42µs +0.48%
Perform Runtime Capabilities Analysis (RCA) on large file sample 8.0±0.12ms 8.0±0.08ms 0.00%
Perform Runtime Capabilities Analysis (RCA) on teleport sample 1426.5±43.96µs 1435.4±55.53µs +0.62%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries 27.8±0.28ms 27.9±0.22ms +0.36%
Teleport evaluation 93.1±3.88µs 94.2±3.73µs +1.18%

@github-actions GitHub Actions
Copy link

Change in memory usage detected by benchmark.

Memory Report for 78971ae

Test This Branch On Main Difference
compile core + standard lib 17991952 bytes 17966344 bytes 25608 bytes

@github-actions GitHub Actions
Copy link

Benchmark for 78971ae

Click to view benchmark
Test Base PR %
Array append evaluation 335.2±2.87µs 326.3±1.78µs -2.66%
Array literal evaluation 197.7±4.63µs 171.6±0.94µs -13.20%
Array update evaluation 416.7±2.79µs 406.4±1.56µs -2.47%
Core + Standard library compilation 22.9±0.68ms 23.0±0.98ms +0.44%
Deutsch-Jozsa evaluation 5.0±0.07ms 4.9±0.05ms -2.00%
Large file parity evaluation 34.6±0.21ms 34.3±0.21ms -0.87%
Large input file compilation 14.2±0.47ms 14.2±0.65ms 0.00%
Large input file compilation (interpreter) 54.3±2.25ms 53.5±2.21ms -1.47%
Large nested iteration 32.9±0.19ms 33.0±0.89ms +0.30%
Perform Runtime Capabilities Analysis (RCA) on Deutsch-Jozsa sample 1569.7±58.77µs 1567.3±74.79µs -0.15%
Perform Runtime Capabilities Analysis (RCA) on large file sample 8.0±0.25ms 7.9±0.07ms -1.25%
Perform Runtime Capabilities Analysis (RCA) on teleport sample 1426.1±33.51µs 1427.8±47.58µs +0.12%
Perform Runtime Capabilities Analysis (RCA) on the core and std libraries 28.0±0.31ms 27.8±0.53ms -0.71%
Teleport evaluation 92.9±3.69µs 93.7±3.59µs +0.86%

@swernli swernli added this pull request to the merge queue Aug 15, 2024
Merged via the queue into main with commit 22c80c9 Aug 15, 2024
19 checks passed
@swernli swernli deleted the swernli/issue1154 branch August 15, 2024 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@billti billti billti approved these changes

@cesarzc cesarzc cesarzc approved these changes

@tcNickolas tcNickolas tcNickolas approved these changes

@DmitryVasilevsky DmitryVasilevsky Awaiting requested review from DmitryVasilevsky DmitryVasilevsky is a code owner

@idavis idavis Awaiting requested review from idavis idavis is a code owner

@minestarks minestarks Awaiting requested review from minestarks minestarks is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support AllowAtMostNCallsCA operation to limit the number of operation calls that can be done in the code
4 participants
@swernli @billti @cesarzc @tcNickolas