Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Test] Skip tokenizer tests if transformers is not in workspace #2744

Merged
merged 5 commits into from
Feb 3, 2025
Merged

[Test] Skip tokenizer tests if transformers is not in workspace #2744

merged 5 commits into from
Feb 3, 2025

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 3, 2025 •
edited
Loading

@vmoens
Update
cd83026 
[ghstack-poisoned]
@vmoens vmoens mentioned this pull request Feb 3, 2025
Merged
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 3, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2744

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job, 19 Pending, 4 Unrelated Failures

As of commit 0c13d93 with merge base ffa99b2 (image):

CANCELLED JOB - The following job was cancelled. Please retry:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

vmoens added a commit that referenced this pull request Feb 3, 2025
@vmoens
[Test] Skip tokenizer tests if transformers is not in workspace
ffc0d1e 
ghstack-source-id: dfb9792ca1680ef4bc00aa3e80eafe456274362d
Pull Request resolved: #2744
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 3, 2025
@vmoens
Update
3c135cd 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Feb 3, 2025
@vmoens
[Test] Skip tokenizer tests if transformers is not in workspace
32f83bf 
ghstack-source-id: 93026680075fed55827f374e026df0ebe5fef636
Pull Request resolved: #2744
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 3, 2025 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 149. Improved: $\large\color{#35bf28}42$. Worsened: $\large\color{#d91a1a}4$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.5083s 0.4296s 2.3275 Ops/s 2.1683 Ops/s $\textbf{\color{#35bf28}+7.34\%}$
test_transformed 1.0138s 0.9145s 1.0935 Ops/s 1.0503 Ops/s $\color{#35bf28}+4.11\%$
test_serial 1.4431s 1.3801s 0.7246 Ops/s 0.7088 Ops/s $\color{#35bf28}+2.22\%$
test_parallel 1.2778s 1.1966s 0.8357 Ops/s 0.7961 Ops/s $\color{#35bf28}+4.98\%$
test_step_mdp_speed[True-True-True-True-True] 0.2184ms 30.7450μs 32.5256 KOps/s 32.2772 KOps/s $\color{#35bf28}+0.77\%$
test_step_mdp_speed[True-True-True-True-False] 50.6050μs 17.6754μs 56.5759 KOps/s 55.3889 KOps/s $\color{#35bf28}+2.14\%$
test_step_mdp_speed[True-True-True-False-True] 60.5420μs 17.1079μs 58.4525 KOps/s 56.3432 KOps/s $\color{#35bf28}+3.74\%$
test_step_mdp_speed[True-True-True-False-False] 31.9600μs 9.9983μs 100.0169 KOps/s 97.7797 KOps/s $\color{#35bf28}+2.29\%$
test_step_mdp_speed[True-True-False-True-True] 79.8090μs 31.7901μs 31.4563 KOps/s 30.2708 KOps/s $\color{#35bf28}+3.92\%$
test_step_mdp_speed[True-True-False-True-False] 59.1400μs 19.7955μs 50.5165 KOps/s 50.7856 KOps/s $\color{#d91a1a}-0.53\%$
test_step_mdp_speed[True-True-False-False-True] 37.2190μs 18.7937μs 53.2094 KOps/s 51.1119 KOps/s $\color{#35bf28}+4.10\%$
test_step_mdp_speed[True-True-False-False-False] 47.1880μs 11.9525μs 83.6648 KOps/s 83.1155 KOps/s $\color{#35bf28}+0.66\%$
test_step_mdp_speed[True-False-True-True-True] 73.3360μs 33.7885μs 29.5958 KOps/s 28.5062 KOps/s $\color{#35bf28}+3.82\%$
test_step_mdp_speed[True-False-True-True-False] 61.9460μs 21.1339μs 47.3174 KOps/s 46.3184 KOps/s $\color{#35bf28}+2.16\%$
test_step_mdp_speed[True-False-True-False-True] 59.7410μs 18.7622μs 53.2987 KOps/s 51.4190 KOps/s $\color{#35bf28}+3.66\%$
test_step_mdp_speed[True-False-True-False-False] 43.8020μs 11.4194μs 87.5701 KOps/s 82.6797 KOps/s $\textbf{\color{#35bf28}+5.91\%}$
test_step_mdp_speed[True-False-False-True-True] 80.0790μs 35.0320μs 28.5453 KOps/s 27.4138 KOps/s $\color{#35bf28}+4.13\%$
test_step_mdp_speed[True-False-False-True-False] 55.4530μs 23.0286μs 43.4243 KOps/s 43.1547 KOps/s $\color{#35bf28}+0.62\%$
test_step_mdp_speed[True-False-False-False-True] 51.6460μs 19.8074μs 50.4861 KOps/s 47.3009 KOps/s $\textbf{\color{#35bf28}+6.73\%}$
test_step_mdp_speed[True-False-False-False-False] 45.4750μs 13.0472μs 76.6447 KOps/s 72.1061 KOps/s $\textbf{\color{#35bf28}+6.29\%}$
test_step_mdp_speed[False-True-True-True-True] 77.8450μs 32.8975μs 30.3974 KOps/s 28.7216 KOps/s $\textbf{\color{#35bf28}+5.83\%}$
test_step_mdp_speed[False-True-True-True-False] 53.5000μs 20.6915μs 48.3291 KOps/s 46.0407 KOps/s $\color{#35bf28}+4.97\%$
test_step_mdp_speed[False-True-True-False-True] 0.5378ms 20.7024μs 48.3036 KOps/s 45.1487 KOps/s $\textbf{\color{#35bf28}+6.99\%}$
test_step_mdp_speed[False-True-True-False-False] 58.3180μs 12.5175μs 79.8884 KOps/s 73.7819 KOps/s $\textbf{\color{#35bf28}+8.28\%}$
test_step_mdp_speed[False-True-False-True-True] 83.3650μs 34.1610μs 29.2731 KOps/s 27.4488 KOps/s $\textbf{\color{#35bf28}+6.65\%}$
test_step_mdp_speed[False-True-False-True-False] 56.4650μs 22.1937μs 45.0578 KOps/s 42.8279 KOps/s $\textbf{\color{#35bf28}+5.21\%}$
test_step_mdp_speed[False-True-False-False-True] 2.6609ms 22.4640μs 44.5157 KOps/s 40.8006 KOps/s $\textbf{\color{#35bf28}+9.11\%}$
test_step_mdp_speed[False-True-False-False-False] 42.9400μs 14.3127μs 69.8679 KOps/s 65.6967 KOps/s $\textbf{\color{#35bf28}+6.35\%}$
test_step_mdp_speed[False-False-True-True-True] 76.8240μs 36.6692μs 27.2708 KOps/s 26.0747 KOps/s $\color{#35bf28}+4.59\%$
test_step_mdp_speed[False-False-True-True-False] 66.6740μs 24.6786μs 40.5210 KOps/s 39.5437 KOps/s $\color{#35bf28}+2.47\%$
test_step_mdp_speed[False-False-True-False-True] 65.8730μs 23.2446μs 43.0207 KOps/s 40.7897 KOps/s $\textbf{\color{#35bf28}+5.47\%}$
test_step_mdp_speed[False-False-True-False-False] 44.3420μs 14.6908μs 68.0699 KOps/s 65.6838 KOps/s $\color{#35bf28}+3.63\%$
test_step_mdp_speed[False-False-False-True-True] 91.2600μs 39.0403μs 25.6145 KOps/s 25.1601 KOps/s $\color{#35bf28}+1.81\%$
test_step_mdp_speed[False-False-False-True-False] 87.5940μs 26.5097μs 37.7221 KOps/s 37.3058 KOps/s $\color{#35bf28}+1.12\%$
test_step_mdp_speed[False-False-False-False-True] 69.2490μs 24.8666μs 40.2146 KOps/s 39.2258 KOps/s $\color{#35bf28}+2.52\%$
test_step_mdp_speed[False-False-False-False-False] 0.2061ms 16.4964μs 60.6194 KOps/s 59.1985 KOps/s $\color{#35bf28}+2.40\%$
test_values[generalized_advantage_estimate-True-True] 11.2101ms 9.6164ms 103.9893 Ops/s 100.5353 Ops/s $\color{#35bf28}+3.44\%$
test_values[vec_generalized_advantage_estimate-True-True] 27.0542ms 23.6507ms 42.2821 Ops/s 41.1556 Ops/s $\color{#35bf28}+2.74\%$
test_values[td0_return_estimate-False-False] 0.2258ms 0.1765ms 5.6651 KOps/s 5.3424 KOps/s $\textbf{\color{#35bf28}+6.04\%}$
test_values[td1_return_estimate-False-False] 26.0633ms 23.8974ms 41.8456 Ops/s 40.4934 Ops/s $\color{#35bf28}+3.34\%$
test_values[vec_td1_return_estimate-False-False] 25.0378ms 23.3971ms 42.7403 Ops/s 40.9897 Ops/s $\color{#35bf28}+4.27\%$
test_values[td_lambda_return_estimate-True-False] 38.2414ms 35.2833ms 28.3420 Ops/s 28.0961 Ops/s $\color{#35bf28}+0.88\%$
test_values[vec_td_lambda_return_estimate-True-False] 26.9252ms 23.5216ms 42.5142 Ops/s 40.6111 Ops/s $\color{#35bf28}+4.69\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 10.7505ms 8.2928ms 120.5870 Ops/s 114.5647 Ops/s $\textbf{\color{#35bf28}+5.26\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.1751ms 1.7714ms 564.5101 Ops/s 523.7417 Ops/s $\textbf{\color{#35bf28}+7.78\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4822ms 0.3544ms 2.8220 KOps/s 2.7005 KOps/s $\color{#35bf28}+4.50\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 37.1338ms 35.7745ms 27.9528 Ops/s 23.0438 Ops/s $\textbf{\color{#35bf28}+21.30\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 4.1807ms 3.2979ms 303.2204 Ops/s 290.3671 Ops/s $\color{#35bf28}+4.43\%$
test_dqn_speed[False-None] 5.8221ms 1.3639ms 733.1891 Ops/s 701.7017 Ops/s $\color{#35bf28}+4.49\%$
test_dqn_speed[False-backward] 1.9459ms 1.8184ms 549.9478 Ops/s 528.2003 Ops/s $\color{#35bf28}+4.12\%$
test_dqn_speed[True-None] 0.8019ms 0.4603ms 2.1727 KOps/s 2.0651 KOps/s $\textbf{\color{#35bf28}+5.21\%}$
test_dqn_speed[True-backward] 0.9473ms 0.8940ms 1.1185 KOps/s 1.0966 KOps/s $\color{#35bf28}+2.00\%$
test_dqn_speed[reduce-overhead-None] 0.7376ms 0.4694ms 2.1304 KOps/s 1.7027 KOps/s $\textbf{\color{#35bf28}+25.12\%}$
test_dqn_speed[reduce-overhead-backward] 0.9805ms 0.8572ms 1.1666 KOps/s 1.0683 KOps/s $\textbf{\color{#35bf28}+9.20\%}$
test_ddpg_speed[False-None] 3.6187ms 2.7683ms 361.2282 Ops/s 342.0438 Ops/s $\textbf{\color{#35bf28}+5.61\%}$
test_ddpg_speed[False-backward] 4.1362ms 3.8625ms 258.9023 Ops/s 246.1023 Ops/s $\textbf{\color{#35bf28}+5.20\%}$
test_ddpg_speed[True-None] 1.6192ms 1.2090ms 827.1142 Ops/s 810.8040 Ops/s $\color{#35bf28}+2.01\%$
test_ddpg_speed[True-backward] 2.4704ms 2.0809ms 480.5637 Ops/s 466.2034 Ops/s $\color{#35bf28}+3.08\%$
test_ddpg_speed[reduce-overhead-None] 1.5348ms 1.1848ms 843.9946 Ops/s 810.3025 Ops/s $\color{#35bf28}+4.16\%$
test_ddpg_speed[reduce-overhead-backward] 2.1668ms 2.0597ms 485.4982 Ops/s 468.1845 Ops/s $\color{#35bf28}+3.70\%$
test_sac_speed[False-None] 8.7108ms 7.6788ms 130.2282 Ops/s 122.8091 Ops/s $\textbf{\color{#35bf28}+6.04\%}$
test_sac_speed[False-backward] 11.4199ms 10.1554ms 98.4701 Ops/s 90.9062 Ops/s $\textbf{\color{#35bf28}+8.32\%}$
test_sac_speed[True-None] 9.7126ms 2.1452ms 466.1476 Ops/s 476.2634 Ops/s $\color{#d91a1a}-2.12\%$
test_sac_speed[True-backward] 3.7934ms 3.6793ms 271.7941 Ops/s 266.3607 Ops/s $\color{#35bf28}+2.04\%$
test_sac_speed[reduce-overhead-None] 2.5648ms 2.0174ms 495.6846 Ops/s 478.6022 Ops/s $\color{#35bf28}+3.57\%$
test_sac_speed[reduce-overhead-backward] 3.8053ms 3.6978ms 270.4319 Ops/s 265.4433 Ops/s $\color{#35bf28}+1.88\%$
test_redq_speed[False-None] 14.5404ms 12.6816ms 78.8542 Ops/s 77.5123 Ops/s $\color{#35bf28}+1.73\%$
test_redq_speed[False-backward] 23.7019ms 21.7936ms 45.8850 Ops/s 45.0598 Ops/s $\color{#35bf28}+1.83\%$
test_redq_speed[True-None] 5.3420ms 4.4951ms 222.4638 Ops/s 207.6458 Ops/s $\textbf{\color{#35bf28}+7.14\%}$
test_redq_speed[True-backward] 13.2131ms 11.4949ms 86.9950 Ops/s 82.6953 Ops/s $\textbf{\color{#35bf28}+5.20\%}$
test_redq_speed[reduce-overhead-None] 5.2355ms 4.4513ms 224.6558 Ops/s 205.2508 Ops/s $\textbf{\color{#35bf28}+9.45\%}$
test_redq_speed[reduce-overhead-backward] 11.9544ms 11.6459ms 85.8675 Ops/s 81.4730 Ops/s $\textbf{\color{#35bf28}+5.39\%}$
test_redq_deprec_speed[False-None] 19.0210ms 12.8747ms 77.6715 Ops/s 77.6124 Ops/s $\color{#35bf28}+0.08\%$
test_redq_deprec_speed[False-backward] 18.6694ms 18.0306ms 55.4613 Ops/s 53.7717 Ops/s $\color{#35bf28}+3.14\%$
test_redq_deprec_speed[True-None] 4.4658ms 3.7613ms 265.8686 Ops/s 260.1219 Ops/s $\color{#35bf28}+2.21\%$
test_redq_deprec_speed[True-backward] 8.2217ms 8.0308ms 124.5199 Ops/s 122.4089 Ops/s $\color{#35bf28}+1.72\%$
test_redq_deprec_speed[reduce-overhead-None] 4.2187ms 3.7645ms 265.6417 Ops/s 261.0413 Ops/s $\color{#35bf28}+1.76\%$
test_redq_deprec_speed[reduce-overhead-backward] 9.1149ms 8.0724ms 123.8795 Ops/s 122.3316 Ops/s $\color{#35bf28}+1.27\%$
test_td3_speed[False-None] 9.4072ms 7.9591ms 125.6423 Ops/s 120.3287 Ops/s $\color{#35bf28}+4.42\%$
test_td3_speed[False-backward] 10.6279ms 10.2854ms 97.2255 Ops/s 89.7607 Ops/s $\textbf{\color{#35bf28}+8.32\%}$
test_td3_speed[True-None] 1.9389ms 1.7355ms 576.2193 Ops/s 552.2561 Ops/s $\color{#35bf28}+4.34\%$
test_td3_speed[True-backward] 3.4684ms 3.3296ms 300.3408 Ops/s 279.2057 Ops/s $\textbf{\color{#35bf28}+7.57\%}$
test_td3_speed[reduce-overhead-None] 1.9180ms 1.7472ms 572.3538 Ops/s 547.9294 Ops/s $\color{#35bf28}+4.46\%$
test_td3_speed[reduce-overhead-backward] 3.4158ms 3.3226ms 300.9674 Ops/s 273.4337 Ops/s $\textbf{\color{#35bf28}+10.07\%}$
test_cql_speed[False-None] 39.3285ms 36.3407ms 27.5174 Ops/s 26.0523 Ops/s $\textbf{\color{#35bf28}+5.62\%}$
test_cql_speed[False-backward] 51.3080ms 46.6134ms 21.4531 Ops/s 20.3915 Ops/s $\textbf{\color{#35bf28}+5.21\%}$
test_cql_speed[True-None] 16.9872ms 15.6449ms 63.9184 Ops/s 60.0539 Ops/s $\textbf{\color{#35bf28}+6.44\%}$
test_cql_speed[True-backward] 23.4381ms 22.4085ms 44.6259 Ops/s 42.0787 Ops/s $\textbf{\color{#35bf28}+6.05\%}$
test_cql_speed[reduce-overhead-None] 16.6102ms 15.6189ms 64.0252 Ops/s 59.7888 Ops/s $\textbf{\color{#35bf28}+7.09\%}$
test_cql_speed[reduce-overhead-backward] 24.4413ms 23.2702ms 42.9734 Ops/s 41.7256 Ops/s $\color{#35bf28}+2.99\%$
test_a2c_speed[False-None] 8.1467ms 7.1587ms 139.6911 Ops/s 129.1625 Ops/s $\textbf{\color{#35bf28}+8.15\%}$
test_a2c_speed[False-backward] 14.3184ms 13.9658ms 71.6035 Ops/s 66.9425 Ops/s $\textbf{\color{#35bf28}+6.96\%}$
test_a2c_speed[True-None] 4.1523ms 3.6587ms 273.3198 Ops/s 248.6069 Ops/s $\textbf{\color{#35bf28}+9.94\%}$
test_a2c_speed[True-backward] 10.3196ms 9.9506ms 100.4961 Ops/s 98.6132 Ops/s $\color{#35bf28}+1.91\%$
test_a2c_speed[reduce-overhead-None] 3.9575ms 3.6882ms 271.1352 Ops/s 266.0779 Ops/s $\color{#35bf28}+1.90\%$
test_a2c_speed[reduce-overhead-backward] 10.3201ms 9.9470ms 100.5332 Ops/s 100.1138 Ops/s $\color{#35bf28}+0.42\%$
test_ppo_speed[False-None] 9.9738ms 7.4648ms 133.9628 Ops/s 132.3753 Ops/s $\color{#35bf28}+1.20\%$
test_ppo_speed[False-backward] 14.9855ms 14.4159ms 69.3680 Ops/s 69.0344 Ops/s $\color{#35bf28}+0.48\%$
test_ppo_speed[True-None] 4.3542ms 4.0243ms 248.4935 Ops/s 254.0330 Ops/s $\color{#d91a1a}-2.18\%$
test_ppo_speed[True-backward] 10.7016ms 9.8099ms 101.9383 Ops/s 100.9897 Ops/s $\color{#35bf28}+0.94\%$
test_ppo_speed[reduce-overhead-None] 4.4102ms 4.0099ms 249.3847 Ops/s 243.7972 Ops/s $\color{#35bf28}+2.29\%$
test_ppo_speed[reduce-overhead-backward] 10.2579ms 9.8572ms 101.4484 Ops/s 100.3521 Ops/s $\color{#35bf28}+1.09\%$
test_reinforce_speed[False-None] 7.6195ms 6.4888ms 154.1114 Ops/s 151.1048 Ops/s $\color{#35bf28}+1.99\%$
test_reinforce_speed[False-backward] 11.2407ms 9.7263ms 102.8141 Ops/s 101.2203 Ops/s $\color{#35bf28}+1.57\%$
test_reinforce_speed[True-None] 3.4299ms 2.9878ms 334.6981 Ops/s 321.4249 Ops/s $\color{#35bf28}+4.13\%$
test_reinforce_speed[True-backward] 9.3123ms 8.8878ms 112.5132 Ops/s 101.8715 Ops/s $\textbf{\color{#35bf28}+10.45\%}$
test_reinforce_speed[reduce-overhead-None] 3.3931ms 2.9999ms 333.3422 Ops/s 329.1557 Ops/s $\color{#35bf28}+1.27\%$
test_reinforce_speed[reduce-overhead-backward] 10.4700ms 8.9010ms 112.3468 Ops/s 114.5076 Ops/s $\color{#d91a1a}-1.89\%$
test_iql_speed[False-None] 33.3258ms 32.0986ms 31.1540 Ops/s 30.1080 Ops/s $\color{#35bf28}+3.47\%$
test_iql_speed[False-backward] 46.6021ms 44.9675ms 22.2383 Ops/s 22.0236 Ops/s $\color{#35bf28}+0.97\%$
test_iql_speed[True-None] 11.9784ms 10.8680ms 92.0129 Ops/s 84.9335 Ops/s $\textbf{\color{#35bf28}+8.34\%}$
test_iql_speed[True-backward] 27.9004ms 21.7748ms 45.9247 Ops/s 45.5951 Ops/s $\color{#35bf28}+0.72\%$
test_iql_speed[reduce-overhead-None] 11.9573ms 10.9041ms 91.7084 Ops/s 90.4499 Ops/s $\color{#35bf28}+1.39\%$
test_iql_speed[reduce-overhead-backward] 23.3823ms 21.5642ms 46.3732 Ops/s 46.2888 Ops/s $\color{#35bf28}+0.18\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.3964ms 4.7099ms 212.3192 Ops/s 212.1988 Ops/s $\color{#35bf28}+0.06\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8423ms 0.5128ms 1.9500 KOps/s 1.9665 KOps/s $\color{#d91a1a}-0.84\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7299ms 0.4871ms 2.0531 KOps/s 2.0570 KOps/s $\color{#d91a1a}-0.19\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.9741ms 4.4746ms 223.4840 Ops/s 223.0055 Ops/s $\color{#35bf28}+0.21\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.9265ms 0.4971ms 2.0117 KOps/s 1.9869 KOps/s $\color{#35bf28}+1.25\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.8055ms 0.4779ms 2.0926 KOps/s 2.1426 KOps/s $\color{#d91a1a}-2.33\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.3588ms 1.6611ms 602.0167 Ops/s 606.3318 Ops/s $\color{#d91a1a}-0.71\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.0880ms 1.6125ms 620.1585 Ops/s 632.1867 Ops/s $\color{#d91a1a}-1.90\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.7623ms 4.6185ms 216.5196 Ops/s 214.1877 Ops/s $\color{#35bf28}+1.09\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2.3138ms 0.6480ms 1.5433 KOps/s 1.5445 KOps/s $\color{#d91a1a}-0.08\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9668ms 0.6213ms 1.6096 KOps/s 1.6286 KOps/s $\color{#d91a1a}-1.17\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.0486ms 4.5191ms 221.2807 Ops/s 219.3616 Ops/s $\color{#35bf28}+0.87\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7843ms 0.5021ms 1.9915 KOps/s 1.9531 KOps/s $\color{#35bf28}+1.96\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.9273ms 0.5201ms 1.9227 KOps/s 2.0578 KOps/s $\textbf{\color{#d91a1a}-6.57\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 7.2514ms 4.5157ms 221.4472 Ops/s 224.3036 Ops/s $\color{#d91a1a}-1.27\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.9488ms 0.4982ms 2.0074 KOps/s 2.0024 KOps/s $\color{#35bf28}+0.25\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.9102ms 0.4798ms 2.0844 KOps/s 2.0523 KOps/s $\color{#35bf28}+1.56\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.1963ms 4.6519ms 214.9647 Ops/s 219.7169 Ops/s $\color{#d91a1a}-2.16\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.0243ms 0.6378ms 1.5678 KOps/s 1.5607 KOps/s $\color{#35bf28}+0.46\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8947ms 0.6193ms 1.6148 KOps/s 1.5988 KOps/s $\color{#35bf28}+1.00\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 5.3340ms 4.1432ms 241.3572 Ops/s 252.9757 Ops/s $\color{#d91a1a}-4.59\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 7.9653ms 2.3463ms 426.2081 Ops/s 449.9969 Ops/s $\textbf{\color{#d91a1a}-5.29\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 4.9850ms 1.3651ms 732.5375 Ops/s 786.7126 Ops/s $\textbf{\color{#d91a1a}-6.89\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 6.0412ms 4.2203ms 236.9483 Ops/s 35.6156 Ops/s $\textbf{\color{#35bf28}+565.29\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 9.0563ms 2.5027ms 399.5671 Ops/s 392.7742 Ops/s $\color{#35bf28}+1.73\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 2.8995ms 1.3286ms 752.6505 Ops/s 781.9027 Ops/s $\color{#d91a1a}-3.74\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.4294s 12.8840ms 77.6158 Ops/s 222.7783 Ops/s $\textbf{\color{#d91a1a}-65.16\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 6.7733ms 2.4660ms 405.5200 Ops/s 403.2102 Ops/s $\color{#35bf28}+0.57\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 6.7978ms 1.4704ms 680.0978 Ops/s 709.3154 Ops/s $\color{#d91a1a}-4.12\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 13.8033ms 11.4106ms 87.6381 Ops/s 82.6170 Ops/s $\textbf{\color{#35bf28}+6.08\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 15.8337ms 14.5839ms 68.5687 Ops/s 68.4305 Ops/s $\color{#35bf28}+0.20\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 21.2673ms 20.2477ms 49.3883 Ops/s 48.5320 Ops/s $\color{#35bf28}+1.76\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 15.7585ms 14.6867ms 68.0888 Ops/s 70.6129 Ops/s $\color{#d91a1a}-3.57\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 22.2395ms 20.4860ms 48.8138 Ops/s 49.3972 Ops/s $\color{#d91a1a}-1.18\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 17.2510ms 16.1533ms 61.9069 Ops/s 64.5250 Ops/s $\color{#d91a1a}-4.06\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 3, 2025 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 149. Improved: $\large\color{#35bf28}12$. Worsened: $\large\color{#d91a1a}12$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.8357s 0.7481s 1.3367 Ops/s 1.3923 Ops/s $\color{#d91a1a}-3.99\%$
test_transformed 1.3984s 1.3105s 0.7631 Ops/s 0.7784 Ops/s $\color{#d91a1a}-1.97\%$
test_serial 2.1630s 2.1573s 0.4636 Ops/s 0.4659 Ops/s $\color{#d91a1a}-0.50\%$
test_parallel 1.8543s 1.8315s 0.5460 Ops/s 0.5433 Ops/s $\color{#35bf28}+0.49\%$
test_step_mdp_speed[True-True-True-True-True] 0.2200ms 40.9872μs 24.3978 KOps/s 25.0667 KOps/s $\color{#d91a1a}-2.67\%$
test_step_mdp_speed[True-True-True-True-False] 66.8210μs 23.6475μs 42.2878 KOps/s 42.3044 KOps/s $\color{#d91a1a}-0.04\%$
test_step_mdp_speed[True-True-True-False-True] 55.8600μs 22.4342μs 44.5748 KOps/s 44.3798 KOps/s $\color{#35bf28}+0.44\%$
test_step_mdp_speed[True-True-True-False-False] 38.2900μs 13.0474μs 76.6435 KOps/s 77.3727 KOps/s $\color{#d91a1a}-0.94\%$
test_step_mdp_speed[True-True-False-True-True] 87.7110μs 43.3215μs 23.0832 KOps/s 23.6358 KOps/s $\color{#d91a1a}-2.34\%$
test_step_mdp_speed[True-True-False-True-False] 58.0300μs 25.8612μs 38.6680 KOps/s 39.1745 KOps/s $\color{#d91a1a}-1.29\%$
test_step_mdp_speed[True-True-False-False-True] 65.2210μs 25.4656μs 39.2687 KOps/s 39.4659 KOps/s $\color{#d91a1a}-0.50\%$
test_step_mdp_speed[True-True-False-False-False] 47.7000μs 15.6038μs 64.0869 KOps/s 64.9085 KOps/s $\color{#d91a1a}-1.27\%$
test_step_mdp_speed[True-False-True-True-True] 80.8910μs 46.0982μs 21.6928 KOps/s 22.2765 KOps/s $\color{#d91a1a}-2.62\%$
test_step_mdp_speed[True-False-True-True-False] 81.3200μs 28.1776μs 35.4892 KOps/s 35.4848 KOps/s $\color{#35bf28}+0.01\%$
test_step_mdp_speed[True-False-True-False-True] 65.3810μs 25.1854μs 39.7056 KOps/s 39.4569 KOps/s $\color{#35bf28}+0.63\%$
test_step_mdp_speed[True-False-True-False-False] 46.8210μs 15.4848μs 64.5794 KOps/s 64.3579 KOps/s $\color{#35bf28}+0.34\%$
test_step_mdp_speed[True-False-False-True-True] 96.9610μs 47.8537μs 20.8970 KOps/s 20.9391 KOps/s $\color{#d91a1a}-0.20\%$
test_step_mdp_speed[True-False-False-True-False] 61.9210μs 30.2643μs 33.0422 KOps/s 33.1387 KOps/s $\color{#d91a1a}-0.29\%$
test_step_mdp_speed[True-False-False-False-True] 70.6700μs 26.8346μs 37.2653 KOps/s 36.4047 KOps/s $\color{#35bf28}+2.36\%$
test_step_mdp_speed[True-False-False-False-False] 48.2400μs 17.5270μs 57.0549 KOps/s 56.2215 KOps/s $\color{#35bf28}+1.48\%$
test_step_mdp_speed[False-True-True-True-True] 90.1510μs 45.6379μs 21.9116 KOps/s 22.0474 KOps/s $\color{#d91a1a}-0.62\%$
test_step_mdp_speed[False-True-True-True-False] 68.0200μs 28.1955μs 35.4666 KOps/s 35.0829 KOps/s $\color{#35bf28}+1.09\%$
test_step_mdp_speed[False-True-True-False-True] 2.5741ms 29.0204μs 34.4585 KOps/s 34.4803 KOps/s $\color{#d91a1a}-0.06\%$
test_step_mdp_speed[False-True-True-False-False] 42.5200μs 17.3773μs 57.5465 KOps/s 58.5507 KOps/s $\color{#d91a1a}-1.72\%$
test_step_mdp_speed[False-True-False-True-True] 78.8600μs 48.2985μs 20.7046 KOps/s 21.0246 KOps/s $\color{#d91a1a}-1.52\%$
test_step_mdp_speed[False-True-False-True-False] 62.4900μs 30.9258μs 32.3355 KOps/s 32.6442 KOps/s $\color{#d91a1a}-0.95\%$
test_step_mdp_speed[False-True-False-False-True] 80.0900μs 31.7696μs 31.4766 KOps/s 31.8517 KOps/s $\color{#d91a1a}-1.18\%$
test_step_mdp_speed[False-True-False-False-False] 53.1800μs 19.8190μs 50.4568 KOps/s 51.5653 KOps/s $\color{#d91a1a}-2.15\%$
test_step_mdp_speed[False-False-True-True-True] 83.6300μs 50.7559μs 19.7021 KOps/s 20.0141 KOps/s $\color{#d91a1a}-1.56\%$
test_step_mdp_speed[False-False-True-True-False] 67.7000μs 33.2289μs 30.0943 KOps/s 30.2863 KOps/s $\color{#d91a1a}-0.63\%$
test_step_mdp_speed[False-False-True-False-True] 60.5200μs 31.3422μs 31.9059 KOps/s 32.0750 KOps/s $\color{#d91a1a}-0.53\%$
test_step_mdp_speed[False-False-True-False-False] 46.7410μs 19.8172μs 50.4613 KOps/s 51.1891 KOps/s $\color{#d91a1a}-1.42\%$
test_step_mdp_speed[False-False-False-True-True] 88.3910μs 51.9006μs 19.2676 KOps/s 19.6137 KOps/s $\color{#d91a1a}-1.76\%$
test_step_mdp_speed[False-False-False-True-False] 61.9400μs 35.6743μs 28.0314 KOps/s 28.6492 KOps/s $\color{#d91a1a}-2.16\%$
test_step_mdp_speed[False-False-False-False-True] 63.3910μs 33.2363μs 30.0875 KOps/s 30.4734 KOps/s $\color{#d91a1a}-1.27\%$
test_step_mdp_speed[False-False-False-False-False] 49.1500μs 21.6280μs 46.2364 KOps/s 46.0874 KOps/s $\color{#35bf28}+0.32\%$
test_values[generalized_advantage_estimate-True-True] 25.7345ms 25.2025ms 39.6786 Ops/s 40.2269 Ops/s $\color{#d91a1a}-1.36\%$
test_values[vec_generalized_advantage_estimate-True-True] 0.1013s 2.9323ms 341.0239 Ops/s 342.1550 Ops/s $\color{#d91a1a}-0.33\%$
test_values[td0_return_estimate-False-False] 0.1078ms 81.4193μs 12.2821 KOps/s 12.6767 KOps/s $\color{#d91a1a}-3.11\%$
test_values[td1_return_estimate-False-False] 56.8206ms 56.5485ms 17.6839 Ops/s 17.9976 Ops/s $\color{#d91a1a}-1.74\%$
test_values[vec_td1_return_estimate-False-False] 1.2967ms 1.0948ms 913.3788 Ops/s 922.0603 Ops/s $\color{#d91a1a}-0.94\%$
test_values[td_lambda_return_estimate-True-False] 94.1584ms 90.6242ms 11.0346 Ops/s 11.4063 Ops/s $\color{#d91a1a}-3.26\%$
test_values[vec_td_lambda_return_estimate-True-False] 1.3890ms 1.0965ms 912.0051 Ops/s 926.2272 Ops/s $\color{#d91a1a}-1.54\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 25.2627ms 25.1103ms 39.8242 Ops/s 40.5864 Ops/s $\color{#d91a1a}-1.88\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1.0503ms 0.7687ms 1.3009 KOps/s 1.3231 KOps/s $\color{#d91a1a}-1.68\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7623ms 0.6782ms 1.4746 KOps/s 1.4899 KOps/s $\color{#d91a1a}-1.03\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5428ms 1.4973ms 667.8744 Ops/s 673.2082 Ops/s $\color{#d91a1a}-0.79\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.7509ms 0.6911ms 1.4470 KOps/s 1.4577 KOps/s $\color{#d91a1a}-0.74\%$
test_dqn_speed[False-None] 1.6071ms 1.5074ms 663.3929 Ops/s 664.6316 Ops/s $\color{#d91a1a}-0.19\%$
test_dqn_speed[False-backward] 2.4024ms 2.1459ms 466.0122 Ops/s 476.2233 Ops/s $\color{#d91a1a}-2.14\%$
test_dqn_speed[True-None] 0.7128ms 0.5666ms 1.7650 KOps/s 1.7857 KOps/s $\color{#d91a1a}-1.16\%$
test_dqn_speed[True-backward] 1.1845ms 1.1292ms 885.5503 Ops/s 881.5158 Ops/s $\color{#35bf28}+0.46\%$
test_dqn_speed[reduce-overhead-None] 0.9841ms 0.5804ms 1.7231 KOps/s 1.7375 KOps/s $\color{#d91a1a}-0.83\%$
test_dqn_speed[reduce-overhead-backward] 1.0413ms 0.9709ms 1.0299 KOps/s 1.0343 KOps/s $\color{#d91a1a}-0.42\%$
test_ddpg_speed[False-None] 3.2485ms 2.8830ms 346.8641 Ops/s 349.9630 Ops/s $\color{#d91a1a}-0.89\%$
test_ddpg_speed[False-backward] 4.6287ms 4.1726ms 239.6604 Ops/s 241.1090 Ops/s $\color{#d91a1a}-0.60\%$
test_ddpg_speed[True-None] 1.5560ms 1.3497ms 740.9175 Ops/s 738.9045 Ops/s $\color{#35bf28}+0.27\%$
test_ddpg_speed[True-backward] 2.5524ms 2.4811ms 403.0469 Ops/s 406.0590 Ops/s $\color{#d91a1a}-0.74\%$
test_ddpg_speed[reduce-overhead-None] 1.7696ms 1.3627ms 733.8131 Ops/s 725.9322 Ops/s $\color{#35bf28}+1.09\%$
test_ddpg_speed[reduce-overhead-backward] 1.9727ms 1.9050ms 524.9310 Ops/s 521.4468 Ops/s $\color{#35bf28}+0.67\%$
test_sac_speed[False-None] 8.5887ms 8.0942ms 123.5445 Ops/s 124.7079 Ops/s $\color{#d91a1a}-0.93\%$
test_sac_speed[False-backward] 11.6577ms 11.0774ms 90.2740 Ops/s 91.4771 Ops/s $\color{#d91a1a}-1.32\%$
test_sac_speed[True-None] 2.0040ms 1.8600ms 537.6304 Ops/s 536.0132 Ops/s $\color{#35bf28}+0.30\%$
test_sac_speed[True-backward] 3.6644ms 3.5938ms 278.2577 Ops/s 276.6120 Ops/s $\color{#35bf28}+0.59\%$
test_sac_speed[reduce-overhead-None] 21.7107ms 12.1761ms 82.1282 Ops/s 82.7106 Ops/s $\color{#d91a1a}-0.70\%$
test_sac_speed[reduce-overhead-backward] 1.8042ms 1.6308ms 613.1979 Ops/s 603.4674 Ops/s $\color{#35bf28}+1.61\%$
test_redq_speed[False-None] 8.0181ms 7.5500ms 132.4504 Ops/s 132.7855 Ops/s $\color{#d91a1a}-0.25\%$
test_redq_speed[False-backward] 11.8825ms 11.3896ms 87.7990 Ops/s 88.4532 Ops/s $\color{#d91a1a}-0.74\%$
test_redq_speed[True-None] 3.2414ms 2.3786ms 420.4156 Ops/s 427.7272 Ops/s $\color{#d91a1a}-1.71\%$
test_redq_speed[True-backward] 4.4471ms 4.2291ms 236.4558 Ops/s 235.4365 Ops/s $\color{#35bf28}+0.43\%$
test_redq_speed[reduce-overhead-None] 2.5673ms 2.3749ms 421.0740 Ops/s 422.0895 Ops/s $\color{#d91a1a}-0.24\%$
test_redq_speed[reduce-overhead-backward] 4.6134ms 4.2506ms 235.2593 Ops/s 234.8337 Ops/s $\color{#35bf28}+0.18\%$
test_redq_deprec_speed[False-None] 9.6192ms 9.2013ms 108.6800 Ops/s 110.1875 Ops/s $\color{#d91a1a}-1.37\%$
test_redq_deprec_speed[False-backward] 12.8076ms 12.3369ms 81.0577 Ops/s 81.8002 Ops/s $\color{#d91a1a}-0.91\%$
test_redq_deprec_speed[True-None] 2.9883ms 2.6971ms 370.7686 Ops/s 372.1553 Ops/s $\color{#d91a1a}-0.37\%$
test_redq_deprec_speed[True-backward] 4.4040ms 4.3425ms 230.2833 Ops/s 215.4373 Ops/s $\textbf{\color{#35bf28}+6.89\%}$
test_redq_deprec_speed[reduce-overhead-None] 3.6237ms 2.6730ms 374.1118 Ops/s 368.2227 Ops/s $\color{#35bf28}+1.60\%$
test_redq_deprec_speed[reduce-overhead-backward] 4.6504ms 4.5525ms 219.6619 Ops/s 218.6575 Ops/s $\color{#35bf28}+0.46\%$
test_td3_speed[False-None] 8.0698ms 7.9695ms 125.4784 Ops/s 125.5925 Ops/s $\color{#d91a1a}-0.09\%$
test_td3_speed[False-backward] 10.9539ms 10.5330ms 94.9395 Ops/s 95.1113 Ops/s $\color{#d91a1a}-0.18\%$
test_td3_speed[True-None] 1.6808ms 1.6579ms 603.1872 Ops/s 583.2071 Ops/s $\color{#35bf28}+3.43\%$
test_td3_speed[True-backward] 3.4730ms 3.4248ms 291.9845 Ops/s 309.0474 Ops/s $\textbf{\color{#d91a1a}-5.52\%}$
test_td3_speed[reduce-overhead-None] 54.8329ms 26.4631ms 37.7884 Ops/s 36.0833 Ops/s $\color{#35bf28}+4.73\%$
test_td3_speed[reduce-overhead-backward] 1.5069ms 1.4384ms 695.2200 Ops/s 720.6480 Ops/s $\color{#d91a1a}-3.53\%$
test_cql_speed[False-None] 17.2888ms 16.8386ms 59.3875 Ops/s 59.0610 Ops/s $\color{#35bf28}+0.55\%$
test_cql_speed[False-backward] 22.4919ms 21.9821ms 45.4915 Ops/s 45.6075 Ops/s $\color{#d91a1a}-0.25\%$
test_cql_speed[True-None] 3.3626ms 3.2849ms 304.4201 Ops/s 301.5323 Ops/s $\color{#35bf28}+0.96\%$
test_cql_speed[True-backward] 6.1743ms 5.7737ms 173.1999 Ops/s 172.6295 Ops/s $\color{#35bf28}+0.33\%$
test_cql_speed[reduce-overhead-None] 20.9743ms 13.2097ms 75.7022 Ops/s 57.6595 Ops/s $\textbf{\color{#35bf28}+31.29\%}$
test_cql_speed[reduce-overhead-backward] 2.1807ms 2.0102ms 497.4646 Ops/s 530.8893 Ops/s $\textbf{\color{#d91a1a}-6.30\%}$
test_a2c_speed[False-None] 3.2605ms 3.1813ms 314.3380 Ops/s 296.7988 Ops/s $\textbf{\color{#35bf28}+5.91\%}$
test_a2c_speed[False-backward] 6.9042ms 6.3692ms 157.0052 Ops/s 159.8793 Ops/s $\color{#d91a1a}-1.80\%$
test_a2c_speed[True-None] 1.5407ms 1.3693ms 730.2795 Ops/s 718.5926 Ops/s $\color{#35bf28}+1.63\%$
test_a2c_speed[True-backward] 3.1485ms 3.0470ms 328.1952 Ops/s 334.4587 Ops/s $\color{#d91a1a}-1.87\%$
test_a2c_speed[reduce-overhead-None] 16.2462ms 9.1589ms 109.1832 Ops/s 110.9047 Ops/s $\color{#d91a1a}-1.55\%$
test_a2c_speed[reduce-overhead-backward] 1.7714ms 1.6239ms 615.8203 Ops/s 672.5844 Ops/s $\textbf{\color{#d91a1a}-8.44\%}$
test_ppo_speed[False-None] 3.9648ms 3.7210ms 268.7418 Ops/s 270.4907 Ops/s $\color{#d91a1a}-0.65\%$
test_ppo_speed[False-backward] 7.5252ms 7.1929ms 139.0261 Ops/s 146.7539 Ops/s $\textbf{\color{#d91a1a}-5.27\%}$
test_ppo_speed[True-None] 1.8204ms 1.4228ms 702.8560 Ops/s 693.4119 Ops/s $\color{#35bf28}+1.36\%$
test_ppo_speed[True-backward] 3.3107ms 3.2102ms 311.5086 Ops/s 317.0710 Ops/s $\color{#d91a1a}-1.75\%$
test_ppo_speed[reduce-overhead-None] 1.1190ms 0.9762ms 1.0244 KOps/s 1.0028 KOps/s $\color{#35bf28}+2.15\%$
test_ppo_speed[reduce-overhead-backward] 1.6879ms 1.5631ms 639.7377 Ops/s 682.3716 Ops/s $\textbf{\color{#d91a1a}-6.25\%}$
test_reinforce_speed[False-None] 2.4649ms 2.2914ms 436.4129 Ops/s 439.6477 Ops/s $\color{#d91a1a}-0.74\%$
test_reinforce_speed[False-backward] 3.8259ms 3.4363ms 291.0130 Ops/s 300.8279 Ops/s $\color{#d91a1a}-3.26\%$
test_reinforce_speed[True-None] 1.7238ms 1.3082ms 764.4377 Ops/s 754.5409 Ops/s $\color{#35bf28}+1.31\%$
test_reinforce_speed[True-backward] 3.1722ms 3.0844ms 324.2140 Ops/s 338.1740 Ops/s $\color{#d91a1a}-4.13\%$
test_reinforce_speed[reduce-overhead-None] 18.1040ms 10.0662ms 99.3421 Ops/s 100.1318 Ops/s $\color{#d91a1a}-0.79\%$
test_reinforce_speed[reduce-overhead-backward] 1.7192ms 1.6273ms 614.5037 Ops/s 654.9444 Ops/s $\textbf{\color{#d91a1a}-6.17\%}$
test_iql_speed[False-None] 9.6858ms 9.2370ms 108.2608 Ops/s 106.6066 Ops/s $\color{#35bf28}+1.55\%$
test_iql_speed[False-backward] 13.7306ms 13.2408ms 75.5243 Ops/s 76.0354 Ops/s $\color{#d91a1a}-0.67\%$
test_iql_speed[True-None] 2.4605ms 2.3503ms 425.4723 Ops/s 431.2798 Ops/s $\color{#d91a1a}-1.35\%$
test_iql_speed[True-backward] 5.0518ms 4.8224ms 207.3646 Ops/s 204.0284 Ops/s $\color{#35bf28}+1.64\%$
test_iql_speed[reduce-overhead-None] 19.0953ms 11.3398ms 88.1846 Ops/s 88.6390 Ops/s $\color{#d91a1a}-0.51\%$
test_iql_speed[reduce-overhead-backward] 1.9583ms 1.9037ms 525.2883 Ops/s 498.4899 Ops/s $\textbf{\color{#35bf28}+5.38\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 8.0935ms 6.4526ms 154.9765 Ops/s 155.5896 Ops/s $\color{#d91a1a}-0.39\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.6453ms 0.3625ms 2.7588 KOps/s 3.7729 KOps/s $\textbf{\color{#d91a1a}-26.88\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.5254ms 0.3280ms 3.0492 KOps/s 4.0128 KOps/s $\textbf{\color{#d91a1a}-24.01\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.4671ms 6.1401ms 162.8632 Ops/s 162.8331 Ops/s $\color{#35bf28}+0.02\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.8319ms 0.2713ms 3.6862 KOps/s 3.0659 KOps/s $\textbf{\color{#35bf28}+20.23\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.4864ms 0.2464ms 4.0583 KOps/s 3.2067 KOps/s $\textbf{\color{#35bf28}+26.56\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.5042ms 1.2774ms 782.8420 Ops/s 752.2309 Ops/s $\color{#35bf28}+4.07\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.4463ms 1.2228ms 817.7783 Ops/s 737.3135 Ops/s $\textbf{\color{#35bf28}+10.91\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.4606ms 6.3024ms 158.6701 Ops/s 157.2741 Ops/s $\color{#35bf28}+0.89\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2.0049ms 0.4110ms 2.4331 KOps/s 2.1081 KOps/s $\textbf{\color{#35bf28}+15.42\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6280ms 0.4381ms 2.2826 KOps/s 2.4354 KOps/s $\textbf{\color{#d91a1a}-6.28\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.4142ms 6.2320ms 160.4618 Ops/s 161.4036 Ops/s $\color{#d91a1a}-0.58\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.8169ms 0.3208ms 3.1171 KOps/s 3.6290 KOps/s $\textbf{\color{#d91a1a}-14.11\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.5981ms 0.3072ms 3.2556 KOps/s 4.1089 KOps/s $\textbf{\color{#d91a1a}-20.77\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.4894ms 6.0740ms 164.6366 Ops/s 162.0212 Ops/s $\color{#35bf28}+1.61\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.4533ms 0.2645ms 3.7813 KOps/s 3.2539 KOps/s $\textbf{\color{#35bf28}+16.21\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6509ms 0.2799ms 3.5722 KOps/s 3.6035 KOps/s $\color{#d91a1a}-0.87\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.4996ms 6.3479ms 157.5334 Ops/s 157.7055 Ops/s $\color{#d91a1a}-0.11\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9637ms 0.4449ms 2.2477 KOps/s 2.3109 KOps/s $\color{#d91a1a}-2.73\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6458ms 0.4184ms 2.3899 KOps/s 2.2817 KOps/s $\color{#35bf28}+4.74\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 7.1645ms 5.5098ms 181.4935 Ops/s 178.3136 Ops/s $\color{#35bf28}+1.78\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 6.0515ms 2.0206ms 494.9018 Ops/s 434.4095 Ops/s $\textbf{\color{#35bf28}+13.93\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 8.9142ms 1.2730ms 785.5654 Ops/s 822.6849 Ops/s $\color{#d91a1a}-4.51\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 8.0009ms 5.6624ms 176.6028 Ops/s 179.6586 Ops/s $\color{#d91a1a}-1.70\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 7.5266ms 2.0057ms 498.5740 Ops/s 430.0193 Ops/s $\textbf{\color{#35bf28}+15.94\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 8.2269ms 1.2978ms 770.5152 Ops/s 880.8263 Ops/s $\textbf{\color{#d91a1a}-12.52\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.5010s 15.6948ms 63.7152 Ops/s 31.4128 Ops/s $\textbf{\color{#35bf28}+102.83\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 9.5942ms 2.1995ms 454.6424 Ops/s 469.9074 Ops/s $\color{#d91a1a}-3.25\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 7.3742ms 1.3756ms 726.9486 Ops/s 721.7189 Ops/s $\color{#35bf28}+0.72\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 13.1965ms 12.9871ms 76.9992 Ops/s 74.6196 Ops/s $\color{#35bf28}+3.19\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 20.7697ms 17.6114ms 56.7814 Ops/s 58.6593 Ops/s $\color{#d91a1a}-3.20\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 18.4712ms 17.7620ms 56.2999 Ops/s 55.5638 Ops/s $\color{#35bf28}+1.32\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 19.0455ms 17.3704ms 57.5692 Ops/s 58.0570 Ops/s $\color{#d91a1a}-0.84\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 17.8103ms 17.5337ms 57.0330 Ops/s 55.6033 Ops/s $\color{#35bf28}+2.57\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 20.0943ms 18.8096ms 53.1643 Ops/s 53.2694 Ops/s $\color{#d91a1a}-0.20\%$

@vmoens
Update
1b4ff1b 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Feb 3, 2025
@vmoens
[Test] Skip tokenizer tests if transformers is not in workspace
1d47433 
ghstack-source-id: a6b7f6defa09ae6a1b2d4a3af15e674cd4a5fb96
Pull Request resolved: #2744
@vmoens
Update
784a05e 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Feb 3, 2025
@vmoens
[Test] Skip tokenizer tests if transformers is not in workspace
a8fc296 
ghstack-source-id: c0570300ba765b59199d58b8fd28581e828b5875
Pull Request resolved: #2744
@vmoens
Update
0c13d93 
[ghstack-poisoned]
vmoens added a commit that referenced this pull request Feb 3, 2025
@vmoens
[Test] Skip tokenizer tests if transformers is not in workspace
f2821a0 
ghstack-source-id: b92facfd14cba62511e7888567c94d3986419ab5
Pull Request resolved: #2744
@vmoens vmoens merged commit 0c13d93 into gh/vmoens/87/base Feb 3, 2025
59 of 69 checks passed
vmoens added a commit that referenced this pull request Feb 3, 2025
@vmoens
[Test] Skip tokenizer tests if transformers is not in workspace
20a19fe 
ghstack-source-id: b92facfd14cba62511e7888567c94d3986419ab5
Pull Request resolved: #2744
@vmoens vmoens deleted the gh/vmoens/87/head branch February 3, 2025 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vmoens @facebook-github-bot