Skip to content

Performance Benchmarks Journal

Jump to bottom
Danilo Lessa Bernardineli edited this page Mar 11, 2024 · 5 revisions

Unless otherwise specified, all runs were relative to a Mac M1 platform, which is a 4+4 core platform

11 March 2024, First PSuU-like runs

Danilo Lessa Bernardineli (BlockScience)

Benchmarks

For varying N_sweep_samples (N_t=2, N_s=2):

  • n=1: 0.09s (212= 4 state measurements, 44P/s)
  • n=10: 0.52s (2102= 40 state measurements, 77P/s)
  • n=100: 5.85s (21002= 400 state measurements, 68P/s)
  • n=200: 12.03s (22002= 800 state measurements, 60P/s)
  • n=400: 32.55s (24002= 1600 state measurements, 50P/s)
  • n=600: error
  • n=1000: error

For varying N_t with N_sweep_samples=10, N_s=2:

  • N_t=2: 0.51s (2102= 40 state measurements, 78P/s)
  • N_t=20: 2.47s (2102= 400 state measurements, 162P/s)
  • N_t=200: 42.72s (2102= 4000 state measurements, 93P/s)

For varying N_t with N_sw=1, N_s=10

  • N_t=10: 0.7s (10110= 100 state measurements, 142P/s)
  • N_t=100: 6.75s (101100 = 1,000 state measurements, 148P/s)
  • N_t=1000: 283.0s (1011000 = 10,000 state measurements, 35P/s)

Last run: N_sw=20, N_s=2, N_t=100: 20.3s (220100= 100 state measurements, 197P/s)

Comments

As per the PSuU workplan, to be able to generate the measurements with an Exploratory coverage, we'll be required to be able to generate measurements across N_t=500, N_s=3 and N_sw = 2048 * 209 = 428,032, giving us a total of 2048*209*3*500 = 642,048,000 state measurements. Assuming a compute speed of 150P/s, we would require 49 days' worth of compute to generate the dataset. Having the Optimal coverage implies 3^9*2^2*50*209*5,000 = 4,113,747,000,000 state measurements, therefore requiring 317,418 days or a 6500x coverage over the Exploratory one.

Assuming a target simulation duration of two hours for the Exploratory coverage, we would require a speed-up of 588x (or ~90,000 P/s). As a hypothesis, we can assume we can have the following stacking performance multipliers by doing the following actions:

  • Disabling deepcopy: 10x improvement
  • Model optimizations: 2x improvement (20x cumulative)
  • Simulation optimizations: 2x improvement (40x cumulative)
  • Running on a cloud VM: 20x improvement (800x cumulative)

For Optimal coverage, assuming a target simulation duration of 7 days, we would require a ~45,000x speed-up. Therefore, we may require employing heuristics or iterative approaches.

Clone this wiki locally