Get build times under 1s #1528
Labels
Comments
|
It seems most of the delay is on the build. I would suggest looking into pre-compiling if possible. Hopefully this helps anyone else looking into the issue. Output of |
|
@adeebshihadeh it looks like you intended to post the output of |
target: a clean build (scons -c && ...) being <1s for anyone willing to dig through profiling data...
import pstats
p = pstats.Stats('build_profile.pstats')
p.sort_stats('cumulative').print_stats() |
|
@adeebshihadeh What is your current build time benchmark in seconds on your machine when running I replaced the C++ regex with RE2 and it sped up the build times from clean build quite significantly on my low-end Mac. |
|
I used ClangBuildAnalyzer and this is what I get - The Regex takes the longest to instantiate. |
So it looks like the most impactful change would be optimizing or replacing regex with a lighter implementation(~25% of the compilation bottleneck in clean builds). Also, what if we split |
|
@adeebshihadeh I believe I've got this one solved: #1602 |
|
I dug into this 1 over the past little bit and go back to work tomorrow so I wanted to share my findings in case anyone wants to move forward on this while I'm preoccupied. Running graph TB;
classDef bottleneck stroke:#f00
Scons[Scons overhead - .1 s]:::bottleneck;
A[g++ -o common.os<br>.3 s<br>];
B[g++ -o dbc.os<br>1.0 s<br><br><br><br><br><br><br><br>];
C[g++ -o parser.os<br>.6 s<br><br><br><br>];
D[g++ -o packer.os<br>.5 s<br><br><br>];
E[g++ -o libdbc.so - .1 s];
F[cythonize packer_pyx.pyx<br>.5 s<br><br><br>];
G[g++ -o packer_pyx.o<br>.7 s<br><br><br><br><br>];
H[g++ -o packer_pyx.so - .1 s];
I[cythonize parser_pyx.pyx<br>.6 s<br><br><br><br>]:::bottleneck;
J[g++ -o parser_pyx.o<br>1.1 s<br><br><br><br><br><br><br><br><br>]:::bottleneck;
K[g++ -o parser_pyx.so - .1 s]:::bottleneck;
L[python3 generator.py - .1 s];
A-->E;
B-->E;
B-->H;
B-->K;
C-->E;
D-->E;
F-->G;
G-->H;
I-->J;
J-->K;
Scons-->A;
Scons-->B;
Scons-->C;
Scons-->D;
Scons-->F;
Scons-->I;
Scons-->L;
Start-->Scons;
Increasing speed of
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have a modern Threadripper with tons of cores - why is building not instant? Is it the Cython? Is it all the C++ regex?
opendbc is a small project, so working in it should feel very snappy and instant. We're open to any changes necessary to get to <1s fresh builds.
It's possible there's one silly thing taking all that time or it's a few things we have to fix. Some notes in #1180.
The goal is to get
scons -c && time scons -j40to report <1s (on my workstation, though it shouldn't be too different on other machines). Here's where we are now:The text was updated successfully, but these errors were encountered: