Titan benchmark

[alain-rs]

Although larger than our focus at this point, it is good to track the progress against these large benchmarks too.

https://www.eecg.utoronto.ca/~kmurray/titan.html

https://dl.acm.org/doi/abs/10.1145/2629579

https://docs.verilogtorouting.org/en/latest/tutorials/titan_benchmarks/

"We found that VPR is at least 2.8 × slower, uses 6.2 × more memory, 2.2 × more wire, and produces critical paths 1.5 × slower compared to Quartus II. Finally, we identified that VPR’s focus on achieving a dense packing and an inability to take apart clusters is responsible for a large portion of the wire length and critical path delay gap."

[alain-rs]

@pegaillardon, @tangxifan, what do you think?

[tangxifan]

@alain-rs

Including the Titan benchmark is a good idea. VTR uses it in their CI for each PR we want to merge.
However, there are a few challenges which have to be resolved

• The BLIF/VQM files are provided. UofT team uses Quartus to synthesis these benchmarks. Their BLIF/VQM are strongly binded to a Stratix-IV architecture. We cannot reuse them for our customized architecture.
• Not every titan benchmark provides HDL. We need someone to collect the information. Note that some of them are created by Qsys. We may need to keep HDL for ourselves.

Regarding QoR, VPR8 has improved a lot on the performance metrics. See the latest paper:
https://dl.acm.org/doi/10.1145/3388617

Note that this is almost 2 year ago. We keep improving runtime on the latest master.

[tangxifan]

Attached resource usage of Titan benchmarks, reported by current VTR master