Benchmarking DDlog optimizations with RSLint.

[ryzhyk]

@Kixiron , could I ask you to help evaluate a couple of improvements I just implemented here: https://github.com/ddlog-dev/differential-datalog/tree/smart_distinct

This features a, hopefully, more efficient implementation of distinct. It's mostly meant to reduce the memory footprint, but I think it may also help with the CPU utilization. In addition, I upgraded DDlog to use the latest version of DD. Unfortunately, our test suite is very limited when it comes to performance measurements.

Do you think you'd be able to measure how this new version affects the CPU and memory utilization of RSLint?

In fact, it would be incredibly useful if we could integrate RSLint into DDlog's test suite. I realize, it is still work in progress and may not even end up in the final version of RSLint, but still it is an interesting and non-trivial benchmark that we'd love to use to test the correctness and performance of DDlog. Would you be able to help with this? There are two ways we can go about it. First, we can extract the DDlog logic plus the minimal set of Rust dependencies from RSLint, along with a recorded .dat file. Second, we can pull the entire RSLint repo as part of the benchmark and run it as a standalone app. The latter is easier to implement, but makes it harder to isolate DDlog bugs and performance issues from the rest of the app.

---

PS. I am also going to make another attempt to figure out why DDlog does not scale with the number of workers. If successful, this may help with your performance issues.

[Kixiron]

Sure, I'd be happy to help with both of those!

[ryzhyk]

Thanks, this is really helpful! Let me know if you have some thoughts on what the best way to create a test case based on RSLint would be.

[Kixiron]

There's actually three benching strategies I can see, the two you mentioned and then a more roundabout but possibly more worthwhile one. RSlint could itself have a benchmark over the ddlog portion of code, so the parsing & other pre-processing could happen outside of said benchmark and then bench ddlog, and then the ddlog repo could pull rslint and run that specific benchmark this would hopefully provide minimal maintenance, stay updated with both projects' progression and provide perf information to both

[ryzhyk]

I like the plan! Once RSLint is in a shape where we can benchmark it by running a few commands, I can wrap it in a test case.

[Kixiron]

At a glance performance seems roughly the same, but I think the relations have shrunk somewhat profile

[ryzhyk]

Yes, indeed, several Threshold arrangements are gone (compared to the last profile you shared), which is exactly what we were going for with this optimization.