Slow compile times

[blast-hardcheese]

In attempting to fully type the interface to Terraform's Datadog provider, I ended up with a 1200 line dhall file full of types and functions.

I have not started to optimize, as I'm still trying to wrap up the last few features I want to express in this PoC, though it's not immediately apparent (other than "have less code to compile!") how to optimize this.

I'm expecting that many different graph types will be required by many different consumers, so I'm reticent to break up the object structure, though that's definitely low hanging fruit (at the expense of making my consumers copy/paste 9 different imports into every file they write).

As the project has gotten to the point where it takes 30 seconds to compile at minimum, up to 1.5m for particularly large dashboards, I figure this is the time to start this discussion. I see some active work in dhall-lang/dhall-haskell#108, is the answer here just "wait for speed improvements"?

[blast-hardcheese]

The current view of the dhall-terraform datadog-provider branch can be seen here: https://github.com/blast-hardcheese/dhall-terraform/compare/datadog-provider

[mbj]

I'm working on an equivalent for cloudformation (nothing works right now, so no code to look at just now), and have started on a different approach: Compiling the CF specification directly to Dhall expressions, avoiding the parser altogether.

This is because I considered the dhall concrete syntax as an unneded intermediary.

I wounder if that approach would help your case also @blast-hardcheese?

I'll post some speed numbers once I got them.

[blast-hardcheese]

@mbj Interesting. I'm trying to stay pure, with the hopes that someone will eventually write go-dhall, which can be added as a dependency to Terraform as an alternate syntax to HCL/.tf.json. It would still be interesting to see the numbers and the equivalent source for the direction you're taking

[Gabriella439]

@blast-hardcheese: I can't say for sure if the ongoing optimizations will help your use case unless we profile what the bottleneck is for your large Dhall expression. I can show you how to do that yourself if you're unable to publicly share the code.

[blast-hardcheese]

@Gabriel439 That would be great! The reusable components are open, but the code that brings everything together is internal. Knowing how to profile would be very helpful, if you've got a link

[Gabriella439]

@blast-hardcheese: How do you currently build or install dhall?

[blast-hardcheese]

@Gabriel439 I've got dhall-json installed in nix and homebrew currently; until 1.2.x is available in LTS, I'm stuck with the package provided by homebrew.

I've got the repo checked out somewhere as well, and I was successfully able to build it, so that's an option as well, though less compelling if I expect whatever procedures we're following to be accessible for those who are not Haskell developers.

[Gabriella439]

@blast-hardcheese: Generally you won't be able to profile it with a prebuilt executable (i.e. one installed via brew, for example). It has to be built with profiling enabled

If you use stack to built it, then you can follow the instructions here for profiling an executable built via stack:

https://stackoverflow.com/questions/32123475/profiling-builds-with-stack

[Gabriella439]

You should check the performance against for the latest release because there have been some large parsing performance improvements in dhall-1.18.0