Parallelization of configure_file

[mitchgrout]

I understand that typically the use case for configure_file is to spit out one or two headers, such as a version.h or config.h based off environment variables or detected compiler/language/runtime features. As such, the fact that configure_file runs sequentially is inconsequential, as the generation of these files is typically pretty quick.

I have a use case in which I want to generate a decent number of headers. For legacy reasons, I really cannot use generator() (think cyclic dependencies, and issues where I cannot link at the library-level but at the executable-level). configure_file seems like a good choice here; my files get generated at configure time, prior to any compilation occuring, so my build configuration is happy.

Here's the problem: I want to generate a few hundred files, and the generation of these files is anywhere between 250ms and 1s. Since configure_file runs sequentially, this will not work for me. A possible solution, what if configure_file could be parallellized? Obviously it does introduce new risks, and could induce a slowdown in the case of single-file generation, however I'm not seeing a clear way in how I should do my build configuration otherwise, besides having it done externally to meson.

[eli-schwartz]

What's the actual problem with using generator()?

Note that parallelizing configure_file would require the ability to run a meson function, get its return value, and wait until later to do the action that actually creates said return value. There are more high-profile scenarios where this would be nice to have, such as compiler checks which are quite slow as they involve forking the compiler and compiling a file.

The line immediately below, may very well be using the result for any reason at all -- compiler checks return booleans usable in if, but even configure_file returns a files object usable for example for run_command. I'd also be additionally concerned that e.g. header generation might be used as an include for a compiler check in which case the compiler check won't receive the files object as an argument, merely the -Idir for it.

It's also not clear to me what's especially slow about configure_file at scale. The major bottleneck that can actually be parallelized is the I/O involved, and the big thing we can do to parallelize this is introduce threads/worker processes. (To contrast this to compiler checks, where we always fork, we'd just have to avoid wait()ing on the subprocess.) Threads are not free, worker processes are even worse. I don't have any numbers on whether it's slower to thread or to wait on writing out all these files though.

Are you using configure_file with a command instead of a configuration_data?

[eli-schwartz]

Thanks for clarifying -- I suspected the answer was going to be related to external commands, yeah...

• The only way to trigger the generation of files is to trigger compilation of the target, which is only achievable via linking
  libraries together (AFAIK)

Wouldn't it work to list both headers as "header inputs" for both targets? If you use custom_target specifically, then you get regular build artifacts that you can reuse in whichever targets you like. Using generator won't work well for this because generator() is a bit magic and assumes you're producing files that are totally internal to that one build target -- this is why all the results are dumped into a hidden private directory.

[mitchgrout]

Hm, I assumed that the result of generator.process should be analogous to custom_target, although it does appear to be slightly different; with generator.process, if I provided the generated list to multiple libraries, I would end up getting a rebuild of each file per target, which is horrendous for my performance (think several hundred test executables depending on these generated headers). I would still need to split the header/source generation phases as if I provided the sources to each library, then I would trigger unnecessary recompilations, but I suppose that's not the worst possible case.

With that in mind, what purpose does configure_file truly serve if custom_target can essentially replace it?

[eli-schwartz]

I would still need to split the header/source generation phases as if I provided the sources to each library, then I would trigger unnecessary recompilations, but I suppose that's not the worst possible case.

You can index the return value:

foo = custom_target(..., output: ['foo.c', 'foo.h'])

foo_c = foo[0]
foo_h = foo[1]

With that in mind, what purpose does configure_file truly serve if custom_target can essentially replace it?

Well, it serves the purpose of applying the moral equivalent of sed to a version.h.in file :) and also producing config.h files based on the results of compiler checks. This is useful in the former case because those are really fast and basically never change, and in the latter case because you simply cannot communicate information out of the setup stage and over to a custom_target without writing it to a file using configure_file. :D

You're entirely correct in concluding that I don't think configure_file is very useful for generalized outputs. There are rare cases where running a custom command at setup time to produce a configure_file output is the best choice, but in general it's not a replacement for custom targets and shouldn't be treated like a replacement.

If nothing else -- any time a configure_file() input is modified, it triggers a full buildsystem reconfigure, which means that configure_file(input: 'version.h.in') should typically be inputs that rarely if ever change.

[mitchgrout]

Out of curiosity, why does the behaviour of generator.process and custom_target differ? My understanding was that generator was to abstract over common arguments, so you could 'bulk process' input files; however, the behaviour differs in that custom_target provides a top-level target, and generator.process provides an object that becomes a top-level target when used (maybe poorly explained).

Could the behaviour be adjusted somewhat so that...

gen   = generator(...)
files = gen.process('foo', 'bar', 'baz')
library('lib1', files)
library('lib2', files)

does not trigger a re-generation of the generated files per-target, making it behave more closely to custom_target?

[mitchgrout]

Hm, actually this can't work either. If we're generating headers, there is an expectation that I could wrap my library up in a dep and create an include-only partial dependency out of it. However, if I partial it, then I won't link the library => won't build it => wont' generate the header. Similarly if you pass the generated header through as a source in the dependency, partial'ing it will also quotient out the header so I won't generate it.

You could argue that I should just carry both include and source in the partial, but the consumer of a dependency shouldn't really need to know that IMO? Further, some of my dependencies would be carrying both genuine C source files to be compiled, plus generated headers, so there's not an easy resolution there. configure_file would be the most correct way for me to deal with this.