Take advantage of lua's module system for package dependencies

[michaelforney]

The current lua build system is not very smart and most functions correspond exactly with writing parts of a ninja manifest. It'd be nice not to have to keep various strings in sync and leverage the lua module system a bit more.

I'm thinking something like this to start:

pkg/zlib/gen.lua:

local libz = lib('libz.a', {...})
return {
	libz=libz,
}

pkg/pigz/gen.lua:

local zlib = require 'pkg/zlib'

exe('pigz', {'pigz.c', 'yarn.c', 'try.c', zlib.libz})

I'm not sure exactly how cflags for headers should be plumbed through. Needs some more thought.

[rnpnr]

The problem with this is that everything that is not scoped in some way will be run on require 'pkg/zlib'. Since multiple invocations of cflags() just append more flags you will automatically get whatever pkg/zlib/gen.lua has for cflags. This will very quickly lead to extremely long command lines and be a mess to debug.

I'm not really sure how you can get around this. For example you could consider changing all the gen.lua to have a gen function which is only called during generation:

local M = {}
M.gen = function()
	-- ...
	M.libz = lib('libz.a', {...})
end
return M

But obviously this doesn't work because M.libz is only set when M.gen() gets called.

You may be able to come up with some metatable shenanigans that allow you to have the effect of M.libz = lib('libz.a', {...}) without any of the side effects that must occur for generation. Most of the things that I can think of would require pretty substantial modifications to the existing files however.

[michaelforney]

I was actually thinking of using a special handler in package.searchers that sets up the appropriate output redirection to local.ninja. I've had the following in my tree for a bit as a proof of concept, but I haven't fully fleshed it out yet: https://github.com/oasislinux/oasis/compare/require

[michaelforney]

One issue is that paths break when they contain variable references like $outdir, so we'd have to switch to using lua variables for paths that will be used outside the package. This isn't necessarily a bad thing, but might be a bit more verbose and is something to think about.

[rnpnr]

I was actually thinking of using a special handler in package.searchers that sets up the appropriate output redirection to local.ninja.

That's interesting, I didn't know that you could do that. I will need to think more on how to use that.

To your first issue, an option is to return a "dependency" table similar to what is done in meson. This would contain both the library and the headers defining the API (I'm pretty sure this is all that is needed from the cflags). The library typically needs to be more than just the archive file because the archive may depend on other archives (the details on how this is handled right now escape me, I know there is something in oasis for handling it).

[michaelforney]

That's interesting, I didn't know that you could do that. I will need to think more on how to use that.

Yeah. I think there is potential here, but we have to do some experimenting to see what interface is most effective and clean.

To your first issue, an option is to return a "dependency" table similar to what is done in meson. This would contain both the library and the headers defining the API (I'm pretty sure this is all that is needed from the cflags). The library typically needs to be more than just the archive file because the archive may depend on other archives (the details on how this is handled right now escape me, I know there is something in oasis for handling it).

Yeah, I was thinking of something along those lines (return a table instead of a file).

Currently, the way static libraries work is a little strange. It dates back to when oasis used rc shell scripts instead of lua. There is an awk script scripts/rsp.awk that flattens out and deduplicates a static library dependency tree. The lib() function builds a libfoo.a.d listing itself and all library dependencies (which may be .d files as well). If the exe() function encounters any .d files, it builds a .rsp file using that script and then adds the file onto the command line with the @ syntax, which causes gcc to read all the libraries from that file. Since the action building .d files has a dependency on the files it lists, the ninja dependencies work out.

It'd be much nicer to track all this in Lua instead, and just generate the build action with all the libraries up front. Previously, when oasis used rc, everything was set up so that it could incrementally generate the manifests (e.g. if only one package changed), since rc was slow. But Lua is much faster and since 8f62705 we always do the whole thing at once in anticipation of wanting to do something like this.