Build infrastructure for facilitating hooking into and customizing worker image builds for incorporation of additional BMI modules

[robertbartel]

At present, all BMI modules supported by the ngen and calibration worker images are incorporated into the primary source Dockerfile for the images. I.e., when someone wishes to use a new BMI module, that has to be fully integrated into the image Dockerfile, requiring the full process of building, testing, review, and merge within the repo.

It would be better to have a way to easily incorporate BMI models into the local images built by a deployment without requiring the BMI model's setup to be part of the repo source code. This would make the process more nimble and allow using BMI models without requiring them to be a component within, or distributed with, NextGen or DMOD. It also would benefit the larger community by making it easier for them to experiment with BMI models without having to ask for them to be committed to the repo. A practical example of the benefit of this relates to NOAA-OWP/ngen#513; with the aforementioned capabilities, Summa could simply be incorporated into the final, customized worker image of a DMOD deployment, without needing to make it a submodule of the repo.

We should develop the mechanisms to support such modular additions of BMI models to DMOD deployment worker images.

Note that #658, #661, and #662 are all closely related.

[aaraney]

Making the assumption that handling a single file (i.e. a complied shared library w/ no external deps) simplifies this effort. Curious if we could use something like pex for this, at least for python bmi modules. Not sure how feasible that is given how NextGen uses pybind11 though.

[robertbartel]

I don't think we have to make that assumption, though we can support it.

What I had in mind was to have another image configuration based on ngen. The only additional build stage for the image would expect a script to be present, and all this modular stage would do is copy the script and any extra local files into the image, and then run that script. The script and those extra files would be in a local directory within the repository tree but ignored by Git. We'll have to make sure that this is all documented, including that this build script will/may need to copy certain things into certain directories in the image (e.g., any shared library into /dmod/shared_libs).

And we might have to do some tweaking to the current build stages for the ngen image, or have a "not-cleaned-up" variant/tag for the ngen image to get this to work. But I think the core solution of a modular build stage powered by an untracked script is the way to go. And the approach still should make it fairly easy to handle a single, already built shared library file as you suggested.

[robertbartel]

Closed by #702.