refactor: Support the move to having build files located in ./files/ in all modules

[gmpinder]

This would be the changes required for supporting blue-build/cli#138.

[xynydev]

Modules requiring refactor

• files
• gschema-overrides
• script
• systemd

General refactor plan

With the setup in config/, we require that each module use a specific subdirectory of config/ for its files. In my view, with a files/ directory in the root of the repository, it should be possible for any module to read from any subdirectory of files/. Each module should check for the existence of /tmp/files/. If that directory exists, /tmp/config/ should be entirely ignored.

(Note: examples below use un-implemented files module syntax from #203)

Without such a refactor, a repo might look like this:

files/
   files/
      usr/
         ...
   gschema-overrides/
      ...
recipes/
   recipe.yml

With the following recipe:

# recipe.yml
...
modules:
  - type: files
    files:
      - source: usr/
        destination: /usr
  - type: gschema-overrides
    include: 
      - zz1-myoverride.gschema.override

With such a refactor, a repo might look like this:

files/
   system/
      usr/
         ...
   gschema/
      ...
recipes/
   recipe.yml

With the following recipe:

# recipe.yml
...
modules:
  - type: files
    files:
      - source: system/
        destination: /
  - type: gschema-overrides
    include: 
      - gschema/zz1-myoverride.gschema.override

Specific refactor details

systemd

The systemd module currently doesn't take in configuration for specifying the units to copy, rather it always copies them from config/systemd/system/ and config/systemd/user/. I am unsure how to refactor this module. It also does something easily doable with the files module.

[fiftydinar]

@xynydev I think that we should keep gschema-overrides name of the folder, to reflect the name of the module.

files module is the exception here with files/system/, but not a rule, because files/files/ can be confusing to the users.

Everything else looks good.

The systemd module currently doesn't take in configuration for specifying the units to copy, rather it always copies them from config/systemd/system/ and config/systemd/user/. I am unsure how to refactor this module. It also does something easily doable with the files module.

Well we can copy systemd units from files/systemd/system/ & files/systemd/user/ if files/systemd/ exists.
If not, use legacy location to copy files from config/systemd/ location.

[xynydev]

@xynydev I think that we should keep gschema-overrides name of the folder, to reflect the name of the module.

files module is the exception here with files/system/, but not a rule, because files/files/ can be confusing to the users.

My point was that those folder names shouldn't be hardcoded IMO. So it could be files/gschema-overrides/and files/files/ if a user wants that.

[fiftydinar]

My point was that those folder names shouldn't be hardcoded IMO. So it could be files/gschema-overrides/and files/files/ if a user wants that.

In that case, I agree.

[gmpinder]

So this idea I had might be too much to implement in the scope of this issue. However, I'm working on the Stages feature (blue-build/cli#173) at the moment and ran into a particular issue of needing to copy between stages. I created the copy module that is pretty much a 1-1 translation of a COPY instruction. It looks something like this:

type: copy
from: ghcr.io/blue-build/cli:latest-installer
src: /out/bluebuild
dest: /usr/bin/

It would end up creating the instruction:

COPY --from=ghcr.io/blue-build/cli:latest-installer /out/bluebuild /usr/bin/

So I essentially re-implemented the "built-in" files module. However, I chose to make a new module cause I though of a potential extension of the files module. How about making the files module a way to perform regular file operations like mv, rm, chmod, chown, etc. Obviously we can adjust the scope to what we feel is appropriate. Just wanted to drop this idea here while you were in the process of refactoring.

[xynydev]

IMV some of those file operations might be possible to commit into git, while most would be more straightforward to do with shell snippets. I wouldn't complicate the files module with them.

Your copy module also looks pretty good, but it has one crucial difference; the bash files module is one layer, but can perform multiple copy operations, while the copy module has to be repeated, and creates a layer for each copy operation. This usecase is something that the containerfile snippets already adequately fills, though, so creating a wrapper for it is really more of a convenience thing.

[gmpinder]

the bash files module is one layer, but can perform multiple copy operations, while the copy module has to be repeated, and creates a layer for each copy operation.

That's a good distinction. Alright fair enough

[prydom]

Being able to use a container image as a file source is something I had to resort to using the containerfile module for. You can see that Containerfile here. It would be convenient if the files module supported a from: argument which templated arbitrary --mount=type=bind,from= arguments.

I have a Github action which builds .srpms and .specs into RPMs and needed to get those RPMs into my overlay (that build container is here).

Thanks @gmpinder for mentioning that stage feature because it could potentially simplify things as long as caching works out. One of the spec files is Mesa and I don't want to be rebuilding that if I don't have to.

[xynydev]

I think that we should keep gschema-overrides name of the folder, to reflect the name of the module.
(@fiftydinar)

Upon further reflection, this might actually be the way.

Many modules currently hardcode a folder name in their code to be copied in to the build, like systemd and gschema-overrides. Refactoring everything would be tedious, and require us to rethink many such features in current modules. Not to mention the pain users would have refactoring into this format. All for what? Less standardization in folder names?

If someone wants a system of 1-1 mapping with a folder in their repo corresponding exactly to the layout in their image, that will be possible with the files module. The module should probably be able to copy files from any directory in files/, to facilitate a user grouping their files in the directory however they want to.

I believe modules that process files in some way before copying them over from their corresponding directory might also be able to support such a structure, by processing the files in the destination folder.

It might be a good thing that I didn't rush to implementing the initial refactor, eventhough it didn't receive much of any pushback from y'all. Maybe some would have been warranted, too.

With that, here's my example from above updated to match my current thoughts.

Filesystem structure:

files/
   system/
      usr/
      etc/
         ...
   gschema-overrides/
      ...
recipes/
   recipe.yml

Recipe:

# recipe.yml
...
modules:
  - type: files # source path is relative to files/
    files:
      - source: system # this feels like it makes sense, and is very similar to the system_files structure used by ublue
        destination: /
  - type: gschema-overrides # copies from files/gschema-overrides/
    include: 
      - zz1-myoverride.gschema.override

[fiftydinar]

@xynydev This looks great to me, I find no issues with it. I am ready to start work on this.

[xynydev]

Great! The only changes needed are is the files module. It should detect if there's /tmp/config or /tmp/files (I think), and change the relative path if the user has switched to the files directory.

I can handle the announcements and stuff, code-wise this should be a small change.

[fiftydinar]

@xynydev I think that we should strictly switch all modules to use "${CONFIG_DIRECTORY}" instead of current hardcoded values in some of the modules.

Is it better to handle the logic of "${CONFIG_DIRECTORY}" in cli, rather then write that for all affected modules?

like, I'm not sure if this is enough:

if [[ -d "/tmp/config/" ]]; then
  CONFIG_DIRECTORY="${CONFIG_DIRECTORY:-/tmp/config}"
elif [[ -d "/tmp/files/" ]]; then
  CONFIG_DIRECTORY="${CONFIG_DIRECTORY:-/tmp/files}"
fi

[fiftydinar]

Oh right, this logic is already handled in CLI.
Nvm, the PR above can get merged.