CFEngine build system

[basvandervlies]

I am just playing with cfbs and it is a nice system for generating your masterfiles setup. What I see is that masterfiles meaning has been shifted the modules are now included in the masterfiles and then copied to /var/cfengine/modules. Templates are included and to force copy package modules get the extension mustache to force the copy.

For me this feels a bit awkward with the cfbs system you could for every standard shortcut a directory:

• masterfiles
• modules
• templates
• data

Then when a policy is added that as a mustache, json or data file iit can be placed in one of the above directories. It is just suggestion

For know cfbs has complete control over the generated directories so local additions to data and templates are not possible

[nickanderson]

Bas van der Vlies ***@***.***> writes: I am just playing with cfbs and it is a nice system for generating your masterfiles setup. Glad you are liking it. What I see is that masterfiles meaning has been shifted the `modules` are now included in the masterfiles and then copied to `/var/cfengine/modules`. Templates are included and to force copy package modules get the extension mustache to force the copy. Oh fun, terminology disambiguation ... <https://docs.cfengine.com/docs/3.18/reference-language-concepts-modules.html> CFEngine has had a thing called "modules" for ever. These modules are programs that emit output that conforms with the module protocol typically found in `/var/cfengine/modules' and used with `usemodule()', or commands type promises where [module => "true"]. The vendoring of those type of modules from Masterfiles is unrelated to `cfbs'. Previously package modules and variables and classes modules were vendored as part of our packages and clients would populate their own `/var/cfengine/modules' by copying them from the hub. But we thought it made more sense to render those from the MPF so that we could account for any differences we might need to based on the binary version present instead of relying on the modules present on the hub to work with all client versions that copy them down. For me this feels a bit awkward with the `cfbs` system you could for every standard shortcut a directory: * masterfiles * modules * templates * data Then when a policy is added that as a mustache, json or data file iit can be placed in one of the above directories. It is just suggestion For know `cfbs` has complete control over the generated directories so local additions to `data` and `templates` are not possible I am unsure of a good way of supporting vendoring assets in a module that are not part of the policy set and not delivered to all agents. At least not without making some tweaks to the MPF and embedded bootstrap policy. We could have some directories inside masterfiles that are excluded from copy to clients by default perhaps. If you have content that is separate, like for example templates, you could have a module that ensures the templates are installed on your hub, that policy file could be run on the hub only, and it could take care of keeping the templates dir up to date with some other source. Similarly, we will soon want to vendor things like compliance reports in cfbs modules and probably we will want to avoid shipping those report specifications to all agents, mostly because it is wasteful and the bandwidth conservation society would not approve. [module => "true"] <https://docs.cfengine.com/docs/3.18/reference-promise-types-commands.html#module>

…

-- Nick Anderson | Doer of Things | (+1) 785-550-1767 | https://northern.tech

[basvandervlies]

I know what modules are ;-) The modules were a separate directory where clients copied it from. This is now changed so they are now copied as template from masterfiles and rendered on the client host in /var/cfengine/modules which is good as you already said. . The current package modules do not have any mustache template variables but that may change i do no know. The package modules are essential and are needed on the clients.

The solution you said about excluding directories for copy to the client is a good one. I also came to the same setup and copied the templates to template/.git/scl so they are exclude in the update,cf on the clients. On the policy server I make an exception and copy it to /var/cfengine/templates.

I have to think about it. The weekend has begun. Have a nice day

[nickanderson]

I am unsure of a good way of supporting vendoring assets in a module
that are not part of the policy set and not delivered to all agents. At
least not without making some tweaks to the MPF and embedded bootstrap
policy. We could have some directories inside masterfiles that are
excluded from copy to clients by default perhaps.

cfengine/masterfiles#2179 contains the MPF tweaks. #4882 has tweaks to the embedded bootstrap/failsafe.

[basvandervlies]

@nickanderson I just realizes that the boostrap mode copies everything from the HUB. The solution to put it in an excluded directory does not work for bootstrap or the config has to be changed.

[nickanderson]

Yep, I know. I brought it up yesterday on an internal ticket where we are discussing vendoring compliance reports.

I don't love the idea, but currently I'm at adding a top level directory like .no-distrib/, .policy_server/ or something and then both in bootstrap and update and perhaps serverd prevent copying of that directory.

I'm hopeful of a better idea materializing.

[nickanderson]

1. Collecting thoughts about adding a top level folder in MPF that is not copied by remote agents
   1. Trying to exclude a subdirectory in access rules
   2. Looking at Changes for bootstrap/failsafe
   3. Looking at changes for update policy

Collecting thoughts about adding a top level folder in MPF that is not copied by remote agents

• Access rules could prevent distribution?
  • Can we do this one?
  • Even if we could, what about multiple masterfiles aka policy channels
    • While we could potentially block using default access rules, that would not really apply to other locations where policy might be copied from.
• Bootstrap policy should exclude the directory
• Update policy should exclude the directory

Trying to exclude a subdirectory in access rules

Even if we could, what about multiple masterfiles aka policy channels

While we could potentially block using default access rules, that would not really apply to other locations where policy might be copied from.

But still, I wanna know, can we share a dir, and then block a subdir of that?

We have deny_* attributes for access type promises

https://docs.cfengine.com/docs/3.18/reference-promise-types-access.html#deny_ips

bundle server access_rules
{
  access:
      "$(def.dir_masterfiles)/.no-distrib"
        comment => "None shall pass!",
        deny => { "0.0.0.0/0" };
}

This does prevent download of the directory, on the client in verbose output I see:

verbose: Server returned error: Unspecified server refusal (see verbose server output)
verbose: Skipping '.no-distrib'

on the hub I see:

verbose: 192.168.33.3>         Received:    STAT masterfiles/.no-distrib
verbose: 192.168.33.3>    Translated to:    STAT /var/cfengine/masterfiles/.no-distrib/
   info: 192.168.33.3>    access denied to STAT: /var/cfengine/masterfiles/.no-distrib/
verbose: 192.168.33.3>    REFUSAL to user='root' of request: SYNCH 1638982997 STAT masterfiles/.no-distrib

So, this could work, but it's likely to cause promises not kept which I think is undesirable.

Looking at Changes for bootstrap/failsafe

This is what we have in failsafe.cf and currently used for exclusion.

body depth_search failsafe_recurse(d)
{
        depth => "$(d)";
        exclude_dirs => { "\.svn", "\.git" };
}


body file_select failsafe_exclude_vcs_files
{
      leaf_name => { "\.git.*", "\.mailmap" };
      file_result => "!leaf_name";
}

So, we could add \.no-distrib to exclude_dirs in failsafe_recurse.

It would apply to other dir copies as well …

… modules …

!policy_server::

  "$(sys.workdir)/modules"
    handle => "failsafe_cfe_internal_bootstrap_update_files_sys_workdir_modules_shortcut",
    copy_from => failsafe_scp("modules"),
    depth_search => failsafe_recurse("inf"),
    file_select => failsafe_exclude_vcs_files,
    classes => failsafe_results("namespace", "modulesdir_update");

from bin for bin-twin

windows::

  # TODO: Remove the use of bin-twin ref: Redmine #7364
  "$(sys.workdir)\\bin-twin\\."
    handle => "failsafe_cfe_internal_bootstrap_update_files_sys_workdir_bin_twin_windows",
    copy_from => failsafe_cp("$(sys.workdir)\\bin\\."),
    depth_search => failsafe_recurse("1"),
    file_select => failsafe_exclude_vcs_files,
    comment => "Make sure we maintain a clone of the binaries and
                libraries for updating";

Looking at changes for update policy

    am_policy_hub|validated_updates_ready::  # policy hub should always put masterfiles in inputs in order to check new policy

      "$(inputs_dir)"
      comment => "Copy policy updates from master source on policy server if a new validation was acquired",
      handle => "cfe_internal_update_policy_files_inputs_dir",
      copy_from => u_rcp("$(master_location)", @(update_def.policy_servers)),
      depth_search => u_recurse("inf"),
      file_select  => u_input_files,
      action => u_immediate,
      classes => u_results("bundle", "update_inputs");
...
body depth_search u_recurse(d)
# @brief Search recursively for files up to `d` levels excluding common version control data
# @param d Maximum depth to search recursively
{
      depth => "$(d)";
      exclude_dirs => { "\.svn", "\.git", "git-core" };
}

If we add on to u_recurse in update policy then it would be applied in other places as well ….

  "$(local_software_dir)"
  comment => "Copy binary updates from master source on policy server",
  handle => "cfe_internal_update_bins_files_pkg_copy",
  copy_from => u_pcp("$(master_software_location)/$(package_dir)", @(update_def.policy_servers)),
  depth_search => u_recurse("1"),  # Nova updates should be in root dir
  action => u_immediate,
  classes => u_if_repaired("bin_newpkg");

  "$(inputs_dir)/modules"
  comment => "Copy any files in modules from master source on policy server if a new validation was acquired",
  handle => "cfe_internal_update_policy_files_modules_dir",
  copy_from => u_rcp("$(modules_dir_source)", @(update_def.policy_servers)),
  depth_search => u_recurse("inf"),
  perms => u_m("755"),
  action => u_immediate;

"$(sys.workdir)/modules"
  comment => "Copy any non-packages modules",
  handle => "cfe_internal_update_policy_files_nonpackages_modules",
  copy_from => u_cp("$(sys.inputdir)$(const.dirsep)modules"),
  if => fileexists("$(sys.inputdir)$(const.dirsep)modules"),
  depth_search => u_recurse("inf"),
  perms => u_mo("755", "root"),
  action => u_immediate,
  file_select => not_vendored_modules("$(_not_vendored_modules_pathname_regex)");

So, seems we could add on to exclude_dirs in u_recurse …

e.g.

exclude_dirs => { "\.svn", "\.git", "git-core", "\.no-distrib" };

But might be better to do that in a new body so as not to adversely affect hte modules and package dirs (though i dont see the same restrictions causing issues if used there.

[basvandervlies]

@nickanderson nice summary and findings. Did not think about the solution to exclude it with a deny access rule!! Just tried it and update/bootstrap works for me and do not see problems with promises not kept or repaired. Maybe I overlooked something.

The other to update.cf/failsafe.cf is also a good one.

[nickanderson]

Yeah, I didn't get any promises not kept from it, was just on the radar to watch out for. A bit unsure about that still since it doesn't extend well to other paths, e.g. when you are serving multiple policy sets from the same hub. So kinda on the fence about blocking it in one place vs just adding the exceptions to the places that copy.

maybe @vpodzime and or @olehermanse have some thoughts.

[nickanderson]

@basvandervlies this one is marked as a question, do you feel like it's been answered or maybe it should be placed in a different category?

[basvandervlies]

Good to hear why not put it outside the masterfiles directory? The cfbs system can include it in the tar.:

• materfiles
• policy_server

the you do not have to update update and bootstrap

[olehermanse]

It is an interesting idea @basvandervlies. cfbs is designed to build multiple modules into 1 "policy set" (folder called masterfiles), which is traditionally placed in /var/cfengine/masterfiles. We have thought about building more things before, but it does add a lot of complexity. Users have 3 things they need to upgrade already - their policy set in /var/cfengine/masterfiles, their CFEngine binaries, and the data in the database.

Although it is understandable to not want the hub specific policy distributed everywhere, there is also something quite nice about the simplicity of the current model (everything in /var/cfengine/masterfiles and copied to inputs on all hosts).

To support multiple targets for cfbs build, we'd have to manage what targets which modules are for, how each target should be installed, etc. It's not impossible, and it's not a bad idea, but it has to be well thought out and I think there are other things we'd like to do with cfbs first.

[basvandervlies]

@olehermanse thanks for the feedback. I can imagine that everything in masterfiles is easy. Masterfiles used to be easy only used for policy files. Now it feels awkward to put dir/files in this directory that are not needed on the client at all. The only way to prevent it from copying (bootstrap/update) is to put in a .git directory. I really find this awkward.

In the CFEngine masterfiles framework there are shortcuts defined for masterfiles, templates, modules and data. So this shortcuts can be used by policy writers as me to write portable code. These shortcuts are not easily configurable (yet) For me it would be nice if the cfbs support the shortcuts definitions.

Maybe this scenario works to put everything under masterfiles and let the module writer determine how it will be copied to the right dirs on the policy server, eg:

• masterfiles/inputs ---> shortcut 'masterfiles'. Clients use this to copy the input files
• masterfiles/templates --> the "module" writer is responsible to copy the needed dirs/files to the shortcut templates, eg /var/cfengine/templates
• masterfiles/data ---> the "module" writer is responsible to copy the needed dirs/files to the shortcut data, eg /var/cfengine/data
• ...

It is a fact of life that policy files need data/script/json files/dirs . These items do not belong in masterfiles and my personal view is that it makes the masterfiles complex for update/bootstrap process with all the "special" exclude/include rules.

Just my 2cents its now cumbersome to distribute these kind of files.

[nickanderson]

I agree that it's currently cumbersome to distribute content with cfbs that is not destined for masterfiles, but I don't think this has really changed (with the exception that we vendor the modules from MPF to the final destination). At least in the default configuration we used to copy all of the modules dir to each host so each host got all modules, just not directly from MPF.

Before cfbs you had to place those files, you still have to place those files today.

I have pondered this a bit and I still haven't arrived at a better idea than to add a directory that is not copied during bootstrap or policy update inside masterfiles.

As I see it, one of the problems with having cfbs mange the non masterfiles areas is that it then requires that cfbs is running on your hub. Maybe I don't want to have cfbs running on my hub. Maybe I want to use cfbs as a local development tool to manage my policy set which I commit and my hubs pull down. Then again, maybe I should embrace cfbs more and expect that it's a requirement to run it with elevated privileges on the hub.

[nickanderson]

As I see it, one of the problems with having cfbs mange the non masterfiles areas is that it then requires that cfbs is running on your hub. Maybe I don't want to have cfbs running on my hub. Maybe I want to use cfbs as a local development tool to manage my policy set which I commit and my hubs pull down. Then again, maybe I should embrace cfbs more and expect that it's a requirement to run it with elevated privileges on the hub.

I might not want to run cfbs on my hub, but I can use masterfiles-stage to do so.

5042503

[basvandervlies]

Just a side note I am will be using to cfbs to test modules. The other is to generate the masterfiles in our repo with our local additions. In this repo we have a script that generates the templates, modules , data and masterfiles for a HUB.