Config system and schemas for R2.0

[o-smirnov]

There's a discussion about configs on the Stimela repo (ok, mostly a monologue, but I'm assuming @SpheMakh will reemerge from his silent holiday retreat eventually) which is spilling over into a Caracal discussion thanks to @gigjozsa's comment, so I thought I'd answer it here.

In summary, I'm playing with OmegaConf as the config file parser and schema. There is an argument to be made for moving Caracal's schemas to it as well, as it is (a) a lot more flexible than pykwalify, (b) does it via the standard (as of 3.7+) Python typing system, and is p[probably a lot more future-proof because of that.

To answer @gigjozsa's question: yes, what you refer to by "overloading" is supported, e.g. Union[str, int]. And fancier things, too, like Dict[str, CustomDataClass] (i.e. a mapping where the subsections all have to follow a specific structure).

It is a bit of a different way to write a schema, in fact the schema is set up within Python itself by declaring dataclasses. As an example, I have translated the first few parameters of the flag worker schema into this (by hand for now, but we can write a script to automate this eventually):

@dataclass
class FlagWorkerConfig:
    enable: bool = False
    field: Enum('target_enum', 'target calibrators') = "calibrators"
    label_in: Optional[str] = ""
    calfields: str = ""
    
    @dataclass 
    class RewindFlags:
        enable: bool = False
        mode: Enum('rewind_flags_enum', 'reset_worker', 'rewind_to_version') = 'reset_worker'
        version: str = "auto"

    rewind_flags: RewindFlags = RewindFlags()

Personally, I find this a lot more readable (and writable! so much less boilerplate junk to type out...) Also a lot more future-proof and flexible. E.g. for something like calfields, we can define a standard enum type, that can be reused in all workers:

    CalFields = Enum("calfields_enum", "bpcal gcal fcal xcal")
    ...
    calfields: List[CalFields] = []

...which makes a lot more sense, and lets the schema check even more things for us.

Note that this doesn't change the config file at all, it's just a different way to specify the schema and validate. Oh, and it does let you access the config as conf.rewind_flags.mode if you want, which I much prefer to conf['rewind_flags']['mode'] (just think of the carbon footprint of all those wasted quotes and brackets! But it still supports the old ways, for those that don't care about the polar bears.)

One downside is that providing documentation is not so elegant. There's no way to write a documentation string right next to the field. We would either have to document them all up front in the dataclass docstring (using one of the standard styles for which automatic parsing is available):

@dataclass
class FlagWorkerConfig:
     """Configures the flag worker
    Parameters
    ----------
    enable:
        Execute the flag worker.
    field:
        Fields that should be flagged. It can be set to either 'target' or 'calibrators' 
        (i.e., all calibrators) as defined in the obsconf worker. Note that this selection is ignored -- i.e., 
    ...
    """
    enable: bool = False

...or else define a dict afterwards, like so:

FlagWorkerConfig_doc = dict(
    enable = "Execute the flag worker.",
    field = """Fields that should be flagged. It can be set to either 'target' or 'calibrators' 
            (i.e., all calibrators) as defined in the obsconf worker. Note that this selection is ignored -- i.e., 
            all fields in the selected .MS file(s) are flagged -- in the flagging step 'flag_mask' (see below). If a 
            user wants to only flag a subset of the calibrators the selection can be further refined using 'calfields' below. 
            The value of 'field' is also used to compose the name of the .MS file(s) that should be flagged, as explained in 
            'label_in' below.""",
    label_in = """This label is added to the input .MS file(s) name, given in the getdata worker, to define the name of the 
            .MS file(s) that should be flagged. These are <input>_<label>.ms if 'field' (see above) is set to 'calibrators', or 
            <input>-<target>_<label>.ms if 'field' is set to 'target' (with one .MS file for each target in the input .MS). If empty, 
            the original .MS is flagged with the field selection explained in 'field' above.""",
    calfields = """If 'field' above is set to 'calibrators', users can specify here what subset of calibrators to process. 
            This should be a comma-separated list of 'xcal' ,'bpcal', 'gcal' and/or 'fcal', which were all set by the obsconf worker. 
            Alternatively, 'auto' selects all calibrators.""",
    rewind_flags = "Rewind flags to specified version.",
    rewind_flags_doc = dict(
          enable = "Enable this segment",
          mode =  """If mode = 'reset_worker' rewind to the flag version before this worker if it exists, or continue if it does not exist; 
                  if mode = 'rewind_to_version' rewind to the flag version given by 'version' below.""",
          version = """Flag version to rewind to. If set to 'auto' it will rewind to the version prefix_workername_before, where 'prefix' is 
                    set in the 'general' worker, and 'workername' is the name of this worker including the suffix '__X' if it is a repeated 
                    instance of this worker in the configuration file. Note that all flag versions that had been saved after this version will 
                    be deleted."""
    )
)

Which is ok I guess, I just dislike that it spreads the info about each parameter into two places. On the up side, this makes the raw documentation section human-readable, unlike the current schema files...

[o-smirnov]

Also, it's worth highlighting that stuff like this becomes easy. You can have a library of standard worker configs, pull them in via _use, then override some specific settings.

crosscal:
  _use: lib.meerkat.crosscal.4k
  secondary:
    order: KGIAKF

inspect:
  _use: lib.meerkat.plots.cal.phaseballs, lib.meerkat.plots.cal.spectra, lib.meerkat.plots.cal.detailed
  shadems:
    plots: 
      my_extra_plots: 
        - some_extra_plots
        - 

[omry]

Hi, OmegaConf author here.

A few points:

1. One downside is that providing documentation is not so elegant. There's no way to write a documentation string right next to the field

This is something that will (hopefully) eventually be supported. Take a look at omry/omegaconf#131.

2. OmegaConf offers runtime type validation as well. Once a config object is instantiated from a dataclass, any runtime mutations are validated against the schema.

3. Take a look at Hydra.
   It builds on OmegaConf and adds a ton value. Here is a pretty solid introduction for the version that predated Structured Configs.
   In particular, I think you will find the config driven composition most interesting for a framework like CARACal.