Default values & weak updates

[psss]

On the last hacking session several alternative approaches were proposed to cover the use case of overriding config values from the command line outlined in #68. Let's compare the available options here and choose the best one. Start a new topic for each implementation. Add relevant comments and feedback there.

Here's description of the model use case to be covered: Write a tmt plan to verify that the current httpd package successfully serves web pages to a client running on the latest fedora release. The distro image for the server is provided on the command line. If not specified, the server should run on centos-stream-9 by default.

[psss]

Weak Update

The following config would be used to define a default value for the provision step image for the server:

provision:
  - how: virtual
    name: client
    image: fedora
  - how: virtual
    name: server
    default:
        image: centos-stream-9

When run in the Testing Farm the server image value would be overriden from the command line in the following way:

tmt run --all provision --how artemis --weak-update --image rhel-9

An alternative syntax could be to use a prefix to mark values which should be modified by the weak update.

provision:
  - how: virtual
    name: client
    image: fedora
  - how: virtual
    name: server
    default-image: centos-stream-9

Btw, what about the how key? Should the provision method be marked with default as well?

provision:
  - how: virtual
    name: client
    image-do-not-change: fedora
  - how: virtual
    name: server
    image: centos-stream-9

[jscotka]

My idea is little bit different. as you mentioned weak update so that defaults, my idea is vice-versa, have something strongly defined. Maybe it also makes sense to implement both of them. it is something similar what is defined in many tools, e.g. in RPM, there are also 3 levels of dependencies.
Implementation should be on FMF level. with special character, e.g. =. The meaning will be value what should not be changed:

provision:
  - how=: virtual
    name: client
    image=: fedora
  - how: virtual
    name: server
    image: centos-stream-9

So that if there is some thing what wants to change it, e.g. from command line image: rhel8 it means that it change just image: not image=: or how=: part

Theoretical issue what has to be solved is how to behave in the case of adjust if the adjust will be ignored if matched. or raised the error. My idea is that it could ignore, maybe just raise warning, to ensure, that there is this rule. so it will behave like constant value instead of variable parameter what you can rewrite. It is directly connected with part what we've discussed on hacking session for HW specification, that there is difference between wish what you want, e.g. machine with USB port what causes that machine e.g. virt one will add some requirements to qemu or openstack and constraints what cause that test will be skipped (e.g. not relevant for s390x)

so that this value will be on same level as +, - mark. But would be harder to implement, becaue +- marks are removed after applying. This should stay here as some metadata for this value.

It reminds me my older idea, that FMF could behave more dynamically, it means to not construct the whole tree, but do it on demand. so that the tree will be held by objects without direct applying all rules, but apply them on demand with some caching to reduce computation. all tree items will contains raw data. (this is not directly connected with this issue, but could help to implement my idea more straightforward)

[psss]

Do I understand it correctly that the = suffix handling would be done on the tmt side only? I believe that context adjust should definitely work somehow with the proposed syntax. In that case special handling in fmf would be needed as well.

[jscotka]

Do I understand it correctly that the = suffix handling would be done on the tmt side only? I believe that context adjust should definitely work somehow with the proposed syntax. In that case special handling in fmf would be needed as well.

• Theoretically possible to implement it on TMT side, as you proposed, that adjust or command line overriding will check that. Implementation of that should be more less straightforward and easy, but in future it leads to big complexity of various operations inside TMT with potential place for corner cases and non-defined behavior.
• Possible also implement it on FMF side and make it part of handling of symbols +, -, = but it could be harder to implement but cleaner in future. e.g. adjust will be just merging of tree nodes (instead of current direct handling inside TMT if I understand it well) or cmd line options what defines something, are also just ajusting -> mergring some key into nodes, and each node will contain some logic, so that in case you want to merge something with x= item it will raise sepcial error what tmt can somehow then handle. So that it mean that FMF will define this basic behaviour, so that some part of TMT code will move to FMF. and TMT will use these features.

[sopos]

In the end, I like this idea more that the default suffix / attribute. It is quite self-explanatory as well. And I like the idea of marking the attributes which I really care of rather specifying those I do not care of.
Example:
It is important for me to have a Fedora on the client side, otherwise it does not matter what provision method will be used.

provision:
  - how: virtual
    name: client
    image=: fedora
  - how: virtual
    name: server
    image: centos-stream-9

It is still questionable if the name shouldn't be rather name= as that is also what I really care of.
Together with system/user/project profile it would also easily override the default coming from the profile.

Implementation on the fmf side would need to introduce a locked boolean to all the attributes. Implementation on the tmt side would be probably easier as it would be applied as last.

[jscotka]

@sopos theoretically yes name=: as value where I do not want to change. But practivally I do not expect changes here so I do not care. But this is questionable.
or second option connected with that could be, and solve by the project config to say: something like, what will say it directly to whole project attributes, or with some selector:
config.fmf:

.constants/:
    - name
    - how:
      where: name='"/plans/.*"
.antoher_config_option/:

or second option, could be something like whitelisting of values, what can be modified by config, to say which values can be changed by scheduler, or CI and system decide if it is possible to satisfy the rule. e.g. you have defined that you need RHEL7 image and upstream CI is unable to provide this system, so that not possible to schedule anything.

[kkaarreell]

My use case is rather modifying values in an adjust. While I understand that being able to override any value in a plan would be tempting, I am also afraid that it would make understanding of a plan more difficult. Maybe a plan should clearly indicate what could be changed eventually. Maybe we should use methods that are already available (adjust, variable plus context expansion implemented in #1590). @psss 's use case with overriding image looks reasonable but I think this is something that could be already done today. It would be nice to gather more real-life use cases before we start adding implementation proposals.

My use case for modifying values through adjust is backed up by a real situation when changes or bugs introduced by new Fedora releases require us to implement or modify workarounds in a prepare step. However, it turned out that for basic modifications using proper structuring enables us to address this through step additions, although it forced me restructure our test plans. For step modifications the current approach won't be probably possible. Some more details are in #1781 along with the proposed approach through aliases.

prepare:
  - how: shell
    alias: my-shell-prepare1
    script:
     - cmd1
     - cmd2

adjust:
  - when alias == my-shell-prepare1
    script+:
      - cmd3

[psss]

The example does not seem to address the use case described in the discussion description. Could you please show how the specific image provided by Testing Farm would override the default one?

[kkaarreell]

$ cat main.fmf 
environment:
  DEFAULT_IMAGE: centos-stream-9
provision:
  - how: virtual
    name: server
    image: ${DEFAULT_IMAGE}
execute:
    script:
      - date
adjust:
  - when: image is defined
    environment+:
      DEFAULT_IMAGE: $@image

$ tmt run -vvv discover provision finish
/var/tmp/tmt/run-206
Found 1 plan.

/
    discover
        how: shell
        order: 50
        summary: 1 test selected
            /script-00
    provision
        how: virtual
        name: server
        order: 50
        user: root
        image: centos-stream-9
        memory: 2048 MB
        disk: 10 GB
        connection: session
        arch: x86_64
        qcow: CentOS-Stream-GenericCloud-9-20221101.0.x86_64.qcow2
        name: tmt-206-IwEnrJxQ
        key: /var/tmp/tmt/run-206/provision/server/id_ecdsa
        progress: booting...
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Local character set is now assumed as: 'utf-8'
        ip: 127.0.0.1
        port: 10028
^C    finish
        guest: stopped
        guest: removed
        prune: Prune plan workdir '/var/tmp/tmt/run-206/'.
        summary: 0 tasks completed

Aborted!

$ tmt run -vvv discover provision -h virtual -i Fedora-36 finish
/var/tmp/tmt/run-207
Found 1 plan.

/
    discover
        how: shell
        order: 50
        summary: 1 test selected
            /script-00
    provision
        how: virtual
        name: server
        order: 50
        user: root
        image: Fedora-36
        memory: 2048 MB
        disk: 10 GB
        connection: session
        arch: x86_64
        qcow: Fedora-Cloud-Base-36-1.5.x86_64.qcow2
        name: tmt-207-ToHdCJbK
        key: /var/tmp/tmt/run-207/provision/server/id_ecdsa
        progress: booting...
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Local character set is now assumed as: 'utf-8'
        ip: 127.0.0.1
        port: 10028
^C    finish
        guest: stopped
        guest: removed
        prune: Prune plan workdir '/var/tmp/tmt/run-207/'.
        summary: 0 tasks completed

Aborted!

[sopos]

I like this idea. It is basically what I was proposing on some of the previous meetings. It is a bit more complicated to write but it does not introduce any new special hard-coded attributes. It just works with the variable evaluation and adjust. Everything is already implemented.

If I understand correctly, the problem with TF is that the cmd line always uses the how and image specification so it basically overrides it completely all the time.

[jscotka]

I like this idea. It is basically what I was proposing on some of the previous meetings. It is a bit more complicated to write but it does not introduce any new special hard-coded attributes. It just works with the variable evaluation and adjust. Everything is already implemented.

If I understand correctly, the problem with TF is that the cmd line always uses the how and image specification so it basically overrides it completely all the time.

I understand. and I also suggest it in my comment, but in more generic than just adjust, so that this merging (computing target value) feature will be implemented inside FMF and adjust will be just semantic keyword using this feature also cmd line option changes, and this default value feature is also just another adjusting = merging of tree nodes. So it leads that this important feature has to be ideally implemented inside FMF. Theoretically it contains also some feature of priority, adjust means higher priority, default value means merge the node in opposite order. I hope I described it well and do not adding goulash here :-)

[kkaarreell]

Just wondering, I would one override image for server in the following definition?

provision:
  - how: virtual
    name: client
    image: fedora
  - how: virtual
    name: server
    default:
        image: centos-stream-9

I see two issues here:

1. Not able to refer specific "phase" in provision (as already mentioned above and in Add support for modifying phases in adjust #1781)
2. Should adjust target image for server directly or as a subtree of default (I hope you get what I mean)?

[sopos]

I think the original idea of weak update is that nothing get overridden. Only missing attributes are added.

[kkaarreell]

Right, but the purpose of adjust block is to modify existing plan so I am asking how one could use adjust to set image for the server.

[sopos]

It seems to me that this topic is more complicated than I originally though.
We need to handle the how as well as the plan can use e.g. local.
One would probably end up with only default definitions for the case "what if".

[jscotka]

It seems to me that this topic is more complicated than I originally though. We need to handle the how as well as the plan can use e.g. local. One would probably end up with only default definitions for the case "what if".

I think so, from my perspective it is not about image name, that's why I also mentioned explicitly how but from my perspective it is very generic (default value) handling.

It reminds me also discussions with configs/profiles for TMT (FMF), so as I remember it is the same what we are discussing here in sense of use-case.
I suggested to use FMF syntax and in this way it was able to handle default values and also something like profiles what will do adjust (just vice-versa order of default)

[sopos]

Hm, in this sense what you are proposing is basically to treat the "normal" attributes as the defaults while the attributes with "=" suffix as the hard ones. If we are gonna implement some additional specific syntax, this one seems to me the least intrusive for the existing plans as it would not require any change.

[kkaarreell]

It seems to me that the discussion is focused on specifying default values but we should be also discussing how subjects of weak updates would be identified. Example presented in #1820 (comment) presents a setup with one "weak default" and one hard coded option but I believe that in reality (interoperability use case) we would be testing more combinations and it won't be practical to have all multiple plans that differ just in the hard coded value. And you can also consider multi-host tests with 3 or more hosts (keylime interoperability test works with 4 host, each can be running on a different release or architecture). What I am trying to say that it won't be sufficient to override all "weak defaults" of one specific attribute but it would be necessary to be able to distinguish between them. And while this may not be the use case for Testing Farm / Artemis, why to implement a feature that fits this one use case and force users to seek for workarounds or request different features for something almost identical?

[jscotka]

It seems to me that the discussion is focused on specifying default values but we should be also discussing how subjects of weak updates would be identified. Example presented in #1820 (comment) presents a setup with one "weak default" and one hard coded option but I believe that in reality (interoperability use case) we would be testing more combinations and it won't be practical to have all multiple plans that differ just in the hard coded value. And you can also consider multi-host tests with 3 or more hosts (keylime interoperability test works with 4 host, each can be running on a different release or architecture). What I am trying to say that it won't be sufficient to override all "weak defaults" of one specific attribute but it would be necessary to be able to distinguish between them. And while this may not be the use case for Testing Farm / Artemis, why to implement a feature that fits this one use case and force users to seek for workarounds or request different features for something almost identical?

Yes, it seems as very natural extension, or generalization of this idea.
And it is exactly what I want as well. Probably some query language with modification, something like XSLT for XML, or something similar, so that you will be able to express some selector and then apply some rule for modification of the item. That's why I think it could be very naturally part of FMF, because it is not anyhow connected with TMT semantics. then TMT will just API of FMF.
Just an examplee could be, that then selector will be something what identifies plan in TMT (in sense so key defined like execute, discover or what is base and then rule for modification like apply image= someimage) (it is more less exactly what adjust do in specific way)

e.g.

selector:
    (execute is not None) and
    (discover is not None) and
    (not execute.image=)             # in sense of my do not rewrite value if defined some rule

modification:
     execute.image = ${redefine value}

also works with karel generalization e.g. add to selector also some e.g. and provison.name == "server"
maybe be able to do some more generic selector, store it and then precise it with additional value.