Add validators list to plugin definition

[mohamedOuladi]

Description
Plugin manifests now have a validators field that performs a post-validation of values defined in enum fields.

Proposal
In the plugin class, validators must be added in order to save and retrieve plugins including this property.

Additional context
See usnistgov/WIPP-frontend#206

[MyleneSimon]

@mohamedOuladi Looking at PolusAI/WIPP-frontend#15 it seems that the validators are added to the UI section of the input, correct? If that is the case, no changes should be required to the backend, are the UI list is a list of Object to allow for flexibility in the UI description.

[MyleneSimon]

The plugin JSON schema should be updated though, see https://github.com/usnistgov/WIPP-Plugins-base-templates/blob/bebb22d4833100dae198548ee5dbacae9089549c/plugin-manifest/schema/wipp-plugin-manifest-schema.json#L469

[mohamedOuladi]

@MyleneSimon The syntax of the plugin changed and so the validators are no longer added to the UI section of the input. validators is now at the same level as inputs, outputs and ui. Hence, the list of validators needed to be added to the plugin definition. The syntax might change in the future and thus I am defining validators as a list of Object just like the UI list.

I will make changes to the plugin JSON schema as well.

[MyleneSimon]

Understood, though with the on-going discussion about changes to the plugin manifest schema, that might change again.
Will PolusAI/WIPP-frontend#15 be updated with the new proposed syntax, or a PR be opened on https://github.com/usnistgov/WIPP-frontend?
Do you have a (small) example of what a plugin manifest would look like with the validators?

On a side note, there was some attempts in the past to add default validators to the job form (such as required fields, etc.) so hopefully we will be able to build on top of what you added to add that!

[mohamedOuladi]

I will update the PR PolusAI/WIPP-frontend#15 as soon as I test on our CI. Here is an example of a plugin manifest with validators (I tried making it as small as possible).

{
    "name": "SegmentationModelsTraining",
    "version": "0.3.1debug1664-new-syntax",
    "containerId": "labshare/polus-smp-training-plugin::0.3.1debug1664",
    "title": "SegmentationModelsTraining",
    "description": "Segmentation models training plugin",
    "author": "Gauhar Bains ([email protected]), Najib Ishaq ([email protected])",
    "institution": "National Center for Advancing Translational Sciences, National Institutes of Health",
    "repository": "https://github.com/polus-au/polus-plugins-dl",
    "website": "https://ncats.nih.gov/preclinical/core/informatics",
    "citation": "",
    "inputs": [
        {
            "name": "pretrainedModel",
            "description": "Path to a model that was previously trained with this plugin. If starting fresh, you must instead provide: 'modelName', 'encoderBaseVariantWeights', and 'optimizerName'. See the README for available options.",
            "type": "genericDataCollection",
            "options": null,
            "required": false
        },
        {
            "name": "modelName",
            "description": "Model architecture to use. Required if starting fresh.",
            "type": "enum",
            "options": {
                "values": [
                    "Unet",
                    "UnetPlusPlus",
                    "MAnet",
                    "Linknet"
                ]
            },
            "required": false
        },
        {
            "name": "encoderBase",
            "description": "The name of the base encoder to use.",
            "type": "enum",
            "options": {
                "values": [
                    "ResNet",
                    "ResNeXt",
                    "ResNeSt"
                ]
            },
            "required": false
        },
        {
            "name": "encoderVariant",
            "description": "The name of the specific variant to use.",
            "type": "enum",
            "options": {
                "values": [
                    "resnet18",
                    "resnet34",
                    "resnet50",
                    "resnet101",
                    "resnet152"
                ]
            },
            "required": false
        },
        {
            "name": "encoderWeights",
            "description": "The name of the pretrained weights to use.",
            "type": "enum",
            "options": {
                "values": [
                    "advprop",
                    "imagenet",
                    "imagenet+5k",
                    "imagenet+background",
                    "instagram",
                    "noisy-student",
                    "random",
                    "ssl",
                    "swsl"
                ]
            },
            "required": false
        }
    ],
    "outputs": [
        {
            "name": "outputDir",
            "description": "Output model",
            "type": "genericDataCollection",
            "options": null,
            "required": true
        }
    ],
    "ui": [
        {
            "key": "inputs.pretrainedModel",
            "title": "pretrainedModel",
            "description": "Path to a model that was previously trained with this plugin. If starting fresh, you must instead provide: 'modelName', 'encoderBaseVariantWeights', and 'optimizerName'. See the README for available options."
        },
        {
            "key": "inputs.modelName",
            "title": "modelName",
            "description": "Model architecture to use. Required if starting fresh.",
            "default": "Unet"
        },
        {
            "key": "inputs.encoderBase",
            "title": "encoderBase",
            "description": "The name of the base encoder to use.",
            "default": "ResNet"
        },
        {
            "key": "inputs.encoderVariant",
            "title": "encoderVariant",
            "description": "The name of the specific variant to use.",
            "default": "resnet34"
        },
        {
            "key": "inputs.encoderWeights",
            "title": "encoderWeights",
            "description": "The name of the pretrained weights to use.",
            "default": "imagenet"
        }
    ],
    "validators": [
        {
            "condition": [
                {
                    "input": "encoderVariant",
                    "value": "resnet18",
                    "eval": "=="
                }
            ],
            "then": [
                {
                    "action": "show",
                    "input": "encoderWeights",
                    "values": [
                        "imagenet",
                        "ssl",
                        "swsl"
                    ]
                }
            ]
        },
        {
            "condition": [
                {
                    "input": "encoderVariant",
                    "value": "resnet34",
                    "eval": "=="
                }
            ],
            "then": [
                {
                    "action": "show",
                    "input": "encoderWeights",
                    "values": [
                        "imagenet"
                    ]
                }
            ]
        },
        {
            "condition": [
                {
                    "input": "encoderVariant",
                    "value": "resnet50",
                    "eval": "=="
                }
            ],
            "then": [
                {
                    "action": "hide",
                    "input": "encoderWeights",
                    "values": [
                        "imagenet",
                        "ssl",
                        "swsl"
                    ]
                }
            ]
        }
    ]
}

Yeah if we can agree on the whole schema validation and come up with a simple syntax that would be great!

[MyleneSimon]

Hi @mohamedOuladi, as discussed offline, if we want to stay consistent with the current manifest format, a better location for the validators might be in the ui section, the same way fieldsets is defined (see https://github.com/usnistgov/MIST/blob/master/wipp-plugin.json#L336), since these two properties are at the same level.
One question might be: are the validators going to be used for something else than UI/display?

[mohamedOuladi]

Yup I will move the validators to the ui section as discussed. Regarding the use of the validators, I think they will be solely used for the UI for now.