No type checking is done on operatorhub.io fields without the --ui_validate_io flag

[jsm84]

Working with the container verification pipeline (CVP), we've noticed several CSV metadata files make it past the operator-courier without failure, and then fail to deploy via OLM as part of the next stage of the pipeline. This seems to occur as we've been instructed not to use --ui_validate_io when linting the CSV & associated metadata files with the operator-courier.

Thus, if a CSV file has any of these extraneous fields defined (however erroneously), the courier ignores these fields altogether (if run without the noted flag), and thus no type checking is performed. When this happens, the operator often fails to deploy from CSV using the OLM, as the culprit field's data type doesn't match what is expected by the OLM for that field.

In one particular example, the capabilities field is defined as a struct/dict in the CSV, however it should be set to type string (as that's what's expected by OLM):

  capabilities:
      description: Adds and remove Couchbase Clusters 
      properties:
        add:
          description: Added capabilities
          items:
            type: string
          type: array
        drop:
          description: Removed capabilities
          items:
            type: string
          type: array

In this case, the CSV passes (as in, no errors) the operator-courier verify ./ with the following warning: WARNING:operatorcourier.validate:csv metadata.annotations.certified not defined.

However, running it through operator-courier verify --ui_validate_io ./ catches the incorrect capabilities data type:

WARNING:operatorcourier.validate:csv metadata.annotations.certified not defined.
WARNING:operatorcourier.validate:csv metadata.annotations.repository not defined.Without this field, the link to the operator source code will not be displayed in the UI.
ERROR:operatorcourier.validate:metadata.annotations.capabilities {'description': 'Adds and remove Couchbase Clusters', 'properties': {'add': {'description': 'Added capabilities', 'items': {'type': 'string'}, 'type': 'array'}, 'drop': {'description': 'Removed capabilities', 'items': {'type': 'string'}, 'type': 'array'}}} is not a valid capabilities level
ERROR:operatorcourier.validate:category database is not a valid category
ERROR:operatorcourier.validate:UI validation failed to verify that required fields for operatorhub.io are properly formatted.
ERROR:operatorcourier.api:Bundle failed validation.
Resulting bundle is invalid, input yaml is improperly defined.

Should these fields at least be type checked by the courier when they exist, even without the --ui_validate_io flag?

[kevinrizza]

@jsm84 There is a difference between "Won't run when deployed to a kube cluster with OLM" and "Won't display well on the operatorhub.io UI". There are probably lots of operators that will be packaged and pushed with the operator-courier that will "work" from a developer perspective even if these fields are formatted incorrectly (as long as the yaml itself is valid). In that case, I don't see the value of failing the validator when the operator itself will run fine on a cluster.