CertificateStore CRD namespace scoping confusion

[cmaclaughlin]

As I was upgrading my team's sample to use Ratify v1.1.0 (from RC-4), I ran into some unexpected issues when applying a Cert Store with the inline provider.

The output of kubectl logs -l app.kubernetes.io/name=ratify --namespace gatekeeper-system out resembling:

{
  "verifierReports": [{
      "isSuccess": false,
      "name": "verifier-notation",
      "type": "notation",
      "message": "Original Error: (Original Error: (unable to fetch certificates for namedStore: certs), Error: verify plugin failure, Code: VERIFY_PLUGIN_FAILURE, Plugin Name: verifier-notation, Component Type: verifier, Documentation: https://ratify.dev/docs/troubleshoot/verifier/notation, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: verifier-notation, Component Type: verifier",
      "artifactType": "application/vnd.cncf.notary.signature"
    }
  ]
}

I tried numerous variations of both the Notation verifier CRD and the certificate store CRD but was not successful until I added namespace: gatekeeper-system to the metadata. And the only reason I guessed namespace needed to be added was because I noticed that, unlike the other CRDs, the certificate store definition is scoped to namespace and not cluster.

1. Purely out of curiosity, why are certificate store CRDs the only ones scoped to a namespace instead of cluster?
2. If there are any plans for using k8s namespace to support multitenancy options, then users will likely assume the namespace needs to be the appropriate namespace where workloads will be running and not the one in which Ratify is deployed. At the very least, please update the appropriate documentation and samples so users know what they need to do without digging into the code and guessing.
3. Since it is reasonable to expect users to configure their required certificate store(s) and handle that separate from helm chart install, please document the steps users will need to perform to update the Notation verifier resource configuration. Inclusion of the best approach e.g.kubectl delete verifiers.config.ratify.deislabs.io/verifier-notation and then kubectl apply -f $notation_verifier_yaml or whatever would be a nice to have.
4. Depending on the complexity the required steps, it may make sense to exclude application of the notation verifier YAML as a part of helm chart install and force users to always create/apply both certificate store(s) and notation verifier configuration to meet their needs.

[cmaclaughlin]

duplicate of #1269; closing