Notation with internal certificate authority

[olopost]

Hi,

We try to activate ratify and pull image with registry where signature is done by notation with our internal CA.

No problem to sign and check signature with notation but on ratify we get the following error

"message": "Original Error: (Original Error: (signature is not produced by a trusted signer), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
      "artifactType": "application/vnd.cncf.notary.signature"

We install ratify with helm and give our CA with this values extract:

notationCert: |
  -----BEGIN CERTIFICATE-----
  CERTDATA
  -----END CERTIFICATE-----

To be more precise, when we launch a signed artefact with

 kubectl run --image internal-registry/library/busybox:1.34.1-glibc test

We get the following error

time=2023-09-28T13:07:00.402419937Z level=info msg=verify result for subject internal-registry/library/busybox@sha256:9231a8a1130e738b1c5c50014fcadb16c460826855eaa8574a18c598642e4ad9: {
  "verifierReports": [
    {
      "isSuccess": false,
      "name": "notation",
      "message": "Original Error: (Original Error: (signature is not produced by a trusted signer), Error: verify signature failure, Code: VERIFY_SIGNATURE_FAILURE, Plugin Name: notation, Component Type: verifier, Documentation: https://github.com/notaryproject/notaryproject/tree/main/specs, Detail: failed to verify signature of digest), Error: verify reference failure, Code: VERIFY_REFERENCE_FAILURE, Plugin Name: notation, Component Type: verifier",
      "artifactType": "application/vnd.cncf.notary.signature"
    },
    {
      "subject": "internal-registry/library/busybox@sha256:9231a8a1130e738b1c5c50014fcadb16c460826855eaa8574a18c598642e4ad9",
      "isSuccess": true,
      "name": "notation",
      "message": "signature verification success",
      "extensions": {
        "Issuer": "",
        "SN": ""
      },
      "artifactType": "application/vnd.cncf.notary.signature"
    }
  ]
} component-type=server go.version=go1.20.8 trace-id=cf28b40e-3318-4bb7-bfc7-fe775a09541a


Can you help us on this topics ?

Best regards.

[akashsinghal]

Hi @olopost,

Thanks for reaching out. Here's a few questions to get started for us to be able to diagnose the issue:

1. Are you using the open-source version of Ratify or managed version of Ratify through a cloud provider?
2. Could you paste the Constraint Template used? You can find this as a resource of type ConstraintTemplate in the templates.gatekeeper.sh definition group.
3. From the error it looks like you might have 2 signatures attached to the image. The failure is likely caused by the extra signature's cert not matching. The notation CLI looks for all signatures attached to the image and returns a successful response if at least one of them is successful. However, Ratify leaves the behavior as a policy decision (thus why the ConstraintTemplate is important). This is likely why notation CLI works fine but Ratify fails. To confirm if you have multiple signatures could you run and share output of:

notation inspect internal-registry/library/busybox:1.34.1-glibc

Note: (please redact any sensitive cert info from the output)

[tesence]

Hi @akashsinghal

1. Yes we are using the open-source version from the official repository (1.10.0)
2. We are using the one provided in the quickstart guide https://ratify.dev/docs/1.0/quickstarts/quickstart-manual#step-3-see-ratify-in-action

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: ratifyverification
spec:
  crd:
    spec:
      names:
        kind: RatifyVerification
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package ratifyverification
        
        # Get data from Ratify
        remote_data := response {
          images := [img | img = input.review.object.spec.containers[_].image]
          response := external_data({"provider": "ratify-provider", "keys": images})
        }

        # Base Gatekeeper violation
        violation[{"msg": msg}] {
          general_violation[{"result": msg}]
        }
        
        # Check if there are any system errors
        general_violation[{"result": result}] {
          err := remote_data.system_error
          err != ""
          result := sprintf("System error calling external data provider: %s", [err])
        }
        
        # Check if there are errors for any of the images
        general_violation[{"result": result}] {
          count(remote_data.errors) > 0
          result := sprintf("Error validating one or more images: %s", remote_data.errors)
        }
        
        # Check if the success criteria is true
        general_violation[{"result": result}] {
          subject_validation := remote_data.responses[_]
          subject_validation[1].isSuccess == false
          result := sprintf("Subject failed verification: %s", [subject_validation[0]])
        }

3. The image has indeed been signed twice, and from the output I can tell that only the second one is correct (the first one being the result of a previous test)

~$ notation inspect internal-registry/library/busybox:1.34.1-glibc --insecure-registry
Warning: Always inspect the artifact using digest(@sha256:...) rather than a tag(:1.34.1-glibc) because resolved digest may not point to the same signed artifact, as tags are mutable.
Inspecting all signatures for signed artifact
internal-registry/library/busybox@sha256:9231a8a1130e738b1c5c50014fcadb16c460826855eaa8574a18c598642e4ad9
└── application/vnd.cncf.notary.signature
    ├── sha256:eceb94d72a87912f88d014b8fd5ace8433088ec5f47c1a66b29ebf1b1a2a33ae
    │   ├── media type: application/jose+json
    │   ├── signature algorithm: RSASSA-PSS-SHA-256
    │   ├── signed attributes
    │   │   ├── signingScheme: notary.x509
    │   │   └── signingTime: Mon Sep 25 12:14:15 2023
    │   ├── user defined attributes
    │   │   └── (empty)
    │   ├── unsigned attributes
    │   │   └── signingAgent: Notation/1.0.0
    │   ├── certificates
    │   │   ├── SHA256 fingerprint: 2f6eb182719abcefdda3291db0c751f6957d3b587a2ae31600d4654fe2a0b479
    │   │   │   ├── issued to: CN=XXXXX,O=OOOOO,L=LLLLL,ST=Ile-de-France,C=FR
    │   │   │   ├── issued by: CN=XXXXX CA,O=OOOOO,L=LLLLL,ST=Ile-de-France,C=FR
    │   │   │   └── expiry: Wed Sep 18 15:46:00 2024
    │   │   └── SHA256 fingerprint: 98fef2ba715708cf39ec1446eeb4105b3d74f6481590b926484fbcfa9bb093ed
    │   │       ├── issued to: CN=XXXXX CA,O=OOOOO,L=LLLLL,ST=Ile-de-France,C=FR
    │   │       ├── issued by: CN=XXXXX CA,O=OOOOO,L=LLLLL,ST=Ile-de-France,C=FR
    │   │       └── expiry: Sun Sep 18 14:58:00 2033
    │   └── signed artifact
    │       ├── media type: application/vnd.docker.distribution.manifest.v2+json
    │       ├── digest: sha256:9231a8a1130e738b1c5c50014fcadb16c460826855eaa8574a18c598642e4ad9
    │       └── size: 428
    └── sha256:073eb85d2b6f41fdb56e16e07565a154acb9a788643ef4cac9f0a2a488c1c2b3
        ├── media type: application/jose+json
        ├── signature algorithm: RSASSA-PSS-SHA-256
        ├── signed attributes
        │   ├── signingScheme: notary.x509
        │   └── signingTime: Tue Sep 26 14:34:33 2023
        ├── user defined attributes
        │   └── (empty)
        ├── unsigned attributes
        │   └── signingAgent: Notation/1.0.0
        ├── certificates
        │   ├── SHA256 fingerprint: dd0269615eb1a04d512d0f3fbbeb5728cda1ae3feded38aa2ef451a55a626750
        │   │   ├── issued to: CN=XXXXX,OU=UUUUU,O=OOOOO,L=LLLLL,ST=Ile de France,C=FR
        │   │   ├── issued by: CN=XXXXX CA,OU=UUUUU,O=OOOOO,L=LLLLL,ST=Ile de France,C=FR
        │   │   └── expiry: Thu Sep 19 07:22:00 2024
        │   └── SHA256 fingerprint: ded365005a8e7eb2530bb903842439d7100c86caaa6b21d4ac622da870462979
        │       ├── issued to: CN=XXXXX CA,OU=UUUUU,O=OOOOO,L=LLLLL,ST=Ile de France,C=FR
        │       ├── issued by: CN=XXXXX CA,OU=UUUUU,O=OOOOO,L=LLLLL,ST=Ile de France,C=FR
        │       └── expiry: Mon Sep 19 07:21:00 2033
        └── signed artifact
            ├── media type: application/vnd.docker.distribution.manifest.v2+json
            ├── digest: sha256:9231a8a1130e738b1c5c50014fcadb16c460826855eaa8574a18c598642e4ad9
            └── size: 428

[tesence]

Our issue definitely came from the fact that the image had 2 signatures from 2 different certificates. Once I made sure that the image I was doing the tests with was only signed by the certificate I want, ratify is working as expected.

Thanks for your help !

[akashsinghal]

@tesence glad you were able to get unblocked. The default policy recommended in the quick start is the most strict policy we have. It requires every single artifact attached to the image verify to true for it to pass admission. We define a few other rego templates depending on the scenario. You can learn more about those here: https://ratify.dev/docs/1.0/reference/rego-templates