Scan registry images before pods are deployed

[cjnosal]

The starboard operator currently watches pods to trigger a scan. In some cases it's desirable to scan an artifact before the code is deployed and running.

A new ArtifactReference CRD could be defined:

kind: ArtifactReference
spec:
  repository: myregistry.com/myrepo
  tag: 1.16
  digest: # optional if tag not specified
  secretName: # optional - for private registries

A new controller would then trigger a scan when a new ArtifactReference instance is created. The initial implementation could rely on the operator to update the CRD when a new image is in the registry.

The functionality could be extended to support polling the registry for new tags or digests if neither is specified. This may require a status field for the reconciler to capture what was scanned:

kind: ArtifactReference
spec:
  repository: myregistry.com/myrepo
  secretName: # optional - for private registries
status:
  scannedDigest: sha256:...

The implementation could largely reuse the logic of VulnerabilityReportReconciler.reconcilePods (read the artifact reference, check ownership, check for existing VulnerabilityReport/Job, check throttling, submit the scan). The usage of corev1.PodSpec and ContainerImages may need to be abstracted as this flow would not have a Pod or a container name.

[danielpacak]

@xtreme-conor-nosal Thank you for starting this discussion. I think this is an interesting idea and some time back we actually tried implementing a similar POC. At that time we considered a CRD to represent a container registry with a subset of repositories. The main reason for that was to use Starboard as an admission controller that could reference existing VulnerabilityReports and reject workloads that use vulnerable images.

We stepped back from this idea though with assumption that such use case is probably more suitable for container registry management service or a separate project. For example, Harbor registry with its pluggable vulnerability scanners or trivy-enforcer.

Beyond that, Starboard associates a VulnerabiltyReport with a K8s workload to inherit the workload's life cycle. This way we could leverage K8s garbage collection, i.e. delete the VulnerabilityReports when a K8s workload is deleted or rescan the workload when the VulnerabiltiyReport is deleted.

VulnerabilityReports are namespace-scoped resources in order to configure RBAC permissions at the same granularity as K8s workloads. With proposed ArtifactReference we'll have to specify what would be the lifecycle of such CRD and how it relates to built-in K8s resources such as Pods or ReplicaSets.

We also consider the Starboard Operator with etcd storage as a runtime security toolkit. We do not store any data that's not pertinent to running K8s workloads Neither we store historical reports.

Anyway we're happy to continue the discussion and see what others think about this approach. /cc @knqyf263 @jerbia @itaysk @amitbis

[cjnosal]

@itaysk The CI/CD pipeline may itself be running in a kubernetes cluster (e.g. using tekton). In this case Starboard could be used as part of the pipeline in a development cluster in addition to monitoring the running workloads in the production cluster.

While there are many possible integration points for Trivy, there is value in having a consistent, kubernetes-native way of triggering scans throughout the pipeline.

[cjnosal]

@danielpacak

In our POC we set the ImageReference to be the owner of the report. External orchestration (e.g. a tekton pipeline) would be responsible for the lifecycle of the ImageReference. Similar to a workload scan, Starboard may set a Deployment to be the owner of a VulnerabilityReport, but is not concerned with the lifecycle of the Deployment.

The namespace and RBAC considerations can also follow the workload approach. The Starboard RBAC is extended to list/get/watch ImageReferences, the ImageReference is deployed to a namespace Starboard is watching, and the VulnerabilityReport saved to that same namespace.

We didn't look at admission controls, but we're also considering triggering scans based on source code or software bill of materials as part of a broader in cluster CI/CD pipeline. With this context, we think Starboard could be viewed as a in-cluster vulnerability scanner, and not limited to only "runtime security". For Trivy's purposes, a Pod is just a pointer to an image in a registry - the implementation isn't impacted by whether that image is running anywhere.

[ncdc]

One way to possibly look at this is to break it into a few distinct pieces:

1. Declarative APIs & tooling to scan things like source code dependencies, images-in-a-registry, and images-referenced-by-pod-like-things. ~ Starboard
2. A place for long-term storage of all the information gathered during scanning (dependencies, OS packages, images, pods, etc.). ~ some type of database
3. Configurable policies for accepting/rejecting pods based on their images and the associated metadata (e.g. reject pod because its image contains an OS package with a CVE and we know there is a newer version of the OS package that fixes the CVE). The policy engine would likely need to evaluate metadata in long-term storage and maybe also in short-term (i.e. the Starboard vulnerability report CRs).

Does this type of separation of concerns sound reasonable?

[itaysk]

@xtreme-conor-nosal and @ncdc thanks for clarifying. I'm going to try and decompose this topic for us to better design the use-case, if I got any part wrong, please correct me!

So I understand now that what you are looking for is a way to trigger an arbitrary image scan that will run inside k8s.

First, we were talking about introducing a new K8s CRD that Starboard Operator will act on. I think it's worth pointing out that we might be looking at k8s CRD from different angles:
In your case, k8s is an infrastructure layer (compute abstraction and API). If you want to run anything (i.e Trivy) you need to ask k8s to do it for you, which oftentimes means creating a k8s resource.
In Starboard Operator's eyes, k8s is the scanning subject (Starboard Operator always scans the cluster). In this case CRDs might be appropriate to extend the state of the cluster. For example we use Node resources that helps kube-bench scan the cluster. Is this analogous to Image resources? Not really since the latter will point to any arbitrary image unrelated to the cluster's workloads.
So I just wanted to point out this incompatibility between the Image CRD with the existing Operator concepts, but nothing we couldn’t adjust if needed (@danielpacak this is in essence the “Scanner API” concept that we originally discussed and didn’t pursue).

The alternative approach is to use Starboard's other mode as a CLI command. If all that's needed is a simple way to trigger an image scan on k8s, starboard scan myimage will create a k8s job to scan this image and report the results back in the same CRD as an automatic cluster scan. My thought is that an imperative command might be better suited for “triggering” things then a declarative resource.

[ncdc]

Thanks @itaysk for your reply. It sounds like the Scanner API concept is more what we're thinking about, and that it could be used for things that are not actively running in your cluster.

For example, you could have a CRD that scans a source code repository. Or another one to scan images that were just built but haven't been deployed yet - for example, images that were built by kpack or s2i or kaniko or even docker build.

I do agree that scans can be imperative, but you can potentially derive additional value if you combine something outside the cluster that is updated periodically (git repo, image tag/digest) with continual scanning.

Please let us know if this is something you'd be interested in discussing more. Thanks!