Security Scan Enhancements

[sam-heilbron]

We have an existing security scanning utility (https://github.com/solo-io/go-utils/tree/master/securityscanutils) which is used by various teams to run Trivy scans against release images on a consistent basis. For example, in Gloo Edge, we run these scans via a GitHub action (https://github.com/solo-io/gloo/blob/master/.github/workflows/trivy-analysis-scheduled.yaml)

In Gloo Edge, we scan open source images, and push the scan results to a google cloud bucket. Those output files are used to generate our docs, where we can document to our users whether the images they rely on contain any high or critical vulnerabilities (https://docs.solo.io/gloo-edge/latest/reference/security-updates/open_source/). In addition to the results being pushed to a google cloud bucket (to be used by our docs), we also push the output to the security tab under Code Scanning (https://github.com/solo-io/gloo/security/code-scanning).

In Gloo Edge, we also scan enterprise images and write results to a google cloud bucket (to be used by the docs). However, due to restrictions on our GitHub account, we cannot write to the security tab, and instead, create github issues in our private solo-projects repository (https://github.com/solo-io/solo-projects/issues/2754).

This method works great, except there are a couple of enhancements that could improve the utility:

• For each vulnerability found in our closed source repo, we produce a new github issue with relevant images included in the description. This is intended to be used by developers to identify which issues need to be updated to fix the vulnerabilities. Each image should be added to our docs, but in terms of developer action, we really just need 1 issue per Major.Minor version, with the list of CVEs of the latest image for the minor version.
• Since we scan images for all LTS branches, when we write to the code scanning tab in Gloo Edge, it appears that each subsequent LTS branch writes over the previous one. We need to figure out how to write to this tab using multiple branches.
• Images are all selected to be scanned as non-prerelease images. That cuts down on a number of images to scan since we don't scan the beta releases for all LTS branches. However, it means we also don't scan the beta releases for the current main branch. It would be useful to be able to scan the beta images for releases that don't yet have stable releases. To allow this, we could just inject a list of images to be scanned, instead of providing min and max versions and version contraints.
• After a scan runs, it's not immediately clear what action needs to be taken. There should be a slack notification that is trigged, with some details about the scan, and a note to developers whether action needs to be taken or not. If a new vulnerability is found in the latest version of each of the LTS branches, we should be made aware. Currently, it requires scrolling through the generated github issues and catching it.
• Scans run serially and can take ~4+ hours. We should be able to improve the performance dramatically using a worker queue and various worker goroutines. Additionally, there are various actions to take as a result of a security scan (ie upload to google cloud, create a github issue...etc). It would be nice to separate the image scanning from the post scan actions using some sort of an event bus.