Gather ideas on how to detect base images and or packages that contain known CVE vulnerabilities

[tschaffter]

@ahayden @thomasyu888 To further tighten the security of our evaluation environment, I'm thinking to scan Docker images for known Common Vulnerabilities and Exposures (CVE) using a service like https://snyk.io/. I experimented with Snyk a few weeks ago and I was able to scan Docker images using one command line.

A strength of Snyk is its integration to CI/CD, however I don't see how we could benefit from this. Instead, we could use the snyk client to scan images.

$ sudo apt install nodejs npm
$ sudo npm install -g snyk
$ snyk test [email protected]
`snyk` requires an authenticated account. Please run `snyk auth` and try again.

Go to https://app.snyk.io/account and get a token, then use it as shown below

$ snyk config set api=xxxxxxxxxxxxxxxxxxxxxx
api updatedting you to our auth page, go ahead and log in,
and once the auth is complete, return to this prompt and you'll

Normally your browser should open to ask you to validate this token so it can be used on your machine. Because I'm running the above command in a Docker container for security reason, the browser page never opening. From Snyk documentation I figured out that I should go to the address https://app.snyk.io/login/cli?token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Back to the terminal:

$ snyk test [email protected]

Testing [email protected]...

✗ Low severity vulnerability found in utile
  Description: Uninitialized Memory Exposure
  Info: https://snyk.io/vuln/npm:utile:20180614
  Introduced through: [email protected], [email protected]
  From: [email protected] > [email protected]
  From: [email protected] > [email protected] > [email protected]

✗ Low severity vulnerability found in serve-static
  Description: Open Redirect
  Info: https://snyk.io/vuln/npm:serve-static:20150113
  Introduced through: [email protected], [email protected]
  From: [email protected]
  From: [email protected] > [email protected]

...

✗ Medium severity vulnerability found in minimist
  Description: Prototype Pollution
  Info: https://snyk.io/vuln/SNYK-JS-MINIMIST-559764
  Introduced through: [email protected], [email protected]
  From: [email protected] > [email protected]
  From: [email protected] > [email protected] > [email protected]
  From: [email protected] > [email protected] > [email protected] > [email protected]
...

✗ High severity vulnerability found in qs
  Description: Prototype Override Protection Bypass
  Info: https://snyk.io/vuln/npm:qs:20170213
  Introduced through: [email protected], [email protected], [email protected], [email protected]
  From: [email protected] > [email protected] > [email protected] > [email protected] > [email protected]
  From: [email protected] > [email protected]
  From: [email protected] > [email protected] > [email protected]
  and 5 more...
...

✗ High severity vulnerability found in bl
  Description: Remote Memory Exposure
  Info: https://snyk.io/vuln/SNYK-JS-BL-608877
  Introduced through: [email protected], [email protected], [email protected]
  From: [email protected] > [email protected]
  From: [email protected] > [email protected] > [email protected]
  From: [email protected] > [email protected] > [email protected]
  and 2 more...



Organization:      thomas.schaffter
Package manager:   npm
Open source:       yes
Project path:      [email protected]

Tested [email protected] for known vulnerabilities, found 46 vulnerabilities, 318 vulnerable paths.

We can use the exit code of snyk test to determine if the test "didn't pass" (i.e. at least one vulnerability?)

$ echo $?
1
$ snyk test debian@latest

Testing debian@latest...

Organization:      thomas.schaffter
Package manager:   npm
Open source:       yes
Project path:      debian@latest

✓ Tested debian@latest for known vulnerabilities, no vulnerable paths found.

Tip: Snyk only tests production dependencies by default. You can try re-running with the `--dev` flag.

tschaffter@docker:~$ echo $?
0

[thomasyu888]

I think we had synk set up at some point. I had brought this up to Aaron at the beginning of this year when we went over vulnerabilities of using docker

[thomasyu888]

@tschaffter , I would leave synk out of the validation of docker images. The reason for this is that they "force" you to login through the console when using their cli (and I think the login token expires over a period of time). Therefore it makes is incredibly difficult to automate this process.

My original thought was for the possibility of all docker images in the docker registry to be scanned. We certainly should implement some service in the new challenge platform that will auto scan all the docker images upon docker push.

[tschaffter]

My original thought was for the possibility of all docker images in the docker registry to be scanned. We certainly should implement some service in the new challenge platform that will auto scan all the docker images upon docker push.

That's definitively a feature that would be great to have in general.

An important note to consider is that the output of the scan is dependent on WHEN the scan occurs. An image that is a few weeks old will likely have CVEs. For the sake of benchmarking, the scan should occurs just before we process the image. We may also need to derogate to this scan if we want to evaluate a past submission on a new dataset that has just been connected.

For the reasons that we have both exposed, let's keep this ticket as Low: Priority and come back to it when it's needed, for example if a Data Site want to be able to add this scan to the validation process.

[thomasyu888]

@tschaffter Now docker scan is a built in command that we can potentially use to scan docker images.

docker pull docker.synapse.org/syn22277123/date-annotator-example:1.2.0
docker scan docker.synapse.org/syn22277123/date-annotator-example:1.2.0

.......
Package manager:   deb
Project name:      docker-image|docker.synapse.org/syn22277123/date-annotator-example
Docker image:      docker.synapse.org/syn22277123/date-annotator-example:1.2.0
Platform:          linux/amd64

Tested 138 dependencies for known vulnerabilities, found 133 vulnerabilities.

For more free scans that keep your images secure, sign up to Snyk at https://dockr.ly/3ePqVcp

[tschaffter]

@thomasyu888 How would you test this feature before rolling it out to production? Can we implement it for submissions that are sent to test submissions queues?

Since the validity of a submission will depends on time - i.e. a submission that was submitted 1 month ago may no longer pass the scan test - I'm afraid that it would be difficult to rerun old submission. Instead, we should consider a model where we email the results of the scan to submitters but don't block submission from being evaluated based on the output of the scan.

[thomasyu888]

@tschaffter Also I just wanted you to see that our date-annotator-example has 133 vulnerabilities and around 10 of high severity.

We could email or upload results of the scan to their log folder, although this is something that they could do on their own now. Since it doesn't seem like we are going to take action on these high severity issues, I wonder if we should even implement this step.

[tschaffter]

Just created nlpsandbox/date-annotator-example#122