Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tooling exists for OSV record creators to validate that they meet the minimum quality bar at record creation time #2187

Open
andrewpollock opened this issue May 9, 2024 · 6 comments
Assignees

Comments

@andrewpollock
Copy link
Contributor

Where possible, relying on JSON schema validation

@andrewpollock
Copy link
Contributor Author

There are valid concerns about the UX of JSON Schema validation.

I am exploring options to address these concerns:

I am not feeling particularly confident they can be fundamentally addressed, though, at this point in time.

In the interests of not succumbing to the tyranny of the or one possibility is to do both JSON Schema validation and replicate the checks in the OSV record linter that will be inevitably necessary.

That also feels not very DRY, though, and introduces other maintenance and behaviour consistency headaches.

@andrewpollock
Copy link
Contributor Author

Decision:

Take a belt-and-suspenders approach: add schema validation (and more human-targeted title) to the OSV Schema where practical, but replicate the checks in the linter.

@oliverchang
Copy link
Collaborator

Decision:

Take a belt-and-suspenders approach: add schema validation (and more human-targeted title) to the OSV Schema where practical, but replicate the checks in the linter.

+1 that makes sense.

Let's do the basic checks where we can in the JSON schema to enable a flexible option that can be re-used everywhere, and offer a more full featured standalone linter separately for users that can integrate that.

@andrewpollock
Copy link
Contributor Author

ossf/osv-schema#246 bolstered the schema validation to the extent I currently believe possible.

@andrewpollock
Copy link
Contributor Author

ossf/osv-schema#246 bolstered the schema validation to the extent I currently believe possible.

I spoke too soon: ossf/osv-schema#251 adds CVSS score validation.

@andrewpollock
Copy link
Contributor Author

ossf/osv-schema#90 is relevant to this work also.

andrewpollock added a commit to ossf/osv-schema that referenced this issue Aug 29, 2024
This is reasonably functional at this point, with multiple checks of two
different aspects:

Ranges:
- `introduced` exists
- don't overlap

Packages:
- plumbing for ecosystem-specific behaviour
- package existence
  - PyPI
  - Go
- package version existence
  - PyPI
  - Go (with some caveats around pseudoversions)
- Basic Purl validity

```
$ go run ./cmd/osv record lint test_data/
Running "osv.dev" check collection on &["test_data/"]
2024/08/07 23:26:14 Found 9 files in "test_data/"
Running "introduced-event-exists" check on "test_data/CVE-2018-5407.json"
Running "range-is-distinct" check on "test_data/CVE-2018-5407.json"
Running "package-exists" check on "test_data/CVE-2018-5407.json"
2024/08/07 23:26:14 "test_data/CVE-2018-5407.json": "package-exists": []checks.CheckError{checks.CheckError{Code:"P0001", Message:": package \"openssl\" not found"}}
Running "package-versions-exist" check on "test_data/CVE-2018-5407.json"
2024/08/07 23:26:14 "test_data/CVE-2018-5407.json": "package-versions-exist": []checks.CheckError{checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}}
Running "package-purl-valid" check on "test_data/CVE-2018-5407.json"
Running "introduced-event-exists" check on "test_data/CVE-2023-41045.json"
Running "range-is-distinct" check on "test_data/CVE-2023-41045.json"
Running "package-exists" check on "test_data/CVE-2023-41045.json"
Running "package-versions-exist" check on "test_data/CVE-2023-41045.json"
Running "package-purl-valid" check on "test_data/CVE-2023-41045.json"
Running "introduced-event-exists" check on "test_data/GHSA-9v2f-6vcg-3hgv.json"
Running "range-is-distinct" check on "test_data/GHSA-9v2f-6vcg-3hgv.json"
Running "package-exists" check on "test_data/GHSA-9v2f-6vcg-3hgv.json"
Running "package-versions-exist" check on "test_data/GHSA-9v2f-6vcg-3hgv.json"
Running "package-purl-valid" check on "test_data/GHSA-9v2f-6vcg-3hgv.json"
Running "introduced-event-exists" check on "test_data/GO-2020-0001.json"
Running "range-is-distinct" check on "test_data/GO-2020-0001.json"
Running "package-exists" check on "test_data/GO-2020-0001.json"
Running "package-versions-exist" check on "test_data/GO-2020-0001.json"
2024/08/07 23:26:16 "test_data/GO-2020-0001.json": "package-versions-exist": []checks.CheckError{checks.CheckError{Code:"P0002", Message:": Failed to find some versions of github.com/gin-gonic/gin: &errors.errorString{s:\"failed to find [1.6] for \\\"github.com/gin-gonic/gin\\\" in [v1.9.0 v1.3.0 v1.7.0 v1.8.0 v1.6.0 v1.8.2 v1.1.1 v1.5.0 v1.7.2 v1.7.1 v1.1.3 v1.1.2 v1.9.1 v1.6.3 v1.10.0 v1.7.3 v1.7.5 v1.4.0 v1.1.4 v1.6.1 v1.7.7 v1.8.1 v1.6.2 v1.7.4 v1.7.6 ]\"}"}}
Running "package-purl-valid" check on "test_data/GO-2020-0001.json"
Running "introduced-event-exists" check on "test_data/GO-2024-2963.json"
Running "range-is-distinct" check on "test_data/GO-2024-2963.json"
Running "package-exists" check on "test_data/GO-2024-2963.json"
Running "package-versions-exist" check on "test_data/GO-2024-2963.json"
Running "package-purl-valid" check on "test_data/GO-2024-2963.json"
Running "introduced-event-exists" check on "test_data/PYSEC-2023-74.json"
Running "range-is-distinct" check on "test_data/PYSEC-2023-74.json"
Running "package-exists" check on "test_data/PYSEC-2023-74.json"
Running "package-versions-exist" check on "test_data/PYSEC-2023-74.json"
Running "package-purl-valid" check on "test_data/PYSEC-2023-74.json"
Running "introduced-event-exists" check on "test_data/nointroduced-CVE-2023-41045.json"
2024/08/07 23:26:18 "test_data/nointroduced-CVE-2023-41045.json": "introduced-event-exists": []checks.CheckError{checks.CheckError{Code:"R0001", Message:": missing 'introduced' object in event"}}
Running "range-is-distinct" check on "test_data/nointroduced-CVE-2023-41045.json"
Running "package-exists" check on "test_data/nointroduced-CVE-2023-41045.json"
Running "package-versions-exist" check on "test_data/nointroduced-CVE-2023-41045.json"
Running "package-purl-valid" check on "test_data/nointroduced-CVE-2023-41045.json"
Running "introduced-event-exists" check on "test_data/nondistinct-CVE-2018-5407.json"
Running "range-is-distinct" check on "test_data/nondistinct-CVE-2018-5407.json"
2024/08/07 23:26:18 "test_data/nondistinct-CVE-2018-5407.json": "range-is-distinct": []checks.CheckError{checks.CheckError{Code:"R0002", Message:": overlapping event: \"e818b74be2170fbe957a07b0da4401c2b694b3b8\""}}
Running "package-exists" check on "test_data/nondistinct-CVE-2018-5407.json"
2024/08/07 23:26:18 "test_data/nondistinct-CVE-2018-5407.json": "package-exists": []checks.CheckError{checks.CheckError{Code:"P0001", Message:": package \"openssl\" not found"}}
Running "package-versions-exist" check on "test_data/nondistinct-CVE-2018-5407.json"
2024/08/07 23:26:18 "test_data/nondistinct-CVE-2018-5407.json": "package-versions-exist": []checks.CheckError{checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}}
Running "package-purl-valid" check on "test_data/nondistinct-CVE-2018-5407.json"
Running "introduced-event-exists" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json"
Running "range-is-distinct" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json"
Running "package-exists" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json"
2024/08/07 23:26:19 "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json": "package-exists": []checks.CheckError{checks.CheckError{Code:"P0001", Message:": package \"Gradi0\" not found"}}
Running "package-versions-exist" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json"
2024/08/07 23:26:19 "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json": "package-versions-exist": []checks.CheckError{checks.CheckError{Code:"P0002", Message:": Failed to find some versions of Gradi0: &errors.errorString{s:\"unable to validate package: fail: \\\"https://pypi.org/pypi/Gradi0/json\\\": bad response: 404\"}"}}
Running "package-purl-valid" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json"
2024/08/07 23:26:19 found errors
exit status 1
```

Part of google/osv.dev#2187

---------

Signed-off-by: Andrew Pollock <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In Progress
Development

No branches or pull requests

2 participants