-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tooling exists for OSV record creators to validate that they meet the minimum quality bar at record creation time #2187
Comments
There are valid concerns about the UX of JSON Schema validation. I am exploring options to address these concerns:
I am not feeling particularly confident they can be fundamentally addressed, though, at this point in time. In the interests of not succumbing to the tyranny of the or one possibility is to do both JSON Schema validation and replicate the checks in the OSV record linter that will be inevitably necessary. That also feels not very DRY, though, and introduces other maintenance and behaviour consistency headaches. |
Decision: Take a belt-and-suspenders approach: add schema validation (and more human-targeted |
+1 that makes sense. Let's do the basic checks where we can in the JSON schema to enable a flexible option that can be re-used everywhere, and offer a more full featured standalone linter separately for users that can integrate that. |
ossf/osv-schema#246 bolstered the schema validation to the extent I currently believe possible. |
I spoke too soon: ossf/osv-schema#251 adds CVSS score validation. |
ossf/osv-schema#90 is relevant to this work also. |
This is reasonably functional at this point, with multiple checks of two different aspects: Ranges: - `introduced` exists - don't overlap Packages: - plumbing for ecosystem-specific behaviour - package existence - PyPI - Go - package version existence - PyPI - Go (with some caveats around pseudoversions) - Basic Purl validity ``` $ go run ./cmd/osv record lint test_data/ Running "osv.dev" check collection on &["test_data/"] 2024/08/07 23:26:14 Found 9 files in "test_data/" Running "introduced-event-exists" check on "test_data/CVE-2018-5407.json" Running "range-is-distinct" check on "test_data/CVE-2018-5407.json" Running "package-exists" check on "test_data/CVE-2018-5407.json" 2024/08/07 23:26:14 "test_data/CVE-2018-5407.json": "package-exists": []checks.CheckError{checks.CheckError{Code:"P0001", Message:": package \"openssl\" not found"}} Running "package-versions-exist" check on "test_data/CVE-2018-5407.json" 2024/08/07 23:26:14 "test_data/CVE-2018-5407.json": "package-versions-exist": []checks.CheckError{checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}} Running "package-purl-valid" check on "test_data/CVE-2018-5407.json" Running "introduced-event-exists" check on "test_data/CVE-2023-41045.json" Running "range-is-distinct" check on "test_data/CVE-2023-41045.json" Running "package-exists" check on "test_data/CVE-2023-41045.json" Running "package-versions-exist" check on "test_data/CVE-2023-41045.json" Running "package-purl-valid" check on "test_data/CVE-2023-41045.json" Running "introduced-event-exists" check on "test_data/GHSA-9v2f-6vcg-3hgv.json" Running "range-is-distinct" check on "test_data/GHSA-9v2f-6vcg-3hgv.json" Running "package-exists" check on "test_data/GHSA-9v2f-6vcg-3hgv.json" Running "package-versions-exist" check on "test_data/GHSA-9v2f-6vcg-3hgv.json" Running "package-purl-valid" check on "test_data/GHSA-9v2f-6vcg-3hgv.json" Running "introduced-event-exists" check on "test_data/GO-2020-0001.json" Running "range-is-distinct" check on "test_data/GO-2020-0001.json" Running "package-exists" check on "test_data/GO-2020-0001.json" Running "package-versions-exist" check on "test_data/GO-2020-0001.json" 2024/08/07 23:26:16 "test_data/GO-2020-0001.json": "package-versions-exist": []checks.CheckError{checks.CheckError{Code:"P0002", Message:": Failed to find some versions of github.com/gin-gonic/gin: &errors.errorString{s:\"failed to find [1.6] for \\\"github.com/gin-gonic/gin\\\" in [v1.9.0 v1.3.0 v1.7.0 v1.8.0 v1.6.0 v1.8.2 v1.1.1 v1.5.0 v1.7.2 v1.7.1 v1.1.3 v1.1.2 v1.9.1 v1.6.3 v1.10.0 v1.7.3 v1.7.5 v1.4.0 v1.1.4 v1.6.1 v1.7.7 v1.8.1 v1.6.2 v1.7.4 v1.7.6 ]\"}"}} Running "package-purl-valid" check on "test_data/GO-2020-0001.json" Running "introduced-event-exists" check on "test_data/GO-2024-2963.json" Running "range-is-distinct" check on "test_data/GO-2024-2963.json" Running "package-exists" check on "test_data/GO-2024-2963.json" Running "package-versions-exist" check on "test_data/GO-2024-2963.json" Running "package-purl-valid" check on "test_data/GO-2024-2963.json" Running "introduced-event-exists" check on "test_data/PYSEC-2023-74.json" Running "range-is-distinct" check on "test_data/PYSEC-2023-74.json" Running "package-exists" check on "test_data/PYSEC-2023-74.json" Running "package-versions-exist" check on "test_data/PYSEC-2023-74.json" Running "package-purl-valid" check on "test_data/PYSEC-2023-74.json" Running "introduced-event-exists" check on "test_data/nointroduced-CVE-2023-41045.json" 2024/08/07 23:26:18 "test_data/nointroduced-CVE-2023-41045.json": "introduced-event-exists": []checks.CheckError{checks.CheckError{Code:"R0001", Message:": missing 'introduced' object in event"}} Running "range-is-distinct" check on "test_data/nointroduced-CVE-2023-41045.json" Running "package-exists" check on "test_data/nointroduced-CVE-2023-41045.json" Running "package-versions-exist" check on "test_data/nointroduced-CVE-2023-41045.json" Running "package-purl-valid" check on "test_data/nointroduced-CVE-2023-41045.json" Running "introduced-event-exists" check on "test_data/nondistinct-CVE-2018-5407.json" Running "range-is-distinct" check on "test_data/nondistinct-CVE-2018-5407.json" 2024/08/07 23:26:18 "test_data/nondistinct-CVE-2018-5407.json": "range-is-distinct": []checks.CheckError{checks.CheckError{Code:"R0002", Message:": overlapping event: \"e818b74be2170fbe957a07b0da4401c2b694b3b8\""}} Running "package-exists" check on "test_data/nondistinct-CVE-2018-5407.json" 2024/08/07 23:26:18 "test_data/nondistinct-CVE-2018-5407.json": "package-exists": []checks.CheckError{checks.CheckError{Code:"P0001", Message:": package \"openssl\" not found"}} Running "package-versions-exist" check on "test_data/nondistinct-CVE-2018-5407.json" 2024/08/07 23:26:18 "test_data/nondistinct-CVE-2018-5407.json": "package-versions-exist": []checks.CheckError{checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}, checks.CheckError{Code:"P0002", Message:": Failed to find some versions of openssl: &errors.errorString{s:\"unsupported ecosystem: Alpine\"}"}} Running "package-purl-valid" check on "test_data/nondistinct-CVE-2018-5407.json" Running "introduced-event-exists" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json" Running "range-is-distinct" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json" Running "package-exists" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json" 2024/08/07 23:26:19 "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json": "package-exists": []checks.CheckError{checks.CheckError{Code:"P0001", Message:": package \"Gradi0\" not found"}} Running "package-versions-exist" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json" 2024/08/07 23:26:19 "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json": "package-versions-exist": []checks.CheckError{checks.CheckError{Code:"P0002", Message:": Failed to find some versions of Gradi0: &errors.errorString{s:\"unable to validate package: fail: \\\"https://pypi.org/pypi/Gradi0/json\\\": bad response: 404\"}"}} Running "package-purl-valid" check on "test_data/nopackage-GHSA-9v2f-6vcg-3hgv.json" 2024/08/07 23:26:19 found errors exit status 1 ``` Part of google/osv.dev#2187 --------- Signed-off-by: Andrew Pollock <[email protected]>
Where possible, relying on JSON schema validation
The text was updated successfully, but these errors were encountered: