Add scoring feature #686
Add scoring feature #686
Conversation
…lations Signed-off-by: Dustin Popp <[email protected]>
…sts paths I found some odd behavior in which Spectral messes with the path strings reported in the "path" array in the rule's context. This led to our code crashing when we couldn't find the string in the API. This change skips the rule and logs an error. In the future, we may want to address the problem on the Spectral side or fix the path in-line and proceed with the check. Signed-off-by: Dustin Popp <[email protected]>
Signed-off-by: Dustin Popp <[email protected]>
Signed-off-by: Dustin Popp <[email protected]>
Signed-off-by: Dustin Popp <[email protected]>
Signed-off-by: Dustin Popp <[email protected]>
Signed-off-by: Dustin Popp <[email protected]>
| `${ruleId}: could not find path string ${pathString} in paths object. Skipping check...` | ||
| ); | ||
| return []; | ||
| } |
There was a problem hiding this comment.
A strange bug I ran into while testing the scoring logic on different APIs. I'll add a test to reproduce before I merge.
| @@ -4,7 +4,7 @@ | |||
| */ | |||
|
|
|||
| /** | |||
| * Takes an unresolved path to a schema and un-resolves it to the format in which | |||
| "find-up": "5.0.0", | ||
| "globby": "^11.0.4", | ||
| "js-yaml": "^3.14.1", | ||
| "json-dup-key-validator": "^1.0.3", | ||
| "lodash": "^4.17.21", | ||
| "nimma": "^0.7.0", |
There was a problem hiding this comment.
"@stoplight/spectral-ref-resolver" and "nimma" were already installed dependencies through Spectral, but this makes it obvious that we need them.
| ); | ||
|
|
||
| // Populate the metrics for the API. | ||
| await metrics.compute(); |
There was a problem hiding this comment.
The code needed for nimma to do this is sorta complicated - this file just makes it look nice and neat for us to define new "metrics" as we need them.
| * SPDX-License-Identifier: Apache2.0 | ||
| */ | ||
|
|
||
| module.exports = { |
There was a problem hiding this comment.
Will be adding some documentation around what this file is, what things mean, and how one should go about updating it when adding a new rule.
| @@ -79,14 +83,12 @@ function convertResults(spectralResults, { config, logger }) { | |||
| finalResultsObject.hasResults = true; | |||
| finalResultsObject[severity].summary.total++; | |||
|
|
|||
| if (!summaryOnly) { | |||
There was a problem hiding this comment.
We don't want to filter this here anymore, because we want to be able to use the rule violations to compute the quality scores. This filtering is moved to the end of run-validator.js, right before printing output.
Signed-off-by: Dustin Popp <[email protected]>
| ...require('./get-captured-text'), | ||
| ...require('./get-captured-text'), | ||
| ...require('./get-message-and-path-from-captured-text'), | ||
| ...require('./strip-ansi'), |
There was a problem hiding this comment.
Purely "syntactic sugar" changes.
|
|
||
| const ibmRuleset = require('@ibm-cloud/openapi-ruleset'); | ||
|
|
||
| describe('scoring-tool rubric tests', function () { |
There was a problem hiding this comment.
These tests are essentially "linters" for the rubric, to make sure it stays up to date and uses the right data.
|
Example output with the new flag: 
|
Signed-off-by: Dustin Popp <[email protected]>
dpopp07
force-pushed
the
dp/integrate-scoring-tool
branch
from
0647ba1
to
19e0365
Compare
This is a big PR to add a feature that translates the rule violations reported by the validator into simple scores that capture the impact of the rule violations on the API.
Any feedback is welcome. The majority of the changes are stable but I'm still adding a couple more pieces of documentation and maybe one or two more tests. That said, it's ready to review.
The commit breakdown is probably not very helpful - I'm going to try and clean it up before merging.