fix: external scanner UX improvements#1692
Open
sergio-sisternes-epam wants to merge 3 commits into
Open
Conversation
- Actionable install command in SkillSpector not-found error with Python >=3.12 constraint and SARIF fallback path - Source column in findings table when external scanners contribute findings, with per-source counts in title - External scanner examples in apm audit --help epilog - --ci constraint visible in --external help text - Idempotent 'already enabled/disabled' uses info level, not warning - Exit code 3 for config/infra errors (feature disabled, scanner not found, malformed SARIF) to distinguish from exit 2 (warnings/usage) - Tests for source-column helpers and updated exit code assertions - Documentation updates for exit code 3 and install commands Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves the external-scanner experience in apm audit by making failures more actionable, adding provenance to mixed findings output, and documenting/validating updated exit-code semantics across code, tests, and docs.
Changes:
- Enhanced findings rendering to show per-scanner provenance (Source column + per-source counts in the table title) when external scanner findings are present.
- Improved SkillSpector “not installed / not on PATH” messaging with explicit install commands and verification guidance.
- Updated exit-code behavior for external-scanner configuration/infrastructure failures to use exit code
3, with corresponding test and documentation updates.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
src/apm_cli/commands/audit.py |
Adds provenance helpers + Source/Category table rendering; updates --external help text; maps external-scanner infra/config errors to exit code 3. |
src/apm_cli/commands/experimental.py |
Downgrades idempotent enable/disable messages from warning to progress/info. |
src/apm_cli/security/external/skillspector.py |
Adds actionable install + verification guidance when SkillSpector is missing. |
tests/unit/test_audit_scan_and_render.py |
Adds unit tests for new provenance helper functions. |
tests/unit/test_external_scanners.py |
Updates assertions to match new SkillSpector availability messaging. |
tests/integration/test_audit_external_scanners.py |
Updates integration expectations for new exit code 3 semantics. |
docs/src/content/docs/integrations/external-scanners.md |
Documents install commands and exit code 3 for config/infra errors. |
docs/src/content/docs/reference/cli/audit.md |
Adds exit code 3 to the CLI reference table. |
packages/apm-guide/.apm/skills/apm-usage/commands.md |
Updates the apm-usage skill reference to mention exit code 3 for scanner infra/config errors. |
Convert mutually exclusive flag checks (--format+--strip, --ci+--strip, --ci+--external, --external+--strip) from sys.exit(1) to click.UsageError, so they exit 2 consistent with the documented exit-code semantics. Also update experimental command tests to expect [i] (info) instead of [!] (warning) for idempotent enable/disable operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three additional test files had the same SystemExit assertion for the --format+--strip incompatibility check. Updated to expect click.UsageError consistent with the exit code contract change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
danielmeppiel
approved these changes
Jun 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
After validating the NVIDIA SkillSpector integration end-to-end and getting an independent UX review, several friction points were identified in the external scanner workflow. Users hit vague error messages when SkillSpector was not installed, findings lacked provenance information when multiple scanners contributed, and config/infra errors were conflated with warning-only exit codes.
Approach
Seven UX improvements implemented in a single pass, validated by running the full integration against a real SkillSpector install:
uv tool install/pip installcommand, notes the Python >=3.12 requirement, and suggests the SARIF fallback path.--helpexamples -- two external scanner usage examples added to the audit command epilog.--ciconstraint visible ----externalhelp text now states "Not supported with --ci."apm experimentaldowngraded from warning to info (progress), since repeating an idempotent operation is not a problem.Testing
_finding_source,_has_external_findings,_findings_title)Files changed
src/apm_cli/commands/audit.pysrc/apm_cli/commands/experimental.pysrc/apm_cli/security/external/skillspector.pytests/unit/test_audit_scan_and_render.pyTestFindingSourceHelpersclasstests/unit/test_external_scanners.pytests/integration/test_audit_external_scanners.pydocs/.../external-scanners.mddocs/.../reference/cli/audit.mdpackages/apm-guide/.../commands.md