Skip to content

fix: external scanner UX improvements#1692

Open
sergio-sisternes-epam wants to merge 3 commits into
mainfrom
sergio-sisternes-epam/fix-external-scanner-ux-improvements
Open

fix: external scanner UX improvements#1692
sergio-sisternes-epam wants to merge 3 commits into
mainfrom
sergio-sisternes-epam/fix-external-scanner-ux-improvements

Conversation

@sergio-sisternes-epam

Copy link
Copy Markdown
Collaborator

Why

After validating the NVIDIA SkillSpector integration end-to-end and getting an independent UX review, several friction points were identified in the external scanner workflow. Users hit vague error messages when SkillSpector was not installed, findings lacked provenance information when multiple scanners contributed, and config/infra errors were conflated with warning-only exit codes.

Approach

Seven UX improvements implemented in a single pass, validated by running the full integration against a real SkillSpector install:

  • Actionable install command -- when SkillSpector is not found, the error now shows the exact uv tool install / pip install command, notes the Python >=3.12 requirement, and suggests the SARIF fallback path.
  • Source column in findings table -- when external scanners contribute findings, the table adds a "Source" column and replaces the dead "Codepoint" column with "Category". The table title shows per-source counts (e.g. "Audit Findings (apm: 2, skillspector: 3)").
  • --help examples -- two external scanner usage examples added to the audit command epilog.
  • --ci constraint visible -- --external help text now states "Not supported with --ci."
  • Idempotent log level -- "already enabled"/"already disabled" messages in apm experimental downgraded from warning to info (progress), since repeating an idempotent operation is not a problem.
  • Exit code 3 -- configuration and infrastructure errors (feature disabled, scanner not found, malformed SARIF) now exit 3, distinguishing them from exit 2 (warning-only findings / usage errors).
  • Documentation -- exit code 3 documented in the external-scanners guide, CLI reference, and apm-usage skill resource.

Testing

  • 7 new unit tests for source-column helpers (_finding_source, _has_external_findings, _findings_title)
  • 6 integration test assertions updated for exit code 3 semantics
  • All 140 affected tests pass
  • Full lint chain clean (ruff check, ruff format, pylint R0801, auth-signals)

Files changed

File Change
src/apm_cli/commands/audit.py Source-column helpers, table rendering, help examples, exit code 3
src/apm_cli/commands/experimental.py Idempotent log level fix
src/apm_cli/security/external/skillspector.py Actionable install error messages
tests/unit/test_audit_scan_and_render.py New TestFindingSourceHelpers class
tests/unit/test_external_scanners.py Updated error message assertion
tests/integration/test_audit_external_scanners.py Exit code 3 assertions
docs/.../external-scanners.md Install commands, exit code 3
docs/.../reference/cli/audit.md Exit code 3 in table
packages/apm-guide/.../commands.md Exit code 3 note

- Actionable install command in SkillSpector not-found error with
  Python >=3.12 constraint and SARIF fallback path
- Source column in findings table when external scanners contribute
  findings, with per-source counts in title
- External scanner examples in apm audit --help epilog
- --ci constraint visible in --external help text
- Idempotent 'already enabled/disabled' uses info level, not warning
- Exit code 3 for config/infra errors (feature disabled, scanner not
  found, malformed SARIF) to distinguish from exit 2 (warnings/usage)
- Tests for source-column helpers and updated exit code assertions
- Documentation updates for exit code 3 and install commands

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 7, 2026 13:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the external-scanner experience in apm audit by making failures more actionable, adding provenance to mixed findings output, and documenting/validating updated exit-code semantics across code, tests, and docs.

Changes:

  • Enhanced findings rendering to show per-scanner provenance (Source column + per-source counts in the table title) when external scanner findings are present.
  • Improved SkillSpector “not installed / not on PATH” messaging with explicit install commands and verification guidance.
  • Updated exit-code behavior for external-scanner configuration/infrastructure failures to use exit code 3, with corresponding test and documentation updates.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/apm_cli/commands/audit.py Adds provenance helpers + Source/Category table rendering; updates --external help text; maps external-scanner infra/config errors to exit code 3.
src/apm_cli/commands/experimental.py Downgrades idempotent enable/disable messages from warning to progress/info.
src/apm_cli/security/external/skillspector.py Adds actionable install + verification guidance when SkillSpector is missing.
tests/unit/test_audit_scan_and_render.py Adds unit tests for new provenance helper functions.
tests/unit/test_external_scanners.py Updates assertions to match new SkillSpector availability messaging.
tests/integration/test_audit_external_scanners.py Updates integration expectations for new exit code 3 semantics.
docs/src/content/docs/integrations/external-scanners.md Documents install commands and exit code 3 for config/infra errors.
docs/src/content/docs/reference/cli/audit.md Adds exit code 3 to the CLI reference table.
packages/apm-guide/.apm/skills/apm-usage/commands.md Updates the apm-usage skill reference to mention exit code 3 for scanner infra/config errors.

Comment thread src/apm_cli/commands/audit.py
Sergio Sisternes and others added 2 commits June 7, 2026 19:22
Convert mutually exclusive flag checks (--format+--strip, --ci+--strip,
--ci+--external, --external+--strip) from sys.exit(1) to
click.UsageError, so they exit 2 consistent with the documented
exit-code semantics.

Also update experimental command tests to expect [i] (info) instead
of [!] (warning) for idempotent enable/disable operations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three additional test files had the same SystemExit assertion for
the --format+--strip incompatibility check. Updated to expect
click.UsageError consistent with the exit code contract change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@danielmeppiel danielmeppiel enabled auto-merge June 8, 2026 21:16
@danielmeppiel danielmeppiel added this pull request to the merge queue Jun 8, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 8, 2026
@sergio-sisternes-epam sergio-sisternes-epam added this pull request to the merge queue Jun 8, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants