Skip to content

Conversation

@jpower432
Copy link
Contributor

⚠️ This probably still needs additional refinement. Sharing early for thoughts.

Summary

This PR introduces support for multiple assessment executors in Layer 4 Evaluation Plans, enabling tools and manual methods to execute assessment procedures with configurable conflict resolution strategies when multiple executors provide results.

This add a new directory specs to documentation how to use the Layer 4 schema for finding determination (how it could be used by Layer 5 tools).

Rationale

  1. Executor Abstraction: Introducing an AssessmentExecutor concept that can represent both automated tools and manual review processes.

  2. Conflict Resolution: Providing three distinct strategies that user can choose based on their security posture and tool maturity:

    • Strict: For organizations requiring zero tolerance and absolute consistency
    • ManualOverride: For organizations that trust human judgment over automated tools
    • AdvisoryRequiresConfirmation: For organizations integrating experimental tools or tools with known false positive patterns
  3. Role-Based Control: The AdvisoryRequiresConfirmation strategy allows users to:

    • Test new tools in production without disrupting workflows
    • Collect data from multiple tools while only triggering findings from trusted sources
    • Gradually promote tools from Advisory to Primary as they prove reliable

Test Data

Run cue vet -d "#EvaluationPlan" -s schemas/layer-4.cue layer4/test-data/multi-tool-plan.yaml

Usage

The new multi-tool-plan.yaml test data demonstrates configuring multiple executors with different roles and conflict resolution strategies:

executors:
  - id: "primary-scanner"
    type: Automated
    role: Primary
  - id: "experimental-tool"
    type: Automated
    role: Advisory
    
conflict_resolution:
  strategy: AdvisoryRequiresConfirmation

Partially resolves #170

Assisted by: Cursor AI (primarily for spec & test data creation)

Introduce support for multiple assessment executors (tools/manual methods) that
can execute assessment procedures, with configurable conflict resolution
strategies when multiple executors provide results.

Assisted by: Cursor Agent
Signed-off-by: Jennifer Power <[email protected]>
Assisted by: Cursor Agent
Signed-off-by: Jennifer Power <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enhance multi-source evaluation support in Layer 4/5

1 participant