feat/add ABEvalFlow as a reusable Konflux IntegrationTestScenario

[ikrispin]

Summary

Adds ABEvalFlow as a reusable Konflux IntegrationTestScenario. Any Konflux application can reference this to run the full ABEvalFlow evaluation (A2A agent testing, security scanning, quality review, scorecard with certification levels) against its built images as part of the CI/CD pipeline.

Tested end-to-end with the google-lightspeed-agent A2A agent on the stone-prod-p02 Konflux cluster. The pipeline parses the Konflux Snapshot, deploys the agent on a workload cluster, runs 3 Harbor A2A trials with LLM-as-judge scoring, computes the unified scorecard with certification, and returns structured TEST_OUTPUT back to Konflux for release gating.

Architecture

Uses a two-cluster model following the standard Konflux pattern for integration tests that need live workloads:

• Konflux cluster (stone-prod-p02) — orchestrates the pipeline, parses Snapshots, computes scorecards, reports TEST_OUTPUT
• Workload cluster (cn-ai-lab / ab-eval-flow) — runs the agent deployment and Harbor evaluation where LiteLLM and eval infrastructure are co-located

The evaluation (Harbor trials + analyze.py) runs as a remote Pod on the workload cluster to avoid cross-cluster networking/SSL issues. Results are extracted back via pod log markers and written to the shared workspace for the scorecard task.

Konflux (stone-prod-p02)                     cn-ai-lab (ab-eval-flow)
├── parse-snapshot                           
├── deploy-agent ──── oc login ─────────────> Agent Deployment + Service
├── prepare (clone ABEvalFlow)               
├── test (security + quality)                
├── evaluate ──── oc login ─────────────────> Eval Pod (Harbor + analyze.py)
│   └─ extracts report.json <───────────────  └─ LiteLLM (svc:4000)
├── analyze-scorecard                          └─ Agent (svc:8000)
├── store                                    
├── emit-result → TEST_OUTPUT                
└── finally: cleanup-agent ─────────────────> Deletes Deployment + eval Pod

What's included

New Tekton tasks (4 files)

• parse-snapshot — extracts container image, git URL, revision from Konflux SNAPSHOT JSON
• deploy-agent — deploys A2A agent on the workload cluster via remote oc login
• emit-result — maps scorecard pass/warn/fail + certification to Konflux TEST_OUTPUT
• cleanup-agent — tears down agent and eval Pods in the finally block

Adapted Tekton tasks (5 files)

Konflux-specific versions of existing phase tasks under pipeline/tasks/konflux/:

• prepare — removed hardcoded namespace, configurable pipeline repo URL
• test — MinIO/DB persistence optional (graceful no-op when secrets missing)
• evaluate — submits eval as remote Pod, runs Harbor + analyze.py on workload cluster, extracts report.json from logs
• analyze-scorecard — detects pre-computed report.json from eval Pod, skips redundant analysis
• store — all external dependencies (MinIO, PostgreSQL, GitHub, Quay) optional

Pipeline + CI

• konflux-eval-pipelinerun.yaml — PipelineRun with 9 tasks chained via bundle resolver, PVC workspace, 4h timeout, finally cleanup
• Makefile — make bundles to push all tasks as Tekton Bundles to Quay
• push-bundles.yaml — GitHub Actions workflow for automated bundle publishing

Submission + Config

• submissions/google-lightspeed-agent/ — POC submission with A2A eval tasks, LLM judge, and Harbor environment
• config/konflux/secrets-template.yaml — templates for workload-cluster-credentials, llm-credentials, compass-facts-api
• config/konflux/integration-test-scenario.yaml — IntegrationTestScenario CR reference

OCI artifacts in Quay

Container image

Image	Purpose
quay.io/rh-ee-ikrispin/abevalflow-eval-base:latest	Pre-built image with Harbor, Python deps, and the A2A adapter. Used by the evaluate task's remote eval Pod. Sourced from the internal OpenShift registry image ab-eval-flow/eval-base:local-env.

Tekton Bundles

All 9 tasks are published as OCI bundles under quay.io/rh-ee-ikrispin/:

Bundle	Source
abevalflow-task-parse-snapshot:0.1	New
abevalflow-task-deploy-agent:0.1	New
abevalflow-task-emit-result:0.1	New
abevalflow-task-cleanup-agent:0.1	New
abevalflow-task-prepare:0.1	Adapted
abevalflow-task-test:0.1	Adapted
abevalflow-task-evaluate:0.1	Adapted
abevalflow-task-analyze-scorecard:0.1	Adapted
abevalflow-task-store:0.1	Adapted

The pipeline/integration/Makefile provides make bundles to push all and make digests to print SHA digests. When moving to production, these should be relocated to a team Quay org (e.g., quay.io/rhecosystemappeng/).

Key design decisions

IntegrationTestScenario, not pipeline migration — ABEvalFlow is integrated as a reusable eval service that Konflux applications reference, not a full migration to OCI Trusted Artifacts. Any project onboards by applying a single CR.

Tekton Bundles — All 9 tasks are published as versioned OCI bundles to Quay and referenced via the bundles resolver. This follows the same pattern as Konflux's own task catalog and allows independent versioning.

Cross-cluster execution — The managed Konflux cluster doesn't allow creating Deployments/Services. The agent and eval workloads are deployed on the workload cluster via oc login with a stored SA token. This matches how other Konflux integration tests handle live workloads.

Remote eval Pod — Harbor and analyze.py run inside a Pod on the workload cluster where LiteLLM and the agent are co-located (cluster-internal svc URLs, no SSL issues). Results are extracted back to Konflux via delimited log markers.

Dual-publish model — The PipelineRun is resolved from git (via the ITS), but the tasks inside it are resolved from Tekton Bundles. Updating task logic requires both a bundle push and a git push. This is documented in the PipelineRun header.

How another project onboards

apiVersion: appstudio.redhat.com/v1beta2
kind: IntegrationTestScenario
metadata:
  name: abevalflow-eval
  namespace: <their-tenant>
spec:
  application: <their-app>
  resolverRef:
    resolver: git
    resourceKind: pipelinerun
    params:
      - name: url
        value: https://github.com/RHEcosystemAppEng/ABEvalFlow
      - name: revision
        value: main
      - name: pathInRepo
        value: pipeline/integration/konflux-eval-pipelinerun.yaml

They also need: a submission directory in this repo, workload-cluster-credentials secret in their tenant, and the agent image accessible from the workload cluster.

Testing

Test environment

• Konflux cluster: stone-prod-p02 (api.stone-prod-p02.hjvn.p1.openshiftapps.com)
• Konflux tenant: ai5-marketplace-tenant
• Workload cluster: cn-ai-lab (api.cn-ai-lab.2vn8.p1.openshiftapps.com)
• Workload namespace: ab-eval-flow
• Target application: google-lightspeed-agent (A2A agent)

Test execution

Testing was done by creating override Snapshots on the Konflux cluster to trigger the IntegrationTestScenario without modifying the upstream agent repo. The Snapshot referenced an existing built image from quay.io/redhat-user-workloads/ai5-marketplace-tenant/google-lightspeed-agent.

Test results

{
  "result": "SUCCESS",
  "note": "ABEvalFlow: recommendation=pass, certification=foundational, gates_passed=5, gates_failed=0",
  "successes": 1,
  "failures": 0,
  "warnings": 0
}

• 3 A2A trials completed via Harbor (mean reward: 0.5)
• 5 gates passed: evaluation (a2a), security (cisco), security (skillmd-scanner), quality (llm-review), quality (skillmd-quality)
• Foundational certification achieved
• Agent deployment + cleanup on workload cluster worked correctly
• Scorecard + certification computed and written to workspace
• TEST_OUTPUT correctly returned to Konflux as SUCCESS

What was validated

• Tekton Bundle resolution from Quay (all 9 tasks)
• SNAPSHOT JSON parsing and component image extraction
• Cross-cluster agent deployment via remote oc login
• Remote eval Pod execution (Harbor + analyze.py on workload cluster)
• Report.json extraction from pod logs via delimited markers
• Pre-computed report detection by analyze-scorecard task
• Scorecard aggregation with all gates
• Certification level computation
• TEST_OUTPUT mapping to Konflux format
• Agent cleanup in finally block (even on failure)
• Graceful handling when optional secrets (MinIO, PostgreSQL, Compass) are missing

What was NOT validated

• Using the actual Konflux snapshot image (private registry auth not configured on workload cluster)
• Compass fact push (SSL cert issue from Konflux cluster — non-blocking, scorecard still computed)
• Full agent responses (agent returns empty content with mock SSO — agent-side limitation)
• SHA-pinned bundle references (currently using :0.1 tags)

Known limitations and TODOs

Item	Status	Details
--insecure-skip-tls-verify	TODO in code	Cross-cluster oc calls skip TLS verification. Should inject workload cluster CA cert as a Secret.
Log-based result transfer	TODO in code	report.json is extracted from eval Pod logs via delimited markers. Should migrate to ConfigMap-based transfer for robustness.
Snapshot image auth	Workaround	The PipelineRun accepts AGENT_IMAGE_OVERRIDE because the Konflux-built image (redhat-user-workloads) is private and the workload cluster can't pull it. Defaults to the snapshot image; set override when needed.
Bundle SHA pinning	Deferred	Bundle references use :0.1 tags (mutable). Should pin by @sha256: digest for production. The Makefile's make digests target provides the SHAs.
Compass fact push	SSL issue	The scorecard task on Konflux can't push facts to compass.stage.redhat.com due to SSL cert verification. Non-blocking — scorecard is computed locally.
Empty agent responses	Agent-side	The lightspeed agent returns 0 chars with mock SSO (SKIP_JWT_VALIDATION=true). Real SSO credentials needed for meaningful eval scores.
Runtime package installs	Deferred	Tasks install jq/pip packages at runtime. Could be pre-baked into a custom base image.
Quay org	Temporary	Bundles and eval-base image are under quay.io/rh-ee-ikrispin/. Should be relocated to a team org for production.

Related

https://redhat.atlassian.net/browse/APPENG-5740?atlOrigin=eyJpIjoiZDE0MDU0ZTE3YWNmNGI2ZDkwMTkxMDFkZDg2MDM5ZDIiLCJwIjoiaiJ9