Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request implements a generic security signals and containment framework (Issue #879) allowing Agent Mesh components to dispatch out-of-band security signals to external supervisors via a non-blocking, bounded ring-buffer queue with support for HMAC-SHA256 signing, mTLS, and event coalescing. The review feedback identifies a critical Denial of Service (DoS) vulnerability in the coalescer due to unbounded map and timer allocation from unauthenticated peers, which should be mitigated with a hard limit on active entries. Additionally, several style guide violations were found, including multiple instances of uncanonicalized peer IDs crossing boundaries (violating Section 1) and an ignored error from 'rand.Read' during UUID generation (violating Section 6.3).
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements an out-of-band security containment signals and webhook mechanism (Issue #879) for Google Agent Mesh, allowing nodes and the control plane to asynchronously dispatch real-time security events (policy denials, transport refusals, identity revocations, and enrollment rejections) to external supervisors. The implementation includes a non-blocking dispatcher with a bounded queue, a sliding-window event coalescer, support for HMAC-SHA256 signing and mTLS, and a runnable reference webhook receiver. Feedback on the changes highlights a bug in both the client and configuration parser where the HMAC secret environment variable fallback is ignored if the secret path is empty, as well as an opportunity to handle oversized payloads more gracefully in the reference webhook receiver by returning a 413 status code.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements the Generic Security Signals and Containment Contract (Issue #879) to enable out-of-band security containment and supervisor feedback loops. It introduces a non-blocking, asynchronous Dispatcher and Coalescer in the internal/containment package to forward security signals (policy denials, transport refusals, identity revocations, and enrollment rejections) to external supervisors over HTTP, HTTPS, or Unix domain sockets. The signals are formatted as CloudEvents 1.0 compliant ProtoJSON payloads and support HMAC-SHA256 signing and mutual TLS (mTLS) for cryptographic integrity. These signals are integrated into both the agentmesh-node (via stream auth gates, middleware, and egress inspection) and the agentmesh-control-plane (via enrollment and refresh handlers). Extensive unit, integration, concurrency, and resilience tests have been added, alongside comprehensive documentation and a runnable reference webhook receiver. The implementation strictly adheres to the repository's style guides, including proper peer ID canonicalization at boundaries, explicit time units, safe loopback-only defaults for plaintext HTTP, and secret hygiene. I have no feedback to provide as the changes are exceptionally well-designed, thoroughly tested, and fully compliant with all style and architectural guidelines.
3b82bdb to
d88dbb9
Compare
…ontract (google#879) Signed-off-by: femi oyesanya <oyesanyf@gmail.com>
…and review findings - Clone SecuritySignalEvent on Dispatch, Coalescer.Record, and Client.Send, and capture metric/log fields before channel handoff to eliminate concurrent read/write data race under -race. - Bound Coalescer active entries to MaxCoalescerEntries (1024) with agentmesh_containment_coalescer_entries gauge and agentmesh_containment_coalescer_overflow_total counter to prevent DoS via spoofed peer IDs. - Canonicalize peer IDs via peer.Decode across EmitPolicyDenied, emitIdentityRevoked, reconcileBannedPeers, emitEnrollmentRejected, and Coalescer.Record, and check crypto/rand.Read errors. - Check error returns on Close/Remove, remove trailing colon in scheme validation error string, and remove unreachable exported wrappers to satisfy golangci-lint and deadcode in hack/lint.sh. - Regenerate sdk/python agentmesh_pb2.py and agentmesh_pb2.pyi matching CI protoc output. Signed-off-by: femi oyesanya <oyesanyf@gmail.com>
biscuit-go defaults to a 2ms Datalog evaluation timeout, which under -race on loaded CI runners exceeds the budget and causes spurious denials in TestIngressDatalogPolicyEvaluation. Match api/datalog_test.go and api/http_grants_test.go (per f1399f6) by passing an explicit 5s world duration budget.
…ad limit - Resolve AGENTMESH_CONTAINMENT_HMAC_SECRET via secrets.FromPathOrEnv in containment.FromProto, EndpointFromProto, NewClient, and NewSingleClient even when hmac_secret_path is omitted, trimming whitespace and preserving fallback across multiple configured endpoints. - Return 413 Payload Too Large (via io.LimitReader(r.Body, maxBodyBytes+1)) and defer body close upfront in the generic-containment-webhook example. - Add TestHMACSecretEnvFallbackWithoutPath unit tests and update oversized payload assertions.
…t bound Add the agentmesh_containment_coalescer_entries gauge and agentmesh_containment_coalescer_overflow_total counter to the containment signals reference metrics table, and replace raw LaTeX delimiters in the coalesced_count description with plain Markdown. Signed-off-by: femi oyesanya <oyesanyf@gmail.com>
d88dbb9 to
d722997
Compare
Resolves #879
Signed-off-by: femi oyesanya oyesanyf@gmail.com
1. Summary & Motivation
Google Agent Mesh operates as the Zero Trust identity, cryptographic task authority, and Policy Enforcement Point (PEP) for multi-agent communications, while delegating OS, pod, and filesystem confinement to external sandbox runtimes (NVIDIA OpenShell, Docker Sandbox
docker sbx, Kubernetesagent-sandboxwithgvisor/kata, or host supervisors), as documented insite/content/docs/contributing/security-architecture.md§3.1:The Gap
When a sandboxed workload is compromised (e.g. via prompt injection or sub-goal hijack) or its identity is revoked:
agentmesh-nodeblocks unauthorized requests in-band (via Biscuit Datalog policy,TaskAuthorizationRuletar_blocknarrowing, orModelArmor/ExtProcinspection) and drops streams from revoked peers (MeshEvent.BANNED).What This PR Delivers
This PR implements an opt-in, 100% generic, runtime-agnostic out-of-band security webhook contract across
agentmesh-nodeandagentmesh-control-plane:SecuritySignalEventenvelope and four discrete security state transitions inapi/agentmesh.proto(protojsonwithUseProtoNames: true, plus CloudEvents 1.0 binary-mode HTTP headers).unix:/path/to.sock,https://(with mTLS / custom CA), or loopback-onlyhttp://127.0.0.1:<port>.git diff go.mod go.sumis empty) and zero vendor-specific or OS-specific concepts. Every protobuf message introduced in this PR is actively wired end-to-end and covered by unit and integration tests.2. Addressing Maintainer Feedback on Issue #879
On issue #879, @aojea noted:
api/agentmesh.proto), Go packages (internal/containment), and node/control-plane hooks contain zero references to specific host engines or OS kernel primitives.protojson(UseProtoNames: true) bodies so Knative/Eventarc sinks, Kubernetes controllers, OpenShell/gVisor supervisors, SIEM pipelines, and host daemons can consume events without custom adapters.agentmesh.policy.denied: Biscuit Datalog denial,tar_blocknarrowing violation, orModelArmor/ExtProcblock, distinguishingLOCAL_WORKLOADvsREMOTE_PEER.agentmesh.transport.refused: libp2p connection/stream auth gate refusal (unauthenticated,not_router,banned), coalesced over a configurable window to prevent scan floods.agentmesh.identity.revoked: Control-plane revocation or node application ofMeshEvent.BANNED/revocation_ids/ ban-rejected/refresh.agentmesh.enrollment.rejected:/enrollor/refreshbootstrap/recovery refusal onagentmesh-control-plane.ext_authz, andInspection(ExtProc/ModelArmor), and every schema message inapi/agentmesh.protohas live emission and test coverage today.3. Signal Flow
sequenceDiagram autonumber participant Agent as Untrusted Agent Workload participant Node as agentmesh-node PEP participant Dispatcher as containment.Dispatcher participant Webhook as External Supervisor participant Host as Sandbox / Orchestrator Runtime Agent->>Node: Unauthorized API / Tool Request Note over Node: Evaluate Biscuit Datalog & tar_block Node-->>Agent: 403 Forbidden (Immediate Data-Path Rejection) Note over Node,Dispatcher: Non-Blocking Out-of-Band Dispatch Node->>Dispatcher: Dispatch(&api.SecuritySignalEvent{...}) alt Signal Coalescing Window Active Dispatcher->>Dispatcher: Aggregate count per (signal, peer, service) else Signal Ready for Dispatch Dispatcher->>Dispatcher: Enqueue to Bounded Ring Buffer Dispatcher->>Dispatcher: Compute HMAC-SHA256 & Format CloudEvents 1.0 Dispatcher->>Webhook: HTTP POST /events (unix:... or https:...) Webhook->>Webhook: Constant-time HMAC verification & ProtoJSON parse Webhook->>Host: Enforce Containment (cgroup freeze / Pod evict / Token revoke) end4. The 4 Outbound Security State Transitions (
api/agentmesh.proto)All signal messages live in
api/agentmesh.protoinside theSecuritySignalEventenvelope and obeyAGENTS.md§1 (google.protobuf.Timestampfields named*_time—event_timeandfirst_seen_time; unset instants rejected):agentmesh.policy.denied(PolicyDeniedSignal):LOCAL_WORKLOADvsREMOTE_PEER.PolicyDenialReason):DATALOG_EVALUATION_DENIED,TASK_AUTHORIZATION_RULE_VIOLATION,INSPECTION_FILTER_BLOCKED,ROLE_MISMATCH,EXPIRED_CREDENTIAL,UNAUTHORIZED_SERVICE,UNAUTHORIZED_PATH_OR_METHOD(classified deterministically via typederrors.Isunwrapping).target_service,requested_operation,requested_resource,caller_peer_id,caller_subject,caller_roles,caller_labels,event_time.agentmesh.transport.refused(TransportRefusedSignal):TransportRefusalReason):PEER_UNAUTHENTICATED,NOT_ROUTER_ROLE,AUTH_DEADLINE_EXCEEDED,PEER_BANNED,PROTOCOL_UNAUTHORIZED.coalesced_count,first_seen_time, andevent_timeper(signal_type, peer_id, protocol_id).agentmesh.identity.revoked(IdentityRevokedSignal):RevocationSource):CONTROL_PLANE_ADMIN,MESH_EVENT_BANNED_GOSSIP,LOCAL_CONTROL_PLANE_SYNC,TOKEN_REFRESH_REJECTION.peer_id,reason,revocation_ids,issuer_peer_id,event_time.agentmesh.enrollment.rejected(EnrollmentRejectedSignal):EnrollmentRejectionReason):INVALID_JWT,EXPIRED_BOOTSTRAP_TOKEN,BOOTSTRAP_TOKEN_USAGE_EXCEEDED,UNATTESTED_LABELS_REJECTED,INVALID_CHALLENGE_SIGNATURE,REQUESTED_ROLE_UNAUTHORIZED,PEER_PREVIOUSLY_BANNED,AUTONOMOUS_RECOVERY_DISABLED.peer_id,requested_role,bootstrap_token_id,remote_address,submitted_labels,is_autonomous_recovery,event_time.5. CloudEvents 1.0 Headers, Transport Security, & Secret Hygiene
5.1 CloudEvents 1.0 Binary Mode & HMAC Signing
Each
POSTcarriesce-specversion: 1.0,ce-id,ce-type,ce-source,ce-time,ce-subject,Content-Type: application/json, and (when configured)X-AgentMesh-Timestamp+ constant-timeX-AgentMesh-Signature: sha256=<hex>over<timestamp>.<body>.5.2 Transports & Anti-SSRF Enforcement
unix:/path/to.sock: Local Unix domain socket transport.https://...: TLS 1.2+ with optionalca_cert_path,client_cert_path, andclient_key_pathfor mTLS.http://127.0.0.1:<port>: Plaintext HTTP is restricted strictly to loopback (net.ParseIP(host).IsLoopback()/localhost); non-loopbackhttp://targets fail closed at initialization.5.3 Secret Hygiene (
AGENTS.md§1)--security-webhook-hmac-path,--security-webhook-ca-cert-path,--security-webhook-client-cert-path,--security-webhook-client-key-path) or environment variable (AGENTMESH_CONTAINMENT_HMAC_SECRET, immediately scrubbed viaos.Unsetenvafter read), never via CLI value flags or echoed in logs.6. Zero New Dependencies, Observability, & Verification
go.mod/go.sumChanges (AGENTS.md§2):git diff go.mod go.sumis0bytes.api/agentmesh.pb.go,sdk/js/src/gen/agentmesh_pb.ts, andsdk/python/src/agent_mesh/_proto/agentmesh_pb2.py/.pyi.promauto):agentmesh_containment_events_queued_totalagentmesh_containment_events_delivered_totalagentmesh_containment_events_dropped_totalagentmesh_containment_events_coalesced_totalagentmesh_containment_delivery_duration_secondsagentmesh_containment_queue_depthsite/content/docs/reference/containment-signals.md,site/content/docs/preview/sandboxed-agents.md,agentmesh-node.md,node-config.md) and runnable pure-Go receiver (development/examples/generic-containment-webhook/).AGENTS.md§3): Standardtestingpackage only, zeroTestify/Ginkgo, zerotime.Sleep,-raceclean,<10sper suite.