Cloud/S3 trajectory serialization loses Claude `thinking`, `tool_use`, `tool_result`, and structured SSE responses

[SSSSSSilly]

Summary

When SAfactory runs LiveCVEBench with Claude Code through the Gateway and stores telemetry in the cloud/S3 landing table, the trajectory returned from S3 is not a lossless representation of the local Claude Code agent.log for the same session.

The Gateway captures the provider-bound request and Anthropic SSE response, but CloudStrategy converts them into the SDK's generic ChatMessage representation before writing the top-level messages and response fields. The current conversion silently drops unsupported Anthropic content blocks and wraps the complete SSE response as a single text item.

As a result, downstream consumers of session_steps.messages and session_steps.response cannot reconstruct the native Claude trajectory without reaching into raw fields in meta_json and implementing a separate Anthropic SSE parser.

Impact

• Claude thinking blocks and signatures are absent from normalized S3 messages.
• tool_use blocks are not converted to ChatMessage.tool_calls.
• tool_result blocks and tool_use_id relationships are lost.
• The response column contains raw SSE wrapped as ordinary text instead of a structured assistant message.
• list_session_steps() does not expose the original request saved in meta_json.request.
• TrajectoryReader reduces responses to text, losing reasoning and tool-call structure again.
• S3 trajectories are unsafe to treat as lossless training, replay, or debugging records.

Reproduction

Configuration:

storage_type: cloud

telemetry:
  mode: strict
  loss_policy: fail_closed
  capture_payload: full

Run LiveCVEBench with Claude Code against the Gateway /v1/messages endpoint, then compare:

1. The local Terminal-Bench sessions/agent.log.
2. S3 landing_test rows for the same job_id and session_id.
3. In particular, compare native Claude content block types and tool IDs against the S3 top-level messages and response fields.

One same-session reproduction is attached as two separate files: one local trajectory and one S3 trajectory.

Task	Session	Local derived assistant turns	S3 inference rows
cve-2025-10116	78845c81-2275-4a08-b1a9-c9089b9debce	28	25

The row/turn counts are diagnostic metadata only. They must not be interpreted as direct proof of dropped LLM calls because local streamed events may reuse a message ID. The definitive issue is the lossy field conversion described below.

Root cause analysis

1. Request-side content blocks are silently discarded

core/data_manager/strategy/cloud_strategy_impl.py::_convert_to_chat_messages() only handles string content, text, and image_url:

if isinstance(content_raw, str):
    content_items.append(ContentItem(type="text", text=content_raw))
elif isinstance(content_raw, list):
    for item in content_raw:
        if item.get("type") == "text":
            ...
        elif item.get("type") == "image_url":
            ...

There is no conversion or preservation path for Anthropic thinking, redacted_thinking, tool_use, tool_result, or signature fields. Unknown blocks are skipped without an error or warning.

The SDK ChatMessage model already exposes tool_calls and tool_call_id, but the conversion does not populate them.

2. The complete Anthropic SSE response is stored as ordinary text

CloudStrategy._build_step_record() constructs:

response_msg = ChatMessage(
    role="assistant",
    content=[ContentItem(type="text", text=response)]
)

For /v1/messages streaming calls, response is the complete Anthropic SSE stream. It is therefore written as one text block rather than a structured assistant message containing thinking, text, tool calls, tool inputs, stop reason, usage, and signatures.

3. The existing response parser is dead and lacks Anthropic SSE support

gateway/storage.py::_assistant_messages_from_response() exists but has no callers.

Even if wired into persistence, _collect_event_text() currently handles OpenAI chat-completion/Responses events but not native Anthropic events such as:

• message_start
• content_block_start
• content_block_delta with thinking_delta, signature_delta, text_delta, or input_json_delta
• content_block_stop
• message_delta
• message_stop

It also returns a normalized text/reasoning message rather than preserving the native ordered content blocks.

4. The standard cloud read path hides the raw request

CloudStrategy._build_step_record() saves the provider request under meta_json.request, but list_session_steps() removes meta_json and only restores env_state, group_id, and dataset.

Consequently, callers using the storage-neutral API cannot access the only lossless request copy.

5. TrajectoryReader performs another text-only conversion

evaluator/trajectory_reader.py::_extract_response_text() extracts plain response text. It does not return ordered Claude content blocks, tool calls/results, thinking signatures, stop metadata, or the raw response.

Expected behavior

With telemetry.capture_payload: full, the persisted and reloaded trajectory should preserve enough information to reconstruct the provider exchange exactly, apart from explicitly documented secret redaction and image URI substitution.

At minimum:

• Preserve the complete provider-bound request as a first-class readable field.
• Preserve the complete raw provider response/SSE as a first-class readable field.
• Preserve ordered Anthropic content blocks and their native fields.
• Preserve tool_use.id ↔ tool_result.tool_use_id relationships.
• Preserve thinking signatures and redacted-thinking blocks when capture policy permits it.
• Do not silently discard unknown block types.
• Keep normalized OpenAI-compatible views separate from raw provider payloads.

Suggested fix

1. Introduce explicit raw columns/fields such as raw_request, raw_response, and provider_format, or reliably expose the existing raw values through list_session_steps().
2. Do not use the generic ChatMessage schema as the only canonical trajectory representation.
3. Add a native Anthropic message/SSE aggregator that preserves ordered blocks, tool inputs, signatures, stop reason, and usage.
4. If an OpenAI-compatible normalized view is required, derive it in addition to the raw representation and document which fields cannot be represented.
5. Make _convert_to_chat_messages() reject or warn on unsupported content block types rather than silently skipping them.
6. Update TrajectoryReader to expose both raw and normalized representations.

Missing test coverage

The existing Claude Gateway test verifies that GatewayTelemetryRecord.request equals the forwarded payload and that GatewayTelemetryRecord.response still contains the SSE signature. It stops before the cloud serialization boundary.

Add an end-to-end round-trip test covering:

Anthropic request/SSE
  -> GatewayTelemetryRecord
  -> GatewayStorage
  -> CloudStrategy/LandingRecord
  -> S3-compatible read/list_session_steps
  -> TrajectoryReader

The fixture should contain thinking, signature, text, tool use with streamed JSON input, tool result, and a final stop event. Assert structural equality after round-trip, excluding only documented redactions.

Acceptance criteria

• A Claude request containing thinking, tool_use, and tool_result can be written and read without losing block type, order, IDs, input, content, or signature.
• A streamed response can be reconstructed from stored data and matches the captured provider stream or its documented canonical representation.
• list_session_steps() exposes the provider-bound request.
• Unknown content types cause a clear warning/error or are preserved as raw blocks.
• SQLite and cloud backends provide equivalent trajectory semantics.

Attachments

• cve-2025-10116-local-trajectory.json
• cve-2025-10116-s3-trajectory.json

Security note: the attachments contain full captured prompts, responses, tool inputs/results, and local paths. Review and sanitize them before uploading to a public GitHub repository.

cve-2025-10116-local-trajectory.json

cve-2025-10116-s3-trajectory.json