Session state does not cross a RemoteA2aAgent boundary in either direction, silently

[iarjunganesh]

Summary

Session state does not cross a RemoteA2aAgent boundary in either direction, and nothing in
the API surface, logs, or docs indicates it. Because the same agent graph works correctly
in-process, this is invisible to unit tests, integration tests and CI, and only appears once an
agent is actually deployed behind A2A.

We hit it twice, in opposite directions, in a SequentialAgent pipeline that ran fine locally for
weeks.

Environment

• google-adk==2.7.1, a2a-sdk==1.1.2, Python 3.12
• Orchestrator is a SequentialAgent on Agent Engine; two peers are RemoteA2aAgents backed by
  Cloud Run services

Direction 1 — output_key on a remote agent never reaches the caller

# runs in the worker process
auditor = LlmAgent(name="auditor", output_schema=Report, output_key="findings", ...)

# runs in the orchestrator
class NextStep(BaseAgent):
    async def _run_async_impl(self, ctx):
        report = ctx.session.state.get("findings")   # populated in-process, always None over A2A

output_key writes into the session of the agent that declares it. In-process the declaring
agent and the reading agent share one session, so this works. Over A2A the worker has its own
session and that state never returns to the caller.

Direction 2 — a caller-side state_delta never reaches the remote agent

class PolicyStep(BaseAgent):
    async def _run_async_impl(self, ctx):
        yield Event(author=self.name, invocation_id=ctx.invocation_id,
                    actions=EventActions(state_delta={"routing": routing}))   # no content

# next in the SequentialAgent
RemoteA2aAgent(name="notifier", ...)   # never sees "routing"

RemoteA2aAgent._construct_message_parts_from_session builds the outgoing message from
event.content.parts and skips events whose content is None. A state-only event therefore
contributes nothing to the A2A request, and the remote agent receives the previous content
event instead — in our case an earlier agent's output, which looked plausible and was silently
acted upon.

Why this is worth a warning rather than only a doc note

The failure is silent, and the two directions fail differently:

• Direction 1 raises, if you happen to have a downstream check. If you don't, an LlmAgent
  reading the missing key will often invent plausible content instead.
• Direction 2 does not raise at all. The remote agent gets some content and proceeds.

Because in-process runs and A2A runs are described as the same composition with a different
transport, the natural assumption is that the hand-off is also the same. Our test suite has 100%
statement and branch coverage and could not have caught either one — the state key is populated in
exactly the topology tests run in.

Suggested remedies, in the order we'd value them

1. Warn when a SequentialAgent contains a RemoteA2aAgent and a sibling declares output_key
   — the combination is almost always a mistake, and it is statically detectable at construction.
2. Warn (or debug-log) when an event with a non-empty state_delta and no content is the last
   event before a RemoteA2aAgent runs, since that state is about to be dropped.
3. Document the boundary explicitly in the A2A guide: state is per-session, sessions are
   per-agent across A2A, and only event content crosses.
4. Optionally, an opt-in forward_state= / state_keys= on RemoteA2aAgent for callers who want
   specific keys serialized into the request.

We worked around it by having the caller read the remote agent's reply from session events
(sound in our case only because output_schema makes that reply validated JSON rather than
prose), and by emitting the hand-off as event content rather than as state_delta. Both are fine
once you know; the cost was entirely in not knowing.

Happy to open a PR for (1) and (3) if that's a direction you'd take.

[ALDRIN121]

Thanks for the thorough report — silent state loss across the A2A boundary in both directions is exactly the kind of footgun that wastes real debugging time, and the write-up is excellent.

I'd like to work on this. Scope I'm planning:

1. Warn at SequentialAgent construction when a sibling declares output_key next to a RemoteA2aAgent (remedy 1)
2. Warn when an event with a non-empty state_delta and no content is dropped before a RemoteA2aAgent runs (remedy 2)
3. Document the per-session/per-agent boundary in the A2A guide (remedy 3)

@iarjunganesh — if you've already started a PR for any of these, let me know and I'll happily take whichever parts remain. Otherwise I'll open one with the fixes and tests.

[ALDRIN121]

Root cause, pinpointed for whoever reviews the PR:

Direction 2 — RemoteA2aAgent._construct_message_parts_from_session (src/google/adk/agents/remote_a2a_agent.py:1196) skips any event without content.parts via a continue. Events carrying only actions.state_delta are dropped there, so nothing crosses to the peer.

Direction 1 — LlmAgent writes output_key into the declaring agent's own session (src/google/adk/agents/llm_agent.py:1045); sessions are per-agent across A2A, so the caller's session never sees the write.

Plan for the PR: warnings at both sites (state-only event dropped → warn; SequentialAgent mixing RemoteA2aAgent with an output_key sibling → warn at construction) + a note in the remote_a2a_agent guide + unit tests for the warn/no-warn paths. Opening the PR shortly.

[iarjunganesh]

Thanks for independently confirming the two code paths @ALDRIN121. Since this came from our production reproducer and I offered to contribute the fix in the original report, I’m going to take the first pass at the upstream PR, including warnings, documentation, and regression tests for both directions.
Please hold off on opening a duplicate for now. I’ll link the draft PR here shortly, and we can adjust or split the scope if the ADK maintainers request it.

[ALDRIN121]

Thanks for the quick response — will hold off on a PR from my side, no duplicate.

Open to splitting if that helps the maintainers: I already have a working branch with the Direction 1 fix (construction-time warning in SequentialAgent when an output_key sibling sits next to a RemoteA2aAgent) plus regression tests covering both directions. Happy to take that slice while you own Direction 2 + docs, or to contribute my tests to your draft if you'd rather keep it as one PR. Link the draft when it's up and we can sort the split.

[ALDRIN121]

Thanks for taking remedy 1 as well in #6862 — no duplicate from my side, glad the whole matrix (outbound drop, inbound rejection, construction-time warning, docs) is covered across the two PRs. Happy to help review/test either if useful. Good luck with the review!