Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/skills/sdk/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -939,7 +939,7 @@ Connections in JavaScript (AI-3378, AI-3387): `api.connections` has `create(conn
Connections in Python (AI-3379, no spec change): `plugins/stream` has `api.connections` with the Go shape: `create(connector_id, *, label=None, user_id=None)`, `authorize(id, *, return_url=None, user_id=None)` returning `Authorization`, `get(id, *, user_id=None)`, `list(*, connector_id=None, user_id=None)` reading every page, `delete(id, *, user_id=None)`; every option is keyword-only, so a user id cannot land in the label's or connector's slot. A user given is `owner.user_id` (create) and `X-Stream-User-Id` on every call, replacing whoever the client acts for and the user its backend was built for (which behind the proxy would sign that user's own token); none is the app's connection, sent as the client is otherwise (a client built for a user still names that user). AI-3387 in Python: `Client.for_user` is deprecated (`DeprecationWarning`, still working) in favour of a keyword-only `user_id=None`, last, on `Client.agent` and `Client.router`, on a client built with `team=`/`usage_categories=`; a device naming a user is refused before any request. Dispatch and `Accelerated` keep `Backend.acting_for` internally.
The session socket sends `reasoning_delta` (`turn_id`, `text`): the next piece of the model's thinking on a written reply, which a kept conversation also streams into the reply's Chat message and an incognito one has nowhere else to show. JavaScript folds it, with `tool_started`, `tool_updated` (not `verbatim`) and `tool_ran`, into each agent turn's `steps` in `sdks/js/src/conversation.ts`: a round of thinking `r1`, `r2`... opens on a delta after anything else and completes when the turn writes text or UI, calls a tool, answers or is cut off; a call runs until `tool_ran`, failed when it carries an `error`, its summary the last non-verbatim update. A turn is `done` on `responded` with no `pending_work`, or `interrupted`. `AgentChat` shows a session with no `conversationId` from these turns (`sdks/js/src/turn-messages.ts`): the agent's turns between two of the person's messages are one reply, shaped as the channel's messages (top level in stream-chat 9, under `custom` in 10) with `ai_reasoning` and `ai_tool_call` steps, rounds numbered across the reply, its whole thinking given to `reasoning`, and a call waiting in `approvals` shown `awaiting_approval` with its question and the session's `user_id` as `target_user_id`. The clients that carry the socket's description (JavaScript types, Python `plugins/stream`) are regenerated. Swift does the same: `Conversation` folds the frames into `Turn.steps` and `Turn.isDone`, and `TurnMessage.messages(of:approvals:userID:isConnected:)` (`session.turnMessages`) builds the replies, each step an attachment type and JSON payload for `AIMessagePart.parts(from:)`, the session's user now `Session.userID`; `swift_assistant`'s `AgentChat` shows an incognito session from them. Kotlin does the same: `Conversation` folds the frames into `Turn.steps` (`TurnStep.Reasoning`, `TurnStep.ToolCall`) and `Turn.isDone`, and `TurnMessage.messages(conversation, approvals, userId, isConnected)` (`AgentSession.turnMessages`, a `Flow`) builds the replies, each step an attachment type and plain-Kotlin fields for `AIMessagePart.parts`; its `target_user_id` is the user `setUser` named (`backend.user`), not the session's `user_id`, which is the name the agent joined the call as; `android_assistant`'s `AgentTransport` shows an incognito session from them. Other SDKs need to update: Dart, whose AI components read the same steps, should fold the same frames into their `Conversation` turns and show a session with no channel through their transports; the server SDKs need nothing.
An agent's UI instructions are kept apart from its instructions, so a session can turn UI off: `AgentConfig`, `AgentConfigRequest`, `AgentConfigPatch` and `SyncAgentRequest` have `ui_instructions` (an additive migration), and `CreateSessionRequest` has `ui`, a boolean a device may send. The router tells the model `ui_instructions` after the instructions (a backend's own session instructions included) unless the session was opened with `ui: false`, which reopening or forking it keeps (`agent_sessions.no_ui`). A sync sends the directory whole, so a sync without `ui_instructions` removes them; a PUT that leaves them out keeps them, so a client that does not know the field does not take the UI away. Go's `Load` compiles agent.yaml's `ui:` into `Folder.UIInstructions` and `syncFolder` sends it as `ui_instructions`; the folder's hash takes it in only when there is UI, so a directory without `ui:` is not synced again, and one with it is synced once more, which moves the UI out of the stored instructions. A router older than the field takes the sync and drops it, so `syncFolder` does not trust a stamp covering UI that the stored config lacks, and syncs again, and the router writes a sync of the same hash that declares UI the config lacks rather than finding it unchanged. Deploy the router before releasing an SDK that sends the field. JavaScript does the same: `loadFolder` puts the compiled `ui:` in `Folder.uiInstructions`, `Agent.sync` sends it as `ui_instructions` (left out without `ui:`, so such a directory keeps its fingerprint), and a stamp covering UI the stored config lacks is synced again. When a sync's answer has no `ui_instructions` though it sent them (a router older than the field), JavaScript syncs once more with the UI joined onto `instructions`, as before, under that body's own fingerprint and without writing the stamp, so the agent keeps its UI and the next sync moves it apart once the router knows the field. That fallback cannot honour `ui: false`: the UI is in `instructions` then, which every session is told, so the Interactive UI switch needs a router that knows the field and a sync it stored apart. Go only syncs again there, so an older router leaves a Go-synced agent with no UI, and Go should fall back the same way. `Agent`'s `instructions` option replaces instructions.md only: the folder's `ui:` is still sent as `ui_instructions`, as in Go. `react_assistant`'s settings have the same Interactive UI switch as `swift_assistant`'s, which opens new conversations with `ui: false`. The shared fixtures are unchanged: they are what the model is told, the instructions and the UI joined by a blank line, as the router joins them. Every generated client is regenerated (Go, JavaScript types, Python `plugins/stream`, Swift, Rust's `types.rs`); Rust compiling and sending `ui:` is still to do, below. Other SDKs need to update: Python, .NET, Ruby, Rust and PHP, when they read `ui:`, should send it as `ui_instructions` the same way; Swift has `SessionOptions.ui`, which `swift_assistant`'s Settings switch sets, and JavaScript's `sessions.create` passes `ui` through with the regenerated types, and Kotlin has `SessionOptions.ui`, which `Sessions.create` sends as it is (null left out); Python, Dart and the other server SDKs should offer `ui` on their create-session options, so an app that shows no UI can say so.
A tool call's state (`ToolCallState`, the `tool_call.started`, `tool_call.progress` and `tool_call.finished` frames) carries `display_title`: what the call is doing, as its tool was declared with (a session's own tool's `display_title`, or a hosted tool's), or "Searching the web" for the router's own search, the same default a Chat reply's tool step uses (`conversation.ToolTitle`). A worker's `host_tools` frame takes `display_title` on each tool, beside `interruptible` and `timeout_ms`. Go has `tools.DisplayTitle(title)`, sent on `host_tools` and on a session's own tools. The generated clients are regenerated (Go, JavaScript types, Python `plugins/stream`, Ruby, PHP, .NET); Swift's generates no `ToolCallState`, so it is unchanged. JavaScript needs nothing: its `host_tools` sends `tools.declared()`, `displayTitle` included, which the router now keeps. Other SDKs need to update: Python, .NET, Ruby, Rust and PHP should send their tool's display title on `host_tools`, their registries taking one where they do not yet; Swift, Kotlin and Dart read `display_title` off the tool call state once regenerated.
A tool call's state (`ToolCallState`, the `tool_call.started`, `tool_call.progress` and `tool_call.finished` frames) carries `display_title`: what the call is doing, as its tool was declared with (a session's own tool's `display_title`, or a hosted tool's), or "Searching the web" for the router's own search, the same default a Chat reply's tool step uses (`conversation.ToolTitle`). A worker's `host_tools` frame takes `display_title` on each tool, beside `interruptible` and `timeout_ms`. Go has `tools.DisplayTitle(title)`, sent on `host_tools` and on a session's own tools. The generated clients are regenerated (Go, JavaScript types, Python `plugins/stream`, Ruby, PHP, .NET); Swift's generates no `ToolCallState`, so it is unchanged. JavaScript needs nothing: its `host_tools` sends `tools.declared()`, `displayTitle` included, which the router now keeps. Every server SDK now declares one: Python `register(display_title=...)` (on `ToolPolicy`, sent by `plugins/stream`'s `_with_policy`), .NET `displayTitle:`, Ruby `display_title:`, PHP `displayTitle:` and Rust's `SessionTool.display_title`; Swift, Kotlin and Dart read it off the tool call state. Nothing is left to update.
Connections in PHP (AI-3381, AI-3387): `$client->connections` (`sdks/php/src/Connections.php`) has `create(NewConnection)`, `authorize(id, AuthorizeOptions)`, `get(id, userId)`, `list(ConnectionFilter)` (every page, `limit` 200, a repeated cursor ends it) and `delete(id, userId)`, Go's shape: the calls with more than one optional take an options object (`NewConnection`, `AuthorizeOptions`, `ConnectionFilter`, `sdks/php/src/`) like Go's `ConnectionRequest`, `AuthorizeOptions` and `ConnectionFilter`, so a user id cannot take the place of a label, return URL or connector (R1.1 of the review: `create('linear', 'ada')` had filed the user's account as the app's). `userId` is optional and last (a parameter of `get` and `delete`, a field of the options objects); given, it is `owner.user_id` and `X-Stream-User-Id` with the server credential (a server token behind the proxy, never a user token); left out or empty, `owner.type: app` and no header, whoever the backend otherwise acts for. `Connections` goes through `Backend::speakingFor` (`@internal`, PHP has no package visibility), which drops the backend's own user. `Dispatch::answer` goes through `Backend::writingFor` (`@internal`, `Dispatch.php:193`), which keeps the backend's own user when the writer has none, as `onBehalfOf` did before; `onBehalfOf` calls `writingFor`. `Backend::onBehalfOf` stays public and working, `@deprecated` in favour of the `userId` parameter; PHP's `Backend::actingFor` is unchanged and still signs a user token (the device path), so its name means something else than `actingFor` in Go and JavaScript. No other public PHP wrapper used `onBehalfOf`.
.NET connections (AI-3382): `VisionAgentsClient.Connections` (`sdks/dotnet/src/GetStream.VisionAgents/Connections.cs`) has the Go shape: `CreateAsync(connectorId, NewConnectionOptions?)`, `AuthorizeAsync(id, AuthorizeOptions?)` and `ListAsync(ConnectionFilter?)` (every page, 200 at a time) take sealed options records with `UserId`, as Go's `ConnectionRequest`, `AuthorizeOptions` and `ConnectionFilter` do, because adjacent optional strings let a user passed in the wrong place become a label; `GetAsync(id, userId)` and `DeleteAsync(id, userId)` take `string? userId` last before the `CancellationToken`. A user given goes through the internal `ActingFor` that Dispatch uses (a server credential plus `X-Stream-User-Id`, replacing a user the client was made for); a null or empty one sends with the client as it is. AI-3387 needed no deprecation in .NET: `Backend` is internal (`Backend.cs:143`) and so is `VisionAgentsClient.ActingFor`, so no public acting helper or wrapper built on one exists.
Ruby connections and the user parameter (AI-3380, AI-3387): `Client#connections` has `create(connector, label:, user_id:)`, `authorize(id, return_url:, user_id:)`, `get(id, user_id:)`, `list(connector:, user_id:)` (every page, `owner_type` app or user) and `delete(id, user_id:)`; a `user_id` is the owner on create and `X-Stream-User-Id` on every call, none is the app's connection with no header. `Client#get`, `post`, `put`, `patch`, `delete` and `socket` take an optional `user_id:` last (nil keeps whoever the backend acts for, empty drops it), and `Backend#acting_for` keeps working with a `@deprecated` tag pointing at it; `Dispatch#answer` acts through `Backend#acting` (`@api private`). Other SDKs follow the same rule (AI-3378..AI-3383).
Expand Down
2 changes: 2 additions & 0 deletions plugins/stream/tests/test_dispatch.py
Original file line number Diff line number Diff line change
Expand Up @@ -1241,6 +1241,7 @@ async def test_a_hosted_functions_policies_are_declared_to_the_router(
interruptible=True,
timeout=90,
needs_approval="Book flight {flight_id}?",
display_title="Booking your flight",
)
async def book_flight(flight_id: str) -> str:
return "booked"
Expand All @@ -1258,6 +1259,7 @@ async def book_flight(flight_id: str) -> str:
assert tool["interruptible"] is True
assert tool["timeout_ms"] == 90000
assert tool["approval"] == {"title": "Book flight {flight_id}?"}
assert tool["display_title"] == "Booking your flight"

async def test_a_hosted_call_the_router_cancels_stops(self, router: Router):
client = stream.Client(url=router.url, customer_id="acme")
Expand Down
2 changes: 2 additions & 0 deletions plugins/stream/vision_agents/plugins/stream/sessions.py
Original file line number Diff line number Diff line change
Expand Up @@ -1159,6 +1159,8 @@ def _with_policy(tool: SessionTool, policy: ToolPolicy) -> SessionTool:
tool.timeout_ms = max(1, int(policy.timeout * 1000))
if policy.approval:
tool.approval = SessionToolApproval(title=policy.approval)
if policy.display_title:
tool.display_title = policy.display_title
return tool


Expand Down
12 changes: 11 additions & 1 deletion sdks/python/vision_agents/core/llm/function_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ class ToolPolicy:
# approval is the question a person answers before each call runs, with {name} for an
# argument. Empty runs calls as they come.
approval: str = ""
# display_title is what a call is doing, in words for the people in the conversation,
# such as "Checking the weather". Empty shows the function's name.
display_title: str = ""


@dataclass
Expand Down Expand Up @@ -99,6 +102,7 @@ def register(
interruptible: bool = False,
timeout: float = 0.0,
needs_approval: str = "",
display_title: str = "",
) -> Callable:
"""
Decorator to register a function with the registry.
Expand All @@ -120,6 +124,9 @@ def register(
call. On a voice call the agent asks it out loud and waits 15 seconds for a
yes; in writing the question goes to the app. Applies where Stream runs the
agent.
display_title: What a call is doing, such as "Checking the weather", at most
80 characters, shown on the reply's tool call. Applies where Stream runs
the agent.

Returns:
Decorator function.
Expand All @@ -134,7 +141,10 @@ def register(
if needs_approval and not approval:
raise ValueError("an approval question cannot be blank")
policy = ToolPolicy(
interruptible=interruptible, timeout=timeout, approval=approval
interruptible=interruptible,
timeout=timeout,
approval=approval,
display_title=display_title,
)

def decorator(func: Callable) -> Callable:
Expand Down
4 changes: 4 additions & 0 deletions sdks/python/vision_agents/core/llm/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -431,6 +431,7 @@ def register_function(
interruptible: bool = False,
timeout: float = 0.0,
needs_approval: str = "",
display_title: str = "",
) -> Callable:
"""
Decorator to register a function with the LLM's function registry.
Expand All @@ -447,6 +448,8 @@ def register_function(
Applies where Stream runs the agent.
needs_approval: A question a person answers before each call runs, with
{name} for an argument. Applies where Stream runs the agent.
display_title: What a call is doing, such as "Checking the weather".
Applies where Stream runs the agent.

Returns:
Decorator function.
Expand All @@ -457,6 +460,7 @@ def register_function(
interruptible=interruptible,
timeout=timeout,
needs_approval=needs_approval,
display_title=display_title,
)

def get_available_functions(self) -> List[ToolSchema]:
Expand Down
Loading