[AIT-209] Message per token examples #3054
Conversation
WalkthroughThis pull request introduces comprehensive examples for AI Transport token streaming in both JavaScript and React, demonstrating real-time LLM token consumption via Ably. Includes complete project scaffolding with documentation, UI components, backend simulation service, and configuration. Changes
Sequence DiagramsequenceDiagram
actor User
participant Client as Client<br/>(JS/React)
participant Ably as Ably Channel
participant Backend as Backend LLM<br/>Service
User->>Client: Select prompt
activate Client
Client->>Client: Generate responseId<br/>Update UI (processing)
Client->>Backend: requestLLMProcessing<br/>(prompt, responseId)
deactivate Client
activate Backend
Backend->>Backend: Tokenize response<br/>into chunks
loop For each token
Backend->>Backend: Randomized delay
Backend->>Ably: Publish token message<br/>(token, responseId header)
end
Backend->>Ably: Publish stream-complete<br/>(responseId header)
deactivate Backend
activate Client
Ably->>Client: Message event (token)
Client->>Client: Filter by responseId<br/>Append to messages
Client->>Client: Aggregate & render
Ably->>Client: Message event (stream-complete)
Client->>Client: Mark processing complete<br/>Update UI
deactivate Client
User->>Client: Disconnect
activate Client
Client->>Ably: Detach channel
deactivate Client
User->>Client: Reconnect
activate Client
Client->>Ably: Re-attach channel
Ably->>Client: Paginated history<br/>(untilAttach)
Client->>Client: Filter & deduplicate<br/>messages by responseId
Client->>Client: Aggregate & render
deactivate Client
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
Pre-merge checks and finishing touches❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing touches
Comment |
matt423
force-pushed
the
ait-148-message-per-token-examples
branch
from
53fafec to
ea06c40
Compare
matt423
changed the base branch from
main
to
AIT-129-AIT-Docs-release-branch
matt423
force-pushed
the
ait-148-message-per-token-examples
branch
from
ea06c40 to
0ba3d4e
Compare
matt423
force-pushed
the
ait-148-message-per-token-examples
branch
from
0ba3d4e to
1c06661
Compare
matt423
force-pushed
the
AIT-129-AIT-Docs-release-branch
branch
from
400eb09 to
f8056cb
Compare
matt423
force-pushed
the
ait-148-message-per-token-examples
branch
from
1c06661 to
73d9273
Compare
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
♻️ Duplicate comments (1)
examples/ai-transport-token-streaming/react/postcss.config.js (1)
1-6: Configuration is correct but depends on missing packages.This PostCSS configuration is standard for Tailwind CSS integration. However, it references
tailwindcssandautoprefixerpackages that are currently missing frompackage.jsondevDependencies. Once those dependencies are added (as flagged in the package.json review), this configuration will work correctly.
🧹 Nitpick comments (12)
examples/ai-transport-token-streaming/javascript/README.md (1)
42-42: Reduce repetitive phrasing in sequential instructions.Lines 42 and 60 use similar structures ("to be your" and "to use your") when describing how to update the
VITE_ABLY_KEYvariable. Varying the phrasing would improve readability across the getting started steps.🔎 Suggested rewording
Line 42 can remain as-is, and line 60 could be revised for clarity:
- In CodeSandbox, rename the `.env.example` file to `.env.local` and update the value of your `VITE_ABLY_KEY` variable to use your Ably API key. + In CodeSandbox, rename the `.env.example` file to `.env.local` and set `VITE_ABLY_KEY` to your Ably API key.Also applies to: 60-60
examples/ai-transport-token-streaming/react/tsconfig.json (1)
12-12: Consider aligning moduleResolution with Vite best practices.The
moduleResolutionis set to"Node"here, whiletsconfig.node.jsonuses"bundler". For consistency and to align with modern Vite practices, consider using"bundler"in both configurations.🔎 Proposed change
- "moduleResolution": "Node", + "moduleResolution": "bundler",examples/ai-transport-token-streaming/javascript/src/config.ts (1)
1-3: Consider standardizing the fallback key across examples.The JavaScript example uses
'YOUR_ABLY_KEY_HERE'as the fallback, while the React example uses'demo-key-for-examples:YOUR_ABLY_KEY_HERE'(which follows Ably's key format pattern). For consistency in reference implementations, both examples should use the same placeholder format.🔎 Standardize with React example format
export const config = { - ABLY_KEY: import.meta.env.VITE_ABLY_KEY || 'YOUR_ABLY_KEY_HERE', + ABLY_KEY: import.meta.env.VITE_ABLY_KEY || 'demo-key-for-examples:YOUR_ABLY_KEY_HERE', };examples/ai-transport-token-streaming/javascript/src/BackendLLMService.ts (2)
10-21: Add disposal method for the Ably client.The Ably Realtime client is instantiated but never closed. For production use, consider adding a
dispose()method to properly close the connection and release resources.🔎 Proposed disposal method
+ dispose(): void { + // Cancel all active streams + this.activeStreams.forEach((timeouts) => { + timeouts.forEach((timeout) => clearTimeout(timeout)); + }); + this.activeStreams.clear(); + + // Close the Ably connection + this.client.close(); + }
81-86: Implement cancellation using the activeStreams map.The
activeStreamsmap stores timeouts but they are never cancelled. Consider adding acancelStream(responseId)method to clear pending timeouts, enabling proper cancellation of in-progress streams.🔎 Proposed cancellation method
+ cancelStream(responseId: string): void { + const timeouts = this.activeStreams.get(responseId); + if (timeouts) { + timeouts.forEach((timeout) => clearTimeout(timeout)); + this.activeStreams.delete(responseId); + } + }examples/ai-transport-token-streaming/react/src/BackendLLMService.ts (2)
10-21: Add disposal method for the Ably client.The Ably Realtime client is instantiated but never closed. For production use, consider adding a
dispose()method to properly close the connection and release resources.🔎 Proposed disposal method
+ dispose(): void { + // Cancel all active streams + this.activeStreams.forEach((timeouts) => { + timeouts.forEach((timeout) => clearTimeout(timeout)); + }); + this.activeStreams.clear(); + + // Close the Ably connection + this.client.close(); + }
81-86: Implement cancellation using the activeStreams map.The
activeStreamsmap stores timeouts but they are never cancelled. Consider adding acancelStream(responseId)method to clear pending timeouts, enabling proper cancellation of in-progress streams.🔎 Proposed cancellation method
+ cancelStream(responseId: string): void { + const timeouts = this.activeStreams.get(responseId); + if (timeouts) { + timeouts.forEach((timeout) => clearTimeout(timeout)); + this.activeStreams.delete(responseId); + } + }examples/ai-transport-token-streaming/react/src/App.tsx (3)
59-77: Consider providing user feedback on error.The error handling in
handlePromptClicksilently fails and only resets theisProcessingflag. For better user experience, consider displaying an error message in the UI whenrequestLLMProcessingfails.🔎 Suggested enhancement
+ const [error, setError] = useState<string | null>(null); const handlePromptClick = async (selectedPrompt: string) => { if (isProcessing || connectionState !== 'connected' || isChannelDetached) { return; } setIsProcessing(true); setMessages([]); setCurrentResponse(''); setPrompt(selectedPrompt); + setError(null); const responseId = `request-${crypto.randomUUID()}`; setCurrentResponseId(responseId); try { await requestLLMProcessing(selectedPrompt, responseId, config.ABLY_KEY, CHANNEL_NAME); } catch (error) { + console.error('Error requesting LLM processing:', error); + setError('Failed to process prompt. Please try again.'); setIsProcessing(false); } };Then display the error in the UI:
<div className="p-4 border border-gray-300 rounded-lg bg-gray-50 h-48 overflow-y-auto whitespace-pre-wrap text-base leading-relaxed"> + {error && <div className="text-red-500 mb-2">{error}</div>} {currentResponse || (isProcessing ? 'Thinking...' : 'Select a prompt below to get started')} {isProcessing && <span className="text-blue-600">▋</span>} </div>
84-117: Consider optimizing history rehydration for large message sets.The current implementation performs deduplication and sorting inside the state setter for each historical message (lines 99-107). For large histories, this could result in O(n²) complexity. Consider collecting all missed messages first, then deduplicating and sorting once before updating state.
🔎 Proposed optimization
const handleReconnect = async () => { setIsChannelDetached(false); await channel.attach(); // Fetch missed messages for current response if (currentResponseId) { + const missedMessages: ProcessedMessage[] = []; + let streamCompleted = false; let page = await channel.history({ untilAttach: true }); // Paginate backwards through history while (page) { for (const message of page.items) { const responseId = message.extras.headers.responseId; if (responseId === currentResponseId) { if (message.name === 'token') { const messageOrder = message.timestamp; - setMessages((prev) => { - // Only add if not already present - if (prev.find((m) => m.messageOrder === messageOrder)) { - return prev; - } - return [...prev, { token: message.data.token, messageOrder }].sort( - (a, b) => a.messageOrder - b.messageOrder, - ); - }); + missedMessages.push({ token: message.data.token, messageOrder }); } else if (message.name === 'stream-complete') { - setIsProcessing(false); + streamCompleted = true; } } } // Move to next page if available page = page.hasNext() ? await page.next() : null; } + + // Merge and deduplicate all messages at once + setMessages((prev) => { + const allMessages = [...prev, ...missedMessages]; + const uniqueMessages = allMessages.filter( + (msg, index, self) => self.findIndex((m) => m.messageOrder === msg.messageOrder) === index, + ); + return uniqueMessages.sort((a, b) => a.messageOrder - b.messageOrder); + }); + + if (streamCompleted) { + setIsProcessing(false); + } } };
29-46: Add defensive checks for message structure.Line 30 accesses
message.extras.headers.responseIdwithout checking ifextrasorheadersexist. While this works for messages published by yourBackendLLMService, it could throw if the channel receives unexpected message formats.🔎 Proposed defensive check
const { channel } = useChannel(CHANNEL_NAME, (message: RealtimeMessage) => { - const responseId = message.extras.headers.responseId; + const responseId = message.extras?.headers?.responseId; if (!currentResponseId || responseId !== currentResponseId) { return; // Ignore messages not for current response } if (message.name === 'token') { const newMessage: ProcessedMessage = { token: message.data.token, messageOrder: message.timestamp, }; setMessages((prev) => [...prev, newMessage]); } else if (message.name === 'stream-complete') { setIsProcessing(false); } });examples/ai-transport-token-streaming/javascript/src/script.ts (2)
185-205: Static analysis warning is a false positive, but consider optimization.The static analysis tool flagged line 186 for XSS risk, but this is a false positive—you're setting
innerHTMLto an empty string and the subsequent button text comes from a hardcoded array. However, recreating all buttons on every UI update is inefficient. Consider rendering buttons once on initialization and updating their disabled state instead.🔎 Proposed optimization
+// Render prompt buttons once on initialization +function renderPromptButtons() { + availablePrompts.forEach((promptText) => { + const button = document.createElement('button'); + button.textContent = promptText; + button.onclick = () => handlePromptClick(promptText); + button.dataset.prompt = promptText; + promptButtons.appendChild(button); + }); +} -// Update prompt buttons +// Update prompt button states function updatePromptButtons() { - promptButtons.innerHTML = ''; - - availablePrompts.forEach((promptText) => { - const button = document.createElement('button'); - button.textContent = promptText; - button.onclick = () => handlePromptClick(promptText); - - const disabled = isProcessing || connectionState !== 'connected' || isChannelDetached; - button.disabled = disabled; - - button.className = `px-3 py-2 text-sm border rounded-md transition-colors ${ - disabled - ? 'bg-gray-100 text-gray-400 cursor-not-allowed border-gray-200' - : 'bg-white hover:bg-blue-50 border-gray-300 hover:border-blue-300 cursor-pointer' - }`; - - promptButtons.appendChild(button); + const disabled = isProcessing || connectionState !== 'connected' || isChannelDetached; + const buttons = promptButtons.querySelectorAll('button'); + buttons.forEach((button) => { + button.disabled = disabled; + button.className = `px-3 py-2 text-sm border rounded-md transition-colors ${ + disabled + ? 'bg-gray-100 text-gray-400 cursor-not-allowed border-gray-200' + : 'bg-white hover:bg-blue-50 border-gray-300 hover:border-blue-300 cursor-pointer' + }`; }); } +// Render buttons on initialization +renderPromptButtons(); + // Initial UI update updateUI();
111-145: Consider optimizing history rehydration for large message sets.The current implementation performs deduplication and sorting for each historical message (lines 128-131). For large histories, this could result in O(n²) complexity. Consider collecting all missed messages first, then deduplicating and sorting once.
🔎 Proposed optimization
async function handleReconnect() { isChannelDetached = false; await channel.attach(); // Fetch missed messages for current response if (currentResponseId) { + const missedMessages: ProcessedMessage[] = []; + let streamCompleted = false; let page = await channel.history({ untilAttach: true }); // Paginate backwards through history while (page) { for (const message of page.items) { const responseId = message.extras.headers.responseId; if (responseId === currentResponseId) { if (message.name === 'token') { const messageOrder = message.timestamp; - // Only add if not already present - if (!messages.find((m) => m.messageOrder === messageOrder)) { - messages.push({ token: message.data.token, messageOrder }); - messages.sort((a, b) => a.messageOrder - b.messageOrder); - } + missedMessages.push({ token: message.data.token, messageOrder }); } else if (message.name === 'stream-complete') { - isProcessing = false; + streamCompleted = true; } } } // Move to next page if available page = page.hasNext() ? await page.next() : null; } + + // Merge and deduplicate all messages at once + const allMessages = [...messages, ...missedMessages]; + const uniqueMessages = allMessages.filter( + (msg, index, self) => self.findIndex((m) => m.messageOrder === msg.messageOrder) === index, + ); + messages = uniqueMessages.sort((a, b) => a.messageOrder - b.messageOrder); + + if (streamCompleted) { + isProcessing = false; + } updateCurrentResponse(); } updateUI(); }
📜 Review details
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Jira integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (23)
examples/ai-transport-token-streaming/javascript/README.mdexamples/ai-transport-token-streaming/javascript/index.htmlexamples/ai-transport-token-streaming/javascript/package.jsonexamples/ai-transport-token-streaming/javascript/src/BackendLLMService.tsexamples/ai-transport-token-streaming/javascript/src/config.tsexamples/ai-transport-token-streaming/javascript/src/script.tsexamples/ai-transport-token-streaming/javascript/src/styles.cssexamples/ai-transport-token-streaming/javascript/tailwind.config.tsexamples/ai-transport-token-streaming/javascript/vite.config.tsexamples/ai-transport-token-streaming/react/README.mdexamples/ai-transport-token-streaming/react/index.htmlexamples/ai-transport-token-streaming/react/package.jsonexamples/ai-transport-token-streaming/react/postcss.config.jsexamples/ai-transport-token-streaming/react/src/App.tsxexamples/ai-transport-token-streaming/react/src/BackendLLMService.tsexamples/ai-transport-token-streaming/react/src/config.tsexamples/ai-transport-token-streaming/react/src/index.tsxexamples/ai-transport-token-streaming/react/src/styles/styles.cssexamples/ai-transport-token-streaming/react/tailwind.config.tsexamples/ai-transport-token-streaming/react/tsconfig.jsonexamples/ai-transport-token-streaming/react/tsconfig.node.jsonexamples/ai-transport-token-streaming/react/vite.config.tssrc/data/examples/index.ts
🧰 Additional context used
🧬 Code graph analysis (3)
examples/ai-transport-token-streaming/react/src/App.tsx (2)
examples/ai-transport-token-streaming/react/src/config.ts (1)
config(1-3)examples/ai-transport-token-streaming/react/src/BackendLLMService.ts (1)
requestLLMProcessing(91-98)
examples/ai-transport-token-streaming/react/src/BackendLLMService.ts (1)
examples/pub-sub-message-annotations/javascript/src/config.ts (1)
channelName(5-5)
examples/ai-transport-token-streaming/javascript/src/BackendLLMService.ts (1)
examples/pub-sub-message-annotations/javascript/src/config.ts (1)
channelName(5-5)
🪛 ast-grep (0.40.3)
examples/ai-transport-token-streaming/javascript/src/script.ts
[warning] 186-186: Direct modification of innerHTML or outerHTML properties detected. Modifying these properties with unsanitized user input can lead to XSS vulnerabilities. Use safe alternatives or sanitize content first.
Context: promptButtons.innerHTML = ''
Note: [CWE-79] Improper Neutralization of Input During Web Page Generation [REFERENCES]
- https://owasp.org/www-community/xss-filter-evasion-cheatsheet
- https://cwe.mitre.org/data/definitions/79.html
(dom-content-modification)
🪛 LanguageTool
examples/ai-transport-token-streaming/javascript/README.md
[style] ~42-~42: This adverb was used twice in the sentence. Consider removing one of them or replacing them with a synonym.
Context: ...the value of VITE_ABLY_KEY to be your Ably API key. 5. Install dependencies: `...
(ADVERB_REPETITION_PREMIUM)
[style] ~60-~60: This adverb was used twice in the sentence. Consider removing one of them or replacing them with a synonym.
Context: ...ur VITE_ABLY_KEY variable to use your Ably API key.
(ADVERB_REPETITION_PREMIUM)
examples/ai-transport-token-streaming/react/README.md
[style] ~42-~42: This adverb was used twice in the sentence. Consider removing one of them or replacing them with a synonym.
Context: ...the value of VITE_ABLY_KEY to be your Ably API key. 5. Install dependencies: `...
(ADVERB_REPETITION_PREMIUM)
[style] ~60-~60: This adverb was used twice in the sentence. Consider removing one of them or replacing them with a synonym.
Context: ...ur VITE_ABLY_KEY variable to use your Ably API key.
(ADVERB_REPETITION_PREMIUM)
🔇 Additional comments (13)
examples/ai-transport-token-streaming/javascript/README.md (1)
1-60: Overall documentation structure is clear and complete.The README provides a well-organized introduction to AI Transport token streaming with clear setup instructions, resource links, and alternative deployment options. The step-by-step getting started guide is easy to follow and the terminology is consistent throughout.
examples/ai-transport-token-streaming/react/tsconfig.node.json (1)
1-10: LGTM!The TypeScript configuration for Vite tooling is correct and follows standard practices for Vite-based projects.
examples/ai-transport-token-streaming/react/index.html (1)
1-12: LGTM!The HTML entry point is correctly structured with proper meta tags and module script configuration for the React application.
examples/ai-transport-token-streaming/react/tsconfig.json (1)
1-11: LGTM!The TypeScript configuration is well-structured with appropriate strict mode settings, JSX configuration, and project references.
Also applies to: 13-20
examples/ai-transport-token-streaming/react/README.md (1)
1-60: LGTM!The README is comprehensive and well-structured, providing clear instructions for setup and usage of the AI Transport token streaming example.
src/data/examples/index.ts (2)
16-25: LGTM!The layout change for the chat-presence example from
double-horizontaltosingle-largelooks intentional and properly configured.
300-302: LGTM!The product key rename from
aitransporttoai_transportis consistent with the new example entry that referencesai_transportat line 10.examples/ai-transport-token-streaming/javascript/src/styles.css (1)
1-3: LGTM!Standard Tailwind CSS configuration is correct.
examples/ai-transport-token-streaming/react/src/styles/styles.css (1)
1-3: LGTM!Standard Tailwind CSS configuration is correct.
examples/ai-transport-token-streaming/react/src/index.tsx (1)
1-9: LGTM!The React entry point follows best practices with StrictMode enabled and correct usage of React 18's
createRootAPI.examples/ai-transport-token-streaming/javascript/vite.config.ts (1)
1-7: LGTM: Consistent configuration pattern.The JavaScript example follows the same Vite configuration pattern as the React example, extending the shared base configuration. This consistency across examples is good for maintainability.
examples/ai-transport-token-streaming/react/src/config.ts (1)
1-3: Better fallback format than JavaScript example.This configuration uses a fallback key format (
'demo-key-for-examples:YOUR_ABLY_KEY_HERE') that follows Ably's key pattern, making it clearer to users what format is expected. This is preferable to the plain placeholder in the JavaScript example.examples/ai-transport-token-streaming/javascript/package.json (1)
10-19: Dependency versions are current and valid for example code.All specified versions exist and have no reported security vulnerabilities. The caret ranges allow automatic updates to newer patch and minor versions within their respective major versions (e.g.,
ably@^2.4.0will use 2.16.0 when installed). While major version upgrades are available for vite (7.3.0) and tailwindcss (4.1.18), these would require code changes and testing. The current pinned versions are safe and appropriate for reference implementation code.
examples/ai-transport-token-streaming/javascript/src/BackendLLMService.ts
Show resolved
Hide resolved
examples/ai-transport-token-streaming/javascript/src/BackendLLMService.ts
Show resolved
Hide resolved
examples/ai-transport-token-streaming/javascript/src/BackendLLMService.ts
Show resolved
Hide resolved
matt423
force-pushed
the
ait-148-message-per-token-examples
branch
from
2ee184a to
b157c70
Compare
matt423
changed the title
matt423
added
review-app
mschristensen
left a comment
mschristensen
left a comment
There was a problem hiding this comment.
Thanks Matt - I have added some comments, mostly on the JS example but same feedback also applies to the React example. Thanks
| description: 'Show online/offline status of users in a chat application.', | ||
| products: ['chat'], | ||
| layout: 'double-horizontal', | ||
| layout: 'single-large', |
There was a problem hiding this comment.
Was this change intentional?
|
|
||
| export const examples: Example[] = [ | ||
| { | ||
| id: 'ai-transport-token-streaming', |
There was a problem hiding this comment.
Can we name this, and the name, description, <title> etc, to indicate that this example is for the message per token pattern (as we will add a similar example for message-per-response pattern separately)
| @@ -0,0 +1,60 @@ | |||
| # AI Transport token streaming | |||
|
|
|||
| Enable real-time streaming of AI/LLM responses by publishing tokens as they arrive from Large Language Model services. | |||
There was a problem hiding this comment.
| Enable real-time streaming of AI/LLM responses by publishing tokens as they arrive from Large Language Model services. | |
| Enable realtime streaming of AI/LLM responses by publishing tokens as they arrive from Large Language Model services. |
|
|
||
| Enable real-time streaming of AI/LLM responses by publishing tokens as they arrive from Large Language Model services. | ||
|
|
||
| AI Transport token streaming allows applications to provide immediate, responsive AI interactions by streaming tokens in real-time rather than waiting for complete responses. This pattern is essential for creating engaging AI-powered experiences where users can see responses being generated as they happen. |
There was a problem hiding this comment.
| AI Transport token streaming allows applications to provide immediate, responsive AI interactions by streaming tokens in real-time rather than waiting for complete responses. This pattern is essential for creating engaging AI-powered experiences where users can see responses being generated as they happen. | |
| AI Transport token streaming allows applications to provide immediate, responsive AI interactions by streaming tokens in realtime rather than waiting for complete responses. This pattern is essential for creating engaging AI-powered experiences where users can see responses being generated as they happen. |
| The streaming approach significantly improves perceived performance and user engagement. Instead of waiting 5-10 seconds for a complete AI response, users see tokens appearing progressively, creating a more natural conversation flow similar to watching someone type in real-time. | ||
|
|
There was a problem hiding this comment.
| The streaming approach significantly improves perceived performance and user engagement. Instead of waiting 5-10 seconds for a complete AI response, users see tokens appearing progressively, creating a more natural conversation flow similar to watching someone type in real-time. | |
| The streaming approach significantly improves perceived performance and user engagement. Instead of waiting 5-10 seconds for a complete AI response, users see tokens appearing progressively, creating a more natural conversation flow similar to watching someone type in realtime. |
| timeouts.push(timeout); | ||
| } else { | ||
| this.channel.publish({ | ||
| name: 'stream-complete', |
There was a problem hiding this comment.
In the message per token docs, we use the event name stop, can we align this approach with the pattern used there? Therefore also use a start event.
| @@ -0,0 +1,98 @@ | |||
| // Backend LLM Service | |||
| // This simulates an independent LLM service that publishes to Ably | |||
There was a problem hiding this comment.
I think it would be clearer to the developer if the abstractions we introduce in the example clearly describe the different components of their backend implementation. So, we can have a specific abstraction for the mock LLM event stream, which is not concerned with publishing those events to the channel. Additionally, I think this abstraction should align closely with the illustrative examples provided in the message per token streaming docs. For example:
// llm.ts
interface StreamEvent {
type: 'message_start' | 'message_delta' | 'message_stop';
text?: string;
responseId: string;
}
export class MockLLM {
responses = {
create: async () => this.createStream(),
};
private async *createStream(): AsyncIterable<StreamEvent> {
const responseId = `resp_${crypto.randomUUID()}`;
yield { type: 'message_start', responseId };
const response =
'Ably AI Transport is a solution for building stateful, steerable, multi-device AI experiences into new or existing applications. You can use AI Transport as the transport layer with any LLM or agent framework, without rebuilding your existing stack or being locked to a particular vendor.';
// Chunk text into tokens
for (let i = 0; i < response.length; i += Math.floor(Math.random() * 8) + 1) {
const token = response.slice(i, i + Math.floor(Math.random() * 8) + 1);
await new Promise((resolve) => setTimeout(resolve, Math.random() * 150 + 50));
yield { type: 'message_delta', text: token, responseId };
}
yield { type: 'message_stop', responseId };
}
}i.e. we implement a "mock" LLM SDK that exposes an async iterable that yields tokens in a way that is similar to the OpenAI SDK API:
For example, in OpenAI we have:
const stream = await openai.responses.create({
model: "gpt-5",
input: prompt,
stream: true,
});
// Iterate through streaming events
for await (const event of stream) {
processEvent(event);
}
We can then have a simple "agent" component which consumes from the stream and publishes to Ably, directly implementing the pattern described in the message per token docs:
// agent.ts
export async function processPrompt() {
const llm = new MockLLM();
const stream = await llm.responses.create();
for await (const event of stream) {
if (event.type === 'message_start') {
// Publish response start
channel.publish({
name: 'start',
extras: {
headers: {
responseId: event.responseId,
},
},
});
} else if (event.type === 'message_delta') {
// Publish tokens
channel.publish({
name: 'token',
data: {
token: event.text,
},
extras: {
headers: {
responseId: event.responseId,
},
},
});
} else if (event.type === 'message_stop') {
// Publish response stop
channel.publish({
name: 'stop',
extras: {
headers: {
responseId: event.responseId,
},
},
});
}
}
}I think this approach would:
- Map more directly to the message-per-token docs which we are aiming to illustrate
- Provide a clear separation of concerns, such that the reader can easily understand what represents mock llm code vs agent code
- Generally simplify the code and reduce boilerplate, making the example more digestible
| // Only add if not already present | ||
| if (!messages.find((m) => m.messageOrder === messageOrder)) { | ||
| messages.push({ token: message.data.token, messageOrder }); | ||
| messages.sort((a, b) => a.messageOrder - b.messageOrder); | ||
| } |
There was a problem hiding this comment.
We shouldn't have to do this - part of the value proposition is guaranteed order and exactly once delivery. Why is this needed?
|
|
||
| interface ProcessedMessage { | ||
| token: string; | ||
| messageOrder: number; |
There was a problem hiding this comment.
We shouldn't need to store an ordering key, can we construct the message string as events arrive, as suggested in the message per token docs? src/pages/docs/ai-transport/features/token-streaming/message-per-token.mdx "Using history for older messages"
| const promptButtons = document.getElementById('prompt-buttons'); | ||
|
|
||
| // Available prompts | ||
| const availablePrompts = ['What is Ably AI Transport?']; |
There was a problem hiding this comment.
Can we simplify by hardcoding support for single prompt? We should simplify this script as much as possible so the code remains focused on the relevant logic
Description
AIT-209
Adds a JS and React example for message per token response streaming. Example mimics a basic
prompt -> answerscenario with aDisconnectbutton to simulate network loss and token recovery using history anduntilAttachReview App
Note: No image for the examples page yet
Checklist
Summary by CodeRabbit
New Features
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.