Overview/ait 189 intro token

[GregHolmes]

Description

Adds overview pages to the documentation covering

the overall AIT product, listing the major features and linking them to other documentation
token streaming, including an overview of the proposed architecture and patterns

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[GregHolmes]

I think this looks good! (I can't approve or anything as I raised it)

@rainbowFi I've left some comments on my thoughts.

I also think we need to be careful and remember that if some of this (such as the full list of agents/frameworks) isn't available on release, we need to remove the TODO comments.

[GregHolmes]

Should these not be "Advanced messaging" and "User input" ? (Maybe user input isn't a helpful title).
But they're the sections defined in the AIT Docs IA Miro.

[GregHolmes]

I'm guessing if this is meant to be where Advanced messaging is, the link would be /docs/ai-transport/features/advanced-messaging Yet to be created though.

[GregHolmes]

We call it capabilities elsewhere in the docs, should we keep with capabilities instead of RBAC?

[GregHolmes]

I'm not sure this paragraph is necessarily correct. We're focusing specifically on streaming per token in this paragraph. But the more preferred way is also valid, streaming per response? Should we talk about that too?

Also, probably an AI addition but it's realtime :D

[rainbowFi]

I am talking about the general definition of token streaming here, rather than anything to do with our recommendations of how to token stream over Ably (which comes later)

[paddybyers]

Suggested change

If an HTTP stream is interrupted, for example because the client loses network connection, then any tokens that were transmitted during the interruption will be lost. Ably AI Transport solves this problem by streaming tokens to a [Pub/Sub channel](docs/channels), which is not tied to the connection state of either the client or the agent. A client that [reconnects](/docs/connect/states#connection-state-recovery) can receive any tokens transmitted while it was disconnected. If a new client connects, for example because the user has moved to a different device, then it is possible to hydrate the new client with all the tokens transmitted for the current request as well as the output from any previous requests. The exact mechanism for doing this will depend on which [token streaming pattern](#patterns) you choose to use.

If an HTTP stream is interrupted, for example because the client loses network connection, then any tokens that were transmitted during the interruption will be lost. Ably AI Transport solves this problem by streaming tokens to a [Pub/Sub channel](docs/channels), which is not tied to the connection state of either the client or the agent. A client that [reconnects](/docs/connect/states#connection-state-recovery) can receive any tokens transmitted while it was disconnected. If a new client connects, for example because the user has moved to a different device, then it is possible to hydrate the new client with all the tokens transmitted for the current request as well as the output from any previous requests. in detail the mechanism for doing this will depend on which [token streaming pattern](#patterns) you choose to use.

[paddybyers]

The other possible reason for using message-per-token is where you want the Ably transport to preserve the specific breakdown of the response into separate fragments. This might be because some higher-level framework is dependent on knowing that breakdown, or is handling token concatenation in some way that is incompatible with Ably performing concatenation of fragments.

[mschristensen]

Thanks for this - I left a few comments. Taking a step back, given this is such a key piece of the offering, I feel that we can do more to describe the value proposition for token streaming over Ably. Are there ways we can explicitly enumerate the key parts of the user experience that constitute a great token streaming experience? We could then contrast those with the complexities of achievng this in a connection-oriented HTTP streaming model, and how Ably solves this out of the box.

I think there is some overlap conceptually with the Sessions & identity overview, but I think it would be okay to repeat some of that here, with a token-streaming rather than session emphasis.

Let's discuss in our catch up tomorrow :)

[mschristensen]

In general, we prefer to use single word "realtime" at Ably.
(This is not what most of the internet seems to do, but alas this is our convention)

[mschristensen]

"This is normally accomplished by streaming the tokens as the response to an HTTP request from the client."

I think this can be moved out into a new paragraph. I think the intro paragraph should focus on the description of what token streaming is before getting into how it is implemented.

Then, I would suggest colocating this statement with the content that follows after the image, since that paragraph starts by describing the weakness of this approach.

[mschristensen]

This is a bit of a wall of text, but there are some nice bits of value prop in there. Can we pull those out, perhaps into bullets?