Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/source/en/model_doc/gemma4_unified_assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,17 @@ limitations under the License.

## Overview

Gemma 4 Unified Assistant is a small, text-only model that enables speculative decoding with for Gemma 4 Unified models using the
Gemma 4 Unified Assistant is a small, text-only model that enables speculative decoding for Gemma 4 Unified models using the
Multi-Token Prediction (MTP) method and associated candidate generator. Pre-trained models are provided for the IT
variants of the Gemma 4 12B model.

For more information, please see [Gemma4 Assistant](./gemma4_assistant.md). Architecturally and conceptually, they
share the same concept and differences to their base model:

* **The entire model uses KV sharing**. This technique, originally introduced with [Gemma 3n](./gemma3n), allows the
model to resuse the KV cache populated by the target model the assistant supports, allowing the assistant to skip
the pre-fille phase entirely, and considerably reducing attention compute during the forward pass.
* **The `position_ids` value are constant**. Since the KV cache is shared and the assistant does not have a mean of
model to reuse the KV cache populated by the target model the assistant supports, allowing the assistant to skip
the pre-fill phase entirely, and considerably reducing attention compute during the forward pass.
* **The `position_ids` value are constant**. Since the KV cache is shared and the assistant does not have a means of
updating the cache, the assistant predicts all tokens from the same position ID.
* **Inputs are the concatenation of embeddings and hidden states**. To adapt for the static KV cache and
`position_ids`, the model takes its inputs as the concatenation of the `embedding` and `hidden_states` for the last
Expand All @@ -43,7 +43,7 @@ share the same concept and differences to their base model:
token will be the last token generated by the assistant (within a drafting round) or the last token accepted by the
target model (between drafting rounds).
* **Cross-attention is used to make the most of the target model's context**. Cross-attention allows the query states
geneated by the assistant to attend to the shared KV cache values from the target model, allowing the assistant to
generated by the assistant to attend to the shared KV cache values from the target model, allowing the assistant to
accurately predict more drafted tokens per drafting round.

## Usage examples
Expand Down
Loading