Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated documentation for Grammar Backend #2545

Merged
merged 9 commits into from
Dec 23, 2024
Merged

Updated documentation for Grammar Backend #2545

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
aa17e0a
updated constrained decoding doc
shuaills Dec 21, 2024
88db12e
rm ipynb outputs
shuaills Dec 21, 2024
48d58aa
updated docs
shuaills Dec 22, 2024
f26eb4f
updated docs
shuaills Dec 22, 2024
0eaf401
remove ipynb outputs
shuaills Dec 22, 2024
ef9c4ac
Merge branch 'main' into docs/structured-decoding-updates
zhaochenyang20 Dec 22, 2024
f9d7349
Update doc
shuaills Dec 22, 2024
a00ecb1
update doc
shuaills Dec 22, 2024
449d810
Merge branch 'main' into docs/structured-decoding-updates
zhaochenyang20 Dec 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docs/backend/openai_api_completions.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,14 @@
"metadata": {},
"source": [
"## Structured decoding (JSON, Regex)\n",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can define a JSON schema or regular expression to constrain the model's output, which depends on the grammar backend.

SGlang has two backends: outlines (default) and Xgrammar. Xgrammar enhances JSON decoding performance but does not support regular expressions. To use Xgrammar, add --grammar-backend xgrammar when launching the server:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and verified the functionality with Xgrammar backend.

"You can specify a JSON schema or a regular expression to constrain the model output. The model output will be guaranteed to follow the given constraints.\n",
"You can define a JSON schema or regular expression to constrain the model's output. The model output will be guaranteed to follow the given constraints and this depends on the grammar backend.\n",
"\n",
"SGlang has two backends: outlines (default) and Xgrammar. Xgrammar enhances JSON decoding performance but does not support regular expressions. To use Xgrammar, add the `--grammar-backend xgrammar` when launching the server:\n",
"\n",
"```bash\n",
"python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
"--port 30000 --host 0.0.0.0 --grammar-backend xgrammar\n",
"```\n",
"\n",
"### JSON"
]
Expand Down
Loading