[Feature] Support new parameter - EBNF in xgrammar

[adarshxs]

Motivation

XGrammar enables efficient structured generation. XGrammar backend already support enforcement of EBNF (Extended Backus-Naur Form) grammars.
adds feature requested in: #2376

Modifications

• added a new parameter called ebnf in sampling_params.py and treat it similar to regex and JSON.
• Implemented the test_ebnf_generate method to validate multiple valid output patterns based on a complex EBNF grammar.

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

TODO: update docs

[zhyncs]

cc @Ubospica @DarkSharpness

[merrymercy]

Great work! Can you also support it with the OpenAI API?

sglang/test/srt/test_openai_server.py

Lines 470 to 480 in 19ba2b0

	response = client.chat.completions.create(
	model=self.model,
	messages=[
	{"role": "system", "content": "You are a helpful AI assistant"},
	{"role": "user", "content": "Introduce the capital of France."},
	],
	temperature=0,
	max_tokens=128,
	extra_body={"regex": regex},
	)
	text = response.choices[0].message.content

[adarshxs]

@merrymercy done

[Ubospica]

It overall looks great for me. Thanks!

[merrymercy]

Thanks for the contribution! It is merged @adarshxs

[adarshxs]

Thank you!