[Feature] In Sglang ，Is chunked-prefill use fused(prefill+decode) batch?

[CSEEduanyu]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

https://arxiv.org/abs/2308.16369 Based on this article, I got the following hypothesis：
1.no prefill batch or decode batch anymore，only fused batch (contains the prefill and decode)
2.The ideal chunk size depends on the current hardware and the computational intensity calculated by the model，on A100+llama13B this num is about 512？

Related resources

No response

[CSEEduanyu]

I can't figure out where the prefill and decode are combined into one batch, and chunked-prefill still seems to perform the prefill and decode separately when enabled. Also, the default chunked-prefill-size=8192, which seems to be a very unreasonable value, and I'm confused about it.Maybe there's something wrong with my perception?

[merrymercy]

By default, it does not mix prefill and decode. However, you can turn on this flag to mix/fuse them

sglang/python/sglang/srt/server_args.py

Lines 418 to 422 in 30b4f77

	parser.add_argument(
	"--enable-mixed-chunk",
	action="store_true",
	help="Enabling mixing prefill and decode in a chunked batch.",
	)

.
It can help reduce the inter-token latency as described in that paper.

We pick chunk size 8192 to favor throughput.

[CSEEduanyu]

Thanks for the reply, I've seen this change in the latest version!

[Desmond819]

does it increase the throughput?

[merrymercy]

In our test, it can reduce inter-token latency for some workloads. It does not increase the peak throughput.

[KylinMountain]

Is there any document described the chunked-prefill-size option? What is this for? Hard to find a document described this.