Support for returning Logits and Calculating Perplexity During Model Evaluation?

[hxer7963]

Hello SGLang Community,

I'm currently exploring the capabilities of the SGLang inference framework for LLMs and I have a couple of questions regarding model evaluation:

• Returning Logits: Does SGLang support returning the logits (instead of return_logprob) during inference? This is often useful for applications requiring a detailed understanding of model predictions on a token level.

• Calculating Perplexity (PPL) During Evaluation: Is there built-in support or recommended practices for calculating the perplexity of a model during evaluation? Perplexity is a crucial metric for assessing the quality of language models, and having an efficient way to calculate it would be valuable.

If these features are not currently supported, are there any plans to include them in future updates?

Any guidance or suggestions on how to implement these functionalities using the current framework would also be greatly appreciated.

Thank you!

Best regards, willhe.

[merrymercy]

They are well supported. Some related docs:

• Return logprob for a generation request

  sglang/docs/en/sampling_params.md

  Lines 23 to 28 in 5ab9418

  	# Whether to return logprobs.
  	return_logprob: Optional[Union[List[bool], bool]] = None
  	# The start location of the prompt for return_logprob.
  	logprob_start_len: Optional[Union[List[int], int]] = None
  	# The number of top logprobs to return.
  	top_logprobs_num: Optional[Union[List[int], int]] = None

• The full OpenAI API spec around logprob is supported.
  

  sglang/test/srt/test_openai_server.py

  Line 72 in 5ab9418

  if logprobs:

• Other ways to use it in the frontend language.
  https://github.com/sgl-project/sglang/blob/main/docs/en/choices_methods.md