How to combine Timestamp, Hotwords, NN-LM shallow fusion in streaming Paraformer?

[duj12]

Now streaming paraformer supports VAD, Punctuation in pipeline(and python websocket runtime), but it seems Timestamp/Hotwords/NN-LM is not combined yet?
How can I add Timestamp Prediction, Contextual Biasing and NN-LM Shallow Fusion in streaming Paraformer?

[LauraGPT]

Short answer: the current Python ParaformerStreaming path cannot combine timestamp prediction, contextual hotwords, and NN-LM shallow fusion by passing inference kwargs. For a supported streaming deployment, use the C++/ONNX 2-pass runtime: return low-latency online partials, then apply timestamps, hotword biasing, and N-gram/WFST decoding to the sentence-final offline result.

Current capability boundary

Path	Streaming partials	Timestamp	Hotword biasing	LM decoding
Python paraformer-zh-streaming	Yes	No	No	No integrated LM scorer
C++/ONNX funasr-wss-server-2pass	2pass-online text	On 2pass-offline, with a timestamp AM	On 2pass-offline; server/client WFST hotwords or a contextual AM	N-gram WFST through --lm-dir

This distinction matters because unknown inference kwargs are accepted by the Python API but are not evidence that the feature is implemented. The streaming inference method currently constructs only {"key": ..., "text": ...} results (source). It can enter beam search when lm_weight and lm_file are set (source), but the shared Paraformer beam-search initializer does not add an LM scorer and explicitly leaves N-gram scoring unsupported there (source).

I also verified this with a real CPU run using FunASR 1.3.14, Torch 2.11.0, paraformer-zh-streaming, the bundled 5.547-second sample, and chunk_size=[0, 10, 5]:

baseline transcript:                   欢迎大家来体验达摩院推出的语音识别模型
hotword + pred_timestamp transcript:  欢迎大家来体验达摩院推出的语音识别模型
result keys for every chunk:          ["key", "text"]
beam-search scorers after lm_weight:  []

So hotword=... and pred_timestamp=True are silently ineffective in this Python streaming path, and lm_weight is not NN-LM shallow fusion.

Supported deployment

Use the official 2-pass server and configure the offline branch with the model/features you need:

bash run_server_2pass.sh \
  --model-dir damo/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-onnx \
  --online-model-dir damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-online-onnx \
  --vad-dir damo/speech_fsmn_vad_zh-cn-16k-common-onnx \
  --punc-dir damo/punc_ct-transformer_zh-cn-common-vad_realtime-vocab272727-onnx \
  --lm-dir damo/speech_ngram_lm_zh-cn-ai-wesp-fst \
  --hotword ../../hotwords.txt

The runtime guide documents the same server options and model choices (server example, model and hotword options). It specifically describes N-gram LM decoding and WFST hotwords for 2pass-offline, not NN-LM fusion (changelog).

Connect with mode="2pass". The client may send per-request hotwords in the initial WebSocket JSON, and the server labels low-latency output as 2pass-online and the corrected sentence-final output as 2pass-offline (protocol). Timestamp fields are returned only when the corresponding acoustic model supports timestamps.

If a neural LM is mandatory, there is no current FunASR flag for it in streaming Paraformer. It requires a custom stateful LM scorer integrated into beam search, or sentence-final N-best rescoring outside the streaming decoder. The latter preserves streaming partials but only changes the final result.