task: Instruct Dataset Creation for Multilingual Speech (Phase 2)

[hahuyhoang411]

Goal

Create a speech instruction finetuning to make Ichigo better in conversation.

Tasklist

• Gathering Vietnamese + English text instruction dataset.
• Clean prompt in instruction dataset (e.g using distillabel)
• Optimizing the pipeline
• Draft research report

[bachvudinh]

Base on my experience, I have concerns about the reliability of Text2Semantic. When I modified the T2S model parameters to stabilize the semantic tokens, it significantly increased the pipeline's processing time compared to the standard Text2Speech + Speech2Semantic pipeline without saving the audio. Therefore, I recommend we proceed with the T2S+ S2S pipeline approach. cc @tuanlda78202

[tuanlda78202]

We can use viXTTS for speech synthesis, that's so good!

[bachvudinh]

Gather all vietnamese instruction data source here:

Data Source	Number of Samples	Note
Viettel x Nvidia dataset	4.5M	instruct data with 55.9% CoT data, 25.7% QnA data and other.
Sailor2 dataset stage 1	TBD	TBD
Sailor2 dataset stage 2	TBD	TBD
Sailor2 dataset preference	TBD	TBD