llama-omni微调相关问题

[sixbenzene]

1，是只能微调LLM吗，我使用模型只能使用文字对话，不能传输语音数据。意思是只能使用LLM，没有speech encoder,speech adaptor?现阶段是否有论文上的stag1的微调，请告知，谢谢。

[Jintao-Huang]

现在支持llm, vit, projector部分

speech_generator不支持训练

[sixbenzene]

现在支持llm, vit, projector部分

speech_generator不支持训练
谢谢！请问默认的数据集有能直接使用的吗，或者该构建怎样的数据集，是[语音指令，文本响应]这样的数据集吗？

[Jintao-Huang]

参考这里：#1653

[sixbenzene]

参考这里：#1653

谢谢！

[learn01one]

hello，没大理解，speech adaptor是支持训练微调的还是不支持呢？

[Jintao-Huang]

支持的, 就是 generator那里是不微调的. 因为loss函数和ms-swift的数据集格式不太兼容

不过可以自己改改源代码来支持

[fromlimbo]

参考这里：#1653

你好，参考的文档里还是有点疑惑，想问下aishell1-zh-mini这个数据集下载下来是可以直接运行的呢(文档里说是可直接运行的demo)，还是说也要自己把它做成下面的格式，可以给一个微调的数据sample看看吗，感谢
[ {"conversations": [ {"from": "user", "value": "<audio>audio_path</audio>11111"}, {"from": "assistant", "value": "22222"} ]}, {"conversations": [ {"from": "user", "value": "<audio>audio_path</audio><audio>audio_path2</audio><audio>audio_path3</audio>aaaaa"}, {"from": "assistant", "value": "bbbbb"}, {"from": "user", "value": "<audio>audio_path</audio>ccccc"}, {"from": "assistant", "value": "ddddd"} ]}, {"conversations": [ {"from": "user", "value": "AAAAA"}, {"from": "assistant", "value": "BBBBB"}, {"from": "user", "value": "CCCCC"}, {"from": "assistant", "value": "DDDDD"} ]} ]

[Shookwu]

@Jintao-Huang 请问可以支持只训练speech adaptor吗，如果要更换自己的模型是需要修改源代码吗，另外infer好像支持文本输入？谢谢🙏

[satheeshkola-532]

支持的, 就是 generator那里是不微调的. 因为loss函数和ms-swift的数据集格式不太兼容

不过可以自己改改源代码来支持

@Jintao-Huang you said that "the generator is not fine-tuned. Because the loss function is not compatible with the ms-swift data set format. However, you can modify the source code yourself to support it." , so could you please modify the Llama omni pipeline code to enable the speech generator fine tuning also..., if you're okay with that we're also willing to collaborate to contribute to this(Then this will be the first end to end speech - speech language model pipeline..)

[fromlimbo]

各位加个讨论群，来看看一起搞个全量版本的微调方案吧

[learn01one]

你好，请问下，lora微调是如何指定微调llm,vit, projector的一部分，如只想微调llm，或者只微调projector，如何设置呢

[enbiwudi]

各位加个讨论群，来看看一起搞个全量版本的微调方案吧

老哥能拉一下吗
enbiwudi123@gmail.com

[satheeshkola-532]

Let's join the discussion group and see how to make a fine-tuning plan for the full version together.

can you add me in the group: satheeshkola532@gmail.com

[UNO-TTS]

参考这里：#1653

@Jintao-Huang
你好，按照qwen-auido的格式构建数据集后，出现了如下bug：
[QUERY]/home3/nlp/llamo/dataset/speech/chunk_00000/0017.mp3
[OUTPUT]Traceback (most recent call last):
File "/home3/tools/anaconda3/envs/py39/lib/python3.9/site-packages/swift/cli/infer.py", line 5, in
infer_main()
File "/home3/tools/anaconda3/envs/py39/lib/python3.9/site-packages/swift/utils/run_utils.py", line 32, in x_main
result = llm_x(args, **kwargs)
File "/home3/tools/anaconda3/envs/py39/lib/python3.9/site-packages/swift/llm/infer.py", line 558, in llm_infer
response, _ = inference(
File "/home3/tools/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home3/tools/anaconda3/envs/py39/lib/python3.9/site-packages/swift/llm/utils/utils.py", line 869, in inference
generate_ids = model.generate(streamer=streamer, generation_config=generation_config, **inputs)
File "/home3/tools/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home3/nlp/llamo/LLaMA-Omni/omni_speech/model/speech_generator/generation.py", line 62, in generate
self._validate_assistant(assistant_model)
TypeError: _validate_assistant() missing 2 required positional arguments: 'tokenizer' and 'assistant_tokenizer'

请问是哪里没有传入tokenizer吗？