Mamba-Shedder: Question about Alpaca prompt consistency between fine-tuning and inference

[gmx-hub]

I’m very interested in your work. However, I have one question I would like to ask. I noticed that Alpaca-style prompts were used during fine-tuning, but the same format was not applied during inference. Is this approach reasonable? Could this discrepancy potentially lead to suboptimal fine-tuning performance?

[Yuan0320]

Hi @gmx-hub, thanks for your interest in our work and for raising this question regarding the consistency of Alpaca-style prompts between fine-tuning and inference.

Indeed, using Alpaca-style prompts consistently could potentially yield better results. Our primary objective was to utilize a dataset to effectively recovery the training-free pruned model, similar to approaches used in other works (such as LLM-Pruner). Alpaca was one of the choices for this purpose; for instance, P2Law employs SlimPajama dataset for post-training.

Additionally, our focus in Mamba-Shedder was to study the sensitivity of the components of Mamba-based models. There might be additional improvements that can be achieved during the accuracy recovery stage.

I hope this clarifies your query. Thanks.

[gmx-hub]

Thanks for your explanation.