Importance-aware Sparse Tuning

The Official PyTorch implementation of Layer-wise Importance Matters: Less Memory for Better Performance in PEFT of LLMs [EMNLP 2024 Findings]. Kai Yao, Penglei Gao, Lichun Li, Yuan Zhao, Xiaofeng Wang, Wei Wang Jianke Zhu

Overview

We introduce a plug-and-play Importance-aware Sparse Tuning (IST) with various PEFT methods that operate on a per-layer basis. Our core idea is to dynamically update the most important layers while keep remained layers unchanged during PEFT training. Extensive experiments on a range of LLMs, PEFTs, and downstream tasks substantiate the effectiveness of our proposed method, showcasing IST's capacity to enhance existing layer-based PEFT methods with less memory cost.

Finetuning LLaMA on commonsense reasoning tasks using IST

This directory includes the IST implementation and guidelines for reproducing the results in our paper.

Setup

1. Install dependencies

conda create -n llm_ist python=3.10
conda activate llm_ist
pip install -r requirements.txt

Datasets

1. Download the complete commonsense datasets from here and download the commonsense 170k finetuning dataset from here, then organize the data as follows

# Store the complete commonsense datasets
./dataset
# rest of the files
./experiment
./peft
# Finetuning commonsense dataset
./commonsense_170k.json
...

Code Structure

Refer to ./ist for the implementation of Importance-aware Sparse Tuning (IST).

Refer to ./rst for the implementation of Random Sparse Tuning (RST).

Refer to ./finetune.py for finetuning LLaMA using DoRA.

Refer to ./commonsense_evaluate.py for the evaluation of the finetuned model.

Finetuning and Evaluation

Finetuning and Evaluating LLaMA-7B with IST & RST

This file contains the code to finetune LLaMA-7B using IST or RST. User can specify different LoRA configuration for finetuning. To be specific, the first argument denotes the rank r, the second argument specifies the corresponding alpha, the third argument indicates the destination for saving the fine-tuned model, and the last argument determines the GPU to use. An example could be:

bash llama_7B_LoRA_IST.sh 32 64 ./finetuned_result/r32_lr2e-4 0
bash llama_7B_LoRA_IST.sh 32 64 ./finetuned_result/r32_lr1e-4 0

Acknowledgement

We greatly appreciate the contributions of two remarkable repositories: LLM-Adapter, PEFT. These projects have significantly benefited our work.

🚩Citation

If this work is helpful, please kindly cite as:

@inproceedings{yao-etal-2024-layer,
    title = "Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models",
    author = "Yao, Kai  and
      Gao, Penglei  and
      Li, Lichun  and
      Zhao, Yuan  and
      Wang, Xiaofeng  and
      Wang, Wei  and
      Zhu, Jianke",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.109",
    pages = "1977--1992",
}