v0.2.1: Variant Models, NEFTune Trick

New features

• Support NEFTune trick for supervised fine-tuning by @anvie in #1252
• Support loading dataset in the sharegpt format - read data/readme for details
• Support generating multiple responses in demo API via the n parameter
• Support caching the pre-processed dataset files via the cache_path argument
• Better LLaMA Board (pagination, controls, etc.)
• Support push_to_hub argument #1088

New models

• Base models
  • ChatGLM3-6B-Base
  • Yi (6B/34B)
  • Mistral-7B
  • BlueLM-7B-Base
  • Skywork-13B-Base
  • XVERSE-65B
  • Falcon-180B
  • Deepseek-Coder-Base (1.3B/6.7B/33B)
• Instruct/Chat models
  • ChatGLM3-6B
  • Mistral-7B-Instruct
  • BlueLM-7B-Chat
  • Zephyr-7B
  • OpenChat-3.5
  • Yayi (7B/13B)
  • Deepseek-Coder-Instruct (1.3B/6.7B/33B)

New datasets

• Pre-training datasets
  • RedPajama V2
  • Pile
• Supervised fine-tuning datasets
  • OpenPlatypus
  • ShareGPT Hyperfiltered
  • ShareGPT4
  • UltraChat 200k
  • AgentInstruct
  • LMSYS Chat 1M
  • Evol Instruct V2

Bug fix

• Fix full-parameter DPO training #1383 #1422 (inspired by @mengban )
• Fix tokenizer config by @lvzii in #1436
• Fix #1197 #1215 #1217 #1218 #1228 #1232 #1285 #1287 #1290 #1316 #1325 #1349 #1356 #1365 #1411 #1418 #1438 #1439 #1446

v0.2.0: Web UI Refactor, LongLoRA

New features

• Support LongLoRA for the LLaMA models
• Support training the Qwen-14B and InternLM-20B models
• Support training state recovery for the all-in-one Web UI
• Support Ascend NPU by @statelesshz in #975
• Integrate MMLU, C-Eval and CMMLU benchmarks

Modifications

• Rename repository to LLaMA Factory (former LLaMA Efficient Tuning)
• Use the cutoff_len argument instead of max_source_length and max_target_length #944
• Add a train_on_prompt option #1184

Bug fix

• Fix numeric error caused by the layer norm dtype in 84b7486 [1]
• Fix bugs in PPO Trainer by @mmbwf in #900
• Fix #424 #762 #814 #887 #913 #1000 #1026 #1032 #1064 #1068 #1074 #1086 #1097 #1176 #1177 #1190 #1191

[1] huggingface/transformers#25598 (comment)

v0.1.8: FlashAttention-2 and Baichuan2

New features

• Support FlashAttention-2 for LLaMA models. (RTX4090, A100, A800 or H100 GPU is required)
• Support training the Baichuan2 models
• Use right-padding to avoid overflow in fp16 training (also mentioned here)
• Align the computation method of the reward score with DeepSpeed-Chat (better generation)
• Support --lora_target all argument which automatically finds the applicable modules for LoRA training

Bug fix

• Use efficient EOS tokens to align with the Baichuan training ( baichuan-inc/Baichuan2#23 )
• Remove PeftTrainer to save model checkpoints in DeepSpeed training
• Fix bugs in web UI by @beat4ocean in #596 by @codemayq in #644 #651 #678 #741 by @kinghuin in #786
• Add dataset explanation by @panpan0000 in #629
• Fix a bug in the DPO data collator
• Fix a bug of the ChatGLM2 tokenizer in right-padding
• #608 #617 #649 #757 #761 #763 #809 #818

v0.1.7: Script Preview and RoPE Scaling

New features

• Preview training script in Web UI by @codemayq in #479 #511
• Support resuming from checkpoints by @niuba in #434 (transformers>=4.31.0 required)
• Two RoPE scaling methods: linear and NTK-aware scaling for LLaMA models (transformers>=4.31.0 required)
• Support training the ChatGLM2-6B model
• Support PPO training in bfloat16 data type #551

Bug fix

• Unusual output of quantized models #278 #391
• Runtime error in distributed DPO training #480
• Unexpected truncation in generation #532
• Dataset streaming error in pre-training #548 #549
• Tensor shape mismatch in PPO training using ChatGLM2 #527 #528
• #475 #476 #478 #481 #494 #551

v0.1.6: DPO Training and Qwen-7B

• Adapt DPO training from the TRL library
• Support fine-tuning the Qwen-7B, Qwen-7B-Chat, XVERSE-13B, and ChatGLM2-6B models
• Implement the "safe" ChatML template for Qwen-7B-Chat
• Better Web UI
• Pretty readme by @codemayq #382
• New features: #395 #451
• Fix InternLM-7B inference #312
• Fix bugs: #351 #354 #361 #376 #408 #417 #420 #423 #426

v0.1.5: Patch release

• Fix LLaMA-2 template #307
• Fix bug in preprocessing 968ce0d
• Fix #294 #296

v0.1.4: Dataset Streaming

• Support dataset streaming
• Fix LLaMA-2 #268
• Fix DeepSpeed ZeRO-3 model save #274
• Fix #242 #284

v0.1.3: Patch release

release v0.1.3

v0.1.2: LLaMA-2 Models

• Support LLaMA-2 (good issue #202 )
• Advanced configurations in Web UI
• Fix API (downgrade pydantic<2.0.0)
• Fix baichuan lora hparam #194 #212
• Fix padding #196
• Fix ZeRO-3 #199
• Allow pass args to app #213
• Code simplification
• Add ShareGPT dataset

v0.1.1

• Web UI: source_prefix, max_length, dev set
• Bug fix: reward token #179
• Update template #171 #177
• Bug fix: replace the Literal type with Enum for pydantic [1] #176
• Add Web demo #180

[1] pydantic/pydantic#5821, fastapi/sqlmodel#67