intel_extension_for_transformers/neural_chat/docs/notebooks/single_card_finetuning_on_habana_gaudi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finetune Your Chatbot on Habana Gaudi \n",
    "\n",
    "NeuralChat is a customizable chat framework designed to create user own chatbot within few minutes on multiple architectures. This notebook will introduce how to finetune your chatbot with the customized data on Intel Habana Gaodi Processor.\n",
    "\n",
    "## Prepare Environment\n",
    "In order to streamline the process, users can construct a Docker image employing a Dockerfile, initiate the Docker container, and then proceed to execute inference or finetuning operations.\n",
    "\n",
    "IMPORTANT: Please note Habana's Gaudi processors(HPU) requires docker environment for running. User needs to manually execute below steps to build docker image and run docker container for inference on Habana HPU. The Jupyter notebook server should be started in the docker container and then run this Jupyter notebook.\n",
    "\n",
    "To run finetuning on Habana HPU, please execute below steps\n",
    "\n",
    "```bash\n",
    "git clone https://github.com/intel/intel-extension-for-transformers.git\n",
    "cd ./intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/docker/\n",
    "docker build --build-arg UBUNTU_VER=22.04 -f Dockerfile -t neuralchat . --target hpu\n",
    "docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all neuralchat:latest\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the Dataset\n",
    "We select 3 kind of datasets to conduct the finetuning process for different tasks.\n",
    "\n",
    "1. Text Generation (General domain instruction): We use the [Alpaca dataset](https://github.com/tatsu-lab/stanford_alpaca) from Stanford University as the general domain dataset to fine-tune the model. This dataset is provided in the form of a JSON file, [alpaca_data.json](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json). In Alpaca, researchers have manually crafted 175 seed tasks to guide `text-davinci-003` in generating 52K instruction data for diverse tasks.\n",
    "\n",
    "2. Summarization: An English-language dataset [cnn_dailymail](https://huggingface.co/datasets/cnn_dailymail) containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail, is used for this task.\n",
    "\n",
    "3. Code Generation: To enhance code performance of LLMs (Large Language Models), we use the [theblackcat102/evol-codealpaca-v1](https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finetune Your Chatbot\n",
    "You can train your chatbot with Alpaca dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.habana import GaudiTrainingArguments\n",
    "from intel_extension_for_transformers.neural_chat.config import (\n",
    "    ModelArguments,\n",
    "    DataArguments,\n",
    "    FinetuningArguments,\n",
    "    TextGenerationFinetuningConfig,\n",
    ")\n",
    "from intel_extension_for_transformers.neural_chat.chatbot import finetune_model\n",
    "model_args = ModelArguments(model_name_or_path=\"meta-llama/Llama-2-7b-chat-hf\")\n",
    "data_args = DataArguments(train_file=\"alpaca_data.json\")\n",
    "training_args = GaudiTrainingArguments(\n",
    "    output_dir='./tmp',\n",
    "    do_train=True,\n",
    "    do_eval=True,\n",
    "    num_train_epochs=3,\n",
    "    overwrite_output_dir=True,\n",
    "    per_device_train_batch_size=4,\n",
    "    per_device_eval_batch_size=4,\n",
    "    gradient_accumulation_steps=1,\n",
    "    save_strategy=\"no\",\n",
    "    log_level=\"info\",\n",
    "    save_total_limit=2,\n",
    "    bf16=True,\n",
    "    use_habana=True,\n",
    "    use_lazy_mode=True,\n",
    ")\n",
    "finetune_args = FinetuningArguments(device=\"habana\")\n",
    "finetune_cfg = TextGenerationFinetuningConfig(\n",
    "            model_args=model_args,\n",
    "            data_args=data_args,\n",
    "            training_args=training_args,\n",
    "            finetune_args=finetune_args,\n",
    "        )\n",
    "finetune_model(finetune_cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train your chatbot on the summarization task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.habana import GaudiTrainingArguments\n",
    "from intel_extension_for_transformers.neural_chat.config import (\n",
    "    ModelArguments,\n",
    "    DataArguments,\n",
    "    FinetuningArguments,\n",
    "    TextGenerationFinetuningConfig,\n",
    ")\n",
    "from intel_extension_for_transformers.neural_chat.chatbot import finetune_model\n",
    "model_args = ModelArguments(model_name_or_path=\"meta-llama/Llama-2-7b-chat-hf\")\n",
    "data_args = DataArguments(dataset_name=\"cnn_dailymail\", dataset_config_name=\"3.0.0\")\n",
    "training_args = GaudiTrainingArguments(\n",
    "    output_dir='./tmp',\n",
    "    do_train=True,\n",
    "    do_eval=True,\n",
    "    num_train_epochs=3,\n",
    "    overwrite_output_dir=True,\n",
    "    per_device_train_batch_size=4,\n",
    "    per_device_eval_batch_size=4,\n",
    "    gradient_accumulation_steps=1,\n",
    "    save_strategy=\"no\",\n",
    "    log_level=\"info\",\n",
    "    save_total_limit=2,\n",
    "    bf16=True,\n",
    "    use_habana=True,\n",
    "    use_lazy_mode=True,\n",
    ")\n",
    "finetune_args = FinetuningArguments(task='summarization', device=\"habana\")\n",
    "finetune_cfg = TextGenerationFinetuningConfig(\n",
    "            model_args=model_args,\n",
    "            data_args=data_args,\n",
    "            training_args=training_args,\n",
    "            finetune_args=finetune_args,\n",
    "        )\n",
    "finetune_model(finetune_cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train your chatbot on the code completion task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from optimum.habana import GaudiTrainingArguments\n",
    "from intel_extension_for_transformers.neural_chat.config import (\n",
    "    ModelArguments,\n",
    "    DataArguments,\n",
    "    FinetuningArguments,\n",
    "    TextGenerationFinetuningConfig,\n",
    ")\n",
    "from intel_extension_for_transformers.neural_chat.chatbot import finetune_model\n",
    "model_args = ModelArguments(model_name_or_path=\"meta-llama/Llama-2-7b-chat-hf\")\n",
    "data_args = DataArguments(dataset_name=\"theblackcat102/evol-codealpaca-v1\")\n",
    "training_args = GaudiTrainingArguments(\n",
    "    output_dir='./tmp',\n",
    "    do_train=True,\n",
    "    do_eval=True,\n",
    "    num_train_epochs=3,\n",
    "    overwrite_output_dir=True,\n",
    "    per_device_train_batch_size=4,\n",
    "    per_device_eval_batch_size=4,\n",
    "    gradient_accumulation_steps=1,\n",
    "    save_strategy=\"no\",\n",
    "    log_level=\"info\",\n",
    "    save_total_limit=2,\n",
    "    bf16=True,\n",
    "    use_habana=True,\n",
    "    use_lazy_mode=True,\n",
    ")\n",
    "finetune_args = FinetuningArguments(task='code-generation', device=\"habana\")\n",
    "finetune_cfg = TextGenerationFinetuningConfig(\n",
    "            model_args=model_args,\n",
    "            data_args=data_args,\n",
    "            training_args=training_args,\n",
    "            finetune_args=finetune_args,\n",
    "        )\n",
    "finetune_model(finetune_cfg)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}