intel_extension_for_transformers/neural_chat/docs/notebooks/customize_chatbot_with_optimized_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generative AI: Using Optimized Model to Customize Chatbot\n",
    "NeuralChat is a customizable chat framework designed to create user own chatbot within few minutes on multiple architectures. This notebook will introduce how to use your optimized model to customize chatbot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Environment 🔩"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install intel extension for transformers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install intel-extension-for-transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install Requirements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/intel/intel-extension-for-transformers.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ./intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/\n",
    "!pip install -r requirements.txt\n",
    "%cd ../../../"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customize Chatbot with Optimized Model 🚀"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Put your optimized model in `model_name_or_path` of `config`, e.g. `/models/Llama-2-7b-chat-hf`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following example to optimize bf16 model on `SPR` server.\n",
    "\n",
    "Refer to notebook ***intel_extension_for_transformers/neural_chat/docs/notebooks/bits_and_bytes_optimization_on_nv_a100.ipynb*** and ***intel_extension_for_transformers/neural_chat/docs/notebooks/amp_optimization_on_habana_gaudi.ipynb*** for `NVIDIA A100` and `Habana Gaudi` servers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intel_extension_for_transformers.neural_chat import build_chatbot, PipelineConfig\n",
    "from intel_extension_for_transformers.transformers import MixedPrecisionConfig\n",
    "config = PipelineConfig(optimization_config=MixedPrecisionConfig(), model_name_or_path='/models/Llama-2-7b-chat-hf')\n",
    "chatbot = build_chatbot(config)\n",
    "response = chatbot.predict(query=\"Tell me about Intel Xeon Scalable Processors.\")\n",
    "print(response)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}