part_mixtral.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "7d9bbf86da5e"
   },
   "outputs": [],
   "source": [
    "# Copyright 2023 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "99c1c3fc2ca5"
   },
   "source": [
    "# Vertex AI Model Garden - Mistral and Mixtral 8x7B Models\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_mistral.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_pytorch_mistral.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/model_garden/model_garden_pytorch_mistral.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "Open in Vertex AI Workbench\n",
    "    </a> (A Python-3 CPU notebook is recommended)\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3de7470326a2"
   },
   "source": [
    "## Overview\n",
    "\n",
    "This notebook demonstrates deploying prebuilt [Mistral](https://mistral.ai/) and Mixtral 8x7B models in Vertex AI.\n",
    "\n",
    "### Objective\n",
    "\n",
    "- Deploy prebuilt [Mistral models](https://huggingface.co/mistralai) with [vLLM](https://github.com/vllm-project/vllm) containers\n",
    "    - [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1): pretrained generative text model with 7 billion parameters\n",
    "    - [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1): Instruction fine-tuned version of the Mistral-7B-v0.1 generative text model\n",
    "- Deploy prebuit [Mixtral 8x7B model](https://huggingface.co/mistralai) with [vLLM](https://github.com/vllm-project/vllm) containers\n",
    "    - [mistralai/Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1): pretrained Mixture of Experts (MoE) model with 8 branches\n",
    "    - [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1): Instruction fine-tuned version of the Mixture of Experts (MoE) model with 8 branches\n",
    "\n",
    "### Costs\n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "\n",
    "* Vertex AI\n",
    "* Cloud Storage\n",
    "\n",
    "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), [Cloud NL API pricing](https://cloud.google.com/natural-language/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "264c07757582"
   },
   "source": [
    "## Before you begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "46d25fe73955"
   },
   "source": [
    "### Install dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "c75c2c1fa6e0",
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: google-cloud-aiplatform in /opt/conda/lib/python3.10/site-packages (1.43.0)\n",
      "Requirement already satisfied: google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1 in /opt/conda/lib/python3.10/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (1.34.1)\n",
      "Requirement already satisfied: google-auth<3.0.0dev,>=2.14.1 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (2.27.0)\n",
      "Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (1.23.0)\n",
      "Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (3.20.3)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (23.2)\n",
      "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (2.14.0)\n",
      "Requirement already satisfied: google-cloud-bigquery<4.0.0dev,>=1.15.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (3.17.2)\n",
      "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (1.12.1)\n",
      "Requirement already satisfied: shapely<3.0.0dev in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (2.0.2)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.56.2 in /opt/conda/lib/python3.10/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (1.62.0)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.10/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (2.31.0)\n",
      "Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /opt/conda/lib/python3.10/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (1.60.1)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.10/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (1.48.2)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from google-auth<3.0.0dev,>=2.14.1->google-cloud-aiplatform) (5.3.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.10/site-packages (from google-auth<3.0.0dev,>=2.14.1->google-cloud-aiplatform) (0.3.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.10/site-packages (from google-auth<3.0.0dev,>=2.14.1->google-cloud-aiplatform) (4.9)\n",
      "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.6.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.4.1)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.7.0)\n",
      "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.10/site-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.8.2)\n",
      "Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /opt/conda/lib/python3.10/site-packages (from google-cloud-resource-manager<3.0.0dev,>=1.3.3->google-cloud-aiplatform) (0.13.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-storage<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.5.0)\n",
      "Requirement already satisfied: numpy>=1.14 in /opt/conda/lib/python3.10/site-packages (from shapely<3.0.0dev->google-cloud-aiplatform) (1.25.2)\n",
      "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /opt/conda/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.0dev,>=2.14.1->google-cloud-aiplatform) (0.5.1)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil<3.0dev,>=2.7.2->google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.16.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (1.26.18)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.34.1->google-cloud-aiplatform) (2024.2.2)\n",
      "Requirement already satisfied: transformers==4.36.0 in /opt/conda/lib/python3.10/site-packages (4.36.0)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (3.13.1)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (0.21.3)\n",
      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (1.25.2)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (6.0.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (2023.12.25)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (2.31.0)\n",
      "Requirement already satisfied: tokenizers<0.19,>=0.14 in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (0.15.2)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (0.4.2)\n",
      "Requirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.10/site-packages (from transformers==4.36.0) (4.66.2)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.36.0) (2024.2.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.36.0) (4.9.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests->transformers==4.36.0) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests->transformers==4.36.0) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests->transformers==4.36.0) (1.26.18)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->transformers==4.36.0) (2024.2.2)\n",
      "Requirement already satisfied: accelerate==0.23.0 in /opt/conda/lib/python3.10/site-packages (0.23.0)\n",
      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.10/site-packages (from accelerate==0.23.0) (1.25.2)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from accelerate==0.23.0) (23.2)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.10/site-packages (from accelerate==0.23.0) (5.9.3)\n",
      "Requirement already satisfied: pyyaml in /opt/conda/lib/python3.10/site-packages (from accelerate==0.23.0) (6.0.1)\n",
      "Requirement already satisfied: torch>=1.10.0 in /opt/conda/lib/python3.10/site-packages (from accelerate==0.23.0) (2.2.1)\n",
      "Requirement already satisfied: huggingface-hub in /opt/conda/lib/python3.10/site-packages (from accelerate==0.23.0) (0.21.3)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (3.13.1)\n",
      "Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (4.9.0)\n",
      "Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (1.12)\n",
      "Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (3.2.1)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (3.1.3)\n",
      "Requirement already satisfied: fsspec in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (2024.2.0)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (8.9.2.26)\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (12.1.3.1)\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (11.0.2.54)\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (10.3.2.106)\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (11.4.5.107)\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (12.1.0.106)\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (2.19.3)\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (12.1.105)\n",
      "Requirement already satisfied: triton==2.2.0 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate==0.23.0) (2.2.0)\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12 in /opt/conda/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.10.0->accelerate==0.23.0) (12.3.101)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from huggingface-hub->accelerate==0.23.0) (2.31.0)\n",
      "Requirement already satisfied: tqdm>=4.42.1 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub->accelerate==0.23.0) (4.66.2)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->accelerate==0.23.0) (2.1.5)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests->huggingface-hub->accelerate==0.23.0) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests->huggingface-hub->accelerate==0.23.0) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests->huggingface-hub->accelerate==0.23.0) (1.26.18)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->huggingface-hub->accelerate==0.23.0) (2024.2.2)\n",
      "Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch>=1.10.0->accelerate==0.23.0) (1.3.0)\n",
      "Requirement already satisfied: gdown in /opt/conda/lib/python3.10/site-packages (5.1.0)\n",
      "Requirement already satisfied: beautifulsoup4 in /opt/conda/lib/python3.10/site-packages (from gdown) (4.12.3)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from gdown) (3.13.1)\n",
      "Requirement already satisfied: requests[socks] in /opt/conda/lib/python3.10/site-packages (from gdown) (2.31.0)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.10/site-packages (from gdown) (4.66.2)\n",
      "Requirement already satisfied: soupsieve>1.2 in /opt/conda/lib/python3.10/site-packages (from beautifulsoup4->gdown) (2.5)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests[socks]->gdown) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests[socks]->gdown) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests[socks]->gdown) (1.26.18)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests[socks]->gdown) (2024.2.2)\n",
      "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /opt/conda/lib/python3.10/site-packages (from requests[socks]->gdown) (1.7.1)\n"
     ]
    }
   ],
   "source": [
    "! pip3 install --upgrade google-cloud-aiplatform\n",
    "! pip3 install transformers==4.36.0\n",
    "! pip3 install accelerate==0.23.0\n",
    "! pip3 install gdown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bb7adab99e41"
   },
   "source": [
    "### Setup Google Cloud project\n",
    "\n",
    "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
    "\n",
    "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
    "\n",
    "1. [Enable the Vertex AI API, Compute Engine API and Cloud Natural Language API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,language.googleapis.com).\n",
    "\n",
    "1. [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs.\n",
    "\n",
    "1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console) with `Vertex AI User` and `Storage Object Admin` roles for deploying fine tuned model to Vertex AI endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6c460088b873"
   },
   "source": [
    "### Define environment variables\n",
    "\n",
    "Set the following variables for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the specified region (`REGION`). Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "855d6b96f291",
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n",
      "Creating gs://ai-sandbox-company-18-asia-notebooks/...\n",
      "ServiceException: 409 A Cloud Storage bucket named 'ai-sandbox-company-18-asia-notebooks' already exists. Try another name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.\n"
     ]
    }
   ],
   "source": [
    "# Please enter your name/initials (no spaces or special characters allowed), ensure that it is unique\n",
    "UNIQUE_PREFIX=\"asia-notebooks\" ### PLEASE UPDATE THIS\n",
    "\n",
    "# Cloud project id.\n",
    "PROJECT_IDS = !(gcloud config get-value core/project)\n",
    "PROJECT_ID = PROJECT_IDS[0]  # @param {type:\"string\"}\n",
    "\n",
    "# The region you want to launch jobs in.\n",
    "# Select region based on the accelerators and regions supported by Vertex AI Prediction\n",
    "# https://cloud.google.com/vertex-ai/docs/predictions/configure-compute.\n",
    "REGION = \"asia-southeast1\"  # @param {type:\"string\"}\n",
    "\n",
    "# The Cloud Storage bucket for storing experiments output.\n",
    "# Remove prefix gs://, e.g. foo_bucket.\n",
    "BUCKET_NAME = f\"{PROJECT_ID}-{UNIQUE_PREFIX}-{REGION}\"\n",
    "\n",
    "# The service account looks like:\n",
    "# '@.iam.gserviceaccount.com'\n",
    "# Please go to https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console\n",
    "# and create service account with `Vertex AI User` and `Storage Object Admin` roles.\n",
    "# The service account for deploying fine tuned model.\n",
    "SERVICE_ACCOUNT = !(gcloud config get-value core/account)  # @param {type:\"string\"}\n",
    "SERVICE_ACCOUNT = SERVICE_ACCOUNT[0]  # @param {type:\"string\"}\n",
    "\n",
    "BUCKET_URI = f\"gs://{BUCKET_NAME}\"  # @param {type:\"string\"}\n",
    "\n",
    "! gcloud config set project $PROJECT_ID\n",
    "! gcloud storage buckets create {BUCKET_URI} --project={PROJECT_ID} --location={REGION}\n",
    "\n",
    "import os\n",
    "\n",
    "STAGING_BUCKET = os.path.join(BUCKET_URI, \"temporal\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e828eb320337"
   },
   "source": [
    "### Initialize Vertex AI API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "12cd25839741",
    "tags": []
   },
   "outputs": [],
   "source": [
    "from google.cloud import aiplatform\n",
    "\n",
    "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=STAGING_BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2cc825514deb"
   },
   "source": [
    "### Define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "b42bd4fa2b2d",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# The pre-built serving docker images with vLLM\n",
    "VLLM_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20240112_0916_RC00\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0c250872074f"
   },
   "source": [
    "### Define common functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "354da31189dc",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from datetime import datetime\n",
    "from typing import Tuple\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "\n",
    "\n",
    "def get_job_name_with_datetime(prefix: str) -> str:\n",
    "    \"\"\"Gets the job name with date time when triggering training or deployment\n",
    "    jobs in Vertex AI.\n",
    "    \"\"\"\n",
    "    return prefix + datetime.now().strftime(\"_%Y%m%d_%H%M%S\")\n",
    "\n",
    "\n",
    "def deploy_model_vllm(\n",
    "    model_name: str,\n",
    "    model_id: str,\n",
    "    service_account: str,\n",
    "    machine_type: str = \"g2-standard-8\",\n",
    "    accelerator_type: str = \"NVIDIA_L4\",\n",
    "    accelerator_count: int = 1,\n",
    "    max_model_len: int = 4096,\n",
    "    use_openai_server: bool = False,\n",
    "    use_chat_completions_if_openai_server: bool = False,\n",
    ") -> Tuple[aiplatform.Model, aiplatform.Endpoint]:\n",
    "    \"\"\"Deploys Mistral models with vLLM on Vertex AI.\n",
    "\n",
    "    Args:\n",
    "        model_name: Display name of the model.\n",
    "        model_id: Model ID or path to model weights.\n",
    "        service_account: Service account for model uploading and deployment.\n",
    "        machine_type: Deployment machine type.\n",
    "        accelerator_type: Deployment accelerator type.\n",
    "        accelerator_count: Number of accelerators to use.\n",
    "        max_model_len: Maximum model length.\n",
    "        use_openai_server: Whether to use the OpenAI-format vLLM model server.\n",
    "        use_chat_completions_if_openai_server: If the OpenAI model server is\n",
    "            used, whether to use the chat completion API as opposed to the text\n",
    "            completion API. The vLLM text completion API mimics the OpenAI text\n",
    "            completion API:\n",
    "            https://platform.openai.com/docs/api-reference/completions/create.\n",
    "            It has two required parameters: the model ID to direct requests to\n",
    "            and the prompt. The response includes a \"choices\" field that\n",
    "            contains the generated text and a \"usage\" field that contains token\n",
    "            counts. The vLLM chat completion API mimics the OpenAI chat\n",
    "            completion API:\n",
    "            https://platform.openai.com/docs/api-reference/chat/create. It has\n",
    "            two required parameters: the model ID to direct requests to and\n",
    "            \"messages\" which is a sequence of system/user/assistant/tool\n",
    "            messages that can represent a multi-turn chat conversation. The\n",
    "            response includes a \"choices\" field that contains the generated\n",
    "            message from a role and a \"usage\" field that contains token counts.\n",
    "\n",
    "    Returns:\n",
    "        Model instance and endpoint instance.\n",
    "    \"\"\"\n",
    "    endpoint = aiplatform.Endpoint.create(display_name=f\"{model_name}-endpoint\")\n",
    "\n",
    "    dtype = \"bfloat16\"\n",
    "    if accelerator_type in [\"NVIDIA_TESLA_T4\", \"NVIDIA_TESLA_V100\"]:\n",
    "        dtype = \"float16\"\n",
    "\n",
    "    vllm_args = [\n",
    "        \"--host=0.0.0.0\",\n",
    "        \"--port=7080\",\n",
    "        f\"--model={model_id}\",\n",
    "        f\"--tensor-parallel-size={accelerator_count}\",\n",
    "        \"--swap-space=16\",\n",
    "        f\"--dtype={dtype}\",\n",
    "        \"--gpu-memory-utilization=0.9\",\n",
    "        f\"--max-model-len={max_model_len}\",\n",
    "        \"--disable-log-stats\",\n",
    "    ]\n",
    "    serving_env = {\n",
    "        \"MODEL_ID\": \"mistralai/Mistral-7B-v0.1\",\n",
    "    }\n",
    "    if use_openai_server:\n",
    "        if use_chat_completions_if_openai_server:\n",
    "            serving_container_predict_route = \"/v1/chat/completions\"\n",
    "        else:\n",
    "            serving_container_predict_route = \"/v1/completions\"\n",
    "    else:\n",
    "        serving_container_predict_route = \"/generate\"\n",
    "    model = aiplatform.Model.upload(\n",
    "        display_name=model_name,\n",
    "        serving_container_image_uri=VLLM_DOCKER_URI,\n",
    "        serving_container_command=[\n",
    "            \"python\",\n",
    "            \"-m\",\n",
    "            (\n",
    "                \"vllm.entrypoints.api_server\"\n",
    "                if not use_openai_server\n",
    "                else \"vllm.entrypoints.openai.api_server\"\n",
    "            ),\n",
    "        ],\n",
    "        serving_container_args=vllm_args,\n",
    "        serving_container_ports=[7080],\n",
    "        serving_container_predict_route=serving_container_predict_route,\n",
    "        serving_container_health_route=\"/health\",\n",
    "        serving_container_environment_variables=serving_env,\n",
    "    )\n",
    "\n",
    "    model.deploy(\n",
    "        endpoint=endpoint,\n",
    "        machine_type=machine_type,\n",
    "        accelerator_type=accelerator_type,\n",
    "        accelerator_count=accelerator_count,\n",
    "        deploy_request_timeout=1800,\n",
    "        service_account=service_account,\n",
    "    )\n",
    "    return model, endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YKZ4CBJ2kYaW",
    "tags": []
   },
   "source": [
    "## Deploy Prebuilt Mistral model with vLLM\n",
    "\n",
    "This section deploys the prebuilt Mistral model with [vLLM](https://github.com/vllm-project/vllm) on a Vertex endpoint. The model deployment step will take ~15 minutes to complete.\n",
    "\n",
    "vLLM is a highly optimized LLM serving framework which can significantly increase serving throughput. The higher QPS you have, the more benefits you get using vLLM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "25b5b3a44cf8"
   },
   "source": [
    "Set the prebuilt model id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "10547af949fc"
   },
   "outputs": [],
   "source": [
    "prebuilt_model_id = \"mistralai/Mistral-7B-Instruct-v0.2\"  # @param [\"mistralai/Mistral-7B-v0.1\", \"mistralai/Mistral-7B-Instruct-v0.1\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "03d504bcd60b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Endpoint\n",
      "Create Endpoint backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512/operations/5816569343051300864\n",
      "Endpoint created. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512\n",
      "To use this Endpoint in another session:\n",
      "endpoint = aiplatform.Endpoint('projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512')\n",
      "Creating Model\n",
      "Create Model backing LRO: projects/877303944797/locations/asia-southeast1/models/7374477264051765248/operations/7028037642813964288\n",
      "Model created. Resource name: projects/877303944797/locations/asia-southeast1/models/7374477264051765248@1\n",
      "To use this Model in another session:\n",
      "model = aiplatform.Model('projects/877303944797/locations/asia-southeast1/models/7374477264051765248@1')\n",
      "Deploying model to Endpoint : projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512\n",
      "Deploy Endpoint model backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512/operations/1263430119779729408\n",
      "Endpoint model deployed. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512\n"
     ]
    }
   ],
   "source": [
    "# Find Vertex AI prediction supported accelerators and regions in\n",
    "# https://cloud.google.com/vertex-ai/docs/predictions/configure-compute.\n",
    "\n",
    "# Sets 1 L4 to deploy Mistral 7B.\n",
    "machine_type = \"g2-standard-8\"\n",
    "accelerator_type = \"NVIDIA_L4\"\n",
    "accelerator_count = 1\n",
    "\n",
    "# Sets 2 V100s to deploy Mistral 7B.\n",
    "# machine_type = \"n1-standard-16\"\n",
    "# accelerator_type = \"NVIDIA_TESLA_V100\"\n",
    "# accelerator_count = 2\n",
    "\n",
    "# Sets 2 T4s to deploy Mistral 7B.\n",
    "# machine_type = \"n1-standard-16\"\n",
    "# accelerator_type = \"NVIDIA_TESLA_T4\"\n",
    "# accelerator_count = 2\n",
    "\n",
    "# Sets 1 A100 (40G) to deploy Mistral 7B.\n",
    "# machine_type = \"a2-highgpu-1g\"\n",
    "# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
    "# accelerator_count = 1\n",
    "\n",
    "# Larger setting of `max-model-len` can lead to higher requirements on\n",
    "# `gpu-memory-utilization` and GPU configuration.\n",
    "max_model_len = 4096\n",
    "\n",
    "def deploy_model_with_config(machine_type, accelerator_type, accelerator_count ):\n",
    "    \n",
    "    model, endpoint = deploy_model_vllm(\n",
    "    model_name=get_job_name_with_datetime(prefix=\"mistral-serve-vllm\"),\n",
    "    model_id=prebuilt_model_id,\n",
    "    service_account=SERVICE_ACCOUNT,\n",
    "    machine_type=machine_type,\n",
    "    accelerator_type=accelerator_type,\n",
    "    accelerator_count=accelerator_count,\n",
    "    max_model_len=max_model_len,\n",
    "    use_openai_server=False,\n",
    "    use_chat_completions_if_openai_server=False,\n",
    ")\n",
    "    \n",
    "    return(model, endpoint)\n",
    "\n",
    "try:\n",
    "    # Code that might potentially cause an error\n",
    "    machine_type = \"a2-highgpu-1g\"\n",
    "    accelerator_type = \"NVIDIA_TESLA_A100\"\n",
    "    accelerator_count = 1\n",
    "    model, endpoint = deploy_model_with_config(machine_type, accelerator_type, accelerator_count )\n",
    "except :  # Replace 'ErrorType' with the specific error you want to catch \n",
    "    try : \n",
    "        print(\"Error may becaused due to machine unavailable \")\n",
    "        machine_type = \"g2-standard-8\"\n",
    "        accelerator_type = \"NVIDIA_L4\"\n",
    "        accelerator_count = 1\n",
    "        model, endpoint = deploy_model_with_config(machine_type, accelerator_type, accelerator_count )\n",
    "    except :\n",
    "        print(\"Error may becaused due to machine unavailable - A100s - trying T4s \")\n",
    "        machine_type = \"n1-standard-16\"\n",
    "        accelerator_type = \"NVIDIA_TESLA_T4\"\n",
    "        accelerator_count = 2\n",
    "        model, endpoint = deploy_model_with_config(machine_type, accelerator_type, accelerator_count )\n",
    "else:\n",
    "    # Code to execute if there's no error in the 'try' block\n",
    "    print(\"Error may becaused due to machine unavailable of any type \")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RRR11SWykYaX"
   },
   "source": [
    "NOTE: If you see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint, the model server is likely still initializing. Please retry later.\n",
    "\n",
    "NOTE: If you receive `InternalServerError: 500 System error` during the deployment, most likely the operation failed due to unavailability of resources. Either retry or use a different accelerator type.\n",
    "\n",
    "Once deployment succeeds, you can send requests to the endpoint with text prompts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3a7948c56e3d"
   },
   "source": [
    "### Run sample prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "3f5a1e1de60d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "My favourite condiment is\n",
      "Output:\n",
      " the classic aioli, so this recipe is a twist on that.  The garlic and lemon in this recipe give it a punch, which you can tone down if you prefer.\n",
      "\n",
      "This dip is also great on top of a salad as well as a side to your favourite BBQ food.\n",
      "\n",
      "Ingredients\n",
      "\n",
      "- 1/2 cup tinned chickpeas (170g)\n",
      "- 1 small clove garlic\n",
      "- 2 Tbsp lemon juice\n",
      "- 1 tsp lemon zest\n",
      "- 1 Tbsp apple cider vinegar\n",
      "- 1 Tbsp dijon mustard\n",
      "- 2 tsp maple syrup\n",
      "- 1/2 tsp sea salt\n",
      "- 1/2 cup olive oil (125ml)\n",
      "- 1-2 Tbsp water\n",
      "\n",
      "Method\n",
      "\n",
      "Place the chickpeas and garlic in a food processor and whiz to create\n"
     ]
    }
   ],
   "source": [
    "# Loads an existing endpoint instance using the endpoint name:\n",
    "# - Using `endpoint_name = endpoint.name` allows us to get the endpoint name of\n",
    "#   the endpoint `endpoint` created in the cell above.\n",
    "# - Alternatively, you can set `endpoint_name = \"1234567890123456789\"` to load\n",
    "#   an existing endpoint with the ID 1234567890123456789.\n",
    "# You may uncomment the code below to load an existing endpoint.\n",
    "\n",
    "# endpoint_name = endpoint_without_peft.name\n",
    "# # endpoint_name = \"\"  # @param {type:\"string\"}\n",
    "# aip_endpoint_name = (\n",
    "#     f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
    "# )\n",
    "# endpoint = aiplatform.Endpoint(aip_endpoint_name)\n",
    "\n",
    "instances = [\n",
    "    {\n",
    "        \"prompt\": \"My favourite condiment is\",\n",
    "        \"n\": 1,\n",
    "        \"max_tokens\": 200,\n",
    "        \"temperature\": 1.0,\n",
    "        \"top_p\": 1.0,\n",
    "        \"top_k\": 10,\n",
    "    },\n",
    "]\n",
    "response = endpoint.predict(instances=instances)\n",
    "\n",
    "for prediction in response.predictions:\n",
    "    print(prediction)\n",
    "\n",
    "# Reference the following code for using the OpenAI vLLM server.\n",
    "# import json\n",
    "# response = endpoint.raw_predict(\n",
    "#     body=json.dumps({\n",
    "#         \"model\": prebuilt_model_id,\n",
    "#         \"prompt\": \"My favourite condiment is\",\n",
    "#         \"n\": 1,\n",
    "#         \"max_tokens\": 200,\n",
    "#         \"temperature\": 1.0,\n",
    "#         \"top_p\": 1.0,\n",
    "#         \"top_k\": 10,\n",
    "#     }),\n",
    "#     headers={\"Content-Type\": \"application/json\"},\n",
    "# )\n",
    "# print(response.json())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wOh9irbqJ-MM",
    "tags": []
   },
   "source": [
    "## Deploy Prebuilt Mixtral 8x7B model with vLLM\n",
    "\n",
    "This section deploys the prebuilt Mixtral 8x7B model with [vLLM](https://github.com/vllm-project/vllm) on a Vertex endpoint. The model deployment step will take ~40 minutes to complete.\n",
    "\n",
    "vLLM is a highly optimized LLM serving framework which can significantly increase serving throughput. The higher QPS you have, the more benefits you get using vLLM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h2uCSnoaJ-MM"
   },
   "source": [
    "Set the prebuilt model id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "-X42gkGYJ-MM"
   },
   "outputs": [],
   "source": [
    "prebuilt_model_id = \"mistralai/Mixtral-8x7B-v0.1\"  # @param [\"mistralai/Mixtral-8x7B-v0.1\", \"mistralai/Mixtral-8x7B-Instruct-v0.1\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "agDw0_7JJ-MM"
   },
   "source": [
    "NOTE: If you see a `ServiceUnavailable: 503 502:Bad Gateway` error when you send requests to the endpoint, the model server is likely still initializing. Please retry later.\n",
    "\n",
    "NOTE: If you receive `InternalServerError: 500 System error` during the deployment, most likely the operation failed due to unavailability of resources. Either retry or use a different accelerator type.\n",
    "\n",
    "Once deployment succeeds, you can send requests to the endpoint with text prompts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Endpoint\n",
      "Create Endpoint backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000/operations/4537547048878080000\n",
      "Endpoint created. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000\n",
      "To use this Endpoint in another session:\n",
      "endpoint = aiplatform.Endpoint('projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000')\n",
      "Creating Model\n",
      "Create Model backing LRO: projects/877303944797/locations/asia-southeast1/models/6906102902805233664/operations/912149348844830720\n",
      "Model created. Resource name: projects/877303944797/locations/asia-southeast1/models/6906102902805233664@1\n",
      "To use this Model in another session:\n",
      "model = aiplatform.Model('projects/877303944797/locations/asia-southeast1/models/6906102902805233664@1')\n",
      "Deploying model to Endpoint : projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000\n",
      "Deploy Endpoint model backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000/operations/8604297512393637888\n",
      "Endpoint model deployed. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000\n",
      "Error may becaused due to machine unavailable of any type \n"
     ]
    }
   ],
   "source": [
    "# Find Vertex AI prediction supported accelerators and regions in\n",
    "# https://cloud.google.com/vertex-ai/docs/predictions/configure-compute.\n",
    "\n",
    "# Sets 8 L4s to deploy Mixtral 8x7B.\n",
    "machine_type = \"g2-standard-96\"\n",
    "accelerator_type = \"NVIDIA_L4\"\n",
    "accelerator_count = 8\n",
    "\n",
    "# Sets 4 A100s (40G) to deploy Mixtral 8x7B.\n",
    "# machine_type = \"a2-highgpu-4g\"\n",
    "# accelerator_type = \"NVIDIA_TESLA_A100\"\n",
    "# accelerator_count = 4\n",
    "def deploy_model_with_config(machine_type, accelerator_type, accelerator_count ):\n",
    "    \n",
    "    model, endpoint = deploy_model_vllm(\n",
    "    model_name=get_job_name_with_datetime(prefix=\"mixtral-serve-vllm\"),\n",
    "    model_id=prebuilt_model_id,\n",
    "    service_account=SERVICE_ACCOUNT,\n",
    "    machine_type=machine_type,\n",
    "    accelerator_type=accelerator_type,\n",
    "    accelerator_count=accelerator_count,\n",
    "    )\n",
    "    return(model, endpoint)\n",
    "\n",
    "try:\n",
    "    # Code that might potentially cause an error\n",
    "    machine_type = \"g2-standard-96\"\n",
    "    accelerator_type = \"NVIDIA_L4\"\n",
    "    accelerator_count = 8\n",
    "    model, endpoint = deploy_model_with_config(machine_type, accelerator_type, accelerator_count )\n",
    "except :  # Replace 'ErrorType' with the specific error you want to catch \n",
    "    try : \n",
    "        print(\"Trying  A100s - in the region \")\n",
    "        machine_type = \"a2-highgpu-4g\"\n",
    "        accelerator_type = \"NVIDIA_TESLA_A100\"\n",
    "        accelerator_count = 4\n",
    "        model, endpoint = deploy_model_with_config(machine_type, accelerator_type, accelerator_count )\n",
    "    except :\n",
    "        print(\"Error may becaused due to machine unavailable - A100s - trying T4s \")\n",
    "        machine_type = \"n1-standard-96\"\n",
    "        accelerator_type = \"NVIDIA_TESLA_T4\"\n",
    "        accelerator_count = 8\n",
    "        model, endpoint = deploy_model_with_config(machine_type, accelerator_type, accelerator_count )\n",
    "else:\n",
    "    # Code to execute if there's no error in the 'try' block\n",
    "    print(\"Error may becaused due to machine unavailable of any type \")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zLyWJK5aJ-MM"
   },
   "source": [
    "### Run sample prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "NYN1Z49SJ-MM"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "What is a car?\n",
      "Output:\n",
      "The automobile (also known as a motor car, or more simply a “car”) is probably one of the most influential inventions of the 20th century and certainly a major force in shaping our current world society. It\n"
     ]
    }
   ],
   "source": [
    "# Loads an existing endpoint instance using the endpoint name:\n",
    "# - Using `endpoint_name = endpoint.name` allows us to get the endpoint name of\n",
    "#   the endpoint `endpoint` created in the cell above.\n",
    "# - Alternatively, you can set `endpoint_name = \"1234567890123456789\"` to load\n",
    "#   an existing endpoint with the ID 1234567890123456789.\n",
    "# You may uncomment the code below to load an existing endpoint.\n",
    "\n",
    "# endpoint_name = endpoint_without_peft.name\n",
    "# # endpoint_name = \"\"  # @param {type:\"string\"}\n",
    "# aip_endpoint_name = (\n",
    "#     f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n",
    "# )\n",
    "# endpoint = aiplatform.Endpoint(aip_endpoint_name)\n",
    "\n",
    "instances = [\n",
    "    {\n",
    "        \"prompt\": \"What is a car?\",\n",
    "        \"max_tokens\": 50,\n",
    "        \"temperature\": 1.0,\n",
    "        \"top_p\": 1.0,\n",
    "        \"top_k\": 10,\n",
    "    },\n",
    "]\n",
    "response = endpoint.predict(instances=instances)\n",
    "\n",
    "for prediction in response.predictions:\n",
    "    print(prediction)\n",
    "\n",
    "# Reference the following code for using the OpenAI vLLM server.\n",
    "# import json\n",
    "# response = endpoint.raw_predict(\n",
    "#     body=json.dumps({\n",
    "#         \"model\": prebuilt_model_id,\n",
    "#         \"prompt\": \"My favourite condiment is\",\n",
    "#         \"n\": 1,\n",
    "#         \"max_tokens\": 200,\n",
    "#         \"temperature\": 1.0,\n",
    "#         \"top_p\": 1.0,\n",
    "#         \"top_k\": 10,\n",
    "#     }),\n",
    "#     headers={\"Content-Type\": \"application/json\"},\n",
    "# )\n",
    "# print(response.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e057b5edcf81",
    "tags": []
   },
   "source": [
    "## Run inferences locally with prebuilt Mistral and Mixtral models\n",
    "\n",
    "You will need at least 24GB of memory to run inference with Mistral-7B. You can run locally or on Vertex AI Prediction endpoints with any of the following specs:\n",
    "- g2-standard-8 with 1 L4 GPU\n",
    "- n1-standard-16 with 2 V100 GPUs\n",
    "- n1-standard-16 with 2 T4 GPUs\n",
    "- a2-highgpu-1g with 1 A100 GPU\n",
    "\n",
    "You will need at least 96GB of memory to run inference with Mixtral 8x7B. You can run locally or on Vertex AI Prediction endpoints with any of the following specs:\n",
    "- g2-standard-96 with 8 L4 GPUs\n",
    "- n1-standard-32 with 8 V100 GPUs\n",
    "- n1-standard-32 with 8 T4 GPUs\n",
    "- a2-highgpu-4g with 4 A100 GPUs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "31f6cc84efdd",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n",
      "/opt/conda/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n",
      "/opt/conda/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
      "  _torch_pytree._register_pytree_node(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2fe181348660482b995b60ba3a742ddf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result: My favourite condiment is the Sriracha Aioli.  I could seriously eat just that on a bun.  If you are not familiar with the Sriracha Chili sauce, it is a spicy Thai hot sauce.  I love it, and you can see a little jar of it on the counter in the picture below.\n",
      "\n",
      "If you live in the Ottawa area, and have not yet tried the gourmet burgers at The Works Gourmet Burger Bistro, you should give it a try.  You can visit them online at:\n",
      "\n",
      "They also have an active Facebook page, where they post lots of great information.  Visit:\n",
      "\n",
      "If you have any questions, I am happy to answer them as best as I can.  I hope you enjoyed reading about this restaurant as much as I did enjoying it. If you would like to see more restaurant photos from my trip to Ottawa, you can also\n",
      "CPU times: user 35.4 s, sys: 14.1 s, total: 49.5 s\n",
      "Wall time: 1min 35s\n"
     ]
    }
   ],
   "source": [
    "# %%time\n",
    "# import torch\n",
    "# import transformers\n",
    "# from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "# device = \"cuda\"  # the device to load the model onto\n",
    "# model_name = \"mistralai/Mistral-7B-v0.1\"  # @param [\"mistralai/Mistral-7B-v0.1\", \"mistralai/Mistral-7B-Instruct-v0.1\", \"mistralai/Mixtral-8x7B-v0.1\", \"mistralai/Mixtral-8x7B-Instruct-v0.1\"]\n",
    "# model = AutoModelForCausalLM.from_pretrained(\n",
    "#     model_name, device_map=\"auto\", return_dict=True, torch_dtype=torch.float16\n",
    "# )\n",
    "# tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "\n",
    "# pipeline = transformers.pipeline(\"text-generation\", model=model, tokenizer=tokenizer)\n",
    "\n",
    "# prompt = \"My favourite condiment is\"\n",
    "\n",
    "# sequences = pipeline(\n",
    "#     prompt,\n",
    "#     max_length=200,\n",
    "#     do_sample=True,\n",
    "#     top_k=10,\n",
    "#     num_return_sequences=1,\n",
    "#     eos_token_id=tokenizer.eos_token_id,\n",
    "# )\n",
    "\n",
    "# for seq in sequences:\n",
    "#     print(f\"Result: {seq['generated_text']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NRaBADRM4JEn"
   },
   "source": [
    "## Cleaning up (for reference, do not run)\n",
    "You can delete the individual resources you created in this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0a9da70d4abc"
   },
   "source": [
    "### Undeploy models and Delete endpoints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set this flag to delete endpoint including undeploying models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "e53749ae6b2c"
   },
   "outputs": [],
   "source": [
    "# delete_endpoint = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "65078f3ec44e"
   },
   "outputs": [],
   "source": [
    "# def list_endpoints():\n",
    "#     return [\n",
    "#         (r.name, r.display_name)\n",
    "#         for r in aiplatform.Endpoint.list()\n",
    "# #        if r.display_name.startswith(\"mistral-serve-vllm\")\n",
    "#     ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete the endpoint using the Vertex AI fully qualified identifier for the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "cf56ac4cc73b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Undeploying all deployed models and deleting endpoint 1428265604481024000 [mixtral-serve-vllm_20240306_034747-endpoint]\n",
      "Undeploying Endpoint model: projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000\n",
      "Undeploy Endpoint model backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000/operations/7545951599961571328\n",
      "Endpoint model undeployed. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000\n",
      "Deleting Endpoint : projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000\n",
      "Delete Endpoint  backing LRO: projects/877303944797/locations/asia-southeast1/operations/2447876821778169856\n",
      "Endpoint deleted. . Resource name: projects/877303944797/locations/asia-southeast1/endpoints/1428265604481024000\n",
      "Undeploying all deployed models and deleting endpoint 4229504572705472512 [mistral-serve-vllm_20240306_033816-endpoint]\n",
      "Undeploying Endpoint model: projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512\n",
      "Undeploy Endpoint model backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512/operations/7059562840205557760\n",
      "Endpoint model undeployed. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512\n",
      "Deleting Endpoint : projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512\n",
      "Delete Endpoint  backing LRO: projects/877303944797/locations/asia-southeast1/operations/3934064698810433536\n",
      "Endpoint deleted. . Resource name: projects/877303944797/locations/asia-southeast1/endpoints/4229504572705472512\n",
      "Undeploying all deployed models and deleting endpoint 275344099874177024 [mixtral-serve-vllm_20240306_022338-endpoint]\n",
      "Undeploying Endpoint model: projects/877303944797/locations/asia-southeast1/endpoints/275344099874177024\n",
      "Undeploy Endpoint model backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/275344099874177024/operations/2051560054569566208\n",
      "Endpoint model undeployed. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/275344099874177024\n",
      "Deleting Endpoint : projects/877303944797/locations/asia-southeast1/endpoints/275344099874177024\n",
      "Delete Endpoint  backing LRO: projects/877303944797/locations/asia-southeast1/operations/1294955317171322880\n",
      "Endpoint deleted. . Resource name: projects/877303944797/locations/asia-southeast1/endpoints/275344099874177024\n",
      "Undeploying all deployed models and deleting endpoint 3774641010341052416 [mistral-serve-vllm_20240306_020314-endpoint]\n",
      "Undeploying Endpoint model: projects/877303944797/locations/asia-southeast1/endpoints/3774641010341052416\n",
      "Undeploy Endpoint model backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/3774641010341052416/operations/2065070853451677696\n",
      "Endpoint model undeployed. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/3774641010341052416\n",
      "Deleting Endpoint : projects/877303944797/locations/asia-southeast1/endpoints/3774641010341052416\n",
      "Delete Endpoint  backing LRO: projects/877303944797/locations/asia-southeast1/operations/5906641335598710784\n",
      "Endpoint deleted. . Resource name: projects/877303944797/locations/asia-southeast1/endpoints/3774641010341052416\n",
      "Undeploying all deployed models and deleting endpoint 5382426077312319488 [mistral-serve-vllm_20240306_021407-endpoint]\n",
      "Undeploying Endpoint model: projects/877303944797/locations/asia-southeast1/endpoints/5382426077312319488\n",
      "Undeploy Endpoint model backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/5382426077312319488/operations/3416150741662826496\n",
      "Endpoint model undeployed. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/5382426077312319488\n",
      "Deleting Endpoint : projects/877303944797/locations/asia-southeast1/endpoints/5382426077312319488\n",
      "Delete Endpoint  backing LRO: projects/877303944797/locations/asia-southeast1/operations/8545750717237821440\n",
      "Endpoint deleted. . Resource name: projects/877303944797/locations/asia-southeast1/endpoints/5382426077312319488\n",
      "Undeploying all deployed models and deleting endpoint 2608771656805515264 [mixtral-serve-vllm_20240305_142435-endpoint]\n",
      "Undeploying Endpoint model: projects/877303944797/locations/asia-southeast1/endpoints/2608771656805515264\n",
      "Undeploy Endpoint model backing LRO: projects/877303944797/locations/asia-southeast1/endpoints/2608771656805515264/operations/3600798326385016832\n",
      "Endpoint model undeployed. Resource name: projects/877303944797/locations/asia-southeast1/endpoints/2608771656805515264\n",
      "Deleting Endpoint : projects/877303944797/locations/asia-southeast1/endpoints/2608771656805515264\n",
      "Delete Endpoint  backing LRO: projects/877303944797/locations/asia-southeast1/operations/6676756871879065600\n",
      "Endpoint deleted. . Resource name: projects/877303944797/locations/asia-southeast1/endpoints/2608771656805515264\n"
     ]
    }
   ],
   "source": [
    "# try:\n",
    "#     if delete_endpoint:\n",
    "#         endpoints = list_endpoints()\n",
    "#         for endpoint_id, endpoint_name in endpoints:\n",
    "#             endpoint = aiplatform.Endpoint(endpoint_id)\n",
    "#             print(\n",
    "#                 f\"Undeploying all deployed models and deleting endpoint {endpoint_id} [{endpoint_name}]\"\n",
    "#             )\n",
    "#             endpoint.delete(force=False)\n",
    "# except Exception as e:\n",
    "#     print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d25a89a34b5e"
   },
   "source": [
    "### Delete Cloud Storage bucket (Not needed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "PAr4UWWx4JEo",
    "tags": []
   },
   "outputs": [],
   "source": [
    "#import os\n",
    "\n",
    "#delete_bucket = False\n",
    "\n",
    "#job.delete()\n",
    "\n",
    "#if delete_bucket or os.getenv(\"ID_TESTING\"):\n",
    "#    ! gsutil rm -rf {BUCKET_URI}"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "model_garden_pytorch_mistral.ipynb",
   "toc_visible": false
  },
  "environment": {
   "kernel": "conda-env-pytorch-pytorch",
   "name": "workbench-notebooks.m113",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/workbench-notebooks:m113"
  },
  "kernelspec": {
   "display_name": "PyTorch 1-13 (Local)",
   "language": "python",
   "name": "conda-env-pytorch-pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}