huggingface-sagemaker-multi-model-endpoint.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Deployed multiple Transformers models using Amazon SageMaker Multi-Model Endpoints </h2>\n",
    "\n",
    "We can hosts up to thousands of models with [Amazon SageMaker multi-model endpoints](https://docs.aws.amazon.com/sagemaker/latest/dg/multi-model-endpoints.html).\n",
    "\n",
    "**Use cases of multi-model endpoints:**\n",
    "\n",
    "- Multiple models which can be served from a common inference container, can be invoked on-demand and we experience some additional latency for infrequently invoked models.\n",
    "- In some cases where the variable latency is tolerable, and cost optimization is more important, we may also decide to use MMEs for A/B/n testing, in place of the more typical production variant based strategy discussed here.\n",
    "\n",
    "**How multi-model endpoints work?**\n",
    "\n",
    "Amazon SageMaker takes care of the loading and unloading of models for a multi-model endpoint in the container’s memory, as they are invoked. When SageMaker receives an inference request for a particular model, it does the following.\n",
    "\n",
    "1. Routes the request to an instance assigned to that model.\n",
    "2. Instead of downloading all of the models from an Amazon S3 bucket to the container, SageMaker downloads only the invoked model from S3 bucket to that instance’s storage volume.\n",
    "3. Also loads the model to the container’s memory on that instance. If the model is already loaded in the container’s memory then invocation is performed immediately because SageMaker already downloaded that model and loaded it.\n",
    "\n",
    "Let’s say SageMaker has already loaded 100 models and needs to load another model into memory but instance’s memory utilization is high then what next?\n",
    "\n",
    "1. SageMaker unloads unused models from that instance’s container memory to ensure that there is enough memory to load the model.\n",
    "2. Models which are unloaded remain on the instance’s storage volume and can be loaded into the container’s memory later without being downloaded again from the S3 bucket.\n",
    "\n",
    "What if the instance’s storage volume reaches its capacity?\n",
    "\n",
    "Then SageMaker deletes the all unused models from the instance storage storage volume. We see SageMaker is much smarter in loading and unloading models to the container’s memory.\n",
    "\n",
    "If we want to add a new model to multi-model endpoint then we upload it to S3 and invoke it. If we want to delete a existing model from multi-model endpoint then stop sending requests and delete it from S3.\n",
    "\n",
    "We will use the Hugging Face Inference DLCs and Amazon SageMaker to deploy multiple transformer models as Multi-Model Endpoint. Amazon SageMaker Multi-Model Endpoint can be used to improve endpoint utilization and optimize costs.\n",
    "\n",
    "This notebook demonstrates how to host 2 pretrained transformers model in one container behind one endpoint.\n",
    "1. Use BERT model to extract embeddings from text.\n",
    "2. Use GPT-2 model to generate synthetic text for a given text.\n",
    "\n",
    "**NOTE**: As the time of writing this only `CPU` Instances are supported for Multi-Model Endpoint.\n",
    "\n",
    "Please refer to [Medium article](https://medium.com/@shwet.prakash97/deployed-multiple-transformers-models-using-amazon-sagemaker-multi-model-endpoints-e73d109dc7cf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Development Environment and Permissions </h2>\n",
    "\n",
    "NOTE: You can run this demo in Sagemaker Studio, your local machine, or Sagemaker Notebook Instances\n",
    "\n",
    "If you are going to use Sagemaker in a local environment (not SageMaker Studio or Notebook Instances). You need access to an IAM Role with the required permissions for Sagemaker. You can find [here](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html) more about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "role = get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "sagemaker_session = sagemaker.session.Session()\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "prefix = 'huggingface-multimodel-deploy'\n",
    "sm_client = boto3.client(\"sagemaker\")\n",
    "\n",
    "\n",
    "print(f\"sagemaker role arn: {role}\")\n",
    "print(f\"sagemaker bucket: {bucket}\")\n",
    "print(f\"sagemaker session region: {region}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Retrieve Model Artifacts </h2>\n",
    "\n",
    "#### `BERT model`\n",
    "\n",
    "First we will download the model artifacts for the pretrained [BERT](https://arxiv.org/abs/1810.04805) model. BERT is a popular natural language processing (NLP) model that extracts meaning and context from text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install transformers==4.17.0 --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('models/bertmodel/model/tokenizer_config.json',\n",
       " 'models/bertmodel/model/special_tokens_map.json',\n",
       " 'models/bertmodel/model/vocab.txt',\n",
       " 'models/bertmodel/model/added_tokens.json')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from transformers import BertTokenizer, BertModel\n",
    "\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "model = BertModel.from_pretrained(\"bert-base-uncased\")\n",
    "\n",
    "model_path = 'models/bertmodel/model'\n",
    "\n",
    "if not os.path.exists(model_path):\n",
    "    os.mkdir(model_path)\n",
    "    \n",
    "model.save_pretrained(save_directory=model_path)\n",
    "tokenizer.save_pretrained(save_directory=model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `GPT-2 model`\n",
    "\n",
    "Second we will download the model artifacts for the pretrained [GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) model. GPT-2 is a popular text generation model that was developed by OpenAI. Given a text prompt it can generate synthetic text that may follow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('models/gptmodel/model/vocab.json', 'models/gptmodel/model/merges.txt')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from transformers import GPT2Tokenizer, GPT2LMHeadModel\n",
    "\n",
    "tokenizer = GPT2Tokenizer.from_pretrained('gpt2')\n",
    "model = GPT2LMHeadModel.from_pretrained('gpt2')\n",
    "\n",
    "model_path = 'models/gptmodel/model'\n",
    "\n",
    "if not os.path.exists(model_path):\n",
    "    os.mkdir(model_path)\n",
    "    \n",
    "model.save_pretrained(save_directory=model_path)\n",
    "tokenizer.save_vocabulary(save_directory=model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Write the Inference Script </h2> \n",
    "\n",
    "#### `BERT model`\n",
    "\n",
    "Since we are bringing a model to SageMaker, we must create an inference script. The script will run inside our HuggingFace container. Our script should include a function for model loading, and optionally functions generating predicitions, and input/output processing. The HuggingFace container provides default implementations for generating a prediction and input/output processing. By including these functions in your script you are overriding the default functions. You can find additional [details here](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#serve-a-pytorch-model).\n",
    "\n",
    "**NOTE**:\n",
    "\n",
    "1. Single model deployment: To install additional libraries at container startup, we can add a requirements.txt text file that specifies the libraries to be installed using pip. Within the archive, the HuggingFace container expects all inference code and requirements.txt file to be inside the code/ directory.\n",
    "\n",
    "2. Multi-model deployment: To install additional libraries on the container, libraries which are part of requirements.txt text file needs to be installed using pip in inference script. Within the archive, the HuggingFace container expects all inference code to be inside the code/ directory.\n",
    "\n",
    "In the next cell we'll see our inference script for BERT model which helps us to extract embeddings from text. \n",
    "\n",
    "You will notice that it uses the [transformers library from Hugging Face](https://huggingface.co/docs/transformers/index) and installed using pip command in inference script, likewise we need to install additional libraries if required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir models/bertmodel/code\n",
    "\n",
    "! cp source_dir/model1/inference.py models/bertmodel/code/inference.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msubprocess\u001b[39;49;00m\n",
      "subprocess.call([\u001b[33m\"\u001b[39;49;00m\u001b[33mpip\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[33m\"\u001b[39;49;00m\u001b[33minstall\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[33m\"\u001b[39;49;00m\u001b[33mtransformers==4.17.0\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtransformers\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m BertTokenizer, BertModel\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mmodel_fn\u001b[39;49;00m(model_dir):\n",
      "    \u001b[33m\"\"\"\u001b[39;49;00m\n",
      "\u001b[33m    Load the model for inference\u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "\n",
      "    model_path = os.path.join(model_dir, \u001b[33m'\u001b[39;49;00m\u001b[33mmodel/\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    \n",
      "    \u001b[37m# Load BERT tokenizer from disk.\u001b[39;49;00m\n",
      "    tokenizer = BertTokenizer.from_pretrained(model_path)\n",
      "\n",
      "    \u001b[37m# Load BERT model from disk.\u001b[39;49;00m\n",
      "    model = BertModel.from_pretrained(model_path)\n",
      "\n",
      "    model_dict = {\u001b[33m'\u001b[39;49;00m\u001b[33mmodel\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m: model, \u001b[33m'\u001b[39;49;00m\u001b[33mtokenizer\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m:tokenizer}\n",
      "    \n",
      "    \u001b[34mreturn\u001b[39;49;00m model_dict\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mpredict_fn\u001b[39;49;00m(input_data, model):\n",
      "    \u001b[33m\"\"\"\u001b[39;49;00m\n",
      "\u001b[33m    Apply model to the incoming request\u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "    \n",
      "    tokenizer = model[\u001b[33m'\u001b[39;49;00m\u001b[33mtokenizer\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m]\n",
      "    bert_model = model[\u001b[33m'\u001b[39;49;00m\u001b[33mmodel\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m]\n",
      "    \n",
      "    encoded_input = tokenizer(input_data, return_tensors=\u001b[33m'\u001b[39;49;00m\u001b[33mpt\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    \n",
      "    \u001b[34mreturn\u001b[39;49;00m bert_model(**encoded_input)\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32minput_fn\u001b[39;49;00m(request_body, request_content_type):\n",
      "    \u001b[33m\"\"\"\u001b[39;49;00m\n",
      "\u001b[33m    Deserialize and prepare the prediction input\u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "    \n",
      "    \u001b[34mif\u001b[39;49;00m request_content_type == \u001b[33m\"\u001b[39;49;00m\u001b[33mapplication/json\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\n",
      "        request = json.loads(request_body)\n",
      "    \u001b[34melse\u001b[39;49;00m:\n",
      "        request = request_body\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m request\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32moutput_fn\u001b[39;49;00m(prediction, response_content_type):\n",
      "    \u001b[33m\"\"\"\u001b[39;49;00m\n",
      "\u001b[33m    Serialize and prepare the prediction output\u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "    \n",
      "    \u001b[34mif\u001b[39;49;00m response_content_type == \u001b[33m\"\u001b[39;49;00m\u001b[33mapplication/json\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\n",
      "        response = \u001b[36mstr\u001b[39;49;00m(prediction)\n",
      "    \u001b[34melse\u001b[39;49;00m:\n",
      "        response = \u001b[36mstr\u001b[39;49;00m(prediction)\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m response\n"
     ]
    }
   ],
   "source": [
    "!pygmentize models/bertmodel/code/inference.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `GPT-2 model`\n",
    "\n",
    "In the next cell we'll see our inference script for GPT-2 model which helps us to generate synthetic text for a given text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir models/gptmodel/code\n",
    "\n",
    "! cp source_dir/model2/inference.py models/gptmodel/code/inference.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msubprocess\u001b[39;49;00m\n",
      "subprocess.call([\u001b[33m\"\u001b[39;49;00m\u001b[33mpip\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[33m\"\u001b[39;49;00m\u001b[33minstall\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[33m\"\u001b[39;49;00m\u001b[33mtransformers==4.17.0\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtransformers\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m GPT2Tokenizer, TextGenerationPipeline, GPT2LMHeadModel\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mmodel_fn\u001b[39;49;00m(model_dir):\n",
      "    \u001b[33m\"\"\"\u001b[39;49;00m\n",
      "\u001b[33m    Load the model for inference\u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "\n",
      "    \u001b[37m# Load GPT2 tokenizer from disk.\u001b[39;49;00m\n",
      "    vocab_path = os.path.join(model_dir, \u001b[33m'\u001b[39;49;00m\u001b[33mmodel/vocab.json\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    merges_path = os.path.join(model_dir, \u001b[33m'\u001b[39;49;00m\u001b[33mmodel/merges.txt\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    \n",
      "    tokenizer = GPT2Tokenizer(vocab_file=vocab_path,\n",
      "                              merges_file=merges_path)\n",
      "\n",
      "    \u001b[37m# Load GPT2 model from disk.\u001b[39;49;00m\n",
      "    model_path = os.path.join(model_dir, \u001b[33m'\u001b[39;49;00m\u001b[33mmodel/\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    model = GPT2LMHeadModel.from_pretrained(model_path)\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m TextGenerationPipeline(model=model, tokenizer=tokenizer)\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mpredict_fn\u001b[39;49;00m(input_data, model):\n",
      "    \u001b[33m\"\"\"\u001b[39;49;00m\n",
      "\u001b[33m    Apply model to the incoming request\u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m model.\u001b[32m__call__\u001b[39;49;00m(input_data)\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32minput_fn\u001b[39;49;00m(request_body, request_content_type):\n",
      "    \u001b[33m\"\"\"\u001b[39;49;00m\n",
      "\u001b[33m    Deserialize and prepare the prediction input\u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "    \n",
      "    \u001b[34mif\u001b[39;49;00m request_content_type == \u001b[33m\"\u001b[39;49;00m\u001b[33mapplication/json\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\n",
      "        request = json.loads(request_body)\n",
      "    \u001b[34melse\u001b[39;49;00m:\n",
      "        request = request_body\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m request\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32moutput_fn\u001b[39;49;00m(prediction, response_content_type):\n",
      "    \u001b[33m\"\"\"\u001b[39;49;00m\n",
      "\u001b[33m    Serialize and prepare the prediction output\u001b[39;49;00m\n",
      "\u001b[33m    \"\"\"\u001b[39;49;00m\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m \u001b[36mstr\u001b[39;49;00m(prediction)\n"
     ]
    }
   ],
   "source": [
    "!pygmentize models/gptmodel/code/inference.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Package Models </h2> \n",
    "\n",
    "For hosting, SageMaker requires that the deployment package be structed in a compatible format. It expects all files to be packaged in a tar archive named \"model.tar.gz\" (We name it as bertmodel.tar.gz an gptmodel.tar.gz) with gzip compression. Within the archive, the HuggingFace container expects all inference code file and requirements.txt to be inside the code/ directory. See the guide here for a thorough explanation of the required directory structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./\n",
      "./code/\n",
      "./code/inference.py\n",
      "./.ipynb_checkpoints/\n",
      "./model/\n",
      "./model/config.json\n",
      "./model/tokenizer_config.json\n",
      "./model/pytorch_model.bin\n",
      "./model/vocab.txt\n",
      "./model/special_tokens_map.json\n",
      "./\n",
      "./code/\n",
      "./code/inference.py\n",
      "./.ipynb_checkpoints/\n",
      "./model/\n",
      "./model/config.json\n",
      "./model/merges.txt\n",
      "./model/vocab.json\n",
      "./model/pytorch_model.bin\n"
     ]
    }
   ],
   "source": [
    "!tar -czvf models/bertmodel.tar.gz -C models/bertmodel/ .\n",
    "!tar -czvf models/gptmodel.tar.gz -C models/gptmodel/ ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Upload multiple HuggingFace models to S3 </h2> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "\n",
    "models_path = 's3://{0}/{1}/models'.format(bucket,prefix)\n",
    "\n",
    "S3Uploader.upload('models/bertmodel.tar.gz', models_path)\n",
    "S3Uploader.upload('models/gptmodel.tar.gz', models_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Create Multi-Model Endpoint </h2> \n",
    "\n",
    "After we upload BERT model to S3 we can deploy our endpoint. To create/deploy a real-time endpoint with boto3 you need to create a \"SageMaker Model\", a \"SageMaker Endpoint Configuration\" and a \"SageMaker Endpoint\". The \"SageMaker Model\" contains our multi-model configuration including our S3 path where we upload/deploy multiple huggingface models. The \"SageMaker Endpoint Configuration\" contains the configuration for the endpoint. The \"SageMaker Endpoint\" is the actual endpoint.\n",
    "\n",
    "Verify `multi-models` LABEL in docker file to indicate any pre-built container is capable of loading and serving multiple models concurrently.\n",
    "\n",
    "LABEL com.amazonaws.sagemaker.capabilities.multi-models=true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create SageMaker Model\n",
    "image_uri = \"763104351884.dkr.ecr.us-east-2.amazonaws.com/huggingface-pytorch-inference:1.10.2-transformers4.17.0-cpu-py38-ubuntu20.04\"\n",
    "multimodels_path = 's3://sagemaker-us-east-2-(account-id)/huggingface-multimodel-deploy/models/'\n",
    "deployment_name = \"huggingface-multi-model\"\n",
    "\n",
    "primary_container = {\n",
    "    'Image': image_uri,\n",
    "    'Mode': 'MultiModel',\n",
    "    'ModelDataUrl': multimodels_path,\n",
    "    'Environment': {\n",
    "        'SAGEMAKER_PROGRAM': 'inference.py',\n",
    "        'SAGEMAKER_REGION': region,\n",
    "        'SAGEMAKER_SUBMIT_DIRECTORY': multimodels_path\n",
    "    }\n",
    "}\n",
    "\n",
    "create_model_response = sm_client.create_model(ModelName = deployment_name,\n",
    "                                              ExecutionRoleArn = get_execution_role(),\n",
    "                                              PrimaryContainer = primary_container)\n",
    "\n",
    "print(create_model_response['ModelArn'])\n",
    "\n",
    "# create SageMaker Endpoint configuration\n",
    "endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName = f\"{deployment_name}-epc\",\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "        'InstanceType':'ml.m5.4xlarge',\n",
    "        'InitialInstanceCount':1,\n",
    "        'ModelName': deployment_name,\n",
    "        'VariantName':'AllTraffic',\n",
    "        'InitialVariantWeight':1\n",
    "        }\n",
    "    ])\n",
    "\n",
    "print('Endpoint configuration arn:  {}'.format(endpoint_config_response['EndpointConfigArn']))\n",
    "\n",
    "# create SageMaker Endpoint\n",
    "endpoint_params = {\n",
    "    'EndpointName': f\"{deployment_name}-ep\",\n",
    "    'EndpointConfigName': f\"{deployment_name}-epc\",\n",
    "}\n",
    "endpoint_response = sm_client.create_endpoint(**endpoint_params)\n",
    "print('EndpointArn = {}'.format(endpoint_response['EndpointArn']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Get Predictions </h2> \n",
    "\n",
    "#### `BERT model`\n",
    "Now that our API endpoint is deployed, we can send it text to get predictions from our BERT model. You can use the SageMaker SDK or the SageMaker Runtime API to invoke the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/\n",
      "  warnings.warn(warning, PythonDeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "b'BaseModelOutputWithPoolingAndCrossAttentions(last_hidden_state=tensor([[[-0.2462, -0.0988,  0.1747,  ..., -0.4059,  0.0966,  0.6564],\\n         [-0.1352, -0.5824, -0.0728,  ..., -0.1726,  0.5765,  0.1273],\\n         [-0.1491, -0.4218,  0.2821,  ...,  0.1332,  0.5053, -0.2813],\\n         ...,\\n         [-0.8054, -0.3126,  0.6776,  ..., -0.0572,  0.0806, -0.0318],\\n         [ 0.7608,  0.1367, -0.2650,  ...,  0.1246, -0.5977, -0.2397],\\n         [ 0.4660,  0.2762,  0.0636,  ...,  0.1112, -0.5502, -0.2997]]],\\n       grad_fn=<NativeLayerNormBackward0>), pooler_output=tensor([[-7.0429e-01, -4.2229e-01, -9.7203e-01,  6.3414e-01,  8.6010e-01,\\n         -3.5008e-01,  3.7999e-02,  2.2652e-01, -8.5239e-01, -9.9980e-01,\\n         -6.4649e-01,  8.4232e-01,  8.9319e-01,  6.2476e-01,  4.8914e-01,\\n         -3.7195e-01,  1.0597e-01, -5.0569e-01,  3.3702e-01,  7.3767e-01,\\n          6.0322e-01,  1.0000e+00, -3.2281e-01,  4.7648e-01,  4.4296e-01,\\n          9.4813e-01, -6.6813e-01,  7.4915e-01,  8.2229e-01,  6.3062e-01,\\n         -1.4025e-01,  2.2783e-01, -9.6329e-01, -2.0670e-01, -9.7215e-01,\\n         -9.3046e-01,  5.0601e-01, -4.7677e-01,  1.6829e-01, -2.7722e-02,\\n         -7.4092e-01,  3.3085e-01,  9.9991e-01, -6.8587e-01,  7.2314e-01,\\n         -1.7279e-01, -1.0000e+00,  2.7953e-01, -6.6518e-01,  9.3125e-01,\\n          8.8504e-01,  9.0542e-01,  2.4233e-01,  4.6617e-01,  4.7114e-01,\\n         -5.9370e-01, -2.0038e-01,  2.7291e-02, -2.8522e-01, -5.5561e-01,\\n         -6.1984e-01,  4.2931e-01, -8.7690e-01, -7.1666e-01,  8.7416e-01,\\n          9.3529e-01, -1.6623e-01, -2.3902e-01, -1.9703e-01, -4.2381e-02,\\n          6.0962e-01,  1.7582e-01, -3.2225e-01, -7.3443e-01,  7.5766e-01,\\n          2.8546e-01, -6.3455e-01,  1.0000e+00, -1.5006e-01, -8.9906e-01,\\n          9.4594e-01,  8.2220e-01,  5.8257e-01, -6.1727e-01,  7.2892e-01,\\n         -1.0000e+00,  5.9967e-01, -1.9553e-01, -9.5505e-01,  2.8234e-01,\\n          6.5812e-01, -3.3026e-01,  9.2597e-01,  7.0949e-01, -6.0489e-01,\\n         -6.2557e-01, -2.1836e-01, -9.4475e-01, -3.5476e-01, -6.5299e-01,\\n          2.4002e-01, -2.9925e-01, -3.6173e-01, -3.2079e-01,  4.4673e-01,\\n         -4.7877e-01,  3.8076e-03,  6.1428e-01,  3.1495e-01,  5.5661e-01,\\n          5.4409e-01, -4.0993e-01,  3.5277e-01, -7.7047e-01,  4.9286e-01,\\n         -3.3508e-01, -9.6048e-01, -6.4067e-01, -9.5352e-01,  4.8227e-01,\\n         -2.5500e-01, -2.7310e-01,  6.2703e-01, -6.0067e-01,  4.3773e-01,\\n         -1.3204e-01, -8.9876e-01, -1.0000e+00, -5.7486e-01, -5.8367e-01,\\n         -3.7799e-02, -3.1076e-01, -9.2927e-01, -8.9301e-01,  5.7826e-01,\\n          8.9921e-01,  3.2762e-01,  9.9950e-01, -2.9298e-01,  7.4470e-01,\\n         -2.8432e-01, -6.9845e-01,  6.4370e-01, -4.0474e-01,  8.6064e-01,\\n         -4.2865e-01, -8.4201e-02,  1.9170e-01, -4.9307e-01,  2.4257e-01,\\n         -7.5579e-01, -1.5230e-01, -8.7137e-01, -7.5294e-01, -3.9612e-01,\\n          7.5623e-01, -7.2463e-01, -9.5676e-01, -2.1992e-01, -7.2237e-03,\\n         -4.2903e-01,  7.3310e-01,  7.8768e-01,  1.8995e-01, -4.1924e-01,\\n          3.3158e-01,  3.7680e-03,  4.9328e-01, -5.4632e-01, -2.5544e-01,\\n          4.2792e-01, -3.4765e-01, -9.5505e-01, -9.1659e-01, -2.7968e-01,\\n          3.3616e-01,  9.3921e-01,  3.2194e-01,  4.4059e-01,  7.8101e-01,\\n         -4.1404e-01,  5.6999e-01, -8.9867e-01,  9.4501e-01, -1.2916e-01,\\n          2.3877e-01, -8.0565e-01,  5.4742e-01, -6.9169e-01,  3.2259e-01,\\n          4.6589e-01, -8.1670e-01, -5.5203e-01, -1.2755e-01, -5.7002e-01,\\n         -4.8809e-01, -8.5480e-01,  3.7383e-01, -2.6327e-01, -4.4017e-01,\\n         -1.7042e-01,  7.8240e-01,  6.5593e-01,  3.9682e-01,  6.1350e-01,\\n          6.6910e-01, -5.8099e-01, -1.2729e-01,  1.3087e-01,  2.4979e-01,\\n          9.0462e-02,  9.5573e-01, -8.7745e-01, -5.3944e-02, -7.9538e-01,\\n         -9.0020e-01, -1.0430e-02, -6.3410e-01, -2.6991e-01, -6.7839e-01,\\n          6.8991e-01, -8.5664e-01,  5.4074e-01,  2.5057e-01, -1.9839e-01,\\n         -5.3468e-01,  2.5042e-01, -5.3122e-01,  4.8517e-01, -2.3039e-01,\\n          9.8263e-01,  9.5669e-01, -6.2871e-01, -3.7287e-01,  9.1808e-01,\\n         -9.5680e-01, -7.3054e-01, -2.0423e-01, -3.7704e-01,  5.3158e-01,\\n         -5.5107e-01,  9.4410e-01,  8.8621e-01,  6.5910e-01, -5.9173e-01,\\n         -8.7734e-01, -4.4452e-01, -6.2441e-01,  1.8345e-02,  2.6674e-01,\\n          9.1898e-01,  7.0334e-01,  4.5215e-01,  1.5477e-01, -3.2292e-01,\\n          6.3614e-01, -9.8578e-01, -8.6722e-01, -9.2116e-01, -2.0438e-01,\\n         -9.5635e-01,  9.1029e-01,  3.3865e-01,  8.6891e-01, -4.8682e-01,\\n         -5.6545e-01, -8.4585e-01,  4.1545e-01,  1.4697e-01,  7.4499e-01,\\n         -7.4186e-01, -4.1167e-01, -6.3849e-01, -7.9425e-01, -7.1474e-02,\\n         -1.6725e-01, -6.2415e-01, -7.7895e-02, -7.4305e-01,  5.2787e-01,\\n          4.8245e-01,  4.6977e-01, -9.3221e-01,  9.6142e-01,  1.0000e+00,\\n          8.7031e-01,  6.2576e-01,  2.6042e-01, -9.9998e-01, -9.3788e-01,\\n          9.9995e-01, -9.8948e-01, -1.0000e+00, -7.3993e-01, -5.3754e-01,\\n          6.8562e-02, -1.0000e+00, -3.0134e-01,  1.0994e-01, -5.8069e-01,\\n          7.9011e-01,  8.6519e-01,  6.6334e-01, -1.0000e+00,  5.5531e-01,\\n          7.9444e-01, -6.8273e-01,  8.9153e-01, -5.4783e-01,  8.5564e-01,\\n          5.5771e-01,  6.9861e-01, -1.2855e-01,  3.9747e-01, -9.7520e-01,\\n         -5.9381e-01, -7.9269e-01, -8.9581e-01,  9.9980e-01,  9.9247e-02,\\n         -7.1784e-01, -5.3887e-01,  7.9213e-01,  7.2672e-03,  1.0741e-01,\\n         -8.6744e-01, -3.0254e-01,  5.2122e-01,  5.6968e-01,  3.0697e-01,\\n          2.7829e-01, -1.3249e-01,  3.7797e-01,  3.5936e-01,  8.6240e-02,\\n          6.5188e-01, -7.8581e-01, -2.3989e-02,  3.9677e-01,  2.9169e-01,\\n         -7.4401e-01, -9.3657e-01,  7.7299e-01, -2.3792e-01,  7.5010e-01,\\n          1.0000e+00,  7.2673e-01, -4.1205e-01,  5.5583e-01,  3.0370e-01,\\n         -5.8846e-01,  1.0000e+00,  7.9738e-01, -9.0979e-01, -6.6592e-01,\\n          7.6298e-01, -6.2132e-01, -7.7240e-01,  9.9498e-01, -3.7270e-01,\\n         -8.7361e-01, -5.2872e-01,  9.4385e-01, -9.5239e-01,  9.9918e-01,\\n         -4.5485e-01, -8.8839e-01,  8.3818e-01,  7.8258e-01, -5.8362e-01,\\n         -5.2211e-01,  2.8821e-02, -7.6797e-01,  3.9479e-01, -4.3449e-01,\\n          4.8557e-01,  1.9137e-01, -4.4434e-03,  6.8384e-01,  1.6296e-01,\\n         -6.7678e-01,  2.0940e-01, -7.0660e-01, -1.5576e-01,  9.6065e-01,\\n          4.1787e-01, -2.0698e-01, -8.8921e-02, -2.7851e-01, -8.6436e-01,\\n         -9.0030e-01,  7.6115e-01,  1.0000e+00, -4.2346e-02,  9.2810e-01,\\n         -3.2609e-01, -1.1601e-01,  7.6556e-02,  6.2215e-01,  5.5409e-01,\\n         -1.5557e-01, -5.8778e-01,  8.2371e-01, -4.4559e-01, -9.5695e-01,\\n          2.3920e-01,  1.5621e-01, -1.8932e-01,  9.9992e-01,  3.4455e-01,\\n          2.7785e-01,  4.8869e-01,  9.7985e-01,  1.2123e-02,  1.5760e-01,\\n          9.4338e-01,  9.5431e-01, -2.4670e-01,  7.1301e-01,  1.6395e-01,\\n         -9.1902e-01, -2.9367e-01, -5.6091e-01,  8.2864e-02, -8.6072e-01,\\n          1.0045e-01, -8.1766e-01,  8.9445e-01,  9.5519e-01,  3.7974e-01,\\n          1.6428e-01,  8.3312e-01,  1.0000e+00, -9.8540e-01,  2.2744e-01,\\n          7.9729e-01, -2.7107e-01, -9.9997e-01, -1.4946e-04, -3.6338e-01,\\n         -1.2888e-01, -8.9895e-01, -3.9675e-01,  9.5274e-02, -8.7694e-01,\\n          8.7259e-01,  7.3736e-01, -6.3866e-01, -8.7836e-01, -5.3952e-01,\\n          3.8029e-01,  8.2281e-02, -9.9608e-01, -4.7665e-01, -2.8731e-01,\\n          5.5650e-01, -3.8107e-01, -6.4363e-01, -4.2192e-01, -3.5137e-01,\\n          4.1607e-01, -1.9183e-01,  6.2999e-01,  9.2827e-01,  7.2474e-01,\\n         -9.6024e-01, -4.7600e-01, -5.6838e-02, -4.2833e-01,  6.1923e-01,\\n         -6.5076e-01, -9.5072e-01, -2.0124e-01,  1.0000e+00, -1.1769e-01,\\n          9.5465e-01,  3.3081e-01,  2.7900e-01, -2.0819e-01,  3.1738e-01,\\n          9.7415e-01,  2.8186e-01, -7.6092e-01, -8.6825e-01,  8.0483e-01,\\n         -5.0071e-01,  5.8275e-01,  9.0219e-01,  8.0464e-01,  4.3357e-01,\\n          9.1644e-01,  1.8520e-01, -1.0471e-02, -6.6956e-02,  9.1749e-01,\\n         -5.6697e-02, -3.6595e-01, -3.5708e-01, -1.4219e-01, -3.1479e-01,\\n          6.1602e-01,  1.0000e+00,  2.6784e-01,  7.7581e-01, -9.4108e-01,\\n         -8.7218e-01, -6.6301e-01,  1.0000e+00,  7.0394e-01, -3.7093e-01,\\n          4.4596e-01,  4.6609e-01, -1.6196e-01,  1.2609e-01, -2.2340e-01,\\n         -3.2369e-01,  2.7568e-01,  1.4218e-01,  7.8667e-01, -4.8022e-01,\\n         -9.3212e-01, -5.9068e-01,  3.4132e-01, -8.4641e-01,  9.9999e-01,\\n         -5.8762e-01, -2.4817e-01, -2.1803e-01, -5.9153e-01, -9.0153e-01,\\n         -6.2096e-02, -8.9386e-01, -1.8735e-01,  2.5682e-01,  7.9349e-01,\\n          3.8490e-01, -6.3735e-01, -6.9540e-01,  8.3634e-01,  7.3290e-01,\\n         -9.4176e-01, -7.7765e-01,  8.1436e-01, -8.3566e-01,  6.4903e-01,\\n          1.0000e+00,  4.8774e-01,  4.4023e-01,  7.5929e-02, -3.2230e-01,\\n          4.4445e-01, -7.9360e-01,  3.6513e-01, -8.2476e-01, -4.4221e-01,\\n         -1.9913e-01,  3.4394e-01, -1.6068e-01, -8.4861e-01,  7.5980e-02,\\n          3.2635e-01, -6.2073e-01, -5.6273e-01, -1.3777e-02,  4.4581e-01,\\n          5.4840e-01, -2.7247e-01, -1.6441e-01,  2.3364e-01,  9.0321e-02,\\n         -4.4573e-01, -4.7720e-01, -5.1088e-01, -1.0000e+00,  3.9913e-01,\\n         -1.0000e+00,  6.3140e-01,  2.6997e-01, -2.0522e-01,  6.4446e-01,\\n          8.9535e-01,  7.9128e-01, -4.2473e-01, -9.2832e-01,  2.9445e-01,\\n          3.8726e-01, -3.5147e-01, -1.4711e-01, -2.4331e-01,  3.6506e-01,\\n         -1.1074e-01,  3.0146e-01, -7.7704e-01,  7.1269e-01, -3.0761e-01,\\n          1.0000e+00,  2.4892e-01, -4.2761e-01, -3.6417e-01,  2.5862e-01,\\n         -2.9382e-01,  1.0000e+00, -9.3796e-02, -8.9436e-01,  3.7350e-01,\\n         -7.8972e-01, -6.0232e-01,  4.7087e-01,  1.4877e-01, -7.0357e-01,\\n         -9.5183e-01,  3.7305e-01,  4.8407e-01, -7.1330e-01,  7.6172e-01,\\n         -3.5378e-01, -4.6132e-01,  8.5328e-02,  9.6855e-01,  9.3842e-01,\\n          6.7007e-01,  4.5927e-01, -8.5576e-01, -5.3646e-01,  8.9675e-01,\\n          2.9143e-01, -4.8709e-01,  1.4774e-01,  1.0000e+00,  4.4341e-01,\\n         -7.8597e-01,  2.6452e-01, -8.2354e-01, -1.3105e-01, -8.2840e-01,\\n          2.9209e-01,  2.1272e-01,  7.6796e-01, -3.4143e-01,  7.3560e-01,\\n         -9.2232e-01,  1.2953e-01, -5.1920e-01, -7.0030e-01,  3.0870e-01,\\n         -7.3938e-01, -9.0843e-01, -9.0789e-01,  6.8067e-01, -2.9995e-01,\\n         -1.3688e-01,  2.9807e-01,  8.8721e-02,  3.8320e-01,  3.7907e-01,\\n         -1.0000e+00,  8.2361e-01,  4.3105e-01,  9.0323e-01,  8.4776e-01,\\n          8.2425e-01,  7.7038e-01,  3.4351e-01, -9.0495e-01, -3.7315e-01,\\n         -3.1706e-01, -3.1218e-01,  5.0647e-01,  6.8359e-01,  6.0040e-01,\\n          1.9903e-01, -5.0796e-01, -7.0890e-01, -8.1465e-01, -9.7855e-01,\\n         -9.6754e-01,  5.1544e-01, -6.5849e-01, -1.4164e-01,  8.9444e-01,\\n         -5.0000e-02, -1.5493e-01, -3.1448e-01, -9.0064e-01,  2.2259e-01,\\n          4.0316e-01, -1.3909e-01,  5.7329e-02,  4.0753e-01,  5.2156e-01,\\n          6.5553e-01,  8.6643e-01, -9.3422e-01,  5.5152e-01, -8.4335e-01,\\n          4.4102e-01,  9.5761e-01, -8.9940e-01,  9.5909e-02,  7.6769e-01,\\n         -2.1159e-01,  3.0813e-01, -2.8968e-01, -2.6445e-01,  7.9634e-01,\\n         -3.5189e-01,  4.6153e-01, -4.2751e-01,  1.2668e-01, -4.3487e-01,\\n         -2.4559e-01, -5.0632e-01, -4.1360e-01,  6.0186e-01, -3.0510e-03,\\n          6.6026e-01,  9.3351e-01, -1.3465e-01, -5.1754e-01, -4.5242e-02,\\n         -8.0939e-01, -7.9774e-01,  4.0735e-01,  3.8867e-02, -2.8360e-01,\\n          8.6788e-01, -6.6490e-02,  9.7830e-01,  4.7661e-01, -3.3393e-01,\\n         -3.7814e-01, -4.2969e-01,  5.3066e-01, -7.7054e-01, -5.9246e-01,\\n         -4.6809e-01,  6.0525e-01,  2.5718e-01,  1.0000e+00, -8.7896e-01,\\n         -9.1013e-01, -7.4345e-01, -3.8568e-01,  4.0054e-01, -4.4582e-01,\\n         -1.0000e+00,  3.2783e-01, -6.4760e-01,  8.0087e-01, -6.8343e-01,\\n          8.9602e-01, -5.9155e-01, -7.5990e-01, -4.8572e-01,  8.6559e-01,\\n          8.8322e-01, -5.0129e-01, -4.9732e-01,  5.2962e-01, -3.4280e-01,\\n          9.8162e-01,  5.7197e-01, -4.3897e-01, -1.5350e-01,  7.1397e-01,\\n         -9.1926e-01, -6.2192e-01,  5.4504e-01]], grad_fn=<TanhBackward0>), hidden_states=None, past_key_values=None, attentions=None, cross_attentions=None)'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import boto3\n",
    "\n",
    "invoke_client = boto3.client('sagemaker-runtime')\n",
    "\n",
    "prompt = \"The best part of Amazon SageMaker is that it makes machine learning easy.\"\n",
    "\n",
    "response = invoke_client.invoke_endpoint(EndpointName=f\"{deployment_name}-ep\",\n",
    "                              TargetModel='bertmodel.tar.gz',\n",
    "                              Body=prompt.encode(encoding='UTF-8'),\n",
    "                              ContentType='text/csv')\n",
    "\n",
    "response['Body'].read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `GPT-2 model`\n",
    "\n",
    "Now that our RESTful API endpoint is deployed, we can send it text to get predictions from our GPT-2 model. You can use the SageMaker Python SDK or the SageMaker Runtime API to invoke the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\\'generated_text\\': \\'\"Working with SageMaker makes machine learning \"a lot easier\" than it used to be.\\\\n\\'}]'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import boto3\n",
    "import json\n",
    "\n",
    "invoke_client = boto3.client('sagemaker-runtime')\n",
    "\n",
    "prompt = \"Working with SageMaker makes machine learning \"\n",
    "\n",
    "response = invoke_client.invoke_endpoint(EndpointName=f\"{deployment_name}-ep\",\n",
    "                              TargetModel='gptmodel.tar.gz', \n",
    "                              Body=json.dumps(prompt),\n",
    "                              ContentType='text/csv')\n",
    "\n",
    "response['Body'].read().decode('utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Dynamically deploying models and Updating a model to the endpoint </h2> \n",
    "\n",
    "To dynamically deploy a model and update a model, you would follow the same approach as above and add it as a new model. For example, if you have retrained the bertmodel.tar.gz model and wanted to start invoking it, you would upload the updated model artifacts behind the following S3 prefix with a new name such as bertmodel_v2.tar.gz, and then change the TargetModel field to invoke bertmodel_v2.tar.gz instead of bertmodel.tar.gz.\n",
    "\n",
    "multimodels_path = 's3://sagemaker-us-east-2-(account-id)/huggingface-multimodel-deploy/models/'\n",
    "\n",
    "You should avoid overwriting model artifacts in Amazon S3, because the old version of the model might still be loaded in the endpoint's running container(s) or on the storage volume of instances on the endpoint: This would lead invocations to still use the old version of the model.\n",
    "\n",
    "Alternatively, you could stop the endpoint and re-deploy a fresh set of models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Delete the Multi-Model Endpoint </h2> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_client.delete_model(ModelName=deployment_name)\n",
    "sm_client.delete_endpoint_config(EndpointConfigName=f\"{deployment_name}-epc\")\n",
    "sm_client.delete_endpoint(EndpointName=f\"{deployment_name}-ep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 style = \"font-size:35px; font-family:Garamond ; font-weight : normal; background-color: #FFC0CB; color :#00008B   ; text-align: center; border-radius: 5px 5px; padding: 5px\"> Conclusion </h2> \n",
    "\n",
    "We successfully deployed two Hugging Face Transformers to Amazon SageMaker for inference using the Multi-Model Endpoint. Multi-Model Endpoints are a great option to optimize compute utilization and costs for your models. Especially when you have independent inference workloads due to use-case differences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks for reading! If you have any questions, feel free to contact me."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}