notebooks/105-language-quantize-bert/105-language-quantize-bert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cacbe6b4",
   "metadata": {
    "id": "rQc-wXjqrEuR"
   },
   "source": [
    "# Quantize NLP models with Post-Training Optimization Tool ​in OpenVINO™\n",
    "This tutorial demonstrates how to apply `INT8` quantization to the Natural Language Processing model known as [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)), using the [Post-Training Optimization Tool API](https://docs.openvino.ai/latest/pot_compression_api_README.html) (part of the [OpenVINO Toolkit](https://docs.openvino.ai/)). A fine-tuned [HuggingFace BERT](https://huggingface.co/transformers/model_doc/bert.html) [PyTorch](https://pytorch.org/) model, trained on the [Microsoft Research Paraphrase Corpus (MRPC)](https://www.microsoft.com/en-us/download/details.aspx?id=52398), will be used. The tutorial is designed to be extendable to custom models and datasets. It consists of the following steps:\n",
    "\n",
    "- Download and prepare the BERT model and MRPC dataset.\n",
    "- Define data loading and accuracy validation functionality.\n",
    "- Prepare the model for quantization.\n",
    "- Run optimization pipeline.\n",
    "- Load and test quantized model.\n",
    "- Compare the performance of the original, converted and quantized models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d6b41e6-132b-40da-b3b9-91bacba29e31",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "771388d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import time\n",
    "import warnings\n",
    "from pathlib import Path\n",
    "from zipfile import ZipFile\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "from addict import Dict\n",
    "from openvino import runtime as ov\n",
    "from openvino.tools.pot import DataLoader as POTDataLoader\n",
    "from openvino.tools.pot import Metric, IEEngine, load_model, save_model, compress_model_weights, create_pipeline\n",
    "from torch.utils.data import TensorDataset\n",
    "from transformers import BertForSequenceClassification, BertTokenizer\n",
    "from transformers import (\n",
    "    glue_convert_examples_to_features as convert_examples_to_features,\n",
    ")\n",
    "from transformers import glue_output_modes as output_modes\n",
    "from transformers import glue_processors as processors\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e66896-d439-4065-868a-65b44d31525a",
   "metadata": {},
   "source": [
    "## Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "284e9a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the data and model directories, source URL and the filename of the model.\n",
    "DATA_DIR = \"data\"\n",
    "MODEL_DIR = \"model\"\n",
    "MODEL_LINK = \"https://download.pytorch.org/tutorial/MRPC.zip\"\n",
    "FILE_NAME = MODEL_LINK.split(\"/\")[-1]\n",
    "\n",
    "os.makedirs(DATA_DIR, exist_ok=True)\n",
    "os.makedirs(MODEL_DIR, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44dc335d",
   "metadata": {
    "id": "YytHDzLE0uOJ",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Prepare the Model\n",
    "Perform the following:\n",
    "- Download and unpack pre-trained BERT model for MRPC by PyTorch.\n",
    "- Convert the model to the ONNX.\n",
    "- Run Model Optimizer to convert the model from the ONNX representation to the OpenVINO Intermediate Representation (OpenVINO IR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be9fc64c",
   "metadata": {
    "id": "f7i6dWUmhloy"
   },
   "outputs": [],
   "source": [
    "download_file(MODEL_LINK, directory=MODEL_DIR, show_progress=True)\n",
    "with ZipFile(f\"{MODEL_DIR}/{FILE_NAME}\", \"r\") as zip_ref:\n",
    "    zip_ref.extractall(MODEL_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dca2fa0",
   "metadata": {
    "id": "ehX7F6KB0uPu"
   },
   "source": [
    "Import all dependencies to load the original PyTorch model and convert it to the ONNX representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb2f6d66",
   "metadata": {
    "id": "r5as0_Yg0uQX",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 1\n",
    "MAX_SEQ_LENGTH = 128\n",
    "\n",
    "\n",
    "def export_model_to_onnx(model, path):\n",
    "    with torch.no_grad():\n",
    "        default_input = torch.ones(1, MAX_SEQ_LENGTH, dtype=torch.int64)\n",
    "        inputs = {\n",
    "            \"input_ids\": default_input,\n",
    "            \"attention_mask\": default_input,\n",
    "            \"token_type_ids\": default_input,\n",
    "        }\n",
    "        symbolic_names = {0: \"batch_size\", 1: \"max_seq_len\"}\n",
    "        torch.onnx.export(\n",
    "            model,\n",
    "            (inputs[\"input_ids\"], inputs[\"attention_mask\"], inputs[\"token_type_ids\"]),\n",
    "            path,\n",
    "            opset_version=11,\n",
    "            do_constant_folding=True,\n",
    "            input_names=[\"input_ids\", \"input_mask\", \"segment_ids\"],\n",
    "            output_names=[\"output\"],\n",
    "            dynamic_axes={\n",
    "                \"input_ids\": symbolic_names,\n",
    "                \"input_mask\": symbolic_names,\n",
    "                \"segment_ids\": symbolic_names,\n",
    "            },\n",
    "        )\n",
    "        print(\"ONNX model saved to {}\".format(path))\n",
    "\n",
    "\n",
    "torch_model = BertForSequenceClassification.from_pretrained(os.path.join(MODEL_DIR, \"MRPC\"))\n",
    "onnx_model_path = Path(MODEL_DIR) / \"bert_mrpc.onnx\"\n",
    "if not onnx_model_path.exists():\n",
    "    export_model_to_onnx(torch_model, onnx_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6911365",
   "metadata": {
    "id": "sNWDAGGd0uRt"
   },
   "source": [
    "## Convert the ONNX Model to OpenVINO IR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20aeea80",
   "metadata": {
    "id": "-6P0c_960uR5",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ir_model_xml = onnx_model_path.with_suffix(\".xml\")\n",
    "ir_model_bin = onnx_model_path.with_suffix(\".bin\")\n",
    "\n",
    "# Convert the ONNX model to OpenVINO IR FP32.\n",
    "if not ir_model_xml.exists():\n",
    "    !mo --input_model $onnx_model_path --output_dir $MODEL_DIR --model_name $ir_model_xml.stem --input input_ids,input_mask,segment_ids --input_shape \"[1,128],[1,128],[1,128]\" --output output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "635f4b0d",
   "metadata": {
    "id": "LBbY7c4NsHzT"
   },
   "source": [
    "## Prepare MRPC Task Dataset\n",
    "\n",
    "To run this tutorial, you will need to download the General Language Understanding Evaluation  (GLUE) data for the MRPC task from HuggingFace. Use the code below to download a script that fetches the MRPC dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43070514",
   "metadata": {
    "id": "NN-qRME1a-Sm"
   },
   "outputs": [],
   "source": [
    "download_file(\n",
    "    \"https://raw.githubusercontent.com/huggingface/transformers/f98ef14d161d7bcdc9808b5ec399981481411cc1/utils/download_glue_data.py\",\n",
    "    show_progress=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af48e548",
   "metadata": {},
   "outputs": [],
   "source": [
    "from download_glue_data import format_mrpc\n",
    "\n",
    "format_mrpc(DATA_DIR, \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da802acd",
   "metadata": {
    "id": "E5hsOsj-0uSc"
   },
   "source": [
    "## Define DataLoader for POT\n",
    "In this step, you define `DataLoader` based on POT API. It will be used to collect statistics for quantization and run model evaluation. \n",
    "Use helper functions from the HuggingFace Transformers to do the data preprocessing. It takes raw text data and encodes sentences and words, producing three model inputs. \n",
    "For more details about the data preprocessing and tokenization, refer to this [description](https://medium.com/@dhartidhami/understanding-bert-word-embeddings-7dc4d2ea54ca)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2a378cb",
   "metadata": {
    "id": "6xnl2PhM0uSn",
    "tags": [],
    "test_replace": {
     "return len(self.dataset)": "return 30"
    }
   },
   "outputs": [],
   "source": [
    "class MRPCDataLoader(POTDataLoader):\n",
    "    # Required methods\n",
    "    def __init__(self, config):\n",
    "        \"\"\"Constructor\n",
    "        :param config: data loader specific config\n",
    "        \"\"\"\n",
    "        if not isinstance(config, Dict):\n",
    "            config = Dict(config)\n",
    "        super().__init__(config)\n",
    "        self._task = config[\"task\"].lower()\n",
    "        self._model_dir = config[\"model_dir\"]\n",
    "        self._data_dir = config[\"data_source\"]\n",
    "        self._batch_size = config[\"batch_size\"]\n",
    "        self._max_length = config[\"max_length\"]\n",
    "        self.examples = []\n",
    "        self._prepare_dataset()\n",
    "\n",
    "    def __len__(self):\n",
    "        \"\"\"Returns size of the dataset\"\"\"\n",
    "        return len(self.dataset)\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        \"\"\"\n",
    "        Returns annotation, data and metadata at the specified index.\n",
    "        Possible formats:\n",
    "        (index, annotation), data\n",
    "        (index, annotation), data, metadata\n",
    "        \"\"\"\n",
    "        if index >= len(self):\n",
    "            raise IndexError\n",
    "\n",
    "        batch = self.dataset[index]\n",
    "        batch = tuple(t.detach().cpu().numpy() for t in batch)\n",
    "        inputs = {\"input_ids\": batch[0], \"input_mask\": batch[1], \"segment_ids\": batch[2]}\n",
    "        labels = batch[3]\n",
    "        return (index, labels), inputs\n",
    "\n",
    "    # Methods specific to the current implementation\n",
    "    def _prepare_dataset(self):\n",
    "        \"\"\"Prepare dataset\"\"\"\n",
    "        tokenizer = BertTokenizer.from_pretrained(self._model_dir, do_lower_case=True)\n",
    "        processor = processors[self._task]()\n",
    "        output_mode = output_modes[self._task]\n",
    "        label_list = processor.get_labels()\n",
    "        examples = processor.get_dev_examples(self._data_dir)\n",
    "        features = convert_examples_to_features(\n",
    "            examples,\n",
    "            tokenizer,\n",
    "            label_list=label_list,\n",
    "            max_length=self._max_length,\n",
    "            output_mode=output_mode,\n",
    "        )\n",
    "        all_input_ids = torch.unsqueeze(torch.tensor([f.input_ids for f in features], dtype=torch.long), 1)\n",
    "        all_attention_mask = torch.unsqueeze(torch.tensor([f.attention_mask for f in features], dtype=torch.long), 1)\n",
    "        all_token_type_ids = torch.unsqueeze(torch.tensor([f.token_type_ids for f in features], dtype=torch.long), 1)\n",
    "        all_labels = torch.unsqueeze(torch.tensor([f.label for f in features], dtype=torch.long), 1)\n",
    "        self.dataset = TensorDataset(\n",
    "            all_input_ids, all_attention_mask, all_token_type_ids, all_labels\n",
    "        )\n",
    "        self.examples = examples"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "71c867d3",
   "metadata": {
    "id": "Re9-YhbBddh3"
   },
   "source": [
    "## Define Accuracy Metric Calculation\n",
    "In this step, the `Metric` interface for MRPC task metrics is implemented. It is used for validating the accuracy of the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f4e7fbc",
   "metadata": {
    "id": "GB8L492ztZEC"
   },
   "outputs": [],
   "source": [
    "class Accuracy(Metric):\n",
    "\n",
    "    # Required methods\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self._name = \"Accuracy\"\n",
    "        self._matches = []\n",
    "\n",
    "    @property\n",
    "    def value(self):\n",
    "        \"\"\"Returns accuracy metric value for the last model output.\"\"\"\n",
    "        return {self._name: self._matches[-1]}\n",
    "\n",
    "    @property\n",
    "    def avg_value(self):\n",
    "        \"\"\"Returns accuracy metric value for all model outputs.\"\"\"\n",
    "        return {self._name: np.ravel(self._matches).mean()}\n",
    "\n",
    "    def update(self, output, target):\n",
    "        \"\"\"\n",
    "        Updates prediction matches.\n",
    "\n",
    "        :param output: model output\n",
    "        :param target: annotations\n",
    "        \"\"\"\n",
    "        if len(output) > 1:\n",
    "            raise Exception(\n",
    "                \"The accuracy metric cannot be calculated \" \"for a model with multiple outputs\"\n",
    "            )\n",
    "        output = np.argmax(output)\n",
    "        match = output == target[0]\n",
    "        self._matches.append(match)\n",
    "\n",
    "    def reset(self):\n",
    "        \"\"\"\n",
    "        Resets collected matches\n",
    "        \"\"\"\n",
    "        self._matches = []\n",
    "\n",
    "    def get_attributes(self):\n",
    "        \"\"\"\n",
    "        Returns a dictionary of metric attributes {metric_name: {attribute_name: value}}.\n",
    "        Required attributes: 'direction': 'higher-better' or 'higher-worse'\n",
    "                             'type': metric type\n",
    "        \"\"\"\n",
    "        return {self._name: {\"direction\": \"higher-better\", \"type\": \"accuracy\"}}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bbbca4a",
   "metadata": {
    "id": "CclWk-fVd9Wi"
   },
   "source": [
    "## Run Quantization Pipeline\n",
    "Define a configuration for the quantization pipeline and run it. Keep in mind that built-in `IEEngine` implementation of `Engine` interface from the POT API for model inference is used here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16457bd4",
   "metadata": {
    "id": "PiAvrwo0tr6Z",
    "tags": [],
    "test_replace": {
     "\"stat_subset_size\": 250,": "\"stat_subset_size\": 25,"
    }
   },
   "outputs": [],
   "source": [
    "warnings.filterwarnings(\"ignore\")  # Suppress accuracychecker warnings.\n",
    "\n",
    "model_config = Dict({\"model_name\": \"bert_mrpc\", \"model\": ir_model_xml, \"weights\": ir_model_bin})\n",
    "engine_config = Dict({\"device\": \"CPU\"})\n",
    "dataset_config = {\n",
    "    \"task\": \"mrpc\",\n",
    "    \"data_source\": os.path.join(DATA_DIR, \"MRPC\"),\n",
    "    \"model_dir\": os.path.join(MODEL_DIR, \"MRPC\"),\n",
    "    \"batch_size\": BATCH_SIZE,\n",
    "    \"max_length\": MAX_SEQ_LENGTH,\n",
    "}\n",
    "algorithms = [\n",
    "    {\n",
    "        \"name\": \"DefaultQuantization\",\n",
    "        \"params\": {\n",
    "            \"target_device\": \"ANY\",\n",
    "            \"model_type\": \"transformer\",\n",
    "            \"preset\": \"performance\",\n",
    "            \"stat_subset_size\": 250,\n",
    "        },\n",
    "    }\n",
    "]\n",
    "\n",
    "\n",
    "# Step 1: Load the model.\n",
    "model = load_model(model_config=model_config)\n",
    "\n",
    "# Step 2: Initialize the data loader.\n",
    "data_loader = MRPCDataLoader(config=dataset_config)\n",
    "\n",
    "# Step 3 (Optional. Required for AccuracyAwareQuantization): Initialize the metric.\n",
    "metric = Accuracy()\n",
    "\n",
    "# Step 4: Initialize the engine for metric calculation and statistics collection.\n",
    "engine = IEEngine(config=engine_config, data_loader=data_loader, metric=metric)\n",
    "\n",
    "# Step 5: Create a pipeline of compression algorithms.\n",
    "pipeline = create_pipeline(algo_config=algorithms, engine=engine)\n",
    "\n",
    "# Step 6 (Optional): Evaluate the original model. Print the results.\n",
    "fp_results = pipeline.evaluate(model=model)\n",
    "if fp_results:\n",
    "    print(\"FP32 model results:\")\n",
    "    for name, value in fp_results.items():\n",
    "        print(f\"{name}: {value:.5f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a05b3999",
   "metadata": {
    "id": "hPj_fcDAG8xG"
   },
   "outputs": [],
   "source": [
    "# Step 7: Execute the pipeline.\n",
    "warnings.filterwarnings(\"ignore\")  # Suppress accuracychecker warnings.\n",
    "print(\n",
    "    f\"Quantizing model with {algorithms[0]['params']['preset']} preset and {algorithms[0]['name']}\"\n",
    ")\n",
    "start_time = time.perf_counter()\n",
    "compressed_model = pipeline.run(model=model)\n",
    "end_time = time.perf_counter()\n",
    "print(f\"Quantization finished in {end_time - start_time:.2f} seconds\")\n",
    "\n",
    "# Step 8 (Optional): Compress model weights to quantized precision\n",
    "#                    in order to reduce the size of the final .bin file.\n",
    "compress_model_weights(model=compressed_model)\n",
    "\n",
    "# Step 9: Save the compressed model to the desired path.\n",
    "compressed_model_paths = save_model(model=compressed_model, save_path=MODEL_DIR, model_name=\"quantized_bert_mrpc\"\n",
    ")\n",
    "compressed_model_xml = compressed_model_paths[0][\"model\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b05dc0de",
   "metadata": {
    "id": "hPj_fcDAG8xG"
   },
   "outputs": [],
   "source": [
    "# Step 10 (Optional): Evaluate the compressed model and print the results.\n",
    "int_results = pipeline.evaluate(model=compressed_model)\n",
    "\n",
    "if int_results:\n",
    "    print(\"INT8 model results:\")\n",
    "    for name, value in int_results.items():\n",
    "        print(f\"{name}: {value:.5f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c30ab44",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Load and Test OpenVINO Model\n",
    "\n",
    "To load and test converted model, perform the following:\n",
    "* Load the model and compile it for CPU.\n",
    "* Prepare the input.\n",
    "* Run the inference.\n",
    "* Get the answer from the model output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d79b1a5",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "core = ov.Core()\n",
    "\n",
    "# Read the model from files.\n",
    "model = core.read_model(model=compressed_model_xml)\n",
    "\n",
    "# Assign dynamic shapes to every input layer.\n",
    "for input_layer in model.inputs:\n",
    "    input_shape = input_layer.partial_shape\n",
    "    input_shape[1] = -1\n",
    "    model.reshape({input_layer: input_shape})\n",
    "\n",
    "# Compile the model for a specific device.\n",
    "compiled_model_int8 = core.compile_model(model=model, device_name=\"CPU\")\n",
    "\n",
    "output_layer = compiled_model_int8.outputs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef1d846e",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The Data Loader returns a pair of sentences (indicated by `sample_idx`) and the inference compares these sentences and outputs whether their meaning is the same. You can test other sentences by changing `sample_idx` to another value (from 0 to 407)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e72504b9",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sample_idx = 5\n",
    "\n",
    "sample = data_loader.examples[sample_idx]\n",
    "inputs = data_loader[sample_idx][1]\n",
    "\n",
    "result = compiled_model_int8(inputs)[output_layer]\n",
    "result = np.argmax(result)\n",
    "\n",
    "print(f\"Text 1: {sample.text_a}\")\n",
    "print(f\"Text 2: {sample.text_b}\")\n",
    "print(f\"The same meaning: {'yes' if result == 1 else 'no'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f67f6a2",
   "metadata": {
    "id": "vQACMfAUo52V",
    "tags": []
   },
   "source": [
    "## Compare Performance of the Original, Converted and Quantized Models\n",
    "\n",
    "Compare the original PyTorch model with OpenVINO converted and quantized models (`FP32`, `INT8`) to see the difference in performance. It is expressed in Sentences Per Second (SPS) measure, which is the same as Frames Per Second (FPS) for images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "734ae69a",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "model = core.read_model(model=ir_model_xml)\n",
    "\n",
    "# Assign dynamic shapes to every input layer.\n",
    "for input_layer in model.inputs:\n",
    "    input_shape = input_layer.partial_shape\n",
    "    input_shape[1] = -1\n",
    "    model.reshape({input_layer: input_shape})\n",
    "\n",
    "# Compile the model for a specific device.\n",
    "compiled_model_fp32 = core.compile_model(model=model, device_name=\"CPU\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f484fff2",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "num_samples = 50\n",
    "inputs = data_loader[0][1]\n",
    "\n",
    "with torch.no_grad():\n",
    "    start = time.perf_counter()\n",
    "    for _ in range(num_samples):\n",
    "        torch_model(torch.as_tensor(list(inputs.values())).squeeze())\n",
    "    end = time.perf_counter()\n",
    "    time_torch = end - start\n",
    "print(\n",
    "    f\"PyTorch model on CPU: {time_torch / num_samples:.3f} seconds per sentence, \"\n",
    "    f\"SPS: {num_samples / time_torch:.2f}\"\n",
    ")\n",
    "\n",
    "start = time.perf_counter()\n",
    "for _ in range(num_samples):\n",
    "    compiled_model_fp32(inputs)\n",
    "end = time.perf_counter()\n",
    "time_ir = end - start\n",
    "print(\n",
    "    f\"IR FP32 model in OpenVINO Runtime/CPU: {time_ir / num_samples:.3f} \"\n",
    "    f\"seconds per sentence, SPS: {num_samples / time_ir:.2f}\"\n",
    ")\n",
    "\n",
    "start = time.perf_counter()\n",
    "for _ in range(num_samples):\n",
    "    compiled_model_int8(inputs)\n",
    "end = time.perf_counter()\n",
    "time_ir = end - start\n",
    "print(\n",
    "    f\"OpenVINO IR INT8 model in OpenVINO Runtime/CPU: {time_ir / num_samples:.3f} \"\n",
    "    f\"seconds per sentence, SPS: {num_samples / time_ir:.2f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "add78af0",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Finally, measure the inference performance of OpenVINO `FP32` and `INT8` models. For this purpose, use [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) in OpenVINO.\n",
    "\n",
    "> **Note**: The `benchmark_app` tool is able to measure the performance of the OpenVINO Intermediate Representation (OpenVINO IR) models only. For more accurate performance, run `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app -m model.xml -d CPU` to benchmark async inference on CPU for one minute. Change `CPU` to `GPU` to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command-line options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f71b38a8",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Inference FP32 model (OpenVINO IR)\n",
    "! benchmark_app -m $ir_model_xml -d CPU -api sync"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdf41525",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": [],
    "test_replace": {
     "benchmark_app": "benchmark-app -t 5"
    }
   },
   "outputs": [],
   "source": [
    "# Inference INT8 model (OpenVINO IR)\n",
    "! benchmark_app -m $compressed_model_xml -d CPU -api sync"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13 (main, Aug 25 2022, 23:51:50) [MSC v.1916 64 bit (AMD64)]"
  },
  "vscode": {
   "interpreter": {
    "hash": "e0404472fd7b5b63117a9fa5c50283296e2708c2449c6090d2cdf8903f95897f"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}