notebooks/112-pytorch-post-training-quantization-nncf/112-pytorch-post-training-quantization-nncf.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Post-Training Quantization of PyTorch models with NNCF\n",
    "\n",
    "The goal of this tutorial is to demonstrate how to use the NNCF (Neural Network Compression Framework) 8-bit quantization in post-training mode (without the fine-tuning pipeline) to optimize a PyTorch model for the high-speed inference via OpenVINO™ Toolkit. The optimization process contains the following steps:\n",
    "\n",
    "1. Evaluate the original model.\n",
    "2. Transform the original model to a quantized one.\n",
    "3. Export optimized and original models to ONNX and then to OpenVINO IR.\n",
    "4. Compare performance of the obtained `FP32` and `INT8` models.\n",
    "\n",
    "This tutorial uses a ResNet-50 model, pre-trained on Tiny ImageNet, which contains 100000 images of 200 classes (500 for each class) downsized to 64×64 colored images. The tutorial will demonstrate that only a tiny part of the dataset is needed for the post-training quantization, not demanding the fine-tuning of the model.\n",
    "\n",
    "\n",
    "> **NOTE**: This notebook requires that a C++ compiler is accessible on the default binary search path of the OS you are running the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# On Windows, this script adds the directory that contains cl.exe to the PATH to enable PyTorch to find the\n",
    "# required C++ tools. This code assumes that Visual Studio 2019 is installed in the default\n",
    "# directory. If you have a different C++ compiler, add the correct path to os.environ[\"PATH\"]\n",
    "# directly.\n",
    "\n",
    "# Adding the path to os.environ[\"LIB\"] is not always required - it depends on the system configuration.\n",
    "\n",
    "import sys\n",
    "\n",
    "if sys.platform == \"win32\":\n",
    "    import distutils.command.build_ext\n",
    "    import os\n",
    "    from pathlib import Path\n",
    "\n",
    "    VS_INSTALL_DIR = r\"C:/Program Files (x86)/Microsoft Visual Studio\"\n",
    "    cl_paths = sorted(list(Path(VS_INSTALL_DIR).glob(\"**/Hostx86/x64/cl.exe\")))\n",
    "    if len(cl_paths) == 0:\n",
    "        raise ValueError(\n",
    "            \"Cannot find Visual Studio. This notebook requires C++. If you installed \"\n",
    "            \"a C++ compiler, please add the directory that contains cl.exe to \"\n",
    "            \"`os.environ['PATH']`\"\n",
    "        )\n",
    "    else:\n",
    "        # If multiple versions of MSVC are installed, get the most recent one.\n",
    "        cl_path = cl_paths[-1]\n",
    "        vs_dir = str(cl_path.parent)\n",
    "        os.environ[\"PATH\"] += f\"{os.pathsep}{vs_dir}\"\n",
    "        # The code for finding the library dirs is from\n",
    "        # https://stackoverflow.com/questions/47423246/get-pythons-lib-path\n",
    "        d = distutils.core.Distribution()\n",
    "        b = distutils.command.build_ext.build_ext(d)\n",
    "        b.finalize_options()\n",
    "        os.environ[\"LIB\"] = os.pathsep.join(b.library_dirs)\n",
    "        print(f\"Added {vs_dir} to PATH\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing model files\n",
    "\n",
    "> **NOTE**: All NNCF logging messages below ERROR level (INFO and WARNING) are disabled to simplify the tutorial. For production use, it is recommended to enable logging by removing ```set_log_level(logging.ERROR)```.\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "import warnings\n",
    "import zipfile\n",
    "from pathlib import Path\n",
    "from typing import List, Tuple\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.utils.data\n",
    "import torchvision.datasets as datasets\n",
    "import torchvision.models as models\n",
    "import torchvision.transforms as transforms\n",
    "from nncf import NNCFConfig  # Important - should be imported directly after torch\n",
    "from nncf.common.logging.logger import set_log_level\n",
    "\n",
    "set_log_level(logging.ERROR)  # Disables all NNCF info and warning messages\n",
    "from nncf.torch import create_compressed_model, register_default_init_args\n",
    "from openvino.runtime import Core\n",
    "from torch.jit import TracerWarning\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "print(f\"Using {device} device\")\n",
    "\n",
    "MODEL_DIR = Path(\"model\")\n",
    "OUTPUT_DIR = Path(\"output\")\n",
    "BASE_MODEL_NAME = \"resnet50\"\n",
    "IMAGE_SIZE = [64, 64]\n",
    "\n",
    "OUTPUT_DIR.mkdir(exist_ok=True)\n",
    "MODEL_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "# Paths where PyTorch, ONNX and OpenVINO IR models will be stored.\n",
    "fp32_checkpoint_filename = Path(BASE_MODEL_NAME + \"_fp32\").with_suffix(\".pth\")\n",
    "fp32_onnx_path = Path(OUTPUT_DIR / (BASE_MODEL_NAME + \"_fp32\")).with_suffix(\".onnx\")\n",
    "fp32_ir_path = fp32_onnx_path.with_suffix(\".xml\")\n",
    "int8_onnx_path = Path(OUTPUT_DIR / (BASE_MODEL_NAME + \"_int8\")).with_suffix(\".onnx\")\n",
    "int8_ir_path = int8_onnx_path.with_suffix(\".xml\")\n",
    "\n",
    "\n",
    "fp32_pth_url = \"https://storage.openvinotoolkit.org/repositories/nncf/openvino_notebook_ckpts/304_resnet50_fp32.pth\"\n",
    "download_file(fp32_pth_url, directory=MODEL_DIR, filename=fp32_checkpoint_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download and Prepare Tiny ImageNet dataset\n",
    "\n",
    "* 100k images of shape 3x64x64,\n",
    "* 200 different classes: snake, spider, cat, truck, grasshopper, gull, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_tiny_imagenet_200(\n",
    "    output_dir: Path,\n",
    "    url: str = \"http://cs231n.stanford.edu/tiny-imagenet-200.zip\",\n",
    "    tarname: str = \"tiny-imagenet-200.zip\",\n",
    "):\n",
    "    archive_path = output_dir / tarname\n",
    "    download_file(url, directory=output_dir, filename=tarname)\n",
    "    zip_ref = zipfile.ZipFile(archive_path, \"r\")\n",
    "    zip_ref.extractall(path=output_dir)\n",
    "    zip_ref.close()\n",
    "    print(f\"Successfully downloaded and extracted dataset to: {output_dir}\")\n",
    "\n",
    "\n",
    "def create_validation_dir(dataset_dir: Path):\n",
    "    VALID_DIR = dataset_dir / \"val\"\n",
    "    val_img_dir = VALID_DIR / \"images\"\n",
    "\n",
    "    fp = open(VALID_DIR / \"val_annotations.txt\", \"r\")\n",
    "    data = fp.readlines()\n",
    "\n",
    "    val_img_dict = {}\n",
    "    for line in data:\n",
    "        words = line.split(\"\\t\")\n",
    "        val_img_dict[words[0]] = words[1]\n",
    "    fp.close()\n",
    "\n",
    "    for img, folder in val_img_dict.items():\n",
    "        newpath = val_img_dir / folder\n",
    "        if not newpath.exists():\n",
    "            os.makedirs(newpath)\n",
    "        if (val_img_dir / img).exists():\n",
    "            os.rename(val_img_dir / img, newpath / img)\n",
    "\n",
    "\n",
    "DATASET_DIR = OUTPUT_DIR / \"tiny-imagenet-200\"\n",
    "if not DATASET_DIR.exists():\n",
    "    download_tiny_imagenet_200(OUTPUT_DIR)\n",
    "    create_validation_dir(DATASET_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helpers classes and functions\n",
    "The code below will help to count accuracy and visualize validation process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AverageMeter(object):\n",
    "    \"\"\"Computes and stores the average and current value\"\"\"\n",
    "\n",
    "    def __init__(self, name: str, fmt: str = \":f\"):\n",
    "        self.name = name\n",
    "        self.fmt = fmt\n",
    "        self.val = 0\n",
    "        self.avg = 0\n",
    "        self.sum = 0\n",
    "        self.count = 0\n",
    "\n",
    "    def update(self, val: float, n: int = 1):\n",
    "        self.val = val\n",
    "        self.sum += val * n\n",
    "        self.count += n\n",
    "        self.avg = self.sum / self.count\n",
    "\n",
    "    def __str__(self):\n",
    "        fmtstr = \"{name} {val\" + self.fmt + \"} ({avg\" + self.fmt + \"})\"\n",
    "        return fmtstr.format(**self.__dict__)\n",
    "\n",
    "\n",
    "class ProgressMeter(object):\n",
    "    \"\"\"Displays the progress of validation process\"\"\"\n",
    "\n",
    "    def __init__(self, num_batches: int, meters: List[AverageMeter], prefix: str = \"\"):\n",
    "        self.batch_fmtstr = self._get_batch_fmtstr(num_batches)\n",
    "        self.meters = meters\n",
    "        self.prefix = prefix\n",
    "\n",
    "    def display(self, batch: int):\n",
    "        entries = [self.prefix + self.batch_fmtstr.format(batch)]\n",
    "        entries += [str(meter) for meter in self.meters]\n",
    "        print(\"\\t\".join(entries))\n",
    "\n",
    "    def _get_batch_fmtstr(self, num_batches: int):\n",
    "        num_digits = len(str(num_batches // 1))\n",
    "        fmt = \"{:\" + str(num_digits) + \"d}\"\n",
    "        return \"[\" + fmt + \"/\" + fmt.format(num_batches) + \"]\"\n",
    "\n",
    "\n",
    "def accuracy(output: torch.Tensor, target: torch.Tensor, topk: Tuple[int] = (1,)):\n",
    "    \"\"\"Computes the accuracy over the k top predictions for the specified values of k\"\"\"\n",
    "    with torch.no_grad():\n",
    "        maxk = max(topk)\n",
    "        batch_size = target.size(0)\n",
    "\n",
    "        _, pred = output.topk(maxk, 1, True, True)\n",
    "        pred = pred.t()\n",
    "        correct = pred.eq(target.view(1, -1).expand_as(pred))\n",
    "\n",
    "        res = []\n",
    "        for k in topk:\n",
    "            correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True)\n",
    "            res.append(correct_k.mul_(100.0 / batch_size))\n",
    "        return res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate(val_loader: torch.utils.data.DataLoader, model: torch.nn.Module):\n",
    "    \"\"\"Compute the metrics using data from val_loader for the model\"\"\"\n",
    "    batch_time = AverageMeter(\"Time\", \":3.3f\")\n",
    "    top1 = AverageMeter(\"Acc@1\", \":2.2f\")\n",
    "    top5 = AverageMeter(\"Acc@5\", \":2.2f\")\n",
    "    progress = ProgressMeter(len(val_loader), [batch_time, top1, top5], prefix=\"Test: \")\n",
    "\n",
    "    # Switch to evaluate mode.\n",
    "    model.eval()\n",
    "    model.to(device)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        end = time.time()\n",
    "        for i, (images, target) in enumerate(val_loader):\n",
    "            images = images.to(device)\n",
    "            target = target.to(device)\n",
    "\n",
    "            # Compute the output.\n",
    "            output = model(images)\n",
    "\n",
    "            # Measure accuracy and record loss.\n",
    "            acc1, acc5 = accuracy(output, target, topk=(1, 5))\n",
    "            top1.update(acc1[0], images.size(0))\n",
    "            top5.update(acc5[0], images.size(0))\n",
    "\n",
    "            # Measure elapsed time.\n",
    "            batch_time.update(time.time() - end)\n",
    "            end = time.time()\n",
    "\n",
    "            print_frequency = 10\n",
    "            if i % print_frequency == 0:\n",
    "                progress.display(i)\n",
    "\n",
    "        print(\n",
    "            \" * Acc@1 {top1.avg:.3f} Acc@5 {top5.avg:.3f}\".format(top1=top1, top5=top5)\n",
    "        )\n",
    "    return top1.avg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create and load original uncompressed model\n",
    "\n",
    "ResNet-50 from the [torchivision repository](https://github.com/pytorch/vision) is pre-trained on ImageNet with more prediction classes than Tiny ImageNet, so the model is adjusted by swapping the last FC layer to one with fewer output values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def create_model(model_path: Path):\n",
    "    \"\"\"Creates the ResNet-50 model and loads the pretrained weights\"\"\"\n",
    "    model = models.resnet50()\n",
    "    # Update the last FC layer for Tiny ImageNet number of classes.\n",
    "    NUM_CLASSES = 200\n",
    "    model.fc = nn.Linear(in_features=2048, out_features=NUM_CLASSES, bias=True)\n",
    "    model.to(device)\n",
    "    if model_path.exists():\n",
    "        checkpoint = torch.load(str(model_path), map_location=\"cpu\")\n",
    "        model.load_state_dict(checkpoint[\"state_dict\"], strict=True)\n",
    "    else:\n",
    "        raise RuntimeError(\"There is no checkpoint to load\")\n",
    "    return model\n",
    "\n",
    "\n",
    "model = create_model(MODEL_DIR / fp32_checkpoint_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create train and validation dataloaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "val_dataset,": "torch.utils.data.Subset(val_dataset, range(50)),"
    }
   },
   "outputs": [],
   "source": [
    "def create_dataloaders(batch_size: int = 128):\n",
    "    \"\"\"Creates train dataloader that is used for quantization initialization and validation dataloader for computing the model accruacy\"\"\"\n",
    "    train_dir = DATASET_DIR / \"train\"\n",
    "    val_dir = DATASET_DIR / \"val\" / \"images\"\n",
    "    normalize = transforms.Normalize(\n",
    "        mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]\n",
    "    )\n",
    "    train_dataset = datasets.ImageFolder(\n",
    "        train_dir,\n",
    "        transforms.Compose(\n",
    "            [\n",
    "                transforms.Resize(IMAGE_SIZE),\n",
    "                transforms.ToTensor(),\n",
    "                normalize,\n",
    "            ]\n",
    "        ),\n",
    "    )\n",
    "    val_dataset = datasets.ImageFolder(\n",
    "        val_dir,\n",
    "        transforms.Compose(\n",
    "            [transforms.Resize(IMAGE_SIZE), transforms.ToTensor(), normalize]\n",
    "        ),\n",
    "    )\n",
    "\n",
    "    train_loader = torch.utils.data.DataLoader(\n",
    "        train_dataset,\n",
    "        batch_size=batch_size,\n",
    "        shuffle=True,\n",
    "        num_workers=0,\n",
    "        pin_memory=True,\n",
    "        sampler=None,\n",
    "    )\n",
    "\n",
    "    val_loader = torch.utils.data.DataLoader(\n",
    "        val_dataset,\n",
    "        batch_size=batch_size,\n",
    "        shuffle=False,\n",
    "        num_workers=0,\n",
    "        pin_memory=True,\n",
    "    )\n",
    "    return train_loader, val_loader\n",
    "\n",
    "\n",
    "train_loader, val_loader = create_dataloaders()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the preparation of the training and validation pipelines, and the model files within this notebook is done, it is time to perform actual post-training quantization with NNCF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I. Evaluate the loaded model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "acc1 = validate(val_loader, model)\n",
    "print(f\"Test accuracy of FP32 model: {acc1:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Export the `FP32` model to the ONNX, which is supported by OpenVINO Toolkit, to benchmark it in comparison with the `INT8` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy_input = torch.randn(1, 3, *IMAGE_SIZE).to(device)\n",
    "torch.onnx.export(model, dummy_input, fp32_onnx_path, opset_version=10)\n",
    "print(f\"FP32 ONNX model was exported to {fp32_onnx_path}.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II. Create and initialize quantization\n",
    "NNCF enables post-training quantization by adding the quantization layers into the model graph and then using a subset of the training dataset to initialize the parameters of these additional quantization layers. The framework is designed so that modifications to your original training code are minor. Quantization is the simplest scenario and requires only three modifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Configure the NNCF parameters to specify compression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "nncf_config_dict = {\n",
    "    \"input_info\": {\"sample_size\": [1, 3, *IMAGE_SIZE]},\n",
    "    \"log_dir\": str(OUTPUT_DIR),\n",
    "    \"compression\": {\n",
    "        \"algorithm\": \"quantization\",\n",
    "        \"initializer\": {\n",
    "            \"range\": {\"num_init_samples\": 15000},\n",
    "            \"batchnorm_adaptation\": {\"num_bn_adaptation_samples\": 4000},\n",
    "        },\n",
    "    },\n",
    "}\n",
    "\n",
    "nncf_config = NNCFConfig.from_dict(nncf_config_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Provide a data loader to initialize the values of quantization ranges and determine which activation should be signed or unsigned from the collected statistics, using a given number of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "train_loader": "val_loader"
    }
   },
   "outputs": [],
   "source": [
    "nncf_config = register_default_init_args(nncf_config, train_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Create a quantized model from a pre-trained `FP32` model and a configuration object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compression_ctrl, model = create_compressed_model(model, nncf_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Evaluate the new model on the validation set after initialization of quantization. The accuracy should be close to the accuracy of the floating-point `FP32` model for a simple case like the one being demonstrated now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc1 = validate(val_loader, model)\n",
    "print(f\"Accuracy of initialized INT8 model: {acc1:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Export an `INT8` model to ONNX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "warnings.filterwarnings(\"ignore\", category=TracerWarning)  # Ignore export warnings\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
    "compression_ctrl.export_model(int8_onnx_path)\n",
    "print(f\"INT8 ONNX model exported to {int8_onnx_path}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III. Convert ONNX models to OpenVINO Intermediate Representation (OpenVINO IR)\n",
    "\n",
    "Use Model Optimizer to convert the ONNX model to OpenVINO IR, with `FP16` precision. The models are saved to the current directory. Then, add the mean values to the model and scale the output with the standard deviation by `--mean_values` and `--scale_values` arguments. It is not necessary to normalize input data before propagating it through the network with these options.\n",
    "\n",
    "For more information about Model Optimizer, refer to the [Model Optimizer Developer Guide](https://docs.openvino.ai/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html).\n",
    "\n",
    "Executing the following command may take a while. There may be some errors or warnings in the output. When Model Optimizer converts the model to OpenVINO IR successfully, the last lines of the output will include: `[ SUCCESS ] Generated IR version 11 model`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_shape = [1, 3, *IMAGE_SIZE]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not fp32_ir_path.exists():\n",
    "    !mo --input_model \"$fp32_onnx_path\" --input_shape \"$input_shape\" --mean_values \"[123.675, 116.28 , 103.53]\" --scale_values \"[58.395, 57.12 , 57.375]\" --compress_to_fp16 --output_dir \"$OUTPUT_DIR\"\n",
    "    assert fp32_ir_path.exists(), \"The OpenVINO IR of FP32 model wasn't created\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not int8_ir_path.exists():\n",
    "    !mo --input_model \"$int8_onnx_path\" --input_shape \"$input_shape\" --mean_values \"[123.675, 116.28 , 103.53]\" --scale_values \"[58.395, 57.12 , 57.375]\" --compress_to_fp16 --output_dir \"$OUTPUT_DIR\"\n",
    "    assert int8_ir_path.exists(), \"The OpenVINO IR of INT8 model wasn't created\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV. Compare perfomance of INT8 model and FP32 model in OpenVINO\n",
    "\n",
    "Finally, measure the inference performance of the `FP32` and `INT8` models, using [Benchmark Tool](https://docs.openvino.ai/latest/openvino_inference_engine_tools_benchmark_tool_README.html) - an inference performance measurement tool in OpenVINO. By default, Benchmark Tool runs inference for 60 seconds in asynchronous mode on CPU. It returns inference speed as latency (milliseconds per image) and throughput (frames per second) values.\n",
    "\n",
    "> **NOTE**: This notebook runs benchmark_app for 15 seconds to give a quick indication of performance. For more accurate performance, it is recommended to run benchmark_app in a terminal/command prompt after closing other applications. Run `benchmark_app -m model.xml -d CPU` to benchmark async inference on CPU for one minute. Change CPU to GPU to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command-line options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "-t 15": "-t 3"
    }
   },
   "outputs": [],
   "source": [
    "def parse_benchmark_output(benchmark_output: str):\n",
    "    \"\"\"Prints the output from benchmark_app in human-readable format\"\"\"\n",
    "    parsed_output = [line for line in benchmark_output if 'FPS' in line]\n",
    "    print(*parsed_output, sep='\\n')\n",
    "\n",
    "\n",
    "print('Benchmark FP32 model (OpenVINO IR)')\n",
    "benchmark_output = ! benchmark_app -m \"$fp32_ir_path\" -d CPU -api async -t 15\n",
    "parse_benchmark_output(benchmark_output)\n",
    "\n",
    "print('Benchmark INT8 model (OpenVINO IR)')\n",
    "benchmark_output = ! benchmark_app -m \"$int8_ir_path\" -d CPU -api async -t 15\n",
    "parse_benchmark_output(benchmark_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show CPU Information for reference:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ie = Core()\n",
    "ie.get_property(\"CPU\", \"FULL_DEVICE_NAME\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "K5HPrY_d-7cV",
    "E01dMaR2_AFL",
    "qMnYsGo9_MA8",
    "L0tH9KdwtHhV"
   ],
   "name": "NNCF Quantization PyTorch Demo (tiny-imagenet/resnet-50)",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}