notebooks/quantizing-model-with-accuracy-control/yolov11-quantization-with-accuracy-control.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convert and Optimize YOLOv11 with OpenVINO™\n",
    "\n",
    "The YOLOv11 algorithm developed by Ultralytics is a cutting-edge, state-of-the-art (SOTA) model that is designed to be fast, accurate, and easy to use, making it an excellent choice for a wide range of object detection, image segmentation, and image classification tasks.\n",
    "More details about its realization can be found in the original model [repository](https://github.com/ultralytics/ultralytics).\n",
    "\n",
    "This tutorial demonstrates step-by-step instructions on how to run apply quantization with accuracy control to PyTorch YOLOv11.\n",
    "The advanced quantization flow allows to apply 8-bit quantization to the model with control of accuracy metric. This is achieved by keeping the most impactful operations within the model in the original precision. The flow is based on the [Basic 8-bit quantization](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/quantizing-models-post-training/basic-quantization-flow.html) and has the following differences:\n",
    "\n",
    "- Besides the calibration dataset, a validation dataset is required to compute the accuracy metric. Both datasets can refer to the same data in the simplest case.\n",
    "- Validation function, used to compute accuracy metric is required. It can be a function that is already available in the source framework or a custom function.\n",
    "- Since accuracy validation is run several times during the quantization process, quantization with accuracy control can take more time than the Basic 8-bit quantization flow.\n",
    "- The resulted model can provide smaller performance improvement than the Basic 8-bit quantization flow because some of the operations are kept in the original precision.\n",
    "\n",
    "> **NOTE**: Currently, 8-bit quantization with accuracy control in NNCF is available only for models in OpenVINO representation.\n",
    "\n",
    "The steps for the quantization with accuracy control are described below.\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Get Pytorch model and OpenVINO IR model](#Get-Pytorch-model-and-OpenVINO-IR-model)\n",
    "    - [Define validator and data loader](#Define-validator-and-data-loader)\n",
    "    - [Prepare calibration and validation datasets](#Prepare-calibration-and-validation-datasets)\n",
    "    - [Prepare validation function](#Prepare-validation-function)\n",
    "- [Run quantization with accuracy control](#Run-quantization-with-accuracy-control)\n",
    "- [Compare Accuracy and Performance of the Original and Quantized Models](#Compare-Accuracy-and-Performance-of-the-Original-and-Quantized-Models)\n",
    "\n",
    "\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/quantizing-model-with-accuracy-control/yolov11-quantization-with-accuracy-control.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Install necessary packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%pip install -q \"openvino>=2024.0.0\"\n",
    "%pip install -q \"nncf>=2.9.0\"\n",
    "%pip install -q \"ultralytics==8.3.59\" tqdm --extra-index-url https://download.pytorch.org/whl/cpu"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Pytorch model and OpenVINO IR model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Generally, PyTorch models represent an instance of the [`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html) class, initialized by a state dictionary with model weights.\n",
    "We will use the YOLOv11 nano model (also known as `yolo11n`) pre-trained on a COCO dataset, which is available in this [repo](https://github.com/ultralytics/ultralytics). Similar steps are also applicable to other YOLOv11 models.\n",
    "Typical steps to obtain a pre-trained model:\n",
    "\n",
    "1. Create an instance of a model class.\n",
    "2. Load a checkpoint state dict, which contains the pre-trained model weights.\n",
    "\n",
    "In this case, the creators of the model provide an API that enables converting the YOLOv11 model to ONNX and then to OpenVINO IR. Therefore, we do not need to do these steps manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "from ultralytics import YOLO\n",
    "from ultralytics.cfg import get_cfg\n",
    "from ultralytics.data.utils import check_det_dataset\n",
    "from ultralytics.engine.validator import BaseValidator as Validator\n",
    "from ultralytics.utils import DEFAULT_CFG\n",
    "from ultralytics.utils import ops\n",
    "from ultralytics.utils.metrics import ConfusionMatrix\n",
    "\n",
    "ROOT = os.path.abspath(\"\")\n",
    "\n",
    "MODEL_NAME = \"yolo11n-seg\"\n",
    "\n",
    "model = YOLO(f\"{ROOT}/{MODEL_NAME}.pt\")\n",
    "args = get_cfg(cfg=DEFAULT_CFG)\n",
    "args.data = \"coco128-seg.yaml\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch the notebook utils script from the openvino_notebooks repo\n",
    "import requests\n",
    "\n",
    "r = requests.get(\n",
    "    url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    ")\n",
    "\n",
    "open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "\n",
    "from notebook_utils import download_file, device_widget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from zipfile import ZipFile\n",
    "\n",
    "from ultralytics.data.utils import DATASETS_DIR\n",
    "\n",
    "DATA_URL = \"https://www.ultralytics.com/assets/coco128-seg.zip\"\n",
    "CFG_URL = \"https://raw.githubusercontent.com/ultralytics/ultralytics/v8.1.0/ultralytics/cfg/datasets/coco.yaml\"\n",
    "\n",
    "OUT_DIR = DATASETS_DIR\n",
    "\n",
    "DATA_PATH = OUT_DIR / \"coco128-seg.zip\"\n",
    "CFG_PATH = OUT_DIR / \"coco128-seg.yaml\"\n",
    "\n",
    "download_file(DATA_URL, DATA_PATH.name, DATA_PATH.parent)\n",
    "download_file(CFG_URL, CFG_PATH.name, CFG_PATH.parent)\n",
    "\n",
    "if not (OUT_DIR / \"coco128/labels\").exists():\n",
    "    with ZipFile(DATA_PATH, \"r\") as zip_ref:\n",
    "        zip_ref.extractall(OUT_DIR)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ultralytics 8.3.0 🚀 Python-3.10.12 torch-2.5.1+cpu CPU (Intel Core(TM) i9-10980XE 3.00GHz)\n",
      "YOLO11n-seg summary (fused): 265 layers, 2,868,664 parameters, 0 gradients, 10.4 GFLOPs\n",
      "\n",
      "\u001b[34m\u001b[1mPyTorch:\u001b[0m starting from '/home/maleksandr/test_notebooks/yolo/openvino_notebooks/notebooks/quantizing-model-with-accuracy-control/yolo11n-seg.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) ((1, 116, 8400), (1, 32, 160, 160)) (5.9 MB)\n",
      "\n",
      "\u001b[34m\u001b[1mOpenVINO:\u001b[0m starting export with openvino 2024.4.0-16579-c3152d32c9c-releases/2024/4...\n",
      "\u001b[34m\u001b[1mOpenVINO:\u001b[0m export success ✅ 2.3s, saved as '/home/maleksandr/test_notebooks/yolo/openvino_notebooks/notebooks/quantizing-model-with-accuracy-control/yolo11n-seg_openvino_model/' (11.3 MB)\n",
      "\n",
      "Export complete (2.6s)\n",
      "Results saved to \u001b[1m/home/maleksandr/test_notebooks/yolo/openvino_notebooks/notebooks/quantizing-model-with-accuracy-control\u001b[0m\n",
      "Predict:         yolo predict task=segment model=/home/maleksandr/test_notebooks/yolo/openvino_notebooks/notebooks/quantizing-model-with-accuracy-control/yolo11n-seg_openvino_model imgsz=640  \n",
      "Validate:        yolo val task=segment model=/home/maleksandr/test_notebooks/yolo/openvino_notebooks/notebooks/quantizing-model-with-accuracy-control/yolo11n-seg_openvino_model imgsz=640 data=/ultralytics/ultralytics/cfg/datasets/coco.yaml  \n",
      "Visualize:       https://netron.app\n"
     ]
    }
   ],
   "source": [
    "import openvino as ov\n",
    "\n",
    "\n",
    "model_path = Path(f\"{ROOT}/{MODEL_NAME}_openvino_model/{MODEL_NAME}.xml\")\n",
    "if not model_path.exists():\n",
    "    model.export(format=\"openvino\", dynamic=True, half=False)\n",
    "\n",
    "ov_model = ov.Core().read_model(model_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define validator and data loader\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The original model repository uses a `Validator` wrapper, which represents the accuracy validation pipeline. It creates dataloader and evaluation metrics and updates metrics on each data batch produced by the dataloader. Besides that, it is responsible for data preprocessing and results postprocessing. For class initialization, the configuration should be provided. We will use the default setup, but it can be replaced with some parameters overriding to test on custom data. The model has connected the `ValidatorClass` method, which creates a validator class instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from ultralytics.data.converter import coco80_to_coco91_class\n",
    "\n",
    "\n",
    "validator = model.task_map[model.task][\"validator\"](args=args)\n",
    "validator.data = check_det_dataset(args.data)\n",
    "validator.stride = 3\n",
    "data_loader = validator.get_dataloader(OUT_DIR / \"coco128-seg\", 1)\n",
    "\n",
    "validator.is_coco = True\n",
    "validator.class_map = coco80_to_coco91_class()\n",
    "validator.names = model.model.names\n",
    "validator.metrics.names = validator.names\n",
    "validator.nc = model.model.model[-1].nc\n",
    "validator.nm = 32\n",
    "validator.process = ops.process_mask\n",
    "validator.plot_masks = []"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare calibration and validation datasets\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "We can use one dataset as calibration and validation datasets. Name it `quantization_dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, openvino\n"
     ]
    }
   ],
   "source": [
    "from typing import Dict\n",
    "\n",
    "import nncf\n",
    "\n",
    "\n",
    "def transform_fn(data_item: Dict):\n",
    "    input_tensor = validator.preprocess(data_item)[\"img\"].numpy()\n",
    "    return input_tensor\n",
    "\n",
    "\n",
    "quantization_dataset = nncf.Dataset(data_loader, transform_fn)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare validation function\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "\n",
    "import torch\n",
    "from nncf.quantization.advanced_parameters import AdvancedAccuracyRestorerParameters\n",
    "\n",
    "\n",
    "def validation_ac(\n",
    "    compiled_model: ov.CompiledModel,\n",
    "    validation_loader: torch.utils.data.DataLoader,\n",
    "    validator: Validator,\n",
    "    num_samples: int = None,\n",
    "    log=True,\n",
    ") -> float:\n",
    "    validator.seen = 0\n",
    "    validator.jdict = []\n",
    "    validator.stats = dict(tp_m=[], tp=[], conf=[], pred_cls=[], target_cls=[], target_img=[])\n",
    "    validator.batch_i = 1\n",
    "    validator.confusion_matrix = ConfusionMatrix(nc=validator.nc)\n",
    "    num_outputs = len(compiled_model.outputs)\n",
    "\n",
    "    counter = 0\n",
    "    for batch_i, batch in enumerate(validation_loader):\n",
    "        if num_samples is not None and batch_i == num_samples:\n",
    "            break\n",
    "        batch = validator.preprocess(batch)\n",
    "        results = compiled_model(batch[\"img\"])\n",
    "        if num_outputs == 1:\n",
    "            preds = torch.from_numpy(results[compiled_model.output(0)])\n",
    "        else:\n",
    "            preds = [\n",
    "                torch.from_numpy(results[compiled_model.output(0)]),\n",
    "                torch.from_numpy(results[compiled_model.output(1)]),\n",
    "            ]\n",
    "        preds = validator.postprocess(preds)\n",
    "        validator.update_metrics(preds, batch)\n",
    "        counter += 1\n",
    "    stats = validator.get_stats()\n",
    "    if num_outputs == 1:\n",
    "        stats_metrics = stats[\"metrics/mAP50-95(B)\"]\n",
    "    else:\n",
    "        stats_metrics = stats[\"metrics/mAP50-95(M)\"]\n",
    "    if log:\n",
    "        print(f\"Validate: dataset length = {counter}, metric value = {stats_metrics:.3f}\")\n",
    "\n",
    "    return stats_metrics\n",
    "\n",
    "\n",
    "validation_fn = partial(validation_ac, validator=validator, log=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run quantization with accuracy control\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "You should provide the calibration dataset and the validation dataset. It can be the same dataset. \n",
    "  - parameter `max_drop` defines the accuracy drop threshold. The quantization process stops when the degradation of accuracy metric on the validation dataset is less than the `max_drop`. The default value is 0.01. NNCF will stop the quantization and report an error if the `max_drop` value can’t be reached.\n",
    "  - `drop_type` defines how the accuracy drop will be calculated: ABSOLUTE (used by default) or RELATIVE.\n",
    "  - `ranking_subset_size` - size of a subset that is used to rank layers by their contribution to the accuracy drop. Default value is 300, and the more samples it has the better ranking, potentially. Here we use the value 25 to speed up the execution. \n",
    "\n",
    "> **NOTE**: Execution can take tens of minutes and requires up to 15 GB of free memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "581a085f97114d09befd8ae89912814c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4cd8ca245544c0db1b29b0ee8b22360",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Validation of initial model was started\n",
      "INFO:nncf:Elapsed Time: 00:00:00\n",
      "INFO:nncf:Elapsed Time: 00:00:04\n",
      "INFO:nncf:Metric of initial model: 0.39269183697877214\n",
      "INFO:nncf:Collecting values for each data item using the initial model\n",
      "INFO:nncf:Elapsed Time: 00:00:04\n",
      "INFO:nncf:Validation of quantized model was started\n",
      "INFO:nncf:Elapsed Time: 00:00:00\n",
      "INFO:nncf:Elapsed Time: 00:00:04\n",
      "INFO:nncf:Metric of quantized model: 0.37315412232099016\n",
      "INFO:nncf:Collecting values for each data item using the quantized model\n",
      "INFO:nncf:Elapsed Time: 00:00:04\n",
      "INFO:nncf:Accuracy drop: 0.01953771465778198 (absolute)\n",
      "INFO:nncf:Accuracy drop: 0.01953771465778198 (absolute)\n",
      "INFO:nncf:Total number of quantized operations in the model: 127\n",
      "INFO:nncf:Number of parallel workers to rank quantized operations: 1\n",
      "INFO:nncf:ORIGINAL metric is used to rank quantizers\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a1eb2333d8e4d8fb6793edaf1b845bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Elapsed Time: 00:02:11\n",
      "INFO:nncf:Changing the scope of quantizer nodes was started\n",
      "INFO:nncf:Reverted 1 operations to the floating-point precision: \n",
      "\t__module.model.23/aten::mul/Multiply_3\n",
      "INFO:nncf:Accuracy drop with the new quantization scope is 0.016468171203124993 (absolute)\n",
      "INFO:nncf:Reverted 1 operations to the floating-point precision: \n",
      "\t__module.model.0.conv/aten::_convolution/Convolution\n",
      "INFO:nncf:Accuracy drop with the new quantization scope is 0.01663718104550027 (absolute)\n",
      "INFO:nncf:Re-calculating ranking scores for remaining groups\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e01d269d85e545b39347740ddf188237",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:Elapsed Time: 00:02:07\n",
      "INFO:nncf:Reverted 2 operations to the floating-point precision: \n",
      "\t__module.model.23/aten::sub/Subtract_1\n",
      "\t__module.model.23/aten::add/Add_7\n",
      "INFO:nncf:Algorithm completed: achieved required accuracy drop 0.006520879829061188 (absolute)\n",
      "INFO:nncf:3 out of 127 were reverted back to the floating-point precision:\n",
      "\t__module.model.23/aten::mul/Multiply_3\n",
      "\t__module.model.23/aten::sub/Subtract_1\n",
      "\t__module.model.23/aten::add/Add_7\n"
     ]
    }
   ],
   "source": [
    "quantized_model = nncf.quantize_with_accuracy_control(\n",
    "    ov_model,\n",
    "    quantization_dataset,\n",
    "    quantization_dataset,\n",
    "    validation_fn=validation_fn,\n",
    "    max_drop=0.01,\n",
    "    preset=nncf.QuantizationPreset.MIXED,\n",
    "    subset_size=128,\n",
    "    advanced_accuracy_restorer_parameters=AdvancedAccuracyRestorerParameters(ranking_subset_size=25),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare Accuracy and Performance of the Original and Quantized Models\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Now we can compare metrics of the Original non-quantized OpenVINO IR model and Quantized OpenVINO IR model to make sure that the `max_drop` is not exceeded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "586908ebc71e413483a46142f2ad74ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = device_widget()\n",
    "\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validate: dataset length = 128, metric value = 0.393\n",
      "Validate: dataset length = 128, metric value = 0.386\n",
      "[Original OpenVINO]: 0.3927\n",
      "[Quantized OpenVINO]: 0.3862\n"
     ]
    }
   ],
   "source": [
    "core = ov.Core()\n",
    "ov_config = {}\n",
    "if device.value != \"CPU\":\n",
    "    quantized_model.reshape({0: [1, 3, 640, 640]})\n",
    "if \"GPU\" in device.value or (\"AUTO\" in device.value and \"GPU\" in core.available_devices):\n",
    "    ov_config = {\"GPU_DISABLE_WINOGRAD_CONVOLUTION\": \"YES\"}\n",
    "quantized_compiled_model = core.compile_model(quantized_model, device.value, ov_config)\n",
    "compiled_ov_model = core.compile_model(ov_model, device.value, ov_config)\n",
    "\n",
    "pt_result = validation_ac(compiled_ov_model, data_loader, validator)\n",
    "quantized_result = validation_ac(quantized_compiled_model, data_loader, validator)\n",
    "\n",
    "\n",
    "print(f\"[Original OpenVINO]: {pt_result:.4f}\")\n",
    "print(f\"[Quantized OpenVINO]: {quantized_result:.4f}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And compare performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Set model directory\n",
    "MODEL_DIR = Path(\"model\")\n",
    "MODEL_DIR.mkdir(exist_ok=True)\n",
    "\n",
    "ir_model_path = MODEL_DIR / \"ir_model.xml\"\n",
    "quantized_model_path = MODEL_DIR / \"quantized_model.xml\"\n",
    "\n",
    "# Save models to use them in the commandline banchmark app\n",
    "ov.save_model(ov_model, ir_model_path, compress_to_fp16=False)\n",
    "ov.save_model(quantized_model, quantized_model_path, compress_to_fp16=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Step 1/11] Parsing and validating input arguments\n",
      "[ INFO ] Parsing input parameters\n",
      "[Step 2/11] Loading OpenVINO Runtime\n",
      "[ WARNING ] Default duration 120 seconds is used for unknown device AUTO\n",
      "[ INFO ] OpenVINO:\n",
      "[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4\n",
      "[ INFO ] \n",
      "[ INFO ] Device info:\n",
      "[ INFO ] AUTO\n",
      "[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4\n",
      "[ INFO ] \n",
      "[ INFO ] \n",
      "[Step 3/11] Setting device configuration\n",
      "[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
      "[Step 4/11] Reading model files\n",
      "[ INFO ] Loading model files\n",
      "[ INFO ] Read model took 15.23 ms\n",
      "[ INFO ] Original model I/O parameters:\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     x (node: x) : f32 / [...] / [?,3,?,?]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     ***NO_NAME*** (node: __module.model.23/aten::cat/Concat_8) : f32 / [...] / [?,116,21..]\n",
      "[ INFO ]     input.255 (node: __module.model.23.cv4.2.1.act/aten::silu_/Swish_46) : f32 / [...] / [?,32,8..,8..]\n",
      "[Step 5/11] Resizing model to match image sizes and given batch\n",
      "[ INFO ] Model batch size: 1\n",
      "[ INFO ] Reshaping model: 'x': [1,3,640,640]\n",
      "[ INFO ] Reshape model took 7.93 ms\n",
      "[Step 6/11] Configuring input of the model\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     x (node: x) : u8 / [N,C,H,W] / [1,3,640,640]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     ***NO_NAME*** (node: __module.model.23/aten::cat/Concat_8) : f32 / [...] / [1,116,8400]\n",
      "[ INFO ]     input.255 (node: __module.model.23.cv4.2.1.act/aten::silu_/Swish_46) : f32 / [...] / [1,32,160,160]\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Compile model took 389.72 ms\n",
      "[Step 8/11] Querying optimal runtime parameters\n",
      "[ INFO ] Model:\n",
      "[ INFO ]   NETWORK_NAME: Model0\n",
      "[ INFO ]   EXECUTION_DEVICES: ['CPU']\n",
      "[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT\n",
      "[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
      "[ INFO ]   MULTI_DEVICE_PRIORITIES: CPU\n",
      "[ INFO ]   CPU:\n",
      "[ INFO ]     AFFINITY: Affinity.CORE\n",
      "[ INFO ]     CPU_DENORMALS_OPTIMIZATION: False\n",
      "[ INFO ]     CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0\n",
      "[ INFO ]     DYNAMIC_QUANTIZATION_GROUP_SIZE: 32\n",
      "[ INFO ]     ENABLE_CPU_PINNING: True\n",
      "[ INFO ]     ENABLE_HYPER_THREADING: True\n",
      "[ INFO ]     EXECUTION_DEVICES: ['CPU']\n",
      "[ INFO ]     EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
      "[ INFO ]     INFERENCE_NUM_THREADS: 36\n",
      "[ INFO ]     INFERENCE_PRECISION_HINT: <Type: 'float32'>\n",
      "[ INFO ]     KV_CACHE_PRECISION: <Type: 'float16'>\n",
      "[ INFO ]     LOG_LEVEL: Level.NO\n",
      "[ INFO ]     MODEL_DISTRIBUTION_POLICY: set()\n",
      "[ INFO ]     NETWORK_NAME: Model0\n",
      "[ INFO ]     NUM_STREAMS: 12\n",
      "[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
      "[ INFO ]     PERFORMANCE_HINT: THROUGHPUT\n",
      "[ INFO ]     PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
      "[ INFO ]     PERF_COUNT: NO\n",
      "[ INFO ]     SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
      "[ INFO ]   MODEL_PRIORITY: Priority.MEDIUM\n",
      "[ INFO ]   LOADED_FROM_CACHE: False\n",
      "[ INFO ]   PERF_COUNT: False\n",
      "[Step 9/11] Creating infer requests and preparing input tensors\n",
      "[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!\n",
      "[ INFO ] Fill input 'x' with random values \n",
      "[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 120000 ms duration)\n",
      "[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
      "[ INFO ] First inference took 45.30 ms\n",
      "[Step 11/11] Dumping statistics report\n",
      "[ INFO ] Execution Devices:['CPU']\n",
      "[ INFO ] Count:            17580 iterations\n",
      "[ INFO ] Duration:         120085.24 ms\n",
      "[ INFO ] Latency:\n",
      "[ INFO ]    Median:        80.74 ms\n",
      "[ INFO ]    Average:       81.80 ms\n",
      "[ INFO ]    Min:           59.61 ms\n",
      "[ INFO ]    Max:           151.88 ms\n",
      "[ INFO ] Throughput:   146.40 FPS\n"
     ]
    }
   ],
   "source": [
    "# Inference Original model (OpenVINO IR)\n",
    "! benchmark_app -m $ir_model_path -shape \"[1,3,640,640]\" -d $device.value -api async"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Step 1/11] Parsing and validating input arguments\n",
      "[ INFO ] Parsing input parameters\n",
      "[Step 2/11] Loading OpenVINO Runtime\n",
      "[ WARNING ] Default duration 120 seconds is used for unknown device AUTO\n",
      "[ INFO ] OpenVINO:\n",
      "[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4\n",
      "[ INFO ] \n",
      "[ INFO ] Device info:\n",
      "[ INFO ] AUTO\n",
      "[ INFO ] Build ................................. 2024.4.0-16579-c3152d32c9c-releases/2024/4\n",
      "[ INFO ] \n",
      "[ INFO ] \n",
      "[Step 3/11] Setting device configuration\n",
      "[ WARNING ] Performance hint was not explicitly specified in command line. Device(AUTO) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
      "[Step 4/11] Reading model files\n",
      "[ INFO ] Loading model files\n",
      "[ INFO ] Read model took 25.39 ms\n",
      "[ INFO ] Original model I/O parameters:\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     x (node: x) : f32 / [...] / [1,3,640,640]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     ***NO_NAME*** (node: __module.model.23/aten::cat/Concat_8) : f32 / [...] / [1,116,8400]\n",
      "[ INFO ]     input.255 (node: __module.model.23.cv4.2.1.act/aten::silu_/Swish_46) : f32 / [...] / [1,32,160,160]\n",
      "[Step 5/11] Resizing model to match image sizes and given batch\n",
      "[ INFO ] Model batch size: 1\n",
      "[ INFO ] Reshaping model: 'x': [1,3,640,640]\n",
      "[ INFO ] Reshape model took 0.05 ms\n",
      "[Step 6/11] Configuring input of the model\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     x (node: x) : u8 / [N,C,H,W] / [1,3,640,640]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     ***NO_NAME*** (node: __module.model.23/aten::cat/Concat_8) : f32 / [...] / [1,116,8400]\n",
      "[ INFO ]     input.255 (node: __module.model.23.cv4.2.1.act/aten::silu_/Swish_46) : f32 / [...] / [1,32,160,160]\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Compile model took 639.12 ms\n",
      "[Step 8/11] Querying optimal runtime parameters\n",
      "[ INFO ] Model:\n",
      "[ INFO ]   NETWORK_NAME: Model0\n",
      "[ INFO ]   EXECUTION_DEVICES: ['CPU']\n",
      "[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT\n",
      "[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
      "[ INFO ]   MULTI_DEVICE_PRIORITIES: CPU\n",
      "[ INFO ]   CPU:\n",
      "[ INFO ]     AFFINITY: Affinity.CORE\n",
      "[ INFO ]     CPU_DENORMALS_OPTIMIZATION: False\n",
      "[ INFO ]     CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0\n",
      "[ INFO ]     DYNAMIC_QUANTIZATION_GROUP_SIZE: 32\n",
      "[ INFO ]     ENABLE_CPU_PINNING: True\n",
      "[ INFO ]     ENABLE_HYPER_THREADING: True\n",
      "[ INFO ]     EXECUTION_DEVICES: ['CPU']\n",
      "[ INFO ]     EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
      "[ INFO ]     INFERENCE_NUM_THREADS: 36\n",
      "[ INFO ]     INFERENCE_PRECISION_HINT: <Type: 'float32'>\n",
      "[ INFO ]     KV_CACHE_PRECISION: <Type: 'float16'>\n",
      "[ INFO ]     LOG_LEVEL: Level.NO\n",
      "[ INFO ]     MODEL_DISTRIBUTION_POLICY: set()\n",
      "[ INFO ]     NETWORK_NAME: Model0\n",
      "[ INFO ]     NUM_STREAMS: 12\n",
      "[ INFO ]     OPTIMAL_NUMBER_OF_INFER_REQUESTS: 12\n",
      "[ INFO ]     PERFORMANCE_HINT: THROUGHPUT\n",
      "[ INFO ]     PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
      "[ INFO ]     PERF_COUNT: NO\n",
      "[ INFO ]     SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
      "[ INFO ]   MODEL_PRIORITY: Priority.MEDIUM\n",
      "[ INFO ]   LOADED_FROM_CACHE: False\n",
      "[ INFO ]   PERF_COUNT: False\n",
      "[Step 9/11] Creating infer requests and preparing input tensors\n",
      "[ WARNING ] No input files were given for input 'x'!. This input will be filled with random values!\n",
      "[ INFO ] Fill input 'x' with random values \n",
      "[Step 10/11] Measuring performance (Start inference asynchronously, 12 inference requests, limits: 120000 ms duration)\n",
      "[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
      "[ INFO ] First inference took 33.20 ms\n",
      "[Step 11/11] Dumping statistics report\n",
      "[ INFO ] Execution Devices:['CPU']\n",
      "[ INFO ] Count:            38424 iterations\n",
      "[ INFO ] Duration:         120039.54 ms\n",
      "[ INFO ] Latency:\n",
      "[ INFO ]    Median:        37.07 ms\n",
      "[ INFO ]    Average:       37.32 ms\n",
      "[ INFO ]    Min:           22.57 ms\n",
      "[ INFO ]    Max:           72.28 ms\n",
      "[ INFO ] Throughput:   320.09 FPS\n"
     ]
    }
   ],
   "source": [
    "# Inference Quantized model (OpenVINO IR)\n",
    "! benchmark_app -m $quantized_model_path -shape \"[1,3,640,640]\" -d $device.value -api async"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "openvino_notebooks": {
   "imageUrl": "",
   "tags": {
    "categories": [
     "Convert",
     "Optimize"
    ],
    "libraries": [],
    "other": [
     "YOLO"
    ],
    "tasks": [
     "Image Segmentation"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}