vision/yolo_v4/pytorch/notebook_yolo.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d2b361bf",
   "metadata": {},
   "source": [
    "Copyright (c) 2023 Graphcore Ltd. All rights reserved."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bdeae4e8",
   "metadata": {},
   "source": [
    "# Object Detection on IPU using YOLO v4 - Inference\n",
    "This notebook demonstrates an object detection task with a YOLO v4 model using an inference pipeline run on Graphcore IPUs. Originally the code of the YOLO v4 model adapted for IPU was published in the [examples GitHub repository](https://github.com/graphcore/examples/tree/master/vision/yolo_v4/pytorch).\n",
    "\n",
    "### Summary table\n",
    "\n",
    "|  Domain | Tasks | Model | Datasets | Workflow |   Number of IPUs   | Execution time |\n",
    "|---------|-------|-------|----------|----------|--------------|--------------|\n",
    "| Vision  | Object detection | YOLO v4 | COCO | Inference | Recommended: POD4 | 2 minutes    |\n",
    "\n",
    "\n",
    "![object detection on IPU](notebook/first_example.png \"Last supper object detection\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "db890f52",
   "metadata": {},
   "source": [
    "\n",
    "## Environment setup\n",
    "\n",
    "The best way to run this demo is on Paperspace Gradient’s cloud IPUs because everything is already set up for you. To improve your experience we preload datasets and pre-install packages. This can take a few minutes. If you experience errors immediately after starting a session please try restarting the kernel before contacting support. If a problem persists or you want to give us feedback on the content of this notebook, please reach out to through our community of developers using our [slack channel](https://www.graphcore.ai/join-community) or raise a [GitHub issue](https://github.com/graphcore/Gradient-PyTorch/issues).\n",
    "\n",
    "\n",
    "To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine.\n",
    "\n",
    "\n",
    "## Requirements\n",
    "Before using the model on IPU you have to build the custom operations for IPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78c567f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "!make"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f1b68706",
   "metadata": {},
   "source": [
    "and install the Python dependencies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c72d24a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r requirements.txt\n",
    "%load_ext graphcore_cloud_tools.notebook_logging.gc_logger"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f332148f",
   "metadata": {},
   "source": [
    "In order to improve usability and support for future users, Graphcore would like to collect information about the applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n",
    "\n",
    "- User progression through the notebook\n",
    "- Notebook details: number of cells, code being run and the output of the cells\n",
    "- Environment details\n",
    "\n",
    "You can disable logging at any time by running `%unload_ext graphcore_cloud_tools.notebook_logging.gc_logger` from any cell."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a0737bab",
   "metadata": {},
   "source": [
    "## COCO dataset\n",
    "\n",
    "This demo of inference with YOLO v4 on IPU uses the checkpoint from a model trained with the [COCO dataset](https://cocodataset.org/). With this we can demonstrate the detection of 80 different classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1fd2135",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ruamel import yaml\n",
    "\n",
    "class_names = yaml.safe_load(open(\"configs/class_name.yaml\"))[\"class_names\"]\n",
    "class_names"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b33a842f",
   "metadata": {},
   "source": [
    "## Model preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c5e71f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import torch\n",
    "\n",
    "from PIL import Image\n",
    "from pathlib import Path\n",
    "\n",
    "import poptorch\n",
    "\n",
    "path_to_detection = Path().parent.resolve()\n",
    "os.environ[\"PYTORCH_APPS_DETECTION_PATH\"] = str(path_to_detection)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "56e23e74",
   "metadata": {},
   "source": [
    "We are using the original YOLO v4 model with a checkpoint trained with the COCO dataset, which we pass as `checkpoint`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aaded57",
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = \"checkpoint/yolov4_p5_reference_weights/yolov4-p5-sd.pt\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4a2b3333",
   "metadata": {},
   "source": [
    "Next we download the checkpoint for the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b63d8db0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "FILE=./checkpoint/yolov4_p5_reference_weights/yolov4-p5-sd.pt; \\\n",
    "if [ -f \"$FILE\" ]; then \\\n",
    "    echo \"$FILE exists, no need to download.\"; \\\n",
    "else \\\n",
    "    mkdir checkpoint; \\\n",
    "    cd checkpoint; \\\n",
    "    curl https://gc-demo-resources.s3.us-west-1.amazonaws.com/yolov4_p5_reference_weights.tar.gz -o yolov4_p5_reference_weights.tar.gz && tar -zxvf yolov4_p5_reference_weights.tar.gz && rm yolov4_p5_reference_weights.tar.gz; \\\n",
    "    cd ..; \\\n",
    "fi "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ce0d4c56",
   "metadata": {},
   "source": [
    "We are using a `YOLOv4InferencePipeline` class to set all the IPU specific options and wrap the PyTorch model in a PopTorch inference model. The pipeline reads the configuration parameters from the [config file](configs/inference-yolov4p5.yaml)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9261c334",
   "metadata": {},
   "outputs": [],
   "source": [
    "from api import YOLOv4InferencePipeline\n",
    "\n",
    "pipeline = YOLOv4InferencePipeline(\n",
    "    checkpoint_path=\"checkpoint/yolov4_p5_reference_weights/yolov4-p5-sd.pt\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "baac6d44",
   "metadata": {},
   "source": [
    "We are using the image of a famous painting (The Last Supper by Leonardo da Vinci) to demonstrate how to use the inference model. This image is stored in the repo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79ab03a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "image = Image.open(\"notebook/last_supper_restored.jpeg\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cd7ca25a",
   "metadata": {},
   "source": [
    "The pipeline call carries out the following steps: image preprocessing; a forward pass of the preprocessed image through the model; model output postprocessing.\n",
    "When used for the first time, a one-time compilation of the model is triggered.\n",
    "For a single IPU it should take about 3 minutes to compile the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8abf9a72",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "processed_batch = pipeline(image)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5213b426",
   "metadata": {},
   "source": [
    "Subsequent calls are much faster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f037e4ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "processed_batch = pipeline(image)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "712092f6",
   "metadata": {},
   "source": [
    "You can now check what has been detected on the input image.\n",
    "The output contains a list of detected objects for each image in the batch.\n",
    "Each result contains five numbers: the first four are the coordinates in XYWH format (x, y, width, and height of bounding box), the fifth number represents the confidence of the prediction, and the sixth number represents the predicted class of the detected object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5424e3fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_batch"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b6a0858c",
   "metadata": {},
   "source": [
    "You can use the coordinates from the of the detected objects to plot bounding boxes on the original image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e314726b",
   "metadata": {},
   "outputs": [],
   "source": [
    "img_paths = pipeline.plot_img(processed_batch, image)\n",
    "img_paths"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9e3de8ba",
   "metadata": {},
   "source": [
    "## Try it on your own\n",
    "\n",
    "Let's check the YOLOv4 model on IPU with some images from the Internet.\n",
    "\n",
    "We can use the `wget` command to store the image as `image.png`. The syntax is `!wget -O image.png [YOUR URL]`, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20b0739a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# UPLOAD YOUR OWN IMAGE HERE BY REPLACING THE URL\n",
    "\n",
    "!wget -O image.png https://www.graphcore.ai/hubfs/assets/images/content/new-team.jpg"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "318ddd7b",
   "metadata": {},
   "source": [
    "Let's now use this image for inference with the same model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b60e451",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "image = Image.open(\"image.png\")\n",
    "\n",
    "processed_batch = pipeline(image)\n",
    "img_paths = pipeline.plot_img(processed_batch, image)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9e2bae0a",
   "metadata": {},
   "source": [
    "### Optional - Release IPUs in use\n",
    "\n",
    "The IPython kernel has a lock on the IPUs used in running the model, preventing other users from using them. For example, if you wish to use other notebooks after working your way through this one, it may be necessary to manually run the below cell to release IPUs from use. This will happen by default if using the Run All option. More information on the topic can be found at [Managing IPU Resources](https://github.com/gradient-ai/Graphcore-HuggingFace/blob/main/useful-tips/managing_ipu_resources.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d4f685f",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.detach()"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}