omnivore_demo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NlE20xLoY_0c"
      },
      "source": [
        "GitHub  \n",
        "https://github.com/facebookresearch/omnivore  \n",
        "論文  \n",
        "https://arxiv.org/abs/2201.08377v1    \n",
        "\n",
        "<a href=\"https://colab.research.google.com/github/kaz12tech/ai_demos/blob/master/omnivore_demo.ipynb\" target=\"_blank\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-hzmmLNNezeY"
      },
      "source": [
        "## 実行前準備\n",
        "「ランタイム」→「ランタイムのタイプを変更」→「ハードウェアアクセラレータ」をGPUに変更\n",
        "## 実行方法\n",
        "「ランタイム」→「すべてのセルを実行」を選択"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x4RkaruBZzM9"
      },
      "source": [
        "## 環境セットアップ\n",
        "pytorchのバージョン変更やライブラリのインストール"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iOLLUZznVR-7",
        "outputId": "5ca798c5-7937-4591-bb46-1ecf90492c14"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "!python -c \"import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.device_count());\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sOy-eHc8Whk4",
        "outputId": "5e321d4a-a2cc-4036-cd5f-11ea2881d465"
      },
      "outputs": [],
      "source": [
        "# Pytorch 1.9.0をインストール\n",
        "#!pip uninstall torch -y\n",
        "#!pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "!pip install einops timm moviepy\n",
        "\n",
        "# インストールしたPytorchの動作確認(バージョン、CUDAの認識可否、GPU数確認)\n",
        "import torch\n",
        "!python -c \"import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.device_count());\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dVbM5eEwYJ5u",
        "outputId": "b5474210-3a6a-42e0-d3b5-81a2e334a310"
      },
      "outputs": [],
      "source": [
        "!pip install \"git+https://github.com/facebookresearch/pytorchvideo.git\"\n",
        "!git clone https://github.com/facebookresearch/omnivore.git"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qlp2jx-ZZ8zI"
      },
      "source": [
        "## モジュールのインポート"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-J1YTpWNXCyR",
        "outputId": "439ccf9f-e137-45f3-bff8-a623fe779df2"
      },
      "outputs": [],
      "source": [
        "%cd /content/omnivore/\n",
        "import json\n",
        "from typing import List\n",
        "\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "import torchvision.transforms as T\n",
        "from PIL import Image\n",
        "from pytorchvideo.data.encoded_video import EncodedVideo\n",
        "from torchvision.transforms._transforms_video import NormalizeVideo\n",
        "from transforms import SpatialCrop, TemporalCrop, DepthNorm\n",
        "\n",
        "from pytorchvideo.transforms import (\n",
        "    ApplyTransformToKey,\n",
        "    ShortSideScale,\n",
        "    UniformTemporalSubsample,\n",
        ")\n",
        "\n",
        "%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.image as mpimg\n",
        "import imageio\n",
        "import matplotlib.animation as animation\n",
        "from IPython.display import HTML\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v1tmf2EDcaQq"
      },
      "source": [
        "## モデルのロード\n",
        "### Model Zoo \n",
        "今回はOmnivore Swin Bを使用\n",
        "\n",
        "\n",
        "| Name      | IN1k Top 1 | Kinetics400 Top 1     | SUN RGBD Top 1     | Model   |\n",
        "| :---        |    :----   |          :--- | :--- |:--- |\n",
        "| Omnivore Swin T      | 81.2       | 78.9   |62.3   | [weights](https://dl.fbaipublicfiles.com/omnivore/models/swinT_checkpoint.torch)   \n",
        "| Omnivore Swin S   | 83.4       | 82.2      |64.6  | [weights](https://dl.fbaipublicfiles.com/omnivore/models/swinS_checkpoint.torch)  |\n",
        "| Omnivore Swin B      | 84.0       | 83.3   |65.4   | [weights](https://dl.fbaipublicfiles.com/omnivore/models/swinB_checkpoint.torch)   |\n",
        "| Omnivore Swin B (IN21k)   | 85.3       | 84.0      |67.2   | [weights](https://dl.fbaipublicfiles.com/omnivore/models/swinB_In21k_checkpoint.torch)   |\n",
        "| Omnivore Swin L (IN21k)      | 86.0       | 84.1   |67.1   | [weights](https://dl.fbaipublicfiles.com/omnivore/models/swinL_In21k_checkpoint.torch) |\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 159,
          "referenced_widgets": [
            "d5ec7ba7b3054166939971f390885fc1",
            "5ea0019fa0a444b9939196734381b027",
            "fc469caba42548df8117749669269b3d",
            "a3b1bd0c8f9a497a8ff91ec99e6cea23",
            "d9aed8ff0b4345dc80d4cbd675f4b873",
            "6ce458de85f04d98bf87287f9cfb274f",
            "ab4ccebe21b34335bb61594c4e3a14eb",
            "28289ecb0fe74774be636104a0783b8a",
            "9c6799f5f89746c58369659f034cd191",
            "225bf7c8fb9742c48df2f7c940e1810c",
            "c091804c18164e21876e142de0020958"
          ]
        },
        "id": "4nHXLlT-ce4n",
        "outputId": "abce91ef-3f04-4c67-a99a-ea5727f08335"
      },
      "outputs": [],
      "source": [
        "# Device on which to run the model\n",
        "# Set to cuda to load on GPU\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "print(\"set device->\", device)\n",
        "\n",
        "# Pick a pretrained model \n",
        "model_name = \"omnivore_swinB\"\n",
        "model = torch.hub.load(\"facebookresearch/omnivore:main\", model=model_name)\n",
        "\n",
        "# Set to eval mode and move to desired device\n",
        "model = model.to(device)\n",
        "model = model.eval()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ArTmUEoHmxTn"
      },
      "source": [
        "## 画像分類"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TB2mFN_Sf2Go"
      },
      "source": [
        "### ID/ラベルのロード\n",
        "予測時に導出されるIDと、IDが意味するラベルのマッピングが示されたImagenet1Kデータセットのラベルファイルをロード  \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6okVDIJTgCaM",
        "outputId": "d295b147-3ef5-4eea-fa9e-2ffc36c80be7"
      },
      "outputs": [],
      "source": [
        "%cd /content/omnivore/\n",
        "# IDとラベルの対応表(imagenet_class_index.json)をダウンロード\n",
        "!wget https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json\n",
        "\n",
        "with open(\"/content/omnivore/imagenet_class_index.json\", \"r\") as f:\n",
        "    imagenet_classnames = json.load(f)\n",
        "\n",
        "# idとlabelをマッピングした辞書型オブジェクトを生成\n",
        "imagenet_id_to_classname = {}\n",
        "for k, v in imagenet_classnames.items():\n",
        "    imagenet_id_to_classname[k] = v[1] "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h_AnEWEOiFHA"
      },
      "source": [
        "### 画像の取得\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 381,
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "ok": true,
              "status": 200,
              "status_text": ""
            }
          }
        },
        "id": "DDiKX97EihVo",
        "outputId": "2002eee8-488a-4e48-b215-98ee8b8f89fd"
      },
      "outputs": [],
      "source": [
        "%cd /content/omnivore/\n",
        "!mkdir images\n",
        "%cd /content/omnivore/images/\n",
        "\n",
        "latent_type='upload_from_pc' #@param ['dawnload_from_web', 'upload_from_pc']\n",
        "if latent_type=='dawnload_from_web':\n",
        "  # サンプル画像のダウンロード\n",
        "  !wget -O library.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/c/c5/13-11-02-olb-by-RalfR-03.jpg/800px-13-11-02-olb-by-RalfR-03.jpg\n",
        "\n",
        "  image_path = \"/content/omnivore/images/library.jpg\"\n",
        "else:\n",
        "  # 画像アップロード\n",
        "  from google.colab import files\n",
        "  uploaded = files.upload()\n",
        "  uploaded = list(uploaded.keys())\n",
        "  file_name = uploaded[0]\n",
        "  image_path = \"/content/omnivore/images/\" + file_name\n",
        "\n",
        "image = Image.open(image_path).convert(\"RGB\")\n",
        "plt.figure(figsize=(6, 6))\n",
        "plt.imshow(image)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GuLuV-0qj2EQ"
      },
      "source": [
        "### 画像のセットアップ\n",
        "画像をモデルが求めるデータ形式に変換"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "71GyPLysjpiW"
      },
      "outputs": [],
      "source": [
        "image_transform = T.Compose(\n",
        "    [\n",
        "        T.Resize(224),\n",
        "        T.CenterCrop(224),\n",
        "        T.ToTensor(),\n",
        "        T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n",
        "    ]\n",
        ")\n",
        "image = image_transform(image)\n",
        "\n",
        "# The model expects inputs of shape: B x C x T x H x W\n",
        "image = image[None, :, None, ...]\n",
        "image = image.to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jA-OBkAYm1Hl"
      },
      "source": [
        "### 予測\n",
        "画像のラベルを予測します"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U7kMRlWDkB1V",
        "outputId": "70b51544-cf00-437d-bb23-521e0b35a44b"
      },
      "outputs": [],
      "source": [
        "prediction = model(image, input_type=\"image\")\n",
        "prediction = F.softmax(prediction, dim=1)\n",
        "values, pred_classes = prediction.topk(k=5)\n",
        "\n",
        "print(\"上位5番目までの検出ラベル\")\n",
        "for value, pred_class in zip(values[0], pred_classes[0]):\n",
        "  print('ラベル:', imagenet_id_to_classname[str(pred_class.item())], ', Score:', value.item())\n",
        "\n",
        "# GPU Memoryがギリギリのため予測が終わったらGPUに配置したimageを削除\n",
        "del image\n",
        "torch.cuda.empty_cache()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nxn4_S6WtVFg"
      },
      "source": [
        "## 動画分類"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nJP7wlK_tuTP"
      },
      "source": [
        "### ID/ラベルのロード\n",
        "予測時に導出されるIDと、IDが意味するラベルのマッピングが示されたKinetics 400データセットのラベルファイルをロード  \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nVv9XKzRtXih",
        "outputId": "fb2cb8b6-6480-4fc0-b9e2-89c3dcfe58dc"
      },
      "outputs": [],
      "source": [
        "%cd /content/omnivore/\n",
        "# IDとラベルの対応表(imagenet_class_index.json)をダウンロード\n",
        "!wget https://dl.fbaipublicfiles.com/pyslowfast/dataset/class_names/kinetics_classnames.json\n",
        "\n",
        "with open(\"kinetics_classnames.json\", \"r\") as f:\n",
        "    kinetics_classnames = json.load(f)\n",
        "\n",
        "# idとlabelをマッピングした辞書型オブジェクトを生成\n",
        "kinetics_id_to_classname = {}\n",
        "for k, v in kinetics_classnames.items():\n",
        "    kinetics_id_to_classname[v] = str(k).replace('\"', \"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u7JgfhxduOR9"
      },
      "source": [
        "### 動画の取得"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "ok": true,
              "status": 200,
              "status_text": ""
            }
          }
        },
        "id": "a5f9RND4udnu",
        "outputId": "669f1d08-d5f5-4945-8c88-796fe9690811"
      },
      "outputs": [],
      "source": [
        "%cd /content/omnivore/\n",
        "!mkdir videos\n",
        "%cd /content/omnivore/videos/\n",
        "\n",
        "latent_type='upload_from_pc' #@param ['dawnload_from_web', 'upload_from_pc']\n",
        "\n",
        "if latent_type=='dawnload_from_web':\n",
        "  # サンプル動画のダウンロード\n",
        "  !wget https://dl.fbaipublicfiles.com/omnivore/example_data/dance.mp4\n",
        "\n",
        "  # 動画のロード\n",
        "  video_path = \"/content/omnivore/videos/dance.mp4\"\n",
        "else:\n",
        "  # 動画アップロード\n",
        "  from google.colab import files\n",
        "  uploaded = files.upload()\n",
        "  uploaded = list(uploaded.keys())\n",
        "  file_name = uploaded[0]\n",
        "  video_path = \"/content/omnivore/videos/\" + file_name  \n",
        "\n",
        "# 動画が大きすぎるとRAMから溢れるためリサイズ\n",
        "from moviepy.editor import *\n",
        "clip = VideoFileClip(video_path)\n",
        "\n",
        "# 高さ80pxにリサイズ\n",
        "clip_resized = clip.resize(height=80)\n",
        "clip_resized.write_videofile(video_path)\n",
        "\n",
        "clip_resized.ipython_display()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2pBFoiZ1vTxT"
      },
      "source": [
        "### 動画のセットアップ\n",
        "動画をモデルが求めるデータ形式に変換"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1XWHcIYkvaQl"
      },
      "outputs": [],
      "source": [
        "num_frames = 160 # default 160\n",
        "sampling_rate = 1 # default 2\n",
        "frames_per_second = 160 # default 30 \n",
        "\n",
        "clip_duration = (num_frames * sampling_rate) / frames_per_second\n",
        "\n",
        "video_transform = ApplyTransformToKey(\n",
        "    key=\"video\",\n",
        "    transform=T.Compose(\n",
        "        [\n",
        "            UniformTemporalSubsample(num_frames), \n",
        "            T.Lambda(lambda x: x / 255.0),  \n",
        "            ShortSideScale(size=224), # default 224\n",
        "            NormalizeVideo(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n",
        "            TemporalCrop(frames_per_clip=16, stride=40), # default 32, 40\n",
        "            SpatialCrop(crop_size=224, num_crops=3), # default 224, 3\n",
        "        ]\n",
        "    ),\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3qVFeyaLzuxS",
        "outputId": "1961b149-32df-4e46-e8cf-15655261f932"
      },
      "outputs": [],
      "source": [
        "# Select the duration of the clip to load by specifying the start and end duration\n",
        "# The start_sec should correspond to where the action occurs in the video\n",
        "start_sec = 0\n",
        "end_sec = start_sec + clip_duration \n",
        "print('start_sec:', start_sec, ', end_sec:', end_sec)\n",
        "\n",
        "# Initialize an EncodedVideo helper class\n",
        "video = EncodedVideo.from_path(video_path)\n",
        "\n",
        "# Load the desired clip\n",
        "video_data = video.get_clip(start_sec=start_sec, end_sec=end_sec)\n",
        "\n",
        "# Apply a transform to normalize the video input\n",
        "video_data = video_transform(video_data)\n",
        "\n",
        "# Move the inputs to the desired device\n",
        "video_inputs = video_data[\"video\"]\n",
        "\n",
        "# Take the first clip \n",
        "# The model expects inputs of shape: B x C x T x H x W\n",
        "video_input = video_inputs[0][None, ...]\n",
        "\n",
        "# GPUメモリに乗り切らないためcpuで実行\n",
        "video_input = video_input.to('cpu')\n",
        "model = model.to('cpu')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zR96HgtD38y9"
      },
      "source": [
        "### 予測\n",
        "動画のラベルを予測します"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_Is7lNP64B1H",
        "outputId": "90d42eb2-dde3-4787-f0b3-7c680c96d964"
      },
      "outputs": [],
      "source": [
        "# Pass the input clip through the model \n",
        "prediction = model(video_input, input_type=\"video\")\n",
        "\n",
        "# Get the predicted classes \n",
        "prediction = F.softmax(prediction, dim=1)\n",
        "values, pred_classes = prediction.topk(k=5)\n",
        "\n",
        "print(\"上位5番目までの検出ラベル\")\n",
        "for value, pred_class in zip(values[0], pred_classes[0]):\n",
        "  print('ラベル:', kinetics_id_to_classname[int(pred_class)], ', Score:', value.item())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fns_o0DtFjD9"
      },
      "source": [
        "## 深度マップ(RGBD画像)による画像分類"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7KXj1fVPFzbV"
      },
      "source": [
        "### ID/ラベルのロード\n",
        "予測時に導出されるIDと、IDが意味するラベルのマッピングが示されたSUN RGBDデータセットのラベルファイルをロード  \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xwzcBB3gF6Gl",
        "outputId": "819fb8b9-1149-41e1-984f-6a5f61ca1ea3"
      },
      "outputs": [],
      "source": [
        "%cd /content/omnivore/\n",
        "!wget https://dl.fbaipublicfiles.com/omnivore/sunrgbd_classnames.json\n",
        "\n",
        "with open(\"sunrgbd_classnames.json\", \"r\") as f:\n",
        "  sunrgbd_id_to_classname = json.load(f)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aU8mHdYIGLPZ"
      },
      "source": [
        "### RGBD画像の取得"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 811
        },
        "id": "uxwCm81MGJuy",
        "outputId": "db45c817-06e9-4c35-977c-64d9e3e97131"
      },
      "outputs": [],
      "source": [
        "%cd /content/omnivore/\n",
        "!mkdir rgbd\n",
        "%cd /content/omnivore/rgbd/\n",
        "\n",
        "# 画像と深度ファイルのダウンロード\n",
        "!wget -O store.png https://upload.wikimedia.org/wikipedia/commons/thumb/f/f4/Interior_of_the_IKEA_B%C4%83neasa_33.jpg/791px-Interior_of_the_IKEA_B%C4%83neasa_33.jpg\n",
        "!wget https://dl.fbaipublicfiles.com/omnivore/example_data/store_disparity.pt\n",
        "\n",
        "image_path = \"/content/omnivore/rgbd/store.png\"\n",
        "depth_path = \"/content/omnivore/rgbd/store_disparity.pt\"\n",
        "image = Image.open(image_path).convert(\"RGB\")\n",
        "depth = torch.load(depth_path)[None, ...]\n",
        "\n",
        "plt.figure(figsize=(20, 10))\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"RGB\", fontsize=20)\n",
        "plt.imshow(image)\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.imshow(depth.numpy().squeeze())\n",
        "plt.title(\"Depth\", fontsize=20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v9mOA6gyGiNy"
      },
      "source": [
        "### 画像のセットアップ\n",
        "画像をモデルが求めるデータ形式に変換"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V9eb9AknGe8q"
      },
      "outputs": [],
      "source": [
        "rgbd_transform = T.Compose(\n",
        "    [\n",
        "        DepthNorm(max_depth=75.0, clamp_max_before_scale=True),\n",
        "        T.Resize(224),\n",
        "        T.CenterCrop(224),\n",
        "        T.Normalize(\n",
        "            mean=[0.485, 0.456, 0.406, 0.0418], \n",
        "            std=[0.229, 0.224, 0.225, 0.0295]\n",
        "        ),\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D1WBwp5QJJUC"
      },
      "outputs": [],
      "source": [
        "# Convert to tensor and transform\n",
        "image = T.ToTensor()(image)\n",
        "rgbd = torch.cat([image, depth], dim=0)\n",
        "rgbd = rgbd_transform(rgbd)\n",
        "\n",
        "# The model expects inputs of shape: B x C x T x H x W\n",
        "rgbd_input = rgbd[None, :, None, ...]\n",
        "rgbd_input = rgbd_input.to(device)\n",
        "model = model.to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KU2dM17SJQ1H"
      },
      "source": [
        "### 予測"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w62nSusmJTJq",
        "outputId": "1ce3cdff-34aa-46f5-c1f3-3994df380641"
      },
      "outputs": [],
      "source": [
        "prediction = model(rgbd_input, input_type=\"rgbd\")\n",
        "prediction = F.softmax(prediction, dim=1)\n",
        "pred_classes = prediction.topk(k=5).indices\n",
        "\n",
        "print(\"上位5番目までの検出ラベル\")\n",
        "for value, pred_class in zip(values[0], pred_classes[0]):\n",
        "  print('ラベル:', sunrgbd_id_to_classname[str(pred_class.item())], ', Score:', value.item())"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "omnivore_demo.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "225bf7c8fb9742c48df2f7c940e1810c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "28289ecb0fe74774be636104a0783b8a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "5ea0019fa0a444b9939196734381b027": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6ce458de85f04d98bf87287f9cfb274f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9c6799f5f89746c58369659f034cd191": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a3b1bd0c8f9a497a8ff91ec99e6cea23": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9c6799f5f89746c58369659f034cd191",
            "max": 478601327,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_28289ecb0fe74774be636104a0783b8a",
            "value": 478601327
          }
        },
        "ab4ccebe21b34335bb61594c4e3a14eb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c091804c18164e21876e142de0020958": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d5ec7ba7b3054166939971f390885fc1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_fc469caba42548df8117749669269b3d",
              "IPY_MODEL_a3b1bd0c8f9a497a8ff91ec99e6cea23",
              "IPY_MODEL_d9aed8ff0b4345dc80d4cbd675f4b873"
            ],
            "layout": "IPY_MODEL_5ea0019fa0a444b9939196734381b027"
          }
        },
        "d9aed8ff0b4345dc80d4cbd675f4b873": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c091804c18164e21876e142de0020958",
            "placeholder": "​",
            "style": "IPY_MODEL_225bf7c8fb9742c48df2f7c940e1810c",
            "value": " 456M/456M [00:13&lt;00:00, 41.1MB/s]"
          }
        },
        "fc469caba42548df8117749669269b3d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ab4ccebe21b34335bb61594c4e3a14eb",
            "placeholder": "​",
            "style": "IPY_MODEL_6ce458de85f04d98bf87287f9cfb274f",
            "value": "100%"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}