submission.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Video Classification\n",
    "The third exercise in the course Neural Networks for Computer Vision.  \n",
    "UCF101 dataset was used to train CNN classification architecture to classify videos.  \n",
    "There are two parts to the exercise:  \n",
    "First, an early fusion video classification architecture is implemented.  \n",
    "Second, a SlowFast video classifier using a ResNet-50 architecture and PytorchVideo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submitted By:\n",
    "\n",
    "* Tal Goldengoren\n",
    "* Guy Kabiri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Table of Contents:\n",
    "* [Data](#data)\n",
    "    * [Data Samples](#data-samples)\n",
    "        * [Basketball](#basketball-samples)\n",
    "        * [Rope Climbing](#ropeclimbing-samples)\n",
    "        * [Biking](#biking-samples)\n",
    "        * [Diving](#diving-samples)\n",
    "        * [Pizza Tossing](#pizzatossing-samples)\n",
    "    * [Data Exploration](#data-exp)\n",
    "* [Part 1 - Early Methods](#part1)\n",
    "    * [Early Fusion](#early-fusion)\n",
    "    * [Training](#part1-training)\n",
    "    * [Results](#part1-results)\n",
    "* [Part 2 - Modern Methods](#part2)\n",
    "    * [SlowFast](#slowfast)\n",
    "    * [Training](#part2-training)\n",
    "    * [Results](#part2-results)\n",
    "* [References](#refs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data <span id=\"data\"><span>\n",
    "\n",
    "The dataset used in this task is [UCF101](https://www.crcv.ucf.edu/data/UCF101.php).  \n",
    "UCF 101 is a set of action recognition videos from YouTube, with 101 action categories.  \n",
    "In this exercise, five classes were used:  \n",
    "Basketball, Rope Climbing, Biking, Diving, Pizza Tossing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Samples <span id=\"data-samples\"></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basketball <span id=\"basketball-samples\"></span>\n",
    "![ezgif com-gif-maker](https://user-images.githubusercontent.com/52006798/146648149-7b2a60dd-c4dc-486d-bf74-6ee36b51e24b.gif)\n",
    "![ezgif com-gif-maker(1)](https://user-images.githubusercontent.com/52006798/146648151-e9b93f3f-75f9-4ba8-af38-b9360b709563.gif)\n",
    "![ezgif com-gif-maker(2)](https://user-images.githubusercontent.com/52006798/146648154-893fdd3f-9f9b-4348-9e62-1ee6f4d24994.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rope Climbing <span id=\"ropeclimbing-samples\"></span>\n",
    "\n",
    "![ezgif com-gif-maker(12)](https://user-images.githubusercontent.com/52006798/146648141-c1d88761-cb3c-44fc-bd1d-adc078c49eb7.gif)\n",
    "![ezgif com-gif-maker(13)](https://user-images.githubusercontent.com/52006798/146648144-bcd00631-6c35-4d2b-8bf8-dbe949401702.gif)\n",
    "![ezgif com-gif-maker(14)](https://user-images.githubusercontent.com/52006798/146648146-e1c00dd1-80d8-4f62-82bd-b861782ea643.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Biking <span id=\"biking-samples\"></span>\n",
    "\n",
    "![ezgif com-gif-maker(3)](https://user-images.githubusercontent.com/52006798/146648156-c329b5d9-271f-49d9-b420-0d88f1d0cad6.gif)\n",
    "![ezgif com-gif-maker(4)](https://user-images.githubusercontent.com/52006798/146648158-ec8451ae-8bba-4808-8783-edfe74f3cec9.gif)\n",
    "![ezgif com-gif-maker(5)](https://user-images.githubusercontent.com/52006798/146648160-2b1e509a-d324-4739-8081-268f6ad2c229.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Diving <span id=\"diving-samples\"></span>\n",
    "\n",
    "![ezgif com-gif-maker(8)](https://user-images.githubusercontent.com/52006798/146648134-1f1960fb-0d1c-40f4-8bde-68e9b07e917a.gif)\n",
    "![ezgif com-gif-maker(6)](https://user-images.githubusercontent.com/52006798/146648167-f6eb1d56-9c7a-4146-95dc-5a49fdc54fd7.gif)\n",
    "![ezgif com-gif-maker(7)](https://user-images.githubusercontent.com/52006798/146648171-d3c9ba37-78b4-4f12-af1f-2bdd28eb22a8.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pizza Tossing <span id=\"pizzatossing-samples\"></span>\n",
    "\n",
    "![ezgif com-gif-maker(9)](https://user-images.githubusercontent.com/52006798/146648135-2ebf956e-e239-4d8d-9d52-57c34070be83.gif)\n",
    "![ezgif com-gif-maker(10)](https://user-images.githubusercontent.com/52006798/146648137-fdb8c9b9-f80b-4bf9-94d2-2a6188312040.gif)\n",
    "![ezgif com-gif-maker(11)](https://user-images.githubusercontent.com/52006798/146648140-6b541150-45c9-4de5-914b-f74867f7a74e.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Exploration <span id=\"data-exp\"></span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "phases = [ 'train', 'valid', 'test' ]\n",
    "paths = { p: 'data/{}'.format(p) for p in phases }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(columns=[ 'name', 'class', 'frames', 'phase' ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "for p in phases:\n",
    "    for cls in os.listdir(paths[p]):     #   iterate over the classes of each phase\n",
    "        class_path = os.path.join(paths[p], cls)\n",
    "        videos = [ f for f in os.listdir(class_path) if os.path.isfile(os.path.join(class_path, f)) ]\n",
    "        for vid in videos:\n",
    "            video_path = os.path.join(class_path, vid)\n",
    "\n",
    "            cap = cv2.VideoCapture(video_path)\n",
    "            df = df.append({\n",
    "                'name': vid,\n",
    "                'class': cls,\n",
    "                'frames': int(cap.get(cv2.CAP_PROP_FRAME_COUNT)),\n",
    "                'phase': p\n",
    "            }, ignore_index=True)\n",
    "            cap.release()\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Samples in train set: 522\n",
      "Samples in valid set: 128\n",
      "Samples in test set: 18\n"
     ]
    }
   ],
   "source": [
    "for p in phases:\n",
    "    print('Samples in {} set: {}'.format(p, len(df[df['phase']==p])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_class_count(df):\n",
    "    fig = plt.figure(figsize=(15, 7))\n",
    "    ax = sns.countplot(x='class', hue='phase', data=df)\n",
    "    for p in ax.patches:\n",
    "        ax.annotate(p.get_height(), (p.get_x()+0.1, p.get_height()+0.15))\n",
    "\n",
    "\n",
    "plot_class_count(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![submission_16_0](https://user-images.githubusercontent.com/52006798/149013103-525d5342-2e2f-4230-88b2-f45ae3c43177.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_frames_dist(df):\n",
    "    # fig = plt.figure(figsize=(20, 10))\n",
    "    ax = sns.displot(df, x='frames', hue='phase', kde=True, element='poly', col='class', )\n",
    "\n",
    "plot_frames_dist(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![submission_17_0](https://user-images.githubusercontent.com/52006798/149013105-ee4d4e51-1f2e-4366-8b41-5c2f2f342db7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1 - Early methods <span id=\"part1\"><span>\n",
    "The first part of the exercise includes two early video processing methods, Early Fusion, and Late Fusion.  \n",
    "\n",
    "Unlike image classification, where a single image has three dimensions, video classification adds a fourth dimension - Time/Frame.  \n",
    "In order to make a CNN model suitable for video classification, a modification is required.  \n",
    "Several methods exist for dealing with this issue, including Early Fusion, Late Fusion, Slow Fusion, and Fast Fusion.  \n",
    "![image](https://user-images.githubusercontent.com/52006798/146635639-db3cd74f-7889-405c-ae82-1e3aaab9c0e8.png)  \n",
    "Implementing this exercise required either Early Fusion or Late Fusion, and Early Fusion was selected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Early Fusion <span id=\"early-fusion\"><span>\n",
    "In a CNN model, the first convolution layer has an input of three channels, representing the three color channels in the image.  \n",
    "During video-CNN processing, the first layer handles 3 channels of color simultaneously in each frame.  \n",
    "\n",
    "How it works:  \n",
    "Video samples representing batch will be [ B, F, 3, H, W ]. Image CNN architectures, however, utilize 2D CNN blocks.  \n",
    "Additionally, all of the frames must be analyzed simultaneously. There is a reshape required, so the first convolution layer will be performed on all frames in the architecture.  \n",
    "Therefore, the new batch size dimensions are [ B, 3F, H, W ].  \n",
    "It means two things, each batch should be modified at runtime, and the first CNN layer in the architecture should be changed in advance to be in the shape of [3F, H, W].  \n",
    "Finally, when dealing with pre-trained architecture, the last fully connected layer should be modified as well to the desired number of classes.  \n",
    "\n",
    "Legend:  \n",
    "* `B` - batch size\n",
    "* `F` - number of frames\n",
    "* `C` - number of channels\n",
    "* `H` - height\n",
    "* `W` - width"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training <span id=\"part1-training\"><span>\n",
    "Various configurations were evaluated to determine which was the best.  \n",
    "EfficientNet, ResNet, and ResNeXt architectures were used.  \n",
    "The Adam and AdamW optimizers were tried, as well as different learning rates and schedulers.  \n",
    "Additionally, two methods of converting video into frames were used: \n",
    "\n",
    "1. Static-Batch: Data preprocessing by selecting a subset of frames from each video and training only on these frames.  \n",
    "As the frames were already stored in the storage and only needed to be loaded, training progress should be incredibly rapid. However, a small dataset may quickly overfit.  \n",
    "Each frame was sampled by a fixed step in this method. step = len(total_frames) / len(subset_frames).  \n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/52006798/147848832-1f1e3890-9aa0-4789-b054-97c17a1d8a84.png\" width=\"30%\"/>    \n",
    "\n",
    "2. Dynamic-Batch: Sample a random group of frames each time.  \n",
    "This would be slower, but it could prevent overfitting since every time a video is loaded, a different set of frames is processed by the network.  \n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/52006798/147848886-b4e54060-7d7c-402a-9fba-ad86eb46df5c.png\" width=\"30%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results <span id=\"part1-results\"><span>\n",
    "The following graphs show a sample of different configurations that were tried.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://user-images.githubusercontent.com/52006798/147827538-4083f42a-1b44-4592-a3ab-043ddd42911d.png\" width=\"30%\"/><img src=\"https://user-images.githubusercontent.com/52006798/147827539-1c5831b5-adb5-4d7d-884b-ebb22a231001.png\" width=\"30%\"/>\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/52006798/147827535-844af00b-dfe3-43e9-81e3-afc93cd5c12c.png\" width=\"30%\"/><img src=\"https://user-images.githubusercontent.com/52006798/147827536-9aef5827-7bac-4869-ace6-36e728cb8a6b.png\" width=\"30%\"/><img src=\"https://user-images.githubusercontent.com/52006798/147827540-7b63f025-0fc1-448e-b5ed-5beea13a3cac.png\" width=\"30%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the preferred configuration was as follows:  \n",
    "\n",
    "*   Pre-trained ResNet-50.\n",
    "*   Pre-processed data (static dataset - the same frames were used in each video during training, instead of random ones).\n",
    "*   32 frames for each video.\n",
    "*   Optimizer: Adam.\n",
    "*   Scheduler: ReduceLROnPlateau (factor=0.8, patience=2).\n",
    "*   Learning rate: 1e-5.\n",
    "*   Batch-size: 8.\n",
    "*   Epochs: 100."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://user-images.githubusercontent.com/52006798/147827800-eb667b7d-d310-4b30-aeab-c0da599c29fe.png\" width=\"45%\"/><img src=\"https://user-images.githubusercontent.com/52006798/147827802-d2ae0176-278b-4359-b85c-497eacaf0ad8.png\" width=\"45%\"/>\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/52006798/147827797-b543147a-7484-4f89-a196-881043f2abf9.png\" width=\"30%\"/><img src=\"https://user-images.githubusercontent.com/52006798/147827798-8d0ca0ac-9f3e-436b-882d-ce220d58152b.png\" width=\"30%\"/><img src=\"https://user-images.githubusercontent.com/52006798/147827799-8c85523d-1104-48cb-8121-db125124737e.png\" width=\"30%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can see above, we achieved accuracy of almost 0.9 and loss a little less than 0.7."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2 - Modern Methods <span id=\"part2\"></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SlowFast <span id=\"slowfast\"></span>\n",
    "\n",
    "The SlowFast architecture handles video in two ways.  \n",
    "In order to capture spatial semantics, the first approach uses a low frame rate, which will be called the Slow pathway.  \n",
    "A high frame rate is used in the second method, called the Fast pathway, to capture motion.  \n",
    "\n",
    "![](https://feichtenhofer.github.io/pubs/slowfast_concept.gif)  \n",
    "\n",
    "While both pathways use ResNet as their backbone, unlike the previous model, these use 3D CNN blocks.  \n",
    "A small number of channels is used to keep the Fast pathway fast since it samples many more frames than the Slow pathway.  \n",
    "In addition, the Fast pathway results are fed into the Slow pathway to improve prediction (taking into account the different shapes in each pathway).  \n",
    "The results from the Slow pathway are not fed into the Fast pathway since the researchers found no improvements while executing it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training <span id=\"part2-training\"></span>\n",
    "\n",
    "SlowFast's training phase required some modifications in the data processing.  \n",
    "Firstly, SlowFast splits into 2 pathways, each of which consumes a different number of frames.  \n",
    "Second, the tensor shape differs from the early method.  \n",
    "While the early methods architecture was fed by [ B, F, 3, H, W ] tensors, the SlowFast architecture was fed by [ B, 3, F, H, W ].  \n",
    "In order to use the same dataset, when handling a SlowFast model, the tensor needs to be reshaped into the correct dimensions and then split into two tensors.  \n",
    "Getting the [ B, 3, F, H, W ] shape for the fast pathway, and the [ B, 3, F/4, H, W ] shape for the slow pathway."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results <span id=\"part2-results\"></span>\n",
    "\n",
    "Configuration used:  \n",
    "\n",
    "*   Pre-trained SlowFast.\n",
    "*   Dynamic data (sample random frames from a video at each epoch).\n",
    "*   32 frames for each video.\n",
    "*   Optimizer: ASGD.\n",
    "*   Scheduler: ReduceLROnPlateau (factor=0.9, patience=4).\n",
    "*   Scheduler: CosineAnnealingLR\n",
    "*   Learning rate: 1e-3.\n",
    "*   Batch-size: 4.\n",
    "*   Epochs: 100.\n",
    "\n",
    "Despite trying several different configurations, we were unable to achieve good results with the SlowFast model.  \n",
    "Therefore, we can't compare the models since SlowFast should yield better results than the simpler methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://user-images.githubusercontent.com/52006798/149008482-d1fa2847-8539-4e8f-818d-edce2274f427.png\" width=\"45%\"/><img src=\"https://user-images.githubusercontent.com/52006798/149008486-76272fc2-bbc6-420e-8964-c69227e58ea6.png\" width=\"45%\"/>\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/52006798/149008476-4e424da6-f4c3-4029-9a31-a0315df3f4ee.png\" width=\"30%\"/><img src=\"https://user-images.githubusercontent.com/52006798/149008492-071af52c-0e36-4082-a422-53a921e8c12e.png\" width=\"30%\"/><img src=\"https://user-images.githubusercontent.com/52006798/149008494-3c0f927b-ef92-434b-ab86-55575f3828ab.png\" width=\"30%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References <span id=\"refs\"><span>\n",
    "* [Video Classification with CNN, RNN, and PyTorch](https://medium.com/howtoai/video-classification-with-cnn-rnn-and-pytorch-abe2f9ee031)\n",
    "* [Video Classification with CNN+LSTM](https://discuss.pytorch.org/t/video-classification-with-cnn-lstm/113413/4)\n",
    "* [Introduction to Video Classification](https://towardsdatascience.com/introduction-to-video-classification-6c6acbc57356)\n",
    "* [Justin Johnson's Lecture](https://web.eecs.umich.edu/~justincj/slides/eecs498/FA2020/598_FA2020_lecture18.pdf)\n",
    "* [Christoph Feichtenhofer](https://feichtenhofer.github.io/)\n",
    "* [SlowFast Explained: Dual-mode CNN for Video Understanding](https://towardsdatascience.com/slowfast-explained-dual-mode-cnn-for-video-understanding-8bf639960256)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "006da77768e8ea81ca7d6c59a79c6d4885e1281e72d4ed14e6e170661bd56a79"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit ('lab3': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}