COLAB_DEMO.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "COLAB_maDLC_TrainNetwork_VideoAnalysis.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/DeepLabCut/DeepLabCut_maDLC_DemoData/blob/master/COLAB_DEMO.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK255E7YoEIt"
      },
      "source": [
        "# DeepLabCut 2.2 Toolbox - COLAB DEMO maDLC DATA\n",
        "![alt text](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1609244670293-I4G1WXUK4R4ELSG20YWN/dlc_logo_whiteBkgnd.png?format=1000w)\n",
        "\n",
        "https://github.com/DeepLabCut/DeepLabCut\n",
        "\n",
        "This notebook illustrates how to, for multi-animal projects, use the cloud-based GPU to:\n",
        "- create a multi-animal training set\n",
        "- train a network\n",
        "- evaluate a network\n",
        "- analyze novel videos\n",
        "- assemble animals and tracklets\n",
        "- create quality check plots!\n",
        "\n",
        "\n",
        "This notebook demonstrates the necessary steps to use DeepLabCut.\n",
        "\n",
        "This shows the most simple code to do so, but many of the functions have additional features, so please check out the docs on GitHub. We also recommend checking out our preprint, which covers the science of maDLC\n",
        "\n",
        "**Lauer et al 2021:** https://www.biorxiv.org/content/10.1101/2021.04.30.442096v1\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "txoddlM8hLKm"
      },
      "source": [
        "## First, go to \"Runtime\" ->\"change runtime type\"->select \"Python3\", and then select \"GPU\"\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bvQ1iTszMYOI"
      },
      "source": [
        "\n",
        "# Clone the entire deeplabcut repo so we can use the demo data:\n",
        "!git clone -l -s git://github.com/DeepLabCut/DeepLabCut_maDLC_DemoData.git cloned-DLC-repo\n",
        "%cd cloned-DLC-repo\n",
        "!ls"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q23BzhA6CXxu"
      },
      "source": [
        "#(this will take a few minutes to install all the dependences!)\n",
        "#!pip install deeplabcut==2.2rc3\n",
        "!pip install git+https://github.com/DeepLabCut/DeepLabCut.git\n",
        "!pip install --upgrade scikit-image\n",
        "%reload_ext numpy\n",
        "%reload_ext scipy\n",
        "%reload_ext matplotlib\n",
        "%reload_ext mpl_toolkits"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqUEb8TBdpWb"
      },
      "source": [
        "After the package is installed, please click \"restart runtime\" for DLC changes to take effect in your COLAB environment. You will see this button at the output of the cell above ^."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oTwAcbq2-FZz"
      },
      "source": [
        "import deeplabcut"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vhENAlQnFENJ"
      },
      "source": [
        "# no need to edit for demo:\n",
        "ProjectFolderName = 'MontBlanc-Daniel-2019-12-16'\n",
        "VideoType = 'mov' #, mp4, MOV, or avi, whatever you uploaded!\n",
        "\n",
        "\n",
        "# No need to edit this, we are going to assume you put videos you want to analyze in the \"videos\" folder, but if this is NOT true, edit below:\n",
        "videofile_path = ['/content/cloned-DLC-repo/'+ProjectFolderName+'/videos/'] #Enter the list of videos or folder to analyze.\n",
        "videofile_path\n",
        "\n",
        "#No need to edit this, as you set it when you passed the ProjectFolderName (above): \n",
        "path_config_file = '/content/cloned-DLC-repo/'+ProjectFolderName+'/config.yaml'\n",
        "path_config_file\n",
        "#This creates a path variable that links to your google drive project"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xNi9s1dboEJN"
      },
      "source": [
        "## Create a multi-animal training dataset:\n",
        "\n",
        "- more info: https://deeplabcut.github.io/DeepLabCut/docs/maDLC_UserGuide.html#create-training-dataset\n",
        "\n",
        "- please check the text below, edit if needed, and then click run (this can take some time):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-AAYSoW313me"
      },
      "source": [
        "#OPTIONAL LEARNING: did you know you can check what each function does by running with a ?\n",
        "deeplabcut.create_multianimaltraining_dataset?"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "eMeUwgxPoEJP"
      },
      "source": [
        "# ATTENTION:\n",
        "\n",
        "#which shuffle do you want to create and train?\n",
        "shuffle = 1 #edit if needed; 1 is the default.\n",
        "\n",
        "#Note, you must run this. If your images are smaller than 400 by 400, please make these numbers smaller.\n",
        "deeplabcut.cropimagesandlabels(path_config_file, size=(400, 400), userfeedback=False)\n",
        "\n",
        "#if you labeled on Windows, please set the windows2linux=True:\n",
        "deeplabcut.create_multianimaltraining_dataset(path_config_file, Shuffles=[shuffle], net_type=\"dlcrnet_ms5\",windows2linux=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c4FczXGDoEJU"
      },
      "source": [
        "## Start training:\n",
        "This function trains the network for a specific shuffle of the training dataset. \n",
        " - more info: https://deeplabcut.github.io/DeepLabCut/docs/maDLC_UserGuide.html#train-the-network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_pOvDq_2oEJW",
        "collapsed": true
      },
      "source": [
        "#let's also change the display and save_iters just in case Colab takes away the GPU... \n",
        "#Typically, you want to train to 50,000 - 200K iterations.\n",
        "#more info and there are more things you can set: https://github.com/DeepLabCut/DeepLabCut/blob/master/docs/functionDetails.md#g-train-the-network\n",
        "\n",
        "deeplabcut.train_network(path_config_file, shuffle=shuffle, displayiters=100,saveiters=1000, maxiters=75000, allow_growth=True)\n",
        "\n",
        "#this will run until you stop it (CTRL+C), or hit \"STOP\" icon, or when it hits the end (default, 50K iterations). \n",
        "#Whichever you chose, you will see what looks like an error message, but it's not an error - don't worry...."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RiDwIVf5-3H_"
      },
      "source": [
        "**When you hit \"STOP\" you will get a KeyInterrupt \"error\"! No worries! :)**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xZygsb2DoEJc"
      },
      "source": [
        "## Start evaluating: \n",
        "\n",
        " - First, we evaluate the pose estimation performance.\n",
        "\n",
        "- This funtion evaluates a trained model for a specific shuffle/shuffles at a particular state or all the states on the data set (images) and stores the results as .5 and .csv file in a subdirectory under **evaluation-results**\n",
        "\n",
        "- If the scoremaps do not look accurate, don't proceed to tracklet assembly; please consider (1) adding more data, (2) adding more bodyparts!\n",
        "\n",
        "- more info: https://deeplabcut.github.io/DeepLabCut/docs/maDLC_UserGuide.html#evaluate-the-trained-network\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nv4zlbrnoEJg"
      },
      "source": [
        "#let's evaluate first:\n",
        "deeplabcut.evaluate_network(path_config_file,Shuffles=[shuffle], plotting=True)\n",
        "#plot a few scoremaps:\n",
        "deeplabcut.extract_save_all_maps(path_config_file, shuffle=shuffle, Indices=[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fYlGbloolDU2"
      },
      "source": [
        "IF these images, numbers, and maps do not look good, do not proceed. You should increase the diversity and number of frames you label, and re-create a training dataset and re-train! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVFLSKKfoEJk"
      },
      "source": [
        "## Start Analyzing videos: \n",
        "This function analyzes the new video. The user can choose the best model from the evaluation results and specify the correct snapshot index for the variable **snapshotindex** in the **config.yaml** file. Otherwise, by default the most recent snapshot is used to analyse the video.\n",
        "\n",
        "The results are stored in a pickle file in the same directory where the video resides. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y_LZiS_0oEJl"
      },
      "source": [
        "print(\"Start Analyzing my video(s)!\")\n",
        "#EDIT OPTION: which video(s) do you want to analyze? You can pass a path or a folder:\n",
        "# currently, if you run \"as is\" it assumes you have a video in the DLC project video folder!\n",
        "\n",
        "deeplabcut.analyze_videos(path_config_file,videofile_path, videotype=VideoType)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91xBLOcBzGxo"
      },
      "source": [
        "Optional: Now you have the option to check the raw dections before animals are assembled. To do so, pass a video path:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "65mWwX5bTc5C"
      },
      "source": [
        "##### PROTIP: #####\n",
        "## look at the output video; if the pose estimation (i.e. key points)\n",
        "##  don't look good, don't proceed with tracking - add more data to your training set and re-train!\n",
        "\n",
        "#EDIT: let's check a specific video (PLEASE EDIT VIDEO PATH):\n",
        "Specific_videofile = '/content/cloned-DLC-repo/MontBlanc-Daniel-2019-12-16/videos/montblanc.mov'\n",
        "\n",
        "#don't edit:\n",
        "deeplabcut.create_video_with_all_detections(path_config_file, [Specific_videofile])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3-OgTJ0Lz20e"
      },
      "source": [
        "If the resutling video (ends in full.mp4) is not good, we highly recommend adding more data and training again. See here: https://deeplabcut.github.io/DeepLabCut/docs/maDLC_UserGuide.html#decision-break-point"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PxRLS2_-r55K"
      },
      "source": [
        "# Next, we will assemble animals using our data-driven optimal graph method:\n",
        "\n",
        "- Here, we will find the optimal graph, which matches the \"data-driven\" method from our paper (Figure adapted from Lauer et al. 2021):\n",
        "![alt text](https://images.squarespace-cdn.com/content/v1/57f6d51c9f74566f55ecf271/1626266017809-XO6NX84QB4FBAZGOTCEY/fig3.jpg?format=700w)\n",
        "\n",
        "\n",
        "- note, you can set the number of animals you expect to see, so check, edit, then click run:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zIvXM7TXIs-U"
      },
      "source": [
        "#Check and edit:\n",
        "numAnimals = 8 #how many animals do you expect to find?\n",
        "tracktype= 'ellipse' #box, skeleton, ellipse -- ellipse is recommended.\n",
        "\n",
        "\n",
        "#Don't edit:\n",
        "deeplabcut.convert_detections2tracklets(path_config_file, videofile_path, videotype=VideoType,\n",
        "                                                    shuffle=shuffle, track_method=tracktype, overwrite=True)\n",
        "\n",
        "deeplabcut.stitch_tracklets(path_config_file, videofile_path, track_method=tracktype, n_tracks=numAnimals)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TqbAnyfL0Q7h"
      },
      "source": [
        "Now let's filter the data to remove any small jitter:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a6izVWX8sdzL"
      },
      "source": [
        "deeplabcut.filterpredictions(path_config_file, \n",
        "                                 videofile_path, \n",
        "                                 videotype=VideoType, \n",
        "                                 track_method = tracktype)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zk4xGb8Ftf3B"
      },
      "source": [
        "## Create plots of your trajectories:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gX21zZbXoEKJ"
      },
      "source": [
        "deeplabcut.plot_trajectories(path_config_file, videofile_path, videotype=VideoType, track_method=tracktype)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pqaCw15v8EmB"
      },
      "source": [
        "Now you can look at the plot-poses file and check the \"plot-likelihood.png\" might want to change the \"p-cutoff\" in the config.yaml file so that you have only high confidnece points plotted in the video. i.e. ~0.8 or 0.9. The current default is 0.4. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCrUvQIvoEKD"
      },
      "source": [
        "## Create labeled video:\n",
        "This funtion is for visualiztion purpose and can be used to create a video in .mp4 format with labels predicted by the network. This video is saved in the same directory where the original video resides. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6aDF7Q7KoEKE"
      },
      "source": [
        "deeplabcut.create_labeled_video(path_config_file,\n",
        "                                videofile_path, \n",
        "                                shuffle=shuffle, \n",
        "                                color_by=\"individual\",\n",
        "                                videotype=VideoType, \n",
        "                                save_frames=False,\n",
        "                                filtered=True, \n",
        "                                track_method = tracktype)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}