flexible_mtl.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "flexible_mtl.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Flexible MTL design for testing different configurations\n",
        "\n",
        "- add comments\n"
      ],
      "metadata": {
        "id": "Wx0-RyBmMikb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Set up\n",
        "\n",
        "First, we need to clone the data. Instead of manually uploading, please clone the repository from Yipeng by running:\n",
        "\n",
        "`!git clone -b oxpet --single-branch https://weisslab.cs.ucl.ac.uk/WEISSTeaching/datasets.git`"
      ],
      "metadata": {
        "id": "c0w1DYWdOC6L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone -b oxpet --single-branch https://weisslab.cs.ucl.ac.uk/WEISSTeaching/datasets.git"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "01E9xLC4Mltp",
        "outputId": "060c66da-cca1-48d0-d414-6601623ea7cb"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "fatal: destination path 'datasets' already exists and is not an empty directory.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then, upload the `library.zip` file. This is a zipped version of the entire library, for is a manual process at the moment, until we work on a better way of exporting our library.\n",
        "\n",
        "Please not that torchvision.models.utils is not supported on the environment run here. A manual fix has been added to patch this. This should be visible when you run the code. Once uploaded, unzip the library using `unzip  library.zip`"
      ],
      "metadata": {
        "id": "t64g4q99QyCD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!unzip -o flexible_mlt.zip "
      ],
      "metadata": {
        "id": "RRmGwjAzNSTm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4efe2e23-3e47-4fe5-8989-9ef00527d000"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Archive:  flexible_mlt.zip\n",
            "  inflating: data/data.py            \n",
            "  inflating: data/__pycache__/data.cpython-38.pyc  \n",
            "  inflating: data/__pycache__/data.cpython-39.pyc  \n",
            "  inflating: data/__pycache__/data.cpython-37.pyc  \n",
            "  inflating: criterion/loss_functions.py  \n",
            "  inflating: criterion/metric_functions.py  \n",
            "  inflating: criterion/criterion.py  \n",
            "  inflating: criterion/__pycache__/criterion.cpython-37.pyc  \n",
            "  inflating: criterion/__pycache__/loss_functions.cpython-37.pyc  \n",
            "  inflating: criterion/__pycache__/loss_functions.cpython-39.pyc  \n",
            "  inflating: criterion/__pycache__/metric_functions.cpython-37.pyc  \n",
            "  inflating: criterion/__pycache__/criterion.cpython-39.pyc  \n",
            "  inflating: main.py                 \n",
            "  inflating: __init__.py             \n",
            "  inflating: utils.py                \n",
            "  inflating: requirements.txt        \n",
            "  inflating: train.py                \n",
            "  inflating: tests/test_pt_continuous_train.py  \n",
            "  inflating: tests/test_loader.py    \n",
            "  inflating: tests/test_model_training_class.py  \n",
            "  inflating: models/bodys.py         \n",
            "  inflating: models/densenet.py      \n",
            "  inflating: models/__init__.py      \n",
            "  inflating: models/model.py         \n",
            "  inflating: models/resnet.py        \n",
            "  inflating: models/utils.py         \n",
            "  inflating: models/heads.py         \n",
            "  inflating: models/__pycache__/bodys.cpython-37.pyc  \n",
            "  inflating: models/__pycache__/resnet.cpython-37.pyc  \n",
            "  inflating: models/__pycache__/model.cpython-37.pyc  \n",
            "  inflating: models/__pycache__/heads.cpython-37.pyc  \n",
            "  inflating: models/__pycache__/utils.cpython-39.pyc  \n",
            "  inflating: models/__pycache__/utils.cpython-38.pyc  \n",
            "  inflating: models/__pycache__/heads.cpython-39.pyc  \n",
            "  inflating: models/__pycache__/utils.cpython-37.pyc  \n",
            "  inflating: models/__pycache__/resnet.cpython-38.pyc  \n",
            "  inflating: models/__pycache__/resnet.cpython-39.pyc  \n",
            "  inflating: models/__pycache__/model.cpython-38.pyc  \n",
            "  inflating: models/__pycache__/model.cpython-39.pyc  \n",
            "  inflating: models/__pycache__/__init__.cpython-37.pyc  \n",
            "  inflating: models/__pycache__/bodys.cpython-39.pyc  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we must install the relevant librries from the requirements file (this should be included in library.zip). Do this by running the following:\n",
        "`!python3 -m venv env`\n",
        "\n",
        "`!source env/bin/activate`\n",
        "\n",
        "`!pip install -r requirements.txt`\n",
        "\n",
        "You may get an error that the latest version of numpy could not be found. You can install this manually, although I believe it should exist by defualt in the first palce."
      ],
      "metadata": {
        "id": "62vB8MGvNmcR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python3 -m venv mtl_test_colab\n",
        "!source mtl_test_colab/bin/activate\n",
        "!which python\n",
        "!pip install -r requirements.txt"
      ],
      "metadata": {
        "id": "HjnJT1XgNwWk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "df6e34cc-8e83-42d3-ddf1-3b7aef43a0be"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Error: Command '['/content/mtl_test_colab/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.\n",
            "/bin/bash: mtl_test_colab/bin/activate: No such file or directory\n",
            "/usr/local/bin/python\n",
            "Requirement already satisfied: certifi==2021.10.8 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 1)) (2021.10.8)\n",
            "Requirement already satisfied: cycler==0.11.0 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 2)) (0.11.0)\n",
            "Collecting fonttools==4.28.5\n",
            "  Using cached fonttools-4.28.5-py3-none-any.whl (890 kB)\n",
            "Collecting h5py==3.6.0\n",
            "  Using cached h5py-3.6.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.1 MB)\n",
            "Collecting imageio==2.13.5\n",
            "  Using cached imageio-2.13.5-py3-none-any.whl (3.3 MB)\n",
            "Requirement already satisfied: joblib==1.1.0 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 6)) (1.1.0)\n",
            "Requirement already satisfied: kiwisolver==1.3.2 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 7)) (1.3.2)\n",
            "Collecting matplotlib==3.5.1\n",
            "  Using cached matplotlib-3.5.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.2 MB)\n",
            "Requirement already satisfied: networkx==2.6.3 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 9)) (2.6.3)\n",
            "\u001b[31mERROR: Could not find a version that satisfies the requirement numpy==1.22.0 (from versions: 1.3.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0.post2, 1.10.1, 1.10.2, 1.10.4, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.12.0, 1.12.1, 1.13.0rc1, 1.13.0rc2, 1.13.0, 1.13.1, 1.13.3, 1.14.0rc1, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0rc1, 1.15.0rc2, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0rc1, 1.16.0rc2, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.17.0rc1, 1.17.0rc2, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.18.0rc1, 1.18.0, 1.18.1, 1.18.2, 1.18.3, 1.18.4, 1.18.5, 1.19.0rc1, 1.19.0rc2, 1.19.0, 1.19.1, 1.19.2, 1.19.3, 1.19.4, 1.19.5, 1.20.0rc1, 1.20.0rc2, 1.20.0, 1.20.1, 1.20.2, 1.20.3, 1.21.0rc1, 1.21.0rc2, 1.21.0, 1.21.1, 1.21.2, 1.21.3, 1.21.4, 1.21.5)\u001b[0m\n",
            "\u001b[31mERROR: No matching distribution found for numpy==1.22.0\u001b[0m\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Now... you design and run the model of your choice!\n",
        "\n",
        "```diff\n",
        "!PLEASE read before running \n",
        "```\n",
        "This notebook is different than the ones I've sent previously. The major difference is that it allows you to quickly and easily design the MTL model of your choice with ease, without needing to change the backend code. It also has added support for storing loss values, validation data, testing data and also the use of custom metrics and custom losses.\n",
        "\n",
        "Please note, it does NOT currently support saving and loading of models. That is a work in progress, but I've put it as a low priority task since it seems like our models are running quickly anyways. If there are urgent issues, then I will get it done ASAP.\n",
        "\n",
        "Please read the instructions here as they will give you a good understanding of how this is intended to be used. Run experiments on it, and if there are any issues notify me and create them on GitHub."
      ],
      "metadata": {
        "id": "cHLSmCgfNXTw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## core libraries to load\n",
        "\n",
        "\n",
        "## define the tasks that you are going to use\n",
        "\n",
        "TASKS = ['class', 'seg', 'bb']"
      ],
      "metadata": {
        "id": "-0H3IoQVQB6f"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Loading the data\n",
        "\n",
        "It is good practice to always laod the data first. The dataloader has been modified to take in a path input from the user, to make it more flexible."
      ],
      "metadata": {
        "id": "RtnHty3dP6wP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## load data libraries\n",
        "from data.data import OxpetDataset, RandomBatchSampler\n",
        "from torch.utils.data import BatchSampler, DataLoader"
      ],
      "metadata": {
        "id": "nmuZWuTuPipB"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "## get your datasets\n",
        "\n",
        "data_path = 'datasets/data_new/' # point to the directory where the data sits\n",
        "trainset = OxpetDataset(data_path + 'train', TASKS)\n",
        "testset = OxpetDataset(data_path + 'test', TASKS)\n",
        "valset = OxpetDataset(data_path + 'val', TASKS)\n",
        "\n",
        "## define your dataloaders\n",
        "TRAIN_BATCH_SIZE = 8\n",
        "TEST_BATCH_SIZE = 8\n",
        "VAL_BATCH_SIZE = 8\n",
        "\n",
        "trainloader = DataLoader(trainset, batch_size=None,\n",
        "                         sampler=BatchSampler(RandomBatchSampler(trainset, TRAIN_BATCH_SIZE), batch_size=TRAIN_BATCH_SIZE, drop_last=False))\n",
        "\n",
        "testloader = DataLoader(testset, batch_size=None,\n",
        "                         sampler=BatchSampler(RandomBatchSampler(testset, TEST_BATCH_SIZE), batch_size=TEST_BATCH_SIZE, drop_last=False))\n",
        "valloader = DataLoader(valset, batch_size=None,\n",
        "                         sampler=BatchSampler(RandomBatchSampler(valset, VAL_BATCH_SIZE), batch_size=VAL_BATCH_SIZE, drop_last=False))\n"
      ],
      "metadata": {
        "id": "4-0FdnAAPhwq"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Defining the model"
      ],
      "metadata": {
        "id": "-XYZR54NRm8N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## imports\n",
        "\n",
        "from models.model import resnet34_bb, resnet34_bb, resnet34_class, resnet34_class_bb, resnet34_seg, resnet34_seg_bb, resnet34_seg_class, resnet34_seg_class_bb\n",
        "from torch.optim import Adam\n",
        "\n",
        "MODEL = resnet34_seg_class_bb()\n",
        "OPTIMIZER = Adam(MODEL.parameters(), lr=1e-04)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uba1y1l4RmAB",
        "outputId": "706f7656-2843-44b8-dcfb-07c816c7c978"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pytorch version is: 1.10.0+cu111. torchvision.models.utils does not exist\n",
            "Importing load_state_dict_from_url from torch.hub instead...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Defining the loss, metrics\n",
        "\n",
        "This is the newest addition to the code. There are some inbuilt losses and meteics, but there is work being done on expanding these. The most important of these are the standard combined loss, which combines the losses with an added weighting. You can also design your own losses. Make sure when doing so, that you subclass from CombinedLoss (see below)."
      ],
      "metadata": {
        "id": "Ret0HTnjSJkM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## example of creating your own loss\n",
        "from main import CombinedLoss # ignore the CPU/GPU messages here..\n",
        "import torch\n",
        "class SimpleCombinedLoss(CombinedLoss):\n",
        "    def __init__(self, loss_dict, weights=None):\n",
        "        super(SimpleCombinedLoss, self).__init__()  # you must inherit superclass\n",
        "        self.loss_dict = loss_dict  # you must define this as such\n",
        "\n",
        "\n",
        "        # feel free to add any configurations for the loss of your choice\n",
        "        # in this case, we've done for a simple weighted loss.\n",
        "        if weights is None:\n",
        "            self.weights = torch.ones(size=(len(self.loss_dict), ))\n",
        "        else:\n",
        "            self.weights = weights\n",
        "\n",
        "    def forward(self, outputs, targets):\n",
        "        # while the contents of the dictionary may vary, you MUST set self.losses to a dictionary that contains your losses\n",
        "        self.loss_values = {\n",
        "            task: weight * loss(outputs[task], targets[task]) for (task, loss), weight in zip(self.loss_dict.items(), self.weights)\n",
        "        }\n",
        "        return sum(self.loss_values.values())"
      ],
      "metadata": {
        "id": "0F9CJsPPSLpg"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "For our case, we are interested in 3 types of losses. A loss with no weights, a loss with weights, and a loss with weights based on a prior. So..."
      ],
      "metadata": {
        "id": "UjRMy81rUYUO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from main import RandomCombinedLoss\n",
        "from criterion.loss_functions import DiceLoss\n",
        "from torch.nn import CrossEntropyLoss, L1Loss\n",
        "\n",
        "LOSSES = [CrossEntropyLoss(), DiceLoss(), L1Loss()]\n",
        "LOSS_DICT = {task: loss for task, loss in zip(TASKS, LOSSES)}\n",
        "# print out loss dict for sanity check\n",
        "print(LOSS_DICT)\n",
        "\n",
        "LOSS1 = SimpleCombinedLoss(LOSS_DICT)  # initialise with no weights\n",
        "#LOSS2 = SimpleCombinedLoss(LOSS_DICT, weights=[0.8, 0.1, 0.1]) # give most weight to class\n",
        "LOSS3 = RandomCombinedLoss(LOSS_DICT, prior='constrained_bernoulli', frequency=70)  #updates every 70 minibatches, e.g. 1 epoch"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n5v1ffS2Ukdq",
        "outputId": "1ad35d72-5214-40de-d8b8-7ff73cc31bbf"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'class': CrossEntropyLoss(), 'seg': DiceLoss(), 'bb': L1Loss()}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# We are now ready to run the model!!!\n",
        "\n",
        "To facilitate an easy way of doing this, I've created a generalised class for running and testing. It is missing some features (e.g. saving) but i'm releasing it now for user testing."
      ],
      "metadata": {
        "id": "1dgEbLvuVizb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from main import RunTorchModel\n",
        "from criterion.metric_functions import Accuracy\n",
        "\n",
        "# initialise run, this is analogous to model.compile() in keras\n",
        "run_instance = RunTorchModel(\n",
        "    model=MODEL, optimizer=OPTIMIZER, loss=LOSS1, metrics={'class':[Accuracy()]}\n",
        ")\n",
        "\n",
        "# training params\n",
        "EPOCHS=5\n",
        "VERBOSE=2\n",
        "TRACK_HISTORY=True # to save model hist\n",
        "\n",
        "run_instance.train(trainloader, valloader=valloader, epochs=EPOCHS, verbose=VERBOSE, track_history=TRACK_HISTORY) # analogous to model.fit() in keras\n",
        "run_instance.get_history() # training history"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GBJdWKUsVhiW",
        "outputId": "a6e159e8-98ff-46e2-d14f-888935239401"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CUDA device not detected. Running on CPU instead.\n",
            "Training model...\n",
            "EPOCH 1 STARTED\n",
            "---------------\n",
            "class tensor(0.2798, dtype=torch.float64, grad_fn=<NllLossBackward0>)\n",
            "seg tensor(0.4760, grad_fn=<RsubBackward1>)\n",
            "bb tensor(117.4887, dtype=torch.float64, grad_fn=<L1LossBackward0>)\n",
            "Batch 1 complete. Time taken: load(0.473), train(20.5), total(21). \n",
            "class tensor(0.2591, dtype=torch.float64, grad_fn=<NllLossBackward0>)\n",
            "seg tensor(0.5373, grad_fn=<RsubBackward1>)\n",
            "bb tensor(111.4976, dtype=torch.float64, grad_fn=<L1LossBackward0>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "run_instance.test(testloader)"
      ],
      "metadata": {
        "id": "MBmRVcNdXMeW"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}