notebook_03_MLP_vs_LogisticRegression_text_classification.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/julurisaichandu/nlp/blob/main/perceptron_vs_logistic_regression.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pfLkIdvqWHsp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fcc1f8c6-f345-4bac-e1ae-a40a18106f1b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import nltk\n",
        "from collections import Counter\n",
        "import numpy as np\n",
        "nltk.download(\"punkt\")\n",
        "from sklearn.metrics import precision_score, recall_score, f1_score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OuFMRYKGWHsw"
      },
      "source": [
        "Notebook 3: Multilayer Perceptron\n",
        "===============\n",
        "\n",
        "CS 6120 Natural Language Processing, Amir\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DX1rm_SiWHsz"
      },
      "source": [
        "Saichandu Juluri"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUZI3Q50WHs0"
      },
      "source": [
        "Saving notebooks as pdfs\n",
        "----------\n",
        "\n",
        "Feel free to add cells to this notebook as you wish. Make sure to leave **code that you've written** and any **answers to questions** that you've written in your notebook. Turn in your notebook as a pdf at the end of lecture's day.\n",
        "\n",
        "\n",
        "To convert your notebook to a pdf for turn in, you'll do the following:\n",
        "1. Kernel -> Restart & Run All (clear your kernel's memory and run all cells)\n",
        "2. File -> Download As -> .html -> open in a browser -> print to pdf\n",
        "\n",
        "(The download as pdf option doesn't preserve formatting and output as nicely as taking the step \"through\" html, but will do if the above doesn't work for you.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "knxiavbxWHs1"
      },
      "source": [
        "Task 1: Implement a Multilayer Perceptron for text classification in Torch\n",
        "-------\n",
        "\n",
        "In this notebook you will get to implement neural text classifiers using [Torch](https://pytorch.org/), a very popular deep learning framework. You may need to consult the documentation but since you will need to use this framework for the upcoming homework assignments this is an opportunity to get familiarized with it.\n",
        "\n",
        "The goal is to build neural binary classifiers to predict the toxicity (i.e., toxic vs non-toxic) of a post using data from the [Jigsaw Unintended Bias in Toxicity Classification competition](https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification) (here we will use a very small subset of the data).\n",
        "\n",
        "Recall that a MLP with $l$ hidden layers makes predictions as\n",
        "\n",
        "$z = g(V^{l}\\ldots g(V^2g(V^1f(x))))\\\\$\n",
        "$P(\\hat{y}|x) = \\text{softmax}(Wz)$\n",
        "\n",
        "where $f(x)$ is a feature representation of the input and hidden layer $j$ produces a new feature vector via a linear transformation paramterized by a weight vector $V^j$ followed by an activation function $g(\\cdot)$ (i.e., an elementwise non-linear transformation). We recommend implementing your network using the [nn.Sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) module but you dont have to.\n",
        "\n",
        "The provided code is based on the feedforward neural network for the XOR problem that we saw in class. We also provide code to read the data and build BOW feature vectors. By default the code subsamples the training/test data to make development faster. Feel free to play with the full dataset if time permits."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tttj9T11WHs1"
      },
      "outputs": [],
      "source": [
        "#NEURAL NETWORK DEFINITION\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch import optim\n",
        "import numpy as np\n",
        "import random\n",
        "# fix the randomness to ensure reproducibility\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "np.random.seed(42)\n",
        "random.seed(42)\n",
        "\n",
        "class MLP(nn.Module):\n",
        "    \"\"\"\n",
        "    Defines the core neural network for doing multiclass classification over a single datapoint at a time. The network can be instantiated with arbitrary architectures (by which we mean number and size of hidden layers)\n",
        "    e.g., architecture = [1000, 50, 2] is a MLP with input layer of size 1000, hidden layer of size 50 and output layer of size 2\n",
        "\n",
        "    Recall that the hidden layer is computed as a linear transformation followed by an activation function (i.e., non-linearity)\n",
        "    Linear transformations are implemented with the nn.Linear()\n",
        "    Note 1: the input layer should have the same size as the input feature vectors and the output layer should be the number of classes.\n",
        "    Note 2: be sure to match the input and output dimensions of all the layers\n",
        "    \"\"\"\n",
        "    def __init__(self, architecture):\n",
        "        \"\"\"\n",
        "        Constructs the computation graph by instantiating the various layers and initializing weights.\n",
        "        :param architecture: dimensions of all the layers (list)\n",
        "        \"\"\"\n",
        "        super(MLP, self).__init__()\n",
        "        self.architecture = architecture\n",
        "        self.layers = nn.Sequential(nn.Linear(architecture[0], architecture[1]),\n",
        "                                    nn.ReLU(),\n",
        "                                    nn.Linear(architecture[1], architecture[2]),\n",
        "                                    nn.LogSoftmax(dim=0)\n",
        "                                    )\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Runs the neural network on the given data and returns log probabilities of the various classes.\n",
        "\n",
        "        :param x: a [inp]-sized tensor of input data\n",
        "        :return: an [out]-sized tensor of log probabilities. (In general your network can be set up to return either log\n",
        "        probabilities or a tuple of (loss, log probability) if you want to pass in y to this function as well\n",
        "        \"\"\"\n",
        "        return self.layers(x)\n",
        "\n",
        "    def predict(self, x):\n",
        "      probs = self.forward(x)\n",
        "      return torch.argmax(probs)\n",
        "\n",
        "def form_input(x) -> torch.Tensor:\n",
        "    \"\"\"\n",
        "    Form the input to the neural network. In general this may be a complex function that synthesizes multiple pieces\n",
        "    of data, does some computation, handles batching, etc.\n",
        "\n",
        "    :param x: a [num_samples x inp] numpy array containing input data\n",
        "    :return: a [num_samples x inp] Tensor\n",
        "    \"\"\"\n",
        "    return torch.from_numpy(x).float()\n",
        "\n",
        "def train_model(model, train_xs, train_ys, num_classes, num_epochs, learning_rate):\n",
        "\n",
        "    optimizer = optim.SGD(model.parameters(), lr=learning_rate)\n",
        "    for epoch in range(0, num_epochs):\n",
        "        ex_indices = [i for i in range(0, len(train_xs))]\n",
        "        random.shuffle(ex_indices)\n",
        "        total_loss = 0.0\n",
        "        for idx in ex_indices:\n",
        "            x = form_input(train_xs[idx])\n",
        "            y = train_ys[idx]\n",
        "            # Build one-hot representation of y. Instead of the label 0 or 1, y_onehot is either [0, 1] or [1, 0]. This\n",
        "            # way we can take the dot product directly with a probability vector to get class probabilities.\n",
        "            y_onehot = torch.zeros(num_classes)\n",
        "            # scatter will write the value of 1 into the position of y_onehot given by y\n",
        "            y_onehot.scatter_(0, torch.from_numpy(np.asarray(y,dtype=np.int64)), 1)\n",
        "            # Zero out the gradients from the model object. *THIS IS VERY IMPORTANT TO DO BEFORE CALLING BACKWARD()*\n",
        "            model.zero_grad()\n",
        "            log_probs = model.forward(x)\n",
        "            # Can also use built-in NLLLoss as a shortcut but we're being explicit here\n",
        "            loss = torch.neg(log_probs).dot(y_onehot)\n",
        "            total_loss += loss\n",
        "            # Computes the gradient and takes the optimizer step\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "        print(\"Total loss on epoch %i: %f\" % (epoch, total_loss))\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RLJK0DyrWHs3"
      },
      "outputs": [],
      "source": [
        "#FRAMEWORK CODE\n",
        "\n",
        "def read_data(path, sample_frac=1):\n",
        "    df = pd.read_csv(path)\n",
        "    df = df.iloc[:int(len(df)*sample_frac)]\n",
        "    y = df[\"label\"]\n",
        "    x = df[\"comment_text\"]\n",
        "    return x, np.array(y).astype(np.float32)\n",
        "\n",
        "def build_vocab(X):\n",
        "    MIN_FREQ = 3\n",
        "    ct = Counter()\n",
        "    for x_i in X:\n",
        "        ct.update(nltk.word_tokenize(x_i.lower().strip()))\n",
        "    #only keep words longer than 2 characters that occur at least MIN_FREQ times\n",
        "    vocab = {k:i for i,k in enumerate([k for (k,v) in ct.most_common() if v > MIN_FREQ and len(k)>1])}\n",
        "    return vocab\n",
        "\n",
        "def build_BOW(X, vocab):\n",
        "    bows = []\n",
        "    for x_i in X:\n",
        "        bow = np.zeros(len(vocab)).astype(np.float32)\n",
        "        tokens = nltk.word_tokenize(x_i.lower().strip())\n",
        "        for t in tokens:\n",
        "            if t in vocab:\n",
        "                bow[vocab[t]]+=1\n",
        "        bows.append(bow)\n",
        "    return np.array(bows)\n",
        "\n",
        "def report_metrics(classifier, test_data, golds):\n",
        "  \"\"\"\n",
        "    Applies the trained classifier to test data and computes performance\n",
        "  \"\"\"\n",
        "#   golds = [data[1] for data in test_data]\n",
        "  classified = [classifier.predict(form_input(data)) for data in test_data]\n",
        "  print(\"Precision:\", precision_score(golds, classified))\n",
        "  print(\"Recall:\", recall_score(golds, classified))\n",
        "  print(\"F1:\", f1_score(golds, classified))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8Hh2i2r0WHs4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3d7378cf-e513-4dc5-c36a-b1bc2b0dbe27"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total loss on epoch 0: 2316.756592\n",
            "Total loss on epoch 1: 1998.807129\n",
            "Total loss on epoch 2: 1658.861694\n",
            "Total loss on epoch 3: 1342.262695\n",
            "Total loss on epoch 4: 1053.362549\n",
            "Total loss on epoch 5: 783.003296\n",
            "Total loss on epoch 6: 590.576477\n",
            "Total loss on epoch 7: 448.241241\n",
            "Total loss on epoch 8: 245.942459\n",
            "Total loss on epoch 9: 248.593674\n",
            "Total loss on epoch 10: 152.918808\n",
            "Total loss on epoch 11: 92.979706\n",
            "Total loss on epoch 12: 71.447441\n",
            "Total loss on epoch 13: 60.748013\n",
            "Total loss on epoch 14: 53.105839\n",
            "Total loss on epoch 15: 49.576965\n",
            "Total loss on epoch 16: 44.354233\n",
            "Total loss on epoch 17: 41.094479\n",
            "Total loss on epoch 18: 36.819019\n",
            "Total loss on epoch 19: 35.837116\n",
            "Total loss on epoch 20: 32.902729\n",
            "Total loss on epoch 21: 31.941465\n",
            "Total loss on epoch 22: 30.842512\n",
            "Total loss on epoch 23: 29.314169\n",
            "Total loss on epoch 24: 28.346806\n",
            "Total loss on epoch 25: 26.615921\n",
            "Total loss on epoch 26: 26.814150\n",
            "Total loss on epoch 27: 25.592453\n",
            "Total loss on epoch 28: 25.482359\n",
            "Total loss on epoch 29: 24.572006\n",
            "Total loss on epoch 30: 23.437565\n",
            "Total loss on epoch 31: 21.837181\n",
            "Total loss on epoch 32: 22.893867\n",
            "Total loss on epoch 33: 21.809732\n",
            "Total loss on epoch 34: 20.407007\n",
            "Total loss on epoch 35: 21.680752\n",
            "Total loss on epoch 36: 21.574190\n",
            "Total loss on epoch 37: 20.299795\n",
            "Total loss on epoch 38: 20.827894\n",
            "Total loss on epoch 39: 19.299057\n",
            "Total loss on epoch 40: 20.411192\n",
            "Total loss on epoch 41: 18.900362\n",
            "Total loss on epoch 42: 20.075029\n",
            "Total loss on epoch 43: 18.917736\n",
            "Total loss on epoch 44: 19.473635\n",
            "Total loss on epoch 45: 19.274620\n",
            "Total loss on epoch 46: 18.464401\n",
            "Total loss on epoch 47: 19.102514\n",
            "Total loss on epoch 48: 19.266754\n",
            "Total loss on epoch 49: 18.179176\n",
            " === Train Set Performance === \n",
            "Precision: 0.9994169096209913\n",
            "Recall: 0.9976717112922002\n",
            "F1: 0.9985435479172736\n",
            " === Test Set Performance === \n",
            "Precision: 0.7892503536067893\n",
            "Recall: 0.7246753246753247\n",
            "F1: 0.7555856465809073\n"
          ]
        }
      ],
      "source": [
        "#read train/test data\n",
        "train_docs, train_ys = read_data(\"data/toxicity_small_train.csv\",sample_frac=0.5)\n",
        "test_x, test_y = read_data(\"data/toxicity_small_test.csv\",sample_frac=0.5)\n",
        "#build vocabulary\n",
        "vocab = build_vocab(train_docs)\n",
        "#extract bag-of-word features\n",
        "train_xs = build_BOW(train_docs, vocab)\n",
        "test_xs = build_BOW(test_x, vocab)\n",
        "\n",
        "#Network definition\n",
        "input_layer_d = train_xs.shape[1]\n",
        "hidden_layer_d = 200\n",
        "num_classes = 2\n",
        "#this just an example of how to instantiate the model\n",
        "model = MLP([input_layer_d, hidden_layer_d, num_classes])\n",
        "\n",
        "# RUN TRAINING AND TEST\n",
        "num_epochs = 50\n",
        "initial_learning_rate = 0.01\n",
        "model = train_model(model, train_xs, train_ys, num_classes, num_epochs,\n",
        "                    initial_learning_rate)\n",
        "\n",
        "print(\" === Train Set Performance === \")\n",
        "report_metrics(model, train_xs, train_ys)\n",
        "\n",
        "print(\" === Test Set Performance === \")\n",
        "report_metrics(model, test_xs, test_y)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Experimenting with double the hidden perceptrons than the initial**"
      ],
      "metadata": {
        "id": "EkclmprnbTDp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Experimenting with double the hidden perceptrons than beginning\n",
        "\n",
        "hidden_layer_d = 400\n",
        "num_classes = 2\n",
        "#this just an example of how to instantiate the model\n",
        "model_doubled = MLP([input_layer_d, hidden_layer_d, num_classes])\n",
        "\n",
        "# RUN TRAINING AND TEST\n",
        "num_epochs = 50\n",
        "initial_learning_rate = 0.01\n",
        "model_doubled = train_model(model_doubled, train_xs, train_ys, num_classes,\n",
        "                            num_epochs, initial_learning_rate)\n",
        "\n",
        "print(\" === Train Set Performance === \")\n",
        "report_metrics(model_doubled, train_xs, train_ys)\n",
        "\n",
        "print(\" === Test Set Performance === \")\n",
        "report_metrics(model_doubled, test_xs, test_y)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Sjo7We-BbNpJ",
        "outputId": "5847f49a-18e9-40ce-ac2c-b7767be9a3d4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total loss on epoch 0: 2332.586426\n",
            "Total loss on epoch 1: 2000.013306\n",
            "Total loss on epoch 2: 1652.435303\n",
            "Total loss on epoch 3: 1320.031250\n",
            "Total loss on epoch 4: 991.057922\n",
            "Total loss on epoch 5: 753.167847\n",
            "Total loss on epoch 6: 494.144012\n",
            "Total loss on epoch 7: 357.883728\n",
            "Total loss on epoch 8: 276.690125\n",
            "Total loss on epoch 9: 137.142593\n",
            "Total loss on epoch 10: 99.156898\n",
            "Total loss on epoch 11: 78.633606\n",
            "Total loss on epoch 12: 66.142441\n",
            "Total loss on epoch 13: 56.996254\n",
            "Total loss on epoch 14: 50.048901\n",
            "Total loss on epoch 15: 45.829216\n",
            "Total loss on epoch 16: 42.156410\n",
            "Total loss on epoch 17: 39.337246\n",
            "Total loss on epoch 18: 38.193947\n",
            "Total loss on epoch 19: 34.715218\n",
            "Total loss on epoch 20: 33.004803\n",
            "Total loss on epoch 21: 31.144903\n",
            "Total loss on epoch 22: 29.691940\n",
            "Total loss on epoch 23: 28.265812\n",
            "Total loss on epoch 24: 27.564047\n",
            "Total loss on epoch 25: 26.822023\n",
            "Total loss on epoch 26: 25.839792\n",
            "Total loss on epoch 27: 25.584017\n",
            "Total loss on epoch 28: 24.097546\n",
            "Total loss on epoch 29: 24.641146\n",
            "Total loss on epoch 30: 23.260279\n",
            "Total loss on epoch 31: 22.703568\n",
            "Total loss on epoch 32: 22.398506\n",
            "Total loss on epoch 33: 21.585571\n",
            "Total loss on epoch 34: 19.994490\n",
            "Total loss on epoch 35: 22.175150\n",
            "Total loss on epoch 36: 21.495594\n",
            "Total loss on epoch 37: 20.215986\n",
            "Total loss on epoch 38: 20.856340\n",
            "Total loss on epoch 39: 20.283066\n",
            "Total loss on epoch 40: 19.294230\n",
            "Total loss on epoch 41: 20.085339\n",
            "Total loss on epoch 42: 18.041498\n",
            "Total loss on epoch 43: 18.693550\n",
            "Total loss on epoch 44: 17.345131\n",
            "Total loss on epoch 45: 18.335068\n",
            "Total loss on epoch 46: 18.386600\n",
            "Total loss on epoch 47: 18.595896\n",
            "Total loss on epoch 48: 18.067453\n",
            "Total loss on epoch 49: 18.217485\n",
            " === Train Set Performance === \n",
            "Precision: 1.0\n",
            "Recall: 0.9970896391152503\n",
            "F1: 0.9985426989215972\n",
            " === Test Set Performance === \n",
            "Precision: 0.8154311649016641\n",
            "Recall: 0.7\n",
            "F1: 0.753319357092942\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Experimenting with triple the hidden perceptrons than the initial**"
      ],
      "metadata": {
        "id": "t1i0fvvT781v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Experimenting with triple hidden perceptrons than initial 200\n",
        "\n",
        "hidden_layer_d = 800\n",
        "num_classes = 2\n",
        "#this just an example of how to instantiate the model\n",
        "model_tripled = MLP([input_layer_d, hidden_layer_d, num_classes])\n",
        "\n",
        "# RUN TRAINING AND TEST\n",
        "num_epochs = 50\n",
        "initial_learning_rate = 0.01\n",
        "model_tripled = train_model(model_tripled, train_xs, train_ys, num_classes,\n",
        "                            num_epochs, initial_learning_rate)\n",
        "\n",
        "print(\" === Train Set Performance === \")\n",
        "report_metrics(model_tripled, train_xs, train_ys)\n",
        "\n",
        "print(\" === Test Set Performance === \")\n",
        "report_metrics(model_tripled, test_xs, test_y)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nblrJ6D_ilJ9",
        "outputId": "d072c707-711c-4d4b-fc75-a77dfa9c0eea"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total loss on epoch 0: 2310.184570\n",
            "Total loss on epoch 1: 1966.795654\n",
            "Total loss on epoch 2: 1592.843262\n",
            "Total loss on epoch 3: 1248.381836\n",
            "Total loss on epoch 4: 952.693970\n",
            "Total loss on epoch 5: 707.831360\n",
            "Total loss on epoch 6: 519.451477\n",
            "Total loss on epoch 7: 253.913666\n",
            "Total loss on epoch 8: 162.996674\n",
            "Total loss on epoch 9: 202.486450\n",
            "Total loss on epoch 10: 229.482941\n",
            "Total loss on epoch 11: 237.042206\n",
            "Total loss on epoch 12: 78.159943\n",
            "Total loss on epoch 13: 61.066521\n",
            "Total loss on epoch 14: 51.797520\n",
            "Total loss on epoch 15: 46.039246\n",
            "Total loss on epoch 16: 41.130043\n",
            "Total loss on epoch 17: 38.697193\n",
            "Total loss on epoch 18: 35.953083\n",
            "Total loss on epoch 19: 32.619350\n",
            "Total loss on epoch 20: 31.112679\n",
            "Total loss on epoch 21: 28.341776\n",
            "Total loss on epoch 22: 29.610992\n",
            "Total loss on epoch 23: 27.882368\n",
            "Total loss on epoch 24: 26.641840\n",
            "Total loss on epoch 25: 25.665567\n",
            "Total loss on epoch 26: 26.084274\n",
            "Total loss on epoch 27: 24.053059\n",
            "Total loss on epoch 28: 24.326094\n",
            "Total loss on epoch 29: 22.896803\n",
            "Total loss on epoch 30: 22.918642\n",
            "Total loss on epoch 31: 22.114370\n",
            "Total loss on epoch 32: 22.330395\n",
            "Total loss on epoch 33: 21.101156\n",
            "Total loss on epoch 34: 21.492338\n",
            "Total loss on epoch 35: 21.230162\n",
            "Total loss on epoch 36: 20.187944\n",
            "Total loss on epoch 37: 19.602694\n",
            "Total loss on epoch 38: 19.731016\n",
            "Total loss on epoch 39: 19.038036\n",
            "Total loss on epoch 40: 19.893347\n",
            "Total loss on epoch 41: 19.039433\n",
            "Total loss on epoch 42: 18.644238\n",
            "Total loss on epoch 43: 19.034292\n",
            "Total loss on epoch 44: 18.678047\n",
            "Total loss on epoch 45: 18.404469\n",
            "Total loss on epoch 46: 17.649704\n",
            "Total loss on epoch 47: 17.334467\n",
            "Total loss on epoch 48: 17.216909\n",
            "Total loss on epoch 49: 17.650494\n",
            " === Train Set Performance === \n",
            "Precision: 0.9994169096209913\n",
            "Recall: 0.9976717112922002\n",
            "F1: 0.9985435479172736\n",
            " === Test Set Performance === \n",
            "Precision: 0.7735849056603774\n",
            "Recall: 0.7454545454545455\n",
            "F1: 0.7592592592592593\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Logistic regression**"
      ],
      "metadata": {
        "id": "4F4MyuABil5f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Network definition\n",
        "input_layer_d = train_xs.shape[1]\n",
        "# hidden_layer_d = 1\n",
        "num_classes = 2\n",
        "\n",
        "# Logistic Regression class inheriting from MLP\n",
        "class LogisticRegression(MLP):\n",
        "    def __init__(self, architecture):\n",
        "        super(LogisticRegression, self).__init__(architecture)\n",
        "        # Overriding the layers for logistic regression\n",
        "        self.layers = nn.Sequential(\n",
        "            nn.Linear(architecture[0], architecture[1]),  # No hidden layers\n",
        "            nn.LogSoftmax(dim=0)\n",
        "        )\n",
        "\n",
        "#this just an example of how to instantiate the model\n",
        "model_logistic = LogisticRegression([input_layer_d, num_classes, num_classes])\n",
        "\n",
        "# RUN TRAINING AND TEST\n",
        "num_epochs = 50\n",
        "initial_learning_rate = 0.01\n",
        "model_logistic = train_model(model_logistic, train_xs, train_ys, num_classes,\n",
        "                             num_epochs, initial_learning_rate)\n",
        "\n",
        "print(\" === Train Set Performance === \")\n",
        "report_metrics(model_logistic, train_xs, train_ys)\n",
        "\n",
        "print(\" === Test Set Performance === \")\n",
        "report_metrics(model_logistic, test_xs, test_y)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UdDOvg4Mi0aY",
        "outputId": "06cd12d5-6330-407c-cae7-1ae49c52684a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total loss on epoch 0: 2372.364502\n",
            "Total loss on epoch 1: 1830.221680\n",
            "Total loss on epoch 2: 1575.410278\n",
            "Total loss on epoch 3: 1413.851196\n",
            "Total loss on epoch 4: 1286.332031\n",
            "Total loss on epoch 5: 1198.283813\n",
            "Total loss on epoch 6: 1121.369507\n",
            "Total loss on epoch 7: 1056.020386\n",
            "Total loss on epoch 8: 1000.851196\n",
            "Total loss on epoch 9: 947.921265\n",
            "Total loss on epoch 10: 914.937744\n",
            "Total loss on epoch 11: 874.240723\n",
            "Total loss on epoch 12: 843.739929\n",
            "Total loss on epoch 13: 815.333374\n",
            "Total loss on epoch 14: 785.691284\n",
            "Total loss on epoch 15: 764.618103\n",
            "Total loss on epoch 16: 737.889709\n",
            "Total loss on epoch 17: 719.100037\n",
            "Total loss on epoch 18: 704.876465\n",
            "Total loss on epoch 19: 681.574036\n",
            "Total loss on epoch 20: 668.119995\n",
            "Total loss on epoch 21: 652.833374\n",
            "Total loss on epoch 22: 635.187866\n",
            "Total loss on epoch 23: 622.627258\n",
            "Total loss on epoch 24: 614.867188\n",
            "Total loss on epoch 25: 597.585693\n",
            "Total loss on epoch 26: 587.629272\n",
            "Total loss on epoch 27: 576.284546\n",
            "Total loss on epoch 28: 565.774231\n",
            "Total loss on epoch 29: 554.011414\n",
            "Total loss on epoch 30: 546.587463\n",
            "Total loss on epoch 31: 535.617615\n",
            "Total loss on epoch 32: 525.720276\n",
            "Total loss on epoch 33: 519.940979\n",
            "Total loss on epoch 34: 511.288239\n",
            "Total loss on epoch 35: 502.359344\n",
            "Total loss on epoch 36: 496.604004\n",
            "Total loss on epoch 37: 489.628143\n",
            "Total loss on epoch 38: 481.220978\n",
            "Total loss on epoch 39: 473.934601\n",
            "Total loss on epoch 40: 468.617554\n",
            "Total loss on epoch 41: 462.160370\n",
            "Total loss on epoch 42: 456.989929\n",
            "Total loss on epoch 43: 450.583405\n",
            "Total loss on epoch 44: 443.660400\n",
            "Total loss on epoch 45: 438.210632\n",
            "Total loss on epoch 46: 433.579376\n",
            "Total loss on epoch 47: 429.512848\n",
            "Total loss on epoch 48: 425.008148\n",
            "Total loss on epoch 49: 419.185394\n",
            " === Train Set Performance === \n",
            "Precision: 0.9875518672199171\n",
            "Recall: 0.969732246798603\n",
            "F1: 0.97856093979442\n",
            " === Test Set Performance === \n",
            "Precision: 0.8304347826086956\n",
            "Recall: 0.7441558441558441\n",
            "F1: 0.7849315068493151\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6GRsQGdUWHs5"
      },
      "source": [
        "#### Q1: Experiment with a couple of different MLP architectures. What do you observe?"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "I have tried by doubling and tripling the number of perceptrons the hidden layer. What I have observed is that, by increasing the number of perceptrons in the hidden layer, the model tries to converge in less ephocs than the model with less perceptrons in the hidden layer during tranining, with the model reaching a lower loss in fewer epochs compared to a model with fewer perceptrons. This behavior tells me that more perceptrons allow the network to learn more complex patterns in the data.\n",
        "\n",
        "Another thing I have observed is that, there is no significant change in both the traning and test metrics after 50 epochs for all the three models."
      ],
      "metadata": {
        "id": "5B--xQK-h48X"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mVtj4YrLWHs6"
      },
      "source": [
        "#### Q2: Compare the performance your MLPs with Logistic Regression (note that this is just a MLP *without* any hidden layers). Are the results what you expected to see?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2fbzO6yVWHs6"
      },
      "source": [
        "When I have used logistic regression instead if MLP, the major difference i have observed is that the covergence of the model is taking lot of epochs.\n",
        "\n",
        "This result was expected because logistic regression is a linear model, which means it can only learn linear decision boundaries. On the other hand, The MLP with hidden layers can learn non-linear relationships in the data, which gives it more ability to model complex patterns.\n",
        "\n",
        "The test accuracy has also not increased a lot even on traning for 100 epochs and is very less compared to the MLP method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YHIK8MNcWHs6"
      },
      "source": [
        "#### Q3: Investigate the impact of [initialization](https://pytorch.org/docs/stable/nn.init.html) of weight matrices using your best performing MLP. Compare Xavier and Glorot initialization with zero initialization."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Model with Xavier and Glorot initialization**"
      ],
      "metadata": {
        "id": "TjpHBZyz-5vj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class MLPXavierInitialization(MLP):\n",
        "    def __init__(self, architecture):\n",
        "        super(MLPXavierInitialization, self).__init__(architecture)\n",
        "        self.V = nn.Linear(architecture[0],  architecture[1])\n",
        "        self.g = nn.ReLU()\n",
        "        self.W = nn.Linear( architecture[1],  architecture[2])\n",
        "        self.log_softmax = nn.LogSoftmax(dim=0)\n",
        "        # Initialize weights according to a formula due to Xavier Glorot.\n",
        "        nn.init.xavier_uniform_(self.V.weight)\n",
        "        nn.init.xavier_uniform_(self.W.weight)\n",
        "\n",
        "        self.layers = nn.Sequential(\n",
        "            self.V,\n",
        "            self.g,\n",
        "            self.W,\n",
        "            self.log_softmax\n",
        "        )\n",
        "\n",
        "hidden_layer_d = 800\n",
        "#this just an example of how to instantiate the model\n",
        "model_xavier = MLPXavierInitialization([input_layer_d, hidden_layer_d,\n",
        "                                        num_classes])\n",
        "\n",
        "# RUN TRAINING AND TEST\n",
        "num_epochs = 50\n",
        "initial_learning_rate = 0.01\n",
        "model_xavier = train_model(model_xavier, train_xs, train_ys, num_classes,\n",
        "                           num_epochs, initial_learning_rate)\n",
        "\n",
        "print(\" === Train Set Performance === \")\n",
        "report_metrics(model_xavier, train_xs, train_ys)\n",
        "\n",
        "print(\" === Test Set Performance === \")\n",
        "report_metrics(model_xavier, test_xs, test_y)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mc5xnwvv8y3H",
        "outputId": "cf8f495f-d280-4b4e-d171-498e67496cc4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total loss on epoch 0: 2309.822754\n",
            "Total loss on epoch 1: 1831.073730\n",
            "Total loss on epoch 2: 1358.197021\n",
            "Total loss on epoch 3: 906.903870\n",
            "Total loss on epoch 4: 606.730713\n",
            "Total loss on epoch 5: 348.369324\n",
            "Total loss on epoch 6: 197.053940\n",
            "Total loss on epoch 7: 136.775101\n",
            "Total loss on epoch 8: 96.476120\n",
            "Total loss on epoch 9: 76.772484\n",
            "Total loss on epoch 10: 63.179474\n",
            "Total loss on epoch 11: 55.180912\n",
            "Total loss on epoch 12: 48.477638\n",
            "Total loss on epoch 13: 43.828606\n",
            "Total loss on epoch 14: 40.266731\n",
            "Total loss on epoch 15: 36.754372\n",
            "Total loss on epoch 16: 35.036648\n",
            "Total loss on epoch 17: 32.324081\n",
            "Total loss on epoch 18: 30.840771\n",
            "Total loss on epoch 19: 29.423429\n",
            "Total loss on epoch 20: 27.430243\n",
            "Total loss on epoch 21: 27.501778\n",
            "Total loss on epoch 22: 26.976696\n",
            "Total loss on epoch 23: 25.792900\n",
            "Total loss on epoch 24: 25.144373\n",
            "Total loss on epoch 25: 23.878721\n",
            "Total loss on epoch 26: 22.979839\n",
            "Total loss on epoch 27: 22.814857\n",
            "Total loss on epoch 28: 22.677376\n",
            "Total loss on epoch 29: 22.289797\n",
            "Total loss on epoch 30: 20.939899\n",
            "Total loss on epoch 31: 21.561167\n",
            "Total loss on epoch 32: 19.688171\n",
            "Total loss on epoch 33: 20.238585\n",
            "Total loss on epoch 34: 19.510420\n",
            "Total loss on epoch 35: 20.354925\n",
            "Total loss on epoch 36: 19.391459\n",
            "Total loss on epoch 37: 19.429457\n",
            "Total loss on epoch 38: 18.820374\n",
            "Total loss on epoch 39: 18.392992\n",
            "Total loss on epoch 40: 18.786961\n",
            "Total loss on epoch 41: 19.307154\n",
            "Total loss on epoch 42: 18.885990\n",
            "Total loss on epoch 43: 18.389536\n",
            "Total loss on epoch 44: 18.662529\n",
            "Total loss on epoch 45: 18.046053\n",
            "Total loss on epoch 46: 17.068609\n",
            "Total loss on epoch 47: 18.054516\n",
            "Total loss on epoch 48: 17.157852\n",
            "Total loss on epoch 49: 16.542810\n",
            " === Train Set Performance === \n",
            "Precision: 1.0\n",
            "Recall: 0.9970896391152503\n",
            "F1: 0.9985426989215972\n",
            " === Test Set Performance === \n",
            "Precision: 0.8062593144560357\n",
            "Recall: 0.7025974025974026\n",
            "F1: 0.7508674531575295\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Model with weights initialized as weights**"
      ],
      "metadata": {
        "id": "kQ-N0szlAMqN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class MLPZeroInitialization(MLP):\n",
        "    def __init__(self, architecture):\n",
        "        super(MLPZeroInitialization, self).__init__(architecture)\n",
        "        self.V = nn.Linear(architecture[0], architecture[1])\n",
        "        self.W = nn.Linear(architecture[1], architecture[2])\n",
        "\n",
        "        # Set the weights to zero\n",
        "        nn.init.zeros_(self.V.weight)\n",
        "        nn.init.zeros_(self.W.weight)\n",
        "\n",
        "        # Activation and output layers\n",
        "        self.g = nn.ReLU()\n",
        "        self.log_softmax = nn.LogSoftmax(dim=0)\n",
        "\n",
        "        # Sequential model\n",
        "        self.layers = nn.Sequential(\n",
        "            self.V,\n",
        "            self.g,\n",
        "            self.W,\n",
        "            self.log_softmax\n",
        "        )\n",
        "\n",
        "hidden_layer_d = 800\n",
        "#this just an example of how to instantiate the model\n",
        "model_zero = MLPZeroInitialization([input_layer_d, hidden_layer_d, num_classes])\n",
        "\n",
        "# RUN TRAINING AND TEST\n",
        "num_epochs = 50\n",
        "initial_learning_rate = 0.01\n",
        "model_zero = train_model(model_zero, train_xs, train_ys, num_classes,\n",
        "                         num_epochs, initial_learning_rate)\n",
        "\n",
        "print(\" === Train Set Performance === \")\n",
        "report_metrics(model_zero, train_xs, train_ys)\n",
        "\n",
        "print(\" === Test Set Performance === \")\n",
        "report_metrics(model_zero, test_xs, test_y)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BTB3iTFQ_HCl",
        "outputId": "9889ebff-3a09-48cd-a5af-2bbb95d66b83"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total loss on epoch 0: 2420.698486\n",
            "Total loss on epoch 1: 2247.050537\n",
            "Total loss on epoch 2: 1980.751343\n",
            "Total loss on epoch 3: 1781.930542\n",
            "Total loss on epoch 4: 1615.297119\n",
            "Total loss on epoch 5: 1383.258179\n",
            "Total loss on epoch 6: 1349.789795\n",
            "Total loss on epoch 7: 1232.670532\n",
            "Total loss on epoch 8: 1131.286011\n",
            "Total loss on epoch 9: 1097.545654\n",
            "Total loss on epoch 10: 965.167419\n",
            "Total loss on epoch 11: 848.700073\n",
            "Total loss on epoch 12: 827.955444\n",
            "Total loss on epoch 13: 874.957764\n",
            "Total loss on epoch 14: 781.959839\n",
            "Total loss on epoch 15: 780.155762\n",
            "Total loss on epoch 16: 747.509399\n",
            "Total loss on epoch 17: 607.706787\n",
            "Total loss on epoch 18: 638.200317\n",
            "Total loss on epoch 19: 692.903748\n",
            "Total loss on epoch 20: 612.919861\n",
            "Total loss on epoch 21: 519.164978\n",
            "Total loss on epoch 22: 575.266174\n",
            "Total loss on epoch 23: 662.061279\n",
            "Total loss on epoch 24: 601.157898\n",
            "Total loss on epoch 25: 469.870026\n",
            "Total loss on epoch 26: 394.388977\n",
            "Total loss on epoch 27: 423.869415\n",
            "Total loss on epoch 28: 353.368683\n",
            "Total loss on epoch 29: 439.027924\n",
            "Total loss on epoch 30: 394.229401\n",
            "Total loss on epoch 31: 330.991760\n",
            "Total loss on epoch 32: 323.620819\n",
            "Total loss on epoch 33: 282.781586\n",
            "Total loss on epoch 34: 249.530167\n",
            "Total loss on epoch 35: 241.067032\n",
            "Total loss on epoch 36: 235.913788\n",
            "Total loss on epoch 37: 228.367325\n",
            "Total loss on epoch 38: 228.098190\n",
            "Total loss on epoch 39: 222.406570\n",
            "Total loss on epoch 40: 222.462677\n",
            "Total loss on epoch 41: 218.783585\n",
            "Total loss on epoch 42: 218.433044\n",
            "Total loss on epoch 43: 216.892487\n",
            "Total loss on epoch 44: 215.031326\n",
            "Total loss on epoch 45: 214.449524\n",
            "Total loss on epoch 46: 213.531097\n",
            "Total loss on epoch 47: 212.230774\n",
            "Total loss on epoch 48: 211.948547\n",
            "Total loss on epoch 49: 210.855957\n",
            " === Train Set Performance === \n",
            "Precision: 1.0\n",
            "Recall: 0.9743888242142026\n",
            "F1: 0.9870283018867925\n",
            " === Test Set Performance === \n",
            "Precision: 0.8107302533532041\n",
            "Recall: 0.7064935064935065\n",
            "F1: 0.7550312283136711\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JBOR2g_KWHs7"
      },
      "source": [
        "When I take my high performing model with weights initialized using Xavier and as zeros, I feel the starting loss is slightly higher for the model intitialized with zeros. And also for running the both for 50 epochs, we can observe that the model with zavier is converging quickly compared to the model initialized with zeros.\n",
        "\n",
        "This happens because when the weights are initialized to zero, all the neurons in a layer end up learning the same things. This makes it harder for the model to learn effectively, leading to slow progress or even getting stuck, as the updates to the model during training don't work well.\n",
        "\n",
        "If we compare the metrics aswell during the traning and testing, for the model intialized with xavier has good metrics compared to model with zeros."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CifxB_UOWHs7"
      },
      "source": [
        "#### Q4: Investigate the impact of [non-linear activations](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity). See Torch documentation on the available non-linear functions and compare the performance of 3 different functions (e.g., sigmoid, relu and tanh)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MUd4TnhGWHs7"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Sigmod activation**"
      ],
      "metadata": {
        "id": "Z9B1bea4A-i6"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RrEoksEYWHs7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6ff90ac8-d928-4efb-a796-d9a51398f6e7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total loss on epoch 0: 2920.830811\n",
            "Total loss on epoch 1: 2601.025879\n",
            "Total loss on epoch 2: 2324.942871\n",
            "Total loss on epoch 3: 2058.722656\n",
            "Total loss on epoch 4: 1796.940918\n",
            "Total loss on epoch 5: 1600.729980\n",
            "Total loss on epoch 6: 1401.272949\n",
            "Total loss on epoch 7: 1252.659912\n",
            "Total loss on epoch 8: 1083.487305\n",
            "Total loss on epoch 9: 987.683350\n",
            "Total loss on epoch 10: 871.386230\n",
            "Total loss on epoch 11: 787.112366\n",
            "Total loss on epoch 12: 714.123413\n",
            "Total loss on epoch 13: 654.508728\n",
            "Total loss on epoch 14: 566.223083\n",
            "Total loss on epoch 15: 500.781921\n",
            "Total loss on epoch 16: 451.632568\n",
            "Total loss on epoch 17: 438.014465\n",
            "Total loss on epoch 18: 404.248138\n",
            "Total loss on epoch 19: 346.210236\n",
            "Total loss on epoch 20: 325.901978\n",
            "Total loss on epoch 21: 304.563660\n",
            "Total loss on epoch 22: 283.918793\n",
            "Total loss on epoch 23: 269.772186\n",
            "Total loss on epoch 24: 267.113556\n",
            "Total loss on epoch 25: 246.037796\n",
            "Total loss on epoch 26: 224.568817\n",
            "Total loss on epoch 27: 197.862778\n",
            "Total loss on epoch 28: 183.935486\n",
            "Total loss on epoch 29: 167.754318\n",
            "Total loss on epoch 30: 162.768250\n",
            "Total loss on epoch 31: 148.769882\n",
            "Total loss on epoch 32: 149.111053\n",
            "Total loss on epoch 33: 138.612152\n",
            "Total loss on epoch 34: 134.576416\n",
            "Total loss on epoch 35: 122.946342\n",
            "Total loss on epoch 36: 118.825172\n",
            "Total loss on epoch 37: 113.878510\n",
            "Total loss on epoch 38: 109.562927\n",
            "Total loss on epoch 39: 104.174080\n",
            "Total loss on epoch 40: 103.881233\n",
            "Total loss on epoch 41: 95.282280\n",
            "Total loss on epoch 42: 92.819786\n",
            "Total loss on epoch 43: 89.158203\n",
            "Total loss on epoch 44: 88.456467\n",
            "Total loss on epoch 45: 84.705170\n",
            "Total loss on epoch 46: 82.158463\n",
            "Total loss on epoch 47: 82.735252\n",
            "Total loss on epoch 48: 76.418961\n",
            "Total loss on epoch 49: 75.414040\n",
            " === Train Set Performance === \n",
            "Precision: 1.0\n",
            "Recall: 0.9906868451688009\n",
            "F1: 0.9953216374269006\n",
            " === Test Set Performance === \n",
            "Precision: 0.839546191247974\n",
            "Recall: 0.6727272727272727\n",
            "F1: 0.7469358327325163\n"
          ]
        }
      ],
      "source": [
        "class MLPSigmoidActivation(MLP):\n",
        "    def __init__(self, architecture):\n",
        "        super(MLPSigmoidActivation, self).__init__(architecture)\n",
        "        self.V = nn.Linear(architecture[0], architecture[1])\n",
        "        self.W = nn.Linear(architecture[1], architecture[2])\n",
        "\n",
        "        # Activation and output layers\n",
        "        self.g = nn.Sigmoid()\n",
        "        self.log_softmax = nn.LogSoftmax(dim=0)\n",
        "\n",
        "        # Sequential model\n",
        "        self.layers = nn.Sequential(\n",
        "            self.V,\n",
        "            self.g,\n",
        "            self.W,\n",
        "            self.log_softmax\n",
        "        )\n",
        "\n",
        "hidden_layer_d = 400\n",
        "\n",
        "#this just an example of how to instantiate the model\n",
        "model_sigmoid = MLPSigmoidActivation([input_layer_d, hidden_layer_d,\n",
        "                                      num_classes])\n",
        "\n",
        "# RUN TRAINING AND TEST\n",
        "num_epochs = 50\n",
        "initial_learning_rate = 0.01\n",
        "model_sigmoid = train_model(model_sigmoid, train_xs, train_ys,\n",
        "                            num_classes, num_epochs, initial_learning_rate)\n",
        "\n",
        "print(\" === Train Set Performance === \")\n",
        "report_metrics(model_sigmoid, train_xs, train_ys)\n",
        "\n",
        "print(\" === Test Set Performance === \")\n",
        "report_metrics(model_sigmoid, test_xs, test_y)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**TanH activation**"
      ],
      "metadata": {
        "id": "bNa7hDheBW1n"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class MLPTanhActivation(MLP):\n",
        "    def __init__(self, architecture):\n",
        "        super(MLPTanhActivation, self).__init__(architecture)\n",
        "        self.V = nn.Linear(architecture[0], architecture[1])\n",
        "        self.W = nn.Linear(architecture[1], architecture[2])\n",
        "\n",
        "        # Activation and output layers\n",
        "        self.g = nn.Tanh()\n",
        "        self.log_softmax = nn.LogSoftmax(dim=0)\n",
        "\n",
        "        # Sequential model\n",
        "        self.layers = nn.Sequential(\n",
        "            self.V,\n",
        "            self.g,\n",
        "            self.W,\n",
        "            self.log_softmax\n",
        "        )\n",
        "\n",
        "hidden_layer_d = 400\n",
        "\n",
        "#this just an example of how to instantiate the model\n",
        "model_tanh = MLPTanhActivation([input_layer_d, hidden_layer_d, num_classes])\n",
        "\n",
        "# RUN TRAINING AND TEST\n",
        "num_epochs = 50\n",
        "initial_learning_rate = 0.01\n",
        "model_tanh = train_model(model_tanh, train_xs, train_ys, num_classes,\n",
        "                         num_epochs, initial_learning_rate)\n",
        "\n",
        "print(\" === Train Set Performance === \")\n",
        "report_metrics(model_tanh, train_xs, train_ys)\n",
        "\n",
        "print(\" === Test Set Performance === \")\n",
        "report_metrics(model_tanh, test_xs, test_y)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gGbe_8n7BSfa",
        "outputId": "d848bc8c-b877-4c0a-eeb8-778b345d23a4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total loss on epoch 0: 2301.167725\n",
            "Total loss on epoch 1: 1955.876709\n",
            "Total loss on epoch 2: 1700.426514\n",
            "Total loss on epoch 3: 1472.700928\n",
            "Total loss on epoch 4: 1231.246216\n",
            "Total loss on epoch 5: 1059.766724\n",
            "Total loss on epoch 6: 896.388672\n",
            "Total loss on epoch 7: 752.691711\n",
            "Total loss on epoch 8: 662.736755\n",
            "Total loss on epoch 9: 585.227234\n",
            "Total loss on epoch 10: 466.390411\n",
            "Total loss on epoch 11: 386.509918\n",
            "Total loss on epoch 12: 301.370300\n",
            "Total loss on epoch 13: 227.525055\n",
            "Total loss on epoch 14: 265.346619\n",
            "Total loss on epoch 15: 203.323212\n",
            "Total loss on epoch 16: 238.941818\n",
            "Total loss on epoch 17: 160.961197\n",
            "Total loss on epoch 18: 106.675034\n",
            "Total loss on epoch 19: 83.824455\n",
            "Total loss on epoch 20: 69.414474\n",
            "Total loss on epoch 21: 62.745872\n",
            "Total loss on epoch 22: 54.857712\n",
            "Total loss on epoch 23: 51.321255\n",
            "Total loss on epoch 24: 47.213696\n",
            "Total loss on epoch 25: 42.860287\n",
            "Total loss on epoch 26: 41.371277\n",
            "Total loss on epoch 27: 38.804268\n",
            "Total loss on epoch 28: 39.515381\n",
            "Total loss on epoch 29: 35.859344\n",
            "Total loss on epoch 30: 35.059090\n",
            "Total loss on epoch 31: 32.728821\n",
            "Total loss on epoch 32: 32.811714\n",
            "Total loss on epoch 33: 31.563568\n",
            "Total loss on epoch 34: 30.727451\n",
            "Total loss on epoch 35: 30.633921\n",
            "Total loss on epoch 36: 28.640457\n",
            "Total loss on epoch 37: 26.732674\n",
            "Total loss on epoch 38: 27.248289\n",
            "Total loss on epoch 39: 26.622957\n",
            "Total loss on epoch 40: 26.962152\n",
            "Total loss on epoch 41: 26.253035\n",
            "Total loss on epoch 42: 25.047544\n",
            "Total loss on epoch 43: 24.350727\n",
            "Total loss on epoch 44: 24.068531\n",
            "Total loss on epoch 45: 23.845131\n",
            "Total loss on epoch 46: 23.106485\n",
            "Total loss on epoch 47: 22.024399\n",
            "Total loss on epoch 48: 23.850414\n",
            "Total loss on epoch 49: 22.560080\n",
            " === Train Set Performance === \n",
            "Precision: 0.9994169096209913\n",
            "Recall: 0.9976717112922002\n",
            "F1: 0.9985435479172736\n",
            " === Test Set Performance === \n",
            "Precision: 0.7891061452513967\n",
            "Recall: 0.7337662337662337\n",
            "F1: 0.7604306864064604\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "-dneWqSqHo_v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "When I observe the three models taking sigmoid, tanh and relu(its defind for answering the 1st question) activations, I observed the following\n",
        "\n",
        "**With Sigmoid**\n",
        "As sigmoid squeezes numbers between 0 and 1, It didn’t work as well because it made the model learn slowly. This is because it can make the updates to the model's weights really small, which slowed down learning.\n",
        "\n",
        "**With Tanh**\n",
        "As Tanh outputs values between -1 and 1, It worked better than Sigmoid because it produced stronger updates to the model, but it still had some of the same issues like slow learning.\n",
        "\n",
        "**With Relu**\n",
        "ReLU performed the best. We know that it sets negative values to zero and leaves positive values as they are. This helped the model learn faster and avoid the problems Sigmoid has."
      ],
      "metadata": {
        "id": "zwAklvcTFpzH"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "GXMnufP1Fsx0"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.7"
    },
    "colab": {
      "provenance": [],
      "machine_shape": "hm",
      "gpuType": "A100",
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}