shallow_tensorflow.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "colab": {
      "name": "shallow-tensorflow.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hikmatfarhat-ndu/CSC645/blob/master/shallow_tensorflow.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B1GbGYpm8pb4"
      },
      "source": [
        "## Using Tensorflow to model the shallow network\n",
        "In this exercise will we redo, using tensorflow the shallow network that we trained from first\n",
        "principles before to recognize the Sonar data\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S7lIYasDLv5c"
      },
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D50ik1a98q88"
      },
      "source": [
        "## The Data\n",
        "\n",
        "The data is from https://archive.ics.uci.edu/ml/datasets/Connectionist+Bench+(Sonar,+Mines+vs.+Rocks) the mines vs rocks sonar data. It is in csv format and very small so download it to your computer. Next we upload it to colab and read it using the pandas package.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ggc_79s_9WAf"
      },
      "source": [
        "### Upload to colab\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dBPAw-TA8qiE",
        "outputId": "44c03617-9d2e-47fa-fb0a-21d8ba0fa35f",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCkgewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwogICAgICBwZXJjZW50LnRleHRDb250ZW50ID0KICAgICAgICAgIGAke01hdGgucm91bmQoKHBvc2l0aW9uIC8gZmlsZURhdGEuYnl0ZUxlbmd0aCkgKiAxMDApfSUgZG9uZWA7CiAgICB9CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 73
        }
      },
      "source": [
        "from google.colab import files\n",
        "file=files.upload()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-3ee24ed3-1686-42f0-84e0-bf64f72dafde\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-3ee24ed3-1686-42f0-84e0-bf64f72dafde\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving sonar.csv to sonar.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tY-5gNfA9jFx"
      },
      "source": [
        "### Read using Pandas"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2lNQeRgwMLpd"
      },
      "source": [
        "import pandas as pd\n",
        "df=pd.read_csv(\"sonar.csv\")\n"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4ULSgk19oN5"
      },
      "source": [
        "### Preprocessing the data\n",
        "\n",
        "We need to perform several operations on the data before we use it. \n",
        "1. The data is sorted: all mines followed by all rocks so we shuffle it using numpy\n",
        "2. We need to break it into train and test sets.\n",
        "3. Make sure that the data in \"float32\" type instead of \"float\". For some reason  Tensorflow is sensitive about that "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0aLmCxiQMfbB",
        "outputId": "be942bd7-da43-44b1-bd72-8baf61e8c9ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#pandas data frame\n",
        "m=df.values\n",
        "# randomize (shuffle) the data\n",
        "np.random.shuffle(m)\n",
        "# Each row has 61 entries, 60 for data and the last one is the label \"M\" or \"R\"\n",
        "# X contains all the data\n",
        "X=m[:,0:60].astype(\"float32\")\n",
        "# Y contains all the labels\n",
        "Y=m[:,60]\n",
        "# convert the labels: \"M\"->1 and \"R\"->0\n",
        "Y=np.array([1.0 if i=='M' else 0.0 for i in Y])\n",
        "Y=Y.reshape((len(Y),1))\n",
        "Y=Y.astype(\"float32\")\n",
        "\n",
        "# split the data and labels into a training and test sets\n",
        "x_train=X[0:175]\n",
        "x_test=X[175:208]\n",
        "y_train=Y[0:175]\n",
        "y_test=Y[175:208]"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'numpy.float32'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yc6230u28pb8"
      },
      "source": [
        "### Important Note\n",
        "Tensorflow stacks the samples row-wise instead of column-wise\n",
        "as we have been doing when we did the gradient descent oursleves. We need to keep that in mind."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ZL7owU98pcP"
      },
      "source": [
        "### Defining the parameters\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TZl1BVG48pcR"
      },
      "source": [
        "learning_rate = 3\n",
        "nb_iterations = 2500\n",
        "\n",
        "# Network Parameters\n",
        "n_h = 64 # number of neurons in hidden layer\n",
        "n_x = X.shape[1] #number of neurons in input\n",
        "n_y = Y.shape[1] #number of neurons in ouput\n"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a_ZUQKDd8pce"
      },
      "source": [
        "### Initialization\n",
        "The forward propagation phase is the same as when we did this exercise from first principles but since tensorflow stacks the data row-wise the forward propagation is slightly different then we are used to.\n",
        "Let $W^0$,$W^1$,$b^0$,$b^1$ be the weights and biases of the first and second layer respectively then forward propagation is given by\n",
        "\\begin{align*}\n",
        "Z^1&=X\\cdot W^0+b^0\\\\\n",
        "A^1 &=\\sigma(Z^1)\\\\\n",
        "Z^2 &=A^1\\cdot W^1+b^1\\\\\n",
        "A^2 &=\\sigma(Z^2)\n",
        "\\end{align*}\n",
        "Compare the above with the equations in the previous exercise. For more details consult the lecture [backpropagation](https://github.com/hikmatfarhat-ndu/CSC645/blob/master/lectures/csc645-lecture-backprop.pdf).\n",
        "\n",
        "According to the above equations we have to define the tensorflow variables that will hold the weights and biases. \n",
        "The biases are initialized to zero  and the weights randomly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MY46Q5c38pch",
        "tags": []
      },
      "source": [
        "\n",
        "initializer = tf.initializers.RandomNormal()\n",
        "\n",
        "W0=tf.Variable(initializer([n_x,n_h]),trainable=True,dtype=tf.float32)\n",
        "W1=tf.Variable(initializer([n_h,n_y]),trainable=True,dtype=tf.float32)\n",
        "\n",
        "b0=tf.Variable(tf.zeros([n_h]))            #biases of the first layer\n",
        "b1=tf.Variable(tf.zeros([n_y]))            #biases of the second layer\n"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W0PPEanu8pct"
      },
      "source": [
        "### Defining the model\n",
        "Our model has two layers. The function \"model\" below should return the ouput of our model for a given input. Note since Tensorflow uses the first index as the sample size the dot product has a different order."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eHUGz6ki8pcv"
      },
      "source": [
        "def model(input):\n",
        "    # Hidden fully connected layer with 256 neurons\n",
        "   \n",
        "    layer_1 = tf.add(tf.matmul(input, W0), b0)\n",
        "    # Output fully connected layer with a neuron for each class\n",
        "    out_layer = tf.matmul(tf.sigmoid(layer_1), W1) + b1\n",
        "    return out_layer"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FYvEYQzY8pc7"
      },
      "source": [
        "Once the model is defined the remaining code is similar to our previous exercise. We define the loss\n",
        "as an average over the cross-entropy but this time since it is binary classification we use the sigmoid instead\n",
        "of the softmax function. Then our optimizer uses gradient descent to minimize the loss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OyOu53Aa8pdC"
      },
      "source": [
        "\n",
        "# Define loss and optimize\n",
        "def loss(pred,label):\n",
        "   return tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=pred, labels=label))\n"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tVCoEjl0wdL2"
      },
      "source": [
        "## Prediction\n",
        "\n",
        "Given input X ( and the parameters we are trying to learn) this function predicts if it is Rock or Mine"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8e8CLkg1utUA"
      },
      "source": [
        "def prediction(X):\n",
        "  a=tf.math.sigmoid(model(X))\n",
        "  return tf.cast((a>0.5),tf.int32)"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "krm_O8dYxC2o"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cyKqGz848pdQ"
      },
      "source": [
        "The model is defined now we run our computation. Recal that our model depends on, the changing, parameters $W^0,W^1,b^0,b^1$ therefore its gradient will change also. The train function is a **single** training step"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XF6wPb6nuM8J"
      },
      "source": [
        "optimizer=tf.optimizers.SGD(learning_rate)\n",
        "\n",
        "def train(data,labels):\n",
        "  with tf.GradientTape() as tape:\n",
        "    diff=loss(model(data),labels)\n",
        "\n",
        "  grad=tape.gradient(diff,[W0,W1,b0,b1])\n",
        "  optimizer.apply_gradients( zip( grad , [W0,W1,b0,b1] ) )\n",
        "  pT=tf.transpose(prediction(data))\n",
        "  correct=np.squeeze(np.dot(pT,labels)+np.dot(1-pT,1-labels))\n",
        "  return diff,correct"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fSK1D5tOuTmY"
      },
      "source": [
        "### Training Loop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v3bNNi0M8pdS",
        "tags": [],
        "outputId": "83a53aa2-3bdc-4fb8-9e65-cfbd4d28aa4a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 103
        }
      },
      "source": [
        "for i in range(nb_iterations):\n",
        " cost,corr=train(x_train,y_train)\n",
        " if(i%500==0):\n",
        "  print(\"cost={},accuracy={}/{}\".format(cost,corr,x_train.shape[0]))\n",
        "  \n"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "cost=0.6938693523406982,accuracy=96.0/175\n",
            "cost=0.2810496687889099,accuracy=147.0/175\n",
            "cost=0.13114233314990997,accuracy=167.0/175\n",
            "cost=0.04477405920624733,accuracy=173.0/175\n",
            "cost=0.032321758568286896,accuracy=175.0/175\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9rPC3U5juYS1"
      },
      "source": [
        "### Accuracy"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "amjTFhXd8pdh",
        "outputId": "aaecd347-48d1-4343-a9f4-da70d88deb63",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        }
      },
      "source": [
        "pT=tf.transpose(prediction(x_test))\n",
        "print(np.dot(pT,y_test))\n",
        "print(np.dot(1-pT,1-y_test))\n",
        "correct=np.dot(pT,y_test)+np.dot(1-pT,1-y_test)\n",
        "accuracy=100*float(np.squeeze(correct))/float(y_test.shape[0])\n",
        "print(\"Accuracy=\"+str(accuracy))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[13.]]\n",
            "[[12.]]\n",
            "Accuracy=78.125\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}