Instagram_Image Analytics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Assignment 4.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "0tdWrGaa7wY_",
        "dKS05f6IIQjB",
        "Ivk6Kgy7Epe6"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZKE7tTVuH_00"
      },
      "source": [
        "<h1><center> Assignment 4 </center></h1>\n",
        "\n",
        "## Group Members\n",
        "\n",
        "*   Harris Azerf\n",
        "*   Jonathan Garcia\n",
        "*   Nadia Florez-Morcote\n",
        "*   Palmer Wenzel\n",
        "*   Saranya Nagarajan"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O6qvPXLV66fa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "80c6011d-9c40-44f0-d8ee-d1ab3b2715f9"
      },
      "source": [
        "!pip install instaloader\n",
        "!pip install --upgrade google-cloud-vision"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting instaloader\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0b/39/170a9e70d344d773a9855fc8d2f2bbe6f0090d79d901058062eb2b7dbdd1/instaloader-4.5.3.tar.gz (47kB)\n",
            "\r\u001b[K     |███████                         | 10kB 16.4MB/s eta 0:00:01\r\u001b[K     |█████████████▉                  | 20kB 16.0MB/s eta 0:00:01\r\u001b[K     |████████████████████▊           | 30kB 14.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▋    | 40kB 14.1MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 51kB 4.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests>=2.4 in /usr/local/lib/python3.6/dist-packages (from instaloader) (2.23.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.4->instaloader) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.4->instaloader) (2020.6.20)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.4->instaloader) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.4->instaloader) (3.0.4)\n",
            "Building wheels for collected packages: instaloader\n",
            "  Building wheel for instaloader (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for instaloader: filename=instaloader-4.5.3-cp36-none-any.whl size=50844 sha256=61d9dff9952f3c8e658873fd2f7af9999f5d1d6ea2c49d29264a223b4d670eb1\n",
            "  Stored in directory: /root/.cache/pip/wheels/3f/5a/0e/bf22dafc60d940c4c30734cb6ed2bb5bbca2784a77961f77da\n",
            "Successfully built instaloader\n",
            "Installing collected packages: instaloader\n",
            "Successfully installed instaloader-4.5.3\n",
            "Collecting google-cloud-vision\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/48/38/754771fa9add8acb4ea79367f55ace7ec5c6da4b13226fcdb736b9015a36/google_cloud_vision-2.0.0-py2.py3-none-any.whl (450kB)\n",
            "\u001b[K     |████████████████████████████████| 460kB 12.2MB/s \n",
            "\u001b[?25hCollecting proto-plus>=1.4.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/75/0d/dd8e8c979eb843b873aeaadcfefd5383da7e2fc5593456a32264cae1cc5c/proto-plus-1.11.0.tar.gz (44kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 5.4MB/s \n",
            "\u001b[?25hCollecting google-api-core[grpc]<2.0.0dev,>=1.22.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/2e/3d/d7af13040ab5b259994a4434ff03d68084a994e709bc8afa4bee1235310e/google_api_core-1.23.0-py2.py3-none-any.whl (91kB)\n",
            "\u001b[K     |████████████████████████████████| 92kB 6.7MB/s \n",
            "\u001b[?25hCollecting libcst>=0.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9f/06/fae14090e8a98e5577aa9b7aac3090d9aba26e9057fb03baa65e38f81183/libcst-0.3.14-py3-none-any.whl (503kB)\n",
            "\u001b[K     |████████████████████████████████| 512kB 20.3MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: protobuf>=3.12.0 in /usr/local/lib/python3.6/dist-packages (from proto-plus>=1.4.0->google-cloud-vision) (3.12.4)\n",
            "Collecting google-auth<2.0dev,>=1.21.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1d/60/81e68e70eea91ef05bb00bcdac243d67b61f826c65aaca6961de622dffd7/google_auth-1.23.0-py2.py3-none-any.whl (114kB)\n",
            "\u001b[K     |████████████████████████████████| 122kB 28.8MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: setuptools>=34.0.0 in /usr/local/lib/python3.6/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (50.3.2)\n",
            "Requirement already satisfied, skipping upgrade: six>=1.13.0 in /usr/local/lib/python3.6/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (1.15.0)\n",
            "Requirement already satisfied, skipping upgrade: requests<3.0.0dev,>=2.18.0 in /usr/local/lib/python3.6/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (2.23.0)\n",
            "Requirement already satisfied, skipping upgrade: pytz in /usr/local/lib/python3.6/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (2018.9)\n",
            "Requirement already satisfied, skipping upgrade: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (1.52.0)\n",
            "Requirement already satisfied, skipping upgrade: grpcio<2.0dev,>=1.29.0; extra == \"grpc\" in /usr/local/lib/python3.6/dist-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (1.33.2)\n",
            "Requirement already satisfied, skipping upgrade: dataclasses>=0.6.0; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from libcst>=0.2.5->google-cloud-vision) (0.7)\n",
            "Collecting typing-inspect>=0.4.0\n",
            "  Downloading https://files.pythonhosted.org/packages/42/1c/66402db44184904a2f14722d317a4da0b5c8c78acfc3faf74362566635c5/typing_inspect-0.6.0-py3-none-any.whl\n",
            "Collecting pyyaml>=5.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz (269kB)\n",
            "\u001b[K     |████████████████████████████████| 276kB 21.0MB/s \n",
            "\u001b[?25hRequirement already satisfied, skipping upgrade: typing-extensions>=3.7.4.2 in /usr/local/lib/python3.6/dist-packages (from libcst>=0.2.5->google-cloud-vision) (3.7.4.3)\n",
            "Requirement already satisfied, skipping upgrade: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.6/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (4.1.1)\n",
            "Requirement already satisfied, skipping upgrade: rsa<5,>=3.1.4; python_version >= \"3.5\" in /usr/local/lib/python3.6/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (4.6)\n",
            "Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.6/dist-packages (from google-auth<2.0dev,>=1.21.1->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (0.2.8)\n",
            "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (2020.6.20)\n",
            "Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (3.0.4)\n",
            "Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (2.10)\n",
            "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (1.24.3)\n",
            "Collecting mypy-extensions>=0.3.0\n",
            "  Downloading https://files.pythonhosted.org/packages/5c/eb/975c7c080f3223a5cdaff09612f3a5221e4ba534f7039db34c35d95fa6a5/mypy_extensions-0.4.3-py2.py3-none-any.whl\n",
            "Requirement already satisfied, skipping upgrade: pyasn1>=0.1.3 in /usr/local/lib/python3.6/dist-packages (from rsa<5,>=3.1.4; python_version >= \"3.5\"->google-auth<2.0dev,>=1.21.1->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-vision) (0.4.8)\n",
            "Building wheels for collected packages: proto-plus, pyyaml\n",
            "  Building wheel for proto-plus (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for proto-plus: filename=proto_plus-1.11.0-cp36-none-any.whl size=41571 sha256=1fb5ed196fa23cdedb7d14b2ae79965dd45172791d0f37758e4e0e97975eee42\n",
            "  Stored in directory: /root/.cache/pip/wheels/78/e3/6b/a14506581b1cde1ac1743f2939dcc06fc06a5af2aa224a334e\n",
            "  Building wheel for pyyaml (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44619 sha256=b34360935b9ef3c5e4bc7ed8d828160b8c8aac58fff0c4a6cd9157814113b523\n",
            "  Stored in directory: /root/.cache/pip/wheels/a7/c1/ea/cf5bd31012e735dc1dfea3131a2d5eae7978b251083d6247bd\n",
            "Successfully built proto-plus pyyaml\n",
            "\u001b[31mERROR: google-colab 1.0.0 has requirement google-auth~=1.17.2, but you'll have google-auth 1.23.0 which is incompatible.\u001b[0m\n",
            "Installing collected packages: proto-plus, google-auth, google-api-core, mypy-extensions, typing-inspect, pyyaml, libcst, google-cloud-vision\n",
            "  Found existing installation: google-auth 1.17.2\n",
            "    Uninstalling google-auth-1.17.2:\n",
            "      Successfully uninstalled google-auth-1.17.2\n",
            "  Found existing installation: google-api-core 1.16.0\n",
            "    Uninstalling google-api-core-1.16.0:\n",
            "      Successfully uninstalled google-api-core-1.16.0\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed google-api-core-1.23.0 google-auth-1.23.0 google-cloud-vision-2.0.0 libcst-0.3.14 mypy-extensions-0.4.3 proto-plus-1.11.0 pyyaml-5.3.1 typing-inspect-0.6.0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "google"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sw9nwF7T7CQm"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UFhWMKdsc1vE"
      },
      "source": [
        "import instaloader\n",
        "from google.cloud import vision"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xp07mHVs7UJ9"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from google.colab import files\n",
        "import os\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import accuracy_score, confusion_matrix\n",
        "\n",
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0tdWrGaa7wY_"
      },
      "source": [
        "## Instagram Scraper"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "doNskKlO70AU"
      },
      "source": [
        "#initializing the scraper and seting the user name as 'natgeo' to look at national geographic instagram account \n",
        "L = instaloader.Instaloader()\n",
        "username = 'natgeo'\n",
        "profile = instaloader.Profile.from_username(L.context, username)\n",
        "\n",
        "# get the posts from the username\n",
        "posts = profile.get_posts()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bL6bW5pE-Dcq"
      },
      "source": [
        "# Creating the dataframe to place the information into\n",
        "instagram_posts = pd.DataFrame(columns=['post_url', 'comment_num', 'like_num', 'post_caption', 'image_url'])\n",
        "\n",
        "\n",
        "# Creating the url by collecting the shortcode and combining with this url string\n",
        "post_url = 'https://www.instagram.com/p/{}/'\n",
        "\n",
        "# Initializing i\n",
        "i = 0\n",
        "for post in posts:\n",
        "    if i < 500:\n",
        "        if post.is_video == False:\n",
        "          i += 1  \n",
        "          # Add the information to the dataframe created before\n",
        "          instagram_posts = instagram_posts.append({'post_url': post_url.format(post.shortcode),\n",
        "                                                    'comment_num': post.comments,\n",
        "                                                    'like_num': post.likes,\n",
        "                                                    'post_caption': post.caption,\n",
        "                                                    'image_url': post.url}, ignore_index=True)\n",
        "    else:\n",
        "        break"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_buvljKK-VZM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "271dc79d-4579-4571-905d-56a497cb524d"
      },
      "source": [
        "instagram_posts.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>post_url</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>like_num</th>\n",
              "      <th>post_caption</th>\n",
              "      <th>image_url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://www.instagram.com/p/CHYOgoNLjwD/</td>\n",
              "      <td>157</td>\n",
              "      <td>24098</td>\n",
              "      <td>Photo by @erintrieb / On Election Day I set up...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>https://www.instagram.com/p/CHXl8P8gT0f/</td>\n",
              "      <td>455</td>\n",
              "      <td>83621</td>\n",
              "      <td>Photo by @tasneemalsultan / A Saudi woman sits...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>https://www.instagram.com/p/CHXXmNpAhHV/</td>\n",
              "      <td>431</td>\n",
              "      <td>184808</td>\n",
              "      <td>Photo by Keith Ladzinski @ladzinski / A kiss o...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://www.instagram.com/p/CHXKiRnAshz/</td>\n",
              "      <td>511</td>\n",
              "      <td>106213</td>\n",
              "      <td>Photo by @dina_litovsky / Most people don’t as...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>https://www.instagram.com/p/CHWudbnoMhb/</td>\n",
              "      <td>554</td>\n",
              "      <td>162146</td>\n",
              "      <td>Photo by @paoloverzone / Fluorescing a ghostly...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                   post_url  ...                                          image_url\n",
              "0  https://www.instagram.com/p/CHYOgoNLjwD/  ...  https://scontent-iad3-1.cdninstagram.com/v/t51...\n",
              "1  https://www.instagram.com/p/CHXl8P8gT0f/  ...  https://scontent-iad3-1.cdninstagram.com/v/t51...\n",
              "2  https://www.instagram.com/p/CHXXmNpAhHV/  ...  https://scontent-iad3-1.cdninstagram.com/v/t51...\n",
              "3  https://www.instagram.com/p/CHXKiRnAshz/  ...  https://scontent-iad3-1.cdninstagram.com/v/t51...\n",
              "4  https://www.instagram.com/p/CHWudbnoMhb/  ...  https://scontent-iad3-1.cdninstagram.com/v/t51...\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FjoqQ6X4GV7P"
      },
      "source": [
        "instagram_posts.to_csv(\"instagram_posts.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dKS05f6IIQjB"
      },
      "source": [
        "## Google Vision"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G2Bs9a2HISuI"
      },
      "source": [
        "#https://colab.research.google.com/github/robeartoe/APIWorkshop/blob/master/MLWorkshop.ipynb#scrollTo=Ho30nQiuLMeQ\n",
        "os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"]= \"UDA-A4-5e230b50c107.json\"\n",
        "\n",
        "def labelAnnotation(url):\n",
        "  # Imports the Google Cloud client library\n",
        "  from google.cloud import vision\n",
        "  #from google.cloud.vision import types\n",
        "  # Instantiates a client\n",
        "  client = vision.ImageAnnotatorClient()\n",
        "\n",
        "  response = client.annotate_image({\n",
        "     'image': {'source': \n",
        "               {'image_uri': url}\n",
        "              },\n",
        "  })\n",
        "\n",
        "  # Detect and Print Labels:\n",
        "  label_list=[]\n",
        "  labels = response.label_annotations\n",
        "  for label in labels:\n",
        "    label_list.append(label.description)\n",
        "  return label_list"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kOvxKAkLPh34"
      },
      "source": [
        "instagram_posts['labels'] = instagram_posts['image_url'].apply(labelAnnotation)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xG2GNxmjPjXS"
      },
      "source": [
        "instagram_posts.to_csv(\"instagram_posts_labeled.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pb2qUDK6EcYo"
      },
      "source": [
        "## Task A: Engagement Metric"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S8YW33W6Eo-s"
      },
      "source": [
        "ig_post_labeled = pd.read_csv(\"instagram_posts_labeled.csv\", index_col=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v40op3XISdkb"
      },
      "source": [
        "# Normalizing the like and comment values\n",
        "ig_post_labeled['like_percent'] = (ig_post_labeled['like_num'] / ig_post_labeled.like_num.max())\n",
        "ig_post_labeled['comment_percent'] = (ig_post_labeled['comment_num'] / ig_post_labeled.comment_num.max())\n",
        "\n",
        "# Calculating the engagement score using specific weights for the likes and the comments\n",
        "like_weight = 0.4\n",
        "comment_weight = 0.6\n",
        "ig_post_labeled['engagement_score'] = ig_post_labeled['like_percent']*like_weight + ig_post_labeled['comment_percent']*comment_weight\n",
        "\n",
        "# Classifying the engagment score as High (1) or Low(0) based on the median value\n",
        "ig_post_labeled['engagement_binary_class'] = np.where(ig_post_labeled['engagement_score'] >= np.median(ig_post_labeled.engagement_score), 1, 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5GSIskhicGWG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 366
        },
        "outputId": "1b5d017c-96b6-4f1d-eb13-542cdf99815f"
      },
      "source": [
        "ig_post_labeled.sort_values(by=['engagement_score']).head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>post_url</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>like_num</th>\n",
              "      <th>post_caption</th>\n",
              "      <th>image_url</th>\n",
              "      <th>labels</th>\n",
              "      <th>like_percent</th>\n",
              "      <th>comment_percent</th>\n",
              "      <th>engagement_score</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://www.instagram.com/p/CHYOgoNLjwD/</td>\n",
              "      <td>157</td>\n",
              "      <td>24098</td>\n",
              "      <td>Photo by @erintrieb / On Election Day I set up...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Hair', 'Face', 'White', 'Black', 'Hairstyle'...</td>\n",
              "      <td>0.014033</td>\n",
              "      <td>0.003340</td>\n",
              "      <td>0.007617</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>https://www.instagram.com/p/CHJKjx4llu1/</td>\n",
              "      <td>215</td>\n",
              "      <td>77932</td>\n",
              "      <td>Photo by @andreabruce / Poll workers Tom Shank...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Table', 'Design', 'Organism', 'Room', 'Event...</td>\n",
              "      <td>0.045382</td>\n",
              "      <td>0.004573</td>\n",
              "      <td>0.020897</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>125</th>\n",
              "      <td>https://www.instagram.com/p/CGk4iNtFt17/</td>\n",
              "      <td>174</td>\n",
              "      <td>89882</td>\n",
              "      <td>Ode to Michigan by James Earl Jones / For many...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Advertising', 'Mural', 'Poster', 'Font', 'Ba...</td>\n",
              "      <td>0.052341</td>\n",
              "      <td>0.003701</td>\n",
              "      <td>0.023157</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     post_url  ...  engagement_binary_class\n",
              "0    https://www.instagram.com/p/CHYOgoNLjwD/  ...                        0\n",
              "39   https://www.instagram.com/p/CHJKjx4llu1/  ...                        0\n",
              "125  https://www.instagram.com/p/CGk4iNtFt17/  ...                        0\n",
              "\n",
              "[3 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b7ay7AWAW-4A",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 349
        },
        "outputId": "d3a1267f-f453-4ea6-eae6-e6c02f932cbd"
      },
      "source": [
        "ig_post_labeled.sort_values(by=['engagement_score'], ascending=False).head(3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>post_url</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>like_num</th>\n",
              "      <th>post_caption</th>\n",
              "      <th>image_url</th>\n",
              "      <th>labels</th>\n",
              "      <th>like_percent</th>\n",
              "      <th>comment_percent</th>\n",
              "      <th>engagement_score</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>117</th>\n",
              "      <td>https://www.instagram.com/p/CGmxGAdM8vA/</td>\n",
              "      <td>47012</td>\n",
              "      <td>371803</td>\n",
              "      <td>Photo by Anastasia Taylor-Lind @anastasiatl / ...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Landmark', 'Medieval architecture', 'Archite...</td>\n",
              "      <td>0.216513</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.686605</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>https://www.instagram.com/p/CHQ-tOlDgpV/</td>\n",
              "      <td>8087</td>\n",
              "      <td>1717229</td>\n",
              "      <td>Photo by @bertiegregory / How many fish can a ...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Beak', 'Fish', 'Bird']</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.172020</td>\n",
              "      <td>0.503212</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>https://www.instagram.com/p/CHUyzWrAmaU/</td>\n",
              "      <td>23345</td>\n",
              "      <td>487668</td>\n",
              "      <td>Photo by Anastasia Taylor-Lind @anastasiatl / ...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Bride', 'Photograph', 'Wedding dress', 'Dres...</td>\n",
              "      <td>0.283985</td>\n",
              "      <td>0.496575</td>\n",
              "      <td>0.411539</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     post_url  ...  engagement_binary_class\n",
              "117  https://www.instagram.com/p/CGmxGAdM8vA/  ...                        1\n",
              "23   https://www.instagram.com/p/CHQ-tOlDgpV/  ...                        1\n",
              "12   https://www.instagram.com/p/CHUyzWrAmaU/  ...                        1\n",
              "\n",
              "[3 rows x 10 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mnVMMgqzEd5I"
      },
      "source": [
        "## Task B: Logistic Regression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IKbFeCP5VPJj"
      },
      "source": [
        "### Word Replacement"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gJnNUXraRz7I"
      },
      "source": [
        "instagram_posts = pd.read_csv('instagram_posts_labeled.csv', index_col=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HSd_20WTVsQn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "f35e0c7b-4da9-42f8-eb02-b456eab180b1"
      },
      "source": [
        "instagram_posts.loc[0,'labels']"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"['Hair', 'Face', 'White', 'Black', 'Hairstyle', 'Shoulder', 'Head', 'Eyebrow', 'Black-and-white', 'T-shirt']\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a9zwgqzD1djJ"
      },
      "source": [
        "labels_df = ig_post_labeled.loc[:,['labels','engagement_binary_class']]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eS8Xi1vx1-9B",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e5d78d00-e490-497c-95b9-ba94a0fbfe19"
      },
      "source": [
        "replace_chars = \"',[]\" # initializing replacement characters (otherwise the labels show as strings)\n",
        "\n",
        "documents = []\n",
        "for i in range((labels_df.shape[0])):\n",
        "    labels =  labels_df.loc[i,'labels'].split()\n",
        "    for label in labels:\n",
        "        for char in replace_chars:\n",
        "            label = label.replace(char,'') # remove the excess characters\n",
        "        content = label.lower() # make it lowercase\n",
        "        documents.append(content) # add it to the document array\n",
        "vectorizer = TfidfVectorizer()\n",
        "vectors = vectorizer.fit_transform(documents)\n",
        "feature_names = vectorizer.get_feature_names()\n",
        "dense = vectors.todense()\n",
        "denselist = dense.tolist()\n",
        "label_tfidf_df = pd.DataFrame(denselist, columns=feature_names).sum(axis=0).sort_values(ascending=False)\n",
        "label_tfidf_df.to_csv(\"label_tfidf.csv\")\n",
        "label_tfidf_df.head(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "sky            142.0\n",
              "photography    125.0\n",
              "tree            80.0\n",
              "plant           72.0\n",
              "water           71.0\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X7g5INBS2Byo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "outputId": "bf2e2a16-0dee-4743-d6ff-06a4f6c4f690"
      },
      "source": [
        "df = pd.read_csv(\"label_tfidf.csv\", header=0, names=['label','tfidf'])\n",
        "df = df[df.tfidf > 5] # using 5 as a threshold for significant tfidf scores\n",
        "\n",
        "label_list = []\n",
        "for index, rows in df.iterrows():  \n",
        "    label_list.append(str(rows.values[0])) # creating a list of all the labels\n",
        "    ## may current repeat labels-- might need to look into changing that\n",
        "\n",
        "# for each label in the label list, create a column in the dataframe for that label\n",
        "# this allows us to use them all as independent variables for the logistic regression\n",
        "for label in label_list:\n",
        "    labels_df[label] = 0 # initializing the columns with all 0s\n",
        "labels_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>labels</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "      <th>sky</th>\n",
              "      <th>photography</th>\n",
              "      <th>tree</th>\n",
              "      <th>plant</th>\n",
              "      <th>water</th>\n",
              "      <th>phenomenon</th>\n",
              "      <th>landscape</th>\n",
              "      <th>natural</th>\n",
              "      <th>mountain</th>\n",
              "      <th>wildlife</th>\n",
              "      <th>cloud</th>\n",
              "      <th>nature</th>\n",
              "      <th>event</th>\n",
              "      <th>people</th>\n",
              "      <th>vehicle</th>\n",
              "      <th>animal</th>\n",
              "      <th>night</th>\n",
              "      <th>sea</th>\n",
              "      <th>adaptation</th>\n",
              "      <th>geological</th>\n",
              "      <th>monochrome</th>\n",
              "      <th>architecture</th>\n",
              "      <th>rock</th>\n",
              "      <th>black</th>\n",
              "      <th>terrestrial</th>\n",
              "      <th>mammal</th>\n",
              "      <th>recreation</th>\n",
              "      <th>white</th>\n",
              "      <th>and</th>\n",
              "      <th>area</th>\n",
              "      <th>environment</th>\n",
              "      <th>grass</th>\n",
              "      <th>fun</th>\n",
              "      <th>street</th>\n",
              "      <th>bird</th>\n",
              "      <th>landforms</th>\n",
              "      <th>human</th>\n",
              "      <th>art</th>\n",
              "      <th>...</th>\n",
              "      <th>morning</th>\n",
              "      <th>uniform</th>\n",
              "      <th>freezing</th>\n",
              "      <th>boat</th>\n",
              "      <th>flame</th>\n",
              "      <th>games</th>\n",
              "      <th>sports</th>\n",
              "      <th>sport</th>\n",
              "      <th>accessory</th>\n",
              "      <th>biome</th>\n",
              "      <th>fox</th>\n",
              "      <th>big</th>\n",
              "      <th>oceanic</th>\n",
              "      <th>ridge</th>\n",
              "      <th>coastal</th>\n",
              "      <th>glacier</th>\n",
              "      <th>dress</th>\n",
              "      <th>beauty</th>\n",
              "      <th>bed</th>\n",
              "      <th>coral</th>\n",
              "      <th>tiger</th>\n",
              "      <th>aeolian</th>\n",
              "      <th>badlands</th>\n",
              "      <th>steppe</th>\n",
              "      <th>botany</th>\n",
              "      <th>metropolitan</th>\n",
              "      <th>lion</th>\n",
              "      <th>standing</th>\n",
              "      <th>herd</th>\n",
              "      <th>festival</th>\n",
              "      <th>pasture</th>\n",
              "      <th>portrait</th>\n",
              "      <th>party</th>\n",
              "      <th>world</th>\n",
              "      <th>wing</th>\n",
              "      <th>hairstyle</th>\n",
              "      <th>motor</th>\n",
              "      <th>protest</th>\n",
              "      <th>sculpture</th>\n",
              "      <th>trail</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>['Hair', 'Face', 'White', 'Black', 'Hairstyle'...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>['Room', 'Textile', 'Tradition', 'Bazaar', 'Fl...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>['Mountainous landforms', 'Mountain', 'Sky', '...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>['Motor vehicle', 'Vehicle', 'Mode of transpor...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>['Water', 'Blue', 'Liquid', 'Aqua', 'Transpare...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 223 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              labels  ...  trail\n",
              "0  ['Hair', 'Face', 'White', 'Black', 'Hairstyle'...  ...      0\n",
              "1  ['Room', 'Textile', 'Tradition', 'Bazaar', 'Fl...  ...      0\n",
              "2  ['Mountainous landforms', 'Mountain', 'Sky', '...  ...      0\n",
              "3  ['Motor vehicle', 'Vehicle', 'Mode of transpor...  ...      0\n",
              "4  ['Water', 'Blue', 'Liquid', 'Aqua', 'Transpare...  ...      0\n",
              "\n",
              "[5 rows x 223 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 124
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pB2YwcMU2PLc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 627
        },
        "outputId": "39394246-b4b8-46be-cc5f-65426427991c"
      },
      "source": [
        "index = 0\n",
        "for label in labels_df['labels']: # for each row in the 'labels' column\n",
        "    label = label.split(sep=None) # split the label string up into words\n",
        "    for item in label: # for each of those words\n",
        "        for char in replace_chars:\n",
        "            item = item.replace(char,'') # remove the excess characters\n",
        "        item = item.lower() # make it lowercase\n",
        "        if item in label_list:\n",
        "            labels_df[item][index] = 1 # set the value of that item to 1, in the corresponding row/column\n",
        "    index = index + 1\n",
        "labels_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:9: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  if __name__ == '__main__':\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>labels</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "      <th>sky</th>\n",
              "      <th>photography</th>\n",
              "      <th>tree</th>\n",
              "      <th>plant</th>\n",
              "      <th>water</th>\n",
              "      <th>phenomenon</th>\n",
              "      <th>landscape</th>\n",
              "      <th>natural</th>\n",
              "      <th>mountain</th>\n",
              "      <th>wildlife</th>\n",
              "      <th>cloud</th>\n",
              "      <th>nature</th>\n",
              "      <th>event</th>\n",
              "      <th>people</th>\n",
              "      <th>vehicle</th>\n",
              "      <th>animal</th>\n",
              "      <th>night</th>\n",
              "      <th>sea</th>\n",
              "      <th>adaptation</th>\n",
              "      <th>geological</th>\n",
              "      <th>monochrome</th>\n",
              "      <th>architecture</th>\n",
              "      <th>rock</th>\n",
              "      <th>black</th>\n",
              "      <th>terrestrial</th>\n",
              "      <th>mammal</th>\n",
              "      <th>recreation</th>\n",
              "      <th>white</th>\n",
              "      <th>and</th>\n",
              "      <th>area</th>\n",
              "      <th>environment</th>\n",
              "      <th>grass</th>\n",
              "      <th>fun</th>\n",
              "      <th>street</th>\n",
              "      <th>bird</th>\n",
              "      <th>landforms</th>\n",
              "      <th>human</th>\n",
              "      <th>art</th>\n",
              "      <th>...</th>\n",
              "      <th>morning</th>\n",
              "      <th>uniform</th>\n",
              "      <th>freezing</th>\n",
              "      <th>boat</th>\n",
              "      <th>flame</th>\n",
              "      <th>games</th>\n",
              "      <th>sports</th>\n",
              "      <th>sport</th>\n",
              "      <th>accessory</th>\n",
              "      <th>biome</th>\n",
              "      <th>fox</th>\n",
              "      <th>big</th>\n",
              "      <th>oceanic</th>\n",
              "      <th>ridge</th>\n",
              "      <th>coastal</th>\n",
              "      <th>glacier</th>\n",
              "      <th>dress</th>\n",
              "      <th>beauty</th>\n",
              "      <th>bed</th>\n",
              "      <th>coral</th>\n",
              "      <th>tiger</th>\n",
              "      <th>aeolian</th>\n",
              "      <th>badlands</th>\n",
              "      <th>steppe</th>\n",
              "      <th>botany</th>\n",
              "      <th>metropolitan</th>\n",
              "      <th>lion</th>\n",
              "      <th>standing</th>\n",
              "      <th>herd</th>\n",
              "      <th>festival</th>\n",
              "      <th>pasture</th>\n",
              "      <th>portrait</th>\n",
              "      <th>party</th>\n",
              "      <th>world</th>\n",
              "      <th>wing</th>\n",
              "      <th>hairstyle</th>\n",
              "      <th>motor</th>\n",
              "      <th>protest</th>\n",
              "      <th>sculpture</th>\n",
              "      <th>trail</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>['Hair', 'Face', 'White', 'Black', 'Hairstyle'...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>['Room', 'Textile', 'Tradition', 'Bazaar', 'Fl...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>['Mountainous landforms', 'Mountain', 'Sky', '...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>['Motor vehicle', 'Vehicle', 'Mode of transpor...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>['Water', 'Blue', 'Liquid', 'Aqua', 'Transpare...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 223 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              labels  ...  trail\n",
              "0  ['Hair', 'Face', 'White', 'Black', 'Hairstyle'...  ...      0\n",
              "1  ['Room', 'Textile', 'Tradition', 'Bazaar', 'Fl...  ...      0\n",
              "2  ['Mountainous landforms', 'Mountain', 'Sky', '...  ...      0\n",
              "3  ['Motor vehicle', 'Vehicle', 'Mode of transpor...  ...      0\n",
              "4  ['Water', 'Blue', 'Liquid', 'Aqua', 'Transpare...  ...      0\n",
              "\n",
              "[5 rows x 223 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 125
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xK45WpH42PpL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "868bcfec-502b-4a8a-ee9d-246336e549cd"
      },
      "source": [
        "# setting up logistic regression\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn import linear_model\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "X = labels_df.iloc[:, 2:].values\n",
        "y = labels_df.iloc[:, 1].values\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.10, random_state=0)\n",
        "\n",
        "lrmodel = LogisticRegression()\n",
        "lrmodel.fit(X_train,y_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
              "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
              "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
              "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
              "                   warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 126
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SlCdPBmx2TEH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "10cb8db4-74f5-459c-b2fe-78db9cc86afd"
      },
      "source": [
        "# using the model to predict the test data\n",
        "y_pred = lrmodel.predict(X_test)\n",
        "\n",
        "# printing misclassifications\n",
        "print('Misclassified samples: %d' %(y_test != y_pred).sum())\n",
        "\n",
        "# printing accuracy score\n",
        "print('Accuracy: %.2f' % accuracy_score(y_test,y_pred))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Misclassified samples: 13\n",
            "Accuracy: 0.74\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8OyGPnih6rpI"
      },
      "source": [
        "#Confusion matrix\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn.utils.multiclass import unique_labels\n",
        "\n",
        "def plot_confusion_matrix(y_true, y_pred, classes,\n",
        "                          normalize=False,\n",
        "                          title=None,\n",
        "                          cmap=plt.cm.Blues):\n",
        "    \"\"\"\n",
        "    This function prints and plots the confusion matrix.\n",
        "    Normalization can be applied by setting `normalize=True`.\n",
        "    \"\"\"\n",
        "    if not title:\n",
        "        if normalize:\n",
        "            title = 'Normalized confusion matrix'\n",
        "        else:\n",
        "            title = 'Confusion matrix, without normalization'\n",
        "\n",
        "    # Compute confusion matrix\n",
        "    cm = confusion_matrix(y_true, y_pred)\n",
        "    # Only use the labels that appear in the data\n",
        "    classes = classes[unique_labels(y_true, y_pred)]\n",
        "    if normalize:\n",
        "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "        print(\"Normalized confusion matrix\")\n",
        "    else:\n",
        "        print('Confusion matrix, without normalization')\n",
        "\n",
        "    #print(cm)\n",
        "\n",
        "    fig, ax = plt.subplots()\n",
        "    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    ax.figure.colorbar(im, ax=ax)\n",
        "    # We want to show all ticks...\n",
        "    ax.set(xticks=np.arange(cm.shape[1]),\n",
        "           yticks=np.arange(cm.shape[0]),\n",
        "           # ... and label them with the respective list entries\n",
        "           xticklabels=classes, yticklabels=classes,\n",
        "           title=title,\n",
        "           ylabel='True label',\n",
        "           xlabel='Predicted label')\n",
        "\n",
        "    # Rotate the tick labels and set their alignment.\n",
        "    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\",\n",
        "             rotation_mode=\"anchor\")\n",
        "\n",
        "    # Loop over data dimensions and create text annotations.\n",
        "    fmt = '.2f' if normalize else 'd'\n",
        "    thresh = cm.max() / 2.\n",
        "    for i in range(cm.shape[0]):\n",
        "        for j in range(cm.shape[1]):\n",
        "            ax.text(j, i, format(cm[i, j], fmt),\n",
        "                    ha=\"center\", va=\"center\",\n",
        "                    color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "    fig.tight_layout()\n",
        "    return ax"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331
        },
        "id": "MzM45SUd90R2",
        "outputId": "997c15d3-23c9-45b3-9e49-90b9c02bd9d4"
      },
      "source": [
        "plot_confusion_matrix(y_true = y_test, \n",
        "                      y_pred = y_pred, classes = np.unique(y_test),\n",
        "                      title = 'Labels: Confusion matrix')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion matrix, without normalization\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f731a114f60>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 129
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATYAAAEYCAYAAADWGtrvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAc7UlEQVR4nO3debwcZZ3v8c83CSEkEAIEFEjYIcAAiRj2AcMikrli1EGQ5Q6bJICKckVGuKNhRIErOijKYoRM2IzsAoosBiEECRAwSNgZ1rCHEJQQCIHf/aPqSHM4p7v6VPfp6jrfN69+0V3d9dSvz+F8eZ5anlJEYGZWJv1aXYCZWaM52MysdBxsZlY6DjYzKx0Hm5mVjoPNzErHwdZgkm6V9JXeXre3SBolaa6kv0s6Jkc750r6biNraxVJJ0o6r9V12AccbN2Q9LSkPVpdR09IGijpJEmPS1qcfpepktZrQPPHA3+KiJUi4syeNhIRR0bEyQ2op2kkjZM0v9bnIuKUiCj0/5D6GgdbOV0BfA44AFgZGA3cC+zegLbXBR5sQDulIGlAq2uwj3Kw1UnSKpJ+J+lVSa+nz0d0+tiGku6W9DdJ10hatWL97SX9WdIiSfdLGtfNdjaSdJukNyQtkHRpxvr2AD4NTIiIeyJiWUS8ERFnRcT56WfWknStpIWSnpB0RMX6J0m6TNKF6XDzQUlj0/duAXYFfiHpTUmbdB4+SzpE0qz0uSSdIemV9GfxgKQt0vemSfpBxXpHpLUsTGtbq+K9kHRk2gNdJOksSerm+58k6XJJF6f1P5DWeUJax3OS9qz4/KGSHk4/+6SkSenyIcAfgLXS7/pm+nM7SdIVaft/Aw5Jl12crrefpKckDU1fj5f0kqTVs/z+rDEcbPXrB/w3Sc9lHWAJ8ItOn/k34DBgTWAZcCaApLWB3wM/AFYFjgOu7OY/+pOBm4BVgBHAzzveSMP0O93Utwdwd0Q8V+U7/AaYD6wF7AOcImm3ivc/l35mGHBtx/eLiN2A24GvRcSKEfFYlW0A7AnsAmxC0nPcF3it84fSbZ+avr8m8Ey6/UqfBbYBtko/95kq290buIjkZ/cX4EaS39vawPeBX1Z89pW07aHAocAZkraOiMXAeOCF9LuuGBEvpOtMIOkVDwMuqdxwRFwK/Bk4U9JqwPnAVyLi1Sr1WoM52OoUEa9FxJUR8VZE/B34IfCpTh+7KCLmpX8c3wX2ldQfOAi4PiKuj4j3I+JmYA7wL11s6l2S8FwrIt6OiFkVNXw2Ik7rpsTVgBe7q1/SSGAn4N/TducC55GEcYdZaY3vkQTE6G5/INW9C6wEbAooIh6OiK5qOxCYGhH3RcQ7wAnADp32CZ4WEYsi4lngT8CYKtu9PSJujIhlwOXA6un675IE5nqShgFExO8j4n8icRvJ/0x2rvG97oyI36a/wyVdvP9VYDfgVuC6iPhdjfaswRxsdZI0WNIvJT2TDkVmAsPS4OpQ2Vt6BlgOGE4SVF9Kh1OLJC0C/pmkl9LZ8YCAu9Ph4GEZS3ytm/Y6rAUsTEO5ssa1K16/VPH8LWBQT/YlRcQtJL29s4BXJE3pGKJ1UdMzFeu9SfI9qtW0YpVNv1zxfAmwIA3pjtd0rJ8OFWenQ+BFJP+TGV7jq1XrDRMRi0gCdQvgJzXasiZwsNXvW8AoYLuIGEoy1IIkhDqMrHi+DknPZQHJH8RFETGs4jGkq95XRLwUEUdExFrAJOBsSRtlqO+PwLZd7Pfr8AKwqqSVOtX4fIa2u7IYGFzx+uOVb0bEmRHxSWBzkiHpt7upad2OF+n+rdVy1JSJpOWBK4EfAx+LiGHA9Xzwu+xu6puqU+JIGkOyK2I66W4I610OtuqWkzSo4jGAZGi1BFiUHhSY3MV6B0naXNJgkn06V6Q9houBvSV9RlL/tM1xXYWQpC9VLH+d5I/p/VoFR8QfgZuBqyV9UtIASSulO98PS/e9/Rk4Nd3+VsDhaW09MRf4YtqT3Shtq+M7bCNpO0nLkQTg2918h+nAoZLGpGFzCnBXRDzdw5qyGggsD7wKLJM0nmS/YIeXgdUkrZy1QUmDSH6WJ5Lss1tb0tGNK9mycLBVdz1JiHU8TgJ+CqxA0gObDdzQxXoXAdNIhk+DgGMA0lCZQPIf/askPbhv0/XvYRvgLklvkuzA/0ZEPAkg6Q+STqxS9z5p7ZcCbwDzgLEkvTmA/YH1SHpKVwOT00DsiTOApSQhcAEf3pk+FPgVSTA/QzK8PL1zA+m2v0vSe3oR2BD4cg/rySwdjh8DXJbWeADJz7rj/UdIQvfJdNfBWl029GGnAs9FxDnp/sKDgB9I2rjhX8C6JU80aWZl4x6bmZWOg83MSsfBZmal42Azs9Ip1AW8GrBCaOBKtT9ohfGJzdZpdQlWh2eeeZoFCxZ0eZ1tT/Ufum7Esq4uwPioWPLqjRGxVyO335ViBdvAlVh+1L6tLsPqcMddnS+TtSLbabuxDW8zli3J/Hf79tyzal3V0RCFCjYza0cCFWuvloPNzPIR0K9/zY/1JgebmeXX9fR4LeNgM7OcPBQ1szJyj83MSkW4x2ZmZSP32MyshHxU1MzKxQcPzKxshIeiZlZC7rGZWbkUbyharGrMrD31U7ZHDZKmSnpF0ryKZWPSWyTOlTRH0rY1y8n5dcysr+u4VjTLo7ZpQOdpjX4E/GdEjAG+l76uykNRM8upcUPRiJgpab3Oi0nueAawMsnd1apysJlZftmPig6XNKfi9ZSImFJjnW8CN0r6Mckoc8daG3GwmVl+2XtsCyKi3tkujwKOjYgrJe0LnA/sUW0F72Mzs3yk7I+eORi4Kn1+OeCDB2bWCxp38KArLwCfSp/vBjxeawUPRc0sp8YdPJA0HRhHsi9uPjAZOAL4maQBwNvAxFrtONjMLL8GXVIVEft389Yn62nHwWZm+Xg+NjMrn+JdUuVgM7P8PLuHmZWOJ5o0s1KRh6JmVkYeippZ2cjBZmZlkswM7mAzszJR+igQB5uZ5ST69fPBAzMrGQ9Fzax0HGxmVi7ex2ZmZSPkHpuZlY8PHphZ6bjHZmbl4n1sZlZG7rGZWan44IGZlZKDzczKRaB+DjYzKxn32MysdBxsZlYqRTx4UKzThc2sPSnjo1Yz0lRJr0ia12n51yU9IulBST+q1Y57bGaWjxo6FJ0G/AK48B/NS7sCE4DREfGOpDVqNeJgM7PcGnWtaETMlLRep8VHAadFxDvpZ16pWU9DqjGzvq1BQ9FubALsLOkuSbdJ2qbWCg62Bjt38oE8M+NU5lx+4j+WbbnJ2tx6wbe457ITueKnk1hpyKAWVmi1LFq0iP3324fRW2zKmC03Y/add7a6pMKTlOkBDJc0p+IxMUPzA4BVge2BbwOXqcbYt6nBJmkvSY9KekLSd5q5raK46LrZTPjqWR9ads73DuA/zryGbfY9hWv/dD/HHrx7i6qzLI479hvsuede3D/vEe6+93423WyzVpdUaFlDLc2iBRExtuIxJcMm5gNXReJu4H1geLUVmhZskvoDZwHjgc2B/SVt3qztFcUd9/0PC99460PLNlpnDWbd+wQAt8x+hM/vPqYVpVkGb7zxBrNmzeSQww4HYODAgQwbNqzFVRVfHcHWE78Fdk23swkwEFhQbYVm9ti2BZ6IiCcjYinwG5IjG33Ow0++yN7jtgLgi5/emhEfW6XFFVl3nn7qKYYPX52Jhx/K9mM/wVETv8LixYtbXVbhNSrYJE0H7gRGSZov6XBgKrBBegrIb4CDIyKqtdPMYFsbeK7i9fx02YdImtgx3o5lS5pYTutMOukSJu67M3dccjwrDl6epe++1+qSrBvLli1j7l/u44hJRzF7zl8YPGQIP/7Raa0uq/DUT5ketUTE/hGxZkQsFxEjIuL8iFgaEQdFxBYRsXVE3FKrnZaf7pGOsacA9Bu8RtUUblePPf0yex+d7HfbaJ01GL/zP7W4IuvO2iNGsPaIEWy73XYAfOFf9+EnDrbqGnseW0M0s8f2PDCy4vWIdFmfs/oqKwLJL/87R3yGX10xq8UVWXc+/vGPM2LESB579FEAbr1lBptuVvpdw7kIkLI9eksze2z3ABtLWp8k0L4MHNDE7RXCBacews6f3Jjhw1bkiRtO5uRzr2fFFZZn0n67AHDNLXO58JrZLa7Sqvmvn/6cQ//tQJYuXcp6G2zAlPP+u9UlFVzxrhVtWrBFxDJJXwNuBPoDUyPiwWZtrygOPmFal8vPmn5rr9ZhPTd6zBjuuGtOq8toKwXLtebuY4uI64Hrm7kNM2sxQT9PNGlmZSIcbGZWQn1qKGpmfUOfOXhgZn1EL5/KkYWDzcxySc5jK1ayOdjMLCf54IGZlY97bGZWLt7HZmZl431sZlZKBcs1B5uZ5ecem5mVi68VNbOy6ZiPrUgcbGaWUx+aj83M+o6C5ZqDzczyc4/NzEpFPnhgZmXkHpuZlU7Bcs3BZmb5Fa3H1sz7ippZX5DxnqJZsk/SVEmvSJrXxXvfkhSShtdqx8FmZrkoPY8tyyODacBeH9mGNBLYE3g2SyMONjPLrX8/ZXrUEhEzgYVdvHUGcDwQWerxPjYzy62OXWzDJVXejXpKREyp3rYmAM9HxP1Z9+U52Mwsl2T/WeZkWxARY7O3rcHAiSTD0MwcbGaWWxPPz90QWB/o6K2NAO6TtG1EvNTdSt0Gm6SfU2U8GxHH9LxWMyuTZp3uEREPAGtUbOdpYGxELKi2XrUe25wq75mZ/UOjck3SdGAcyb64+cDkiDi/3na6DbaIuKDTBgdHxFv1bsDMyk1A/wYlW0TsX+P99bK0U/N0D0k7SHoIeCR9PVrS2VkaN7M+IOM5bL15dUKW89h+CnwGeA0gIu4HdmlmUWbWXhp15UGjZDoqGhHPdUrb95pTjpm1GwH9CnataJZge07SjkBIWg74BvBwc8sys3ZSsFzLFGxHAj8D1gZeAG4EvtrMosysfbTlRJPp+SIH9kItZtamijYUzXJUdANJ10l6NZ1O5BpJG/RGcWbWHpTx0VuyHBX9NXAZsCawFnA5ML2ZRZlZe2nH0z0GR8RFEbEsfVwMDGp2YWbWHpKjotkevaXataKrpk//IOk7wG9Irh3dD7i+F2ozs3bQy72xLKodPLiXJMg6Kp5U8V4AJzSrKDNrL21zVDQi1u/NQsysPXUMRYsk05UHkrYANqdi31pEXNisosysvbTTUBQASZNJphHZnGTf2nhgFuBgMzOgd0/lyCLLUdF9gN2BlyLiUGA0sHJTqzKztiElJ+hmefSWLEPRJRHxvqRlkoYCrwAjm1yXmbWRgo1EMwXbHEnDgF+RHCl9E7izqVWZWVtpm6OiHSLi6PTpuZJuAIZGxF+bW5aZtQvRu8PMLKqdoLt1tfci4r7mlGRmbaWXJ5HMolqP7SdV3gtgtwbXwpajRnLTbWc0ullrolX2+n+tLsHq8M7j3d6xLpe2Od0jInbtzULMrH1lOb2iN/mGyWaWi2ijHpuZWVYDCtZlc7CZWS7JHaiK1WPLMoOuJB0k6Xvp63Ukbdv80sysXTRqPjZJU9OZuudVLDtd0iOS/irp6vS82ur1ZKj5bGAHoOMOzX8Hzsqwnpn1EQ28r+g0YK9Oy24GtoiIrYDHyDBlWpZg2y4ivgq8DRARrwMDM5VoZqXXcV/RRlwrGhEzgYWdlt0UEcvSl7OBEbXaybKP7V1J/UnOXUPS6sD7GdYzsz6if+/tYjsMuLTWh7IE25nA1cAakn5IMtvHf+SrzczKQvXN3DFc0pyK11MiYkrG7fxfYBlwSa3PZrlW9BJJ95JMXSTg8xHhO8Gb2T/UcVB0QUSMrb99HQJ8Ftg9IqLW57NMNLkO8BZwXeWyiHi23uLMrJyaObmHpL2A44FPRcRbWdbJMhT9PR/c1GUQsD7wKPBPPazTzEqk4+BBQ9qSppPM2D1c0nxgMslR0OWBm9Pz5WZHxJHV2skyFN2y04a3Bo7u5uNm1gc16vzciNi/i8Xn19tO3VceRMR9krardz0zKylB/4JdeZBlH9v/qXjZD9gaeKFpFZlZW2nX2++tVPF8Gck+tyubU46ZtaO2Crb0xNyVIuK4XqrHzNpQ0S6CrzY1+ICIWCZpp94syMzaS7sNRe8m2Z82V9K1wOXA4o43I+KqJtdmZu2gze550GEQ8BrJPQ46zmcLwMFmZggYULAuW7VgWyM9IjqPDwKtQ81LGsys72inHlt/YEU+HGgdHGxmlhL9uoyJ1qkWbC9GxPd7rRIza0vJzVxaXcWHVQu2gpVqZoWUcdrv3lQt2HbvtSrMrG0J6F+wZKt2w+SF3b1nZlapUbN7NIpvv2dmuRUs1xxsZpaPyHZXqN7kYDOzfAp4w2QHm5nlVqxYc7CZWU6iDSeaNDOrpWC55mAzs7zkfWxmVi4+KmpmpeQem5mVTrFizcFmZjmpHW+/Z2ZWS9GGokXb52dmbUgZHzXbkaZKekXSvIplq0q6WdLj6b9XqdWOg83McpOyPTKYBuzVadl3gBkRsTEwI31dlYPNzHJJTvdQpkctETET6Dxl2gTggvT5BcDna7XjfWxmllsdu9iGS5pT8XpKREypsc7HIuLF9PlLwMdqbcTBZmY5qZ6JJhdExNiebikiQlLNm0l5KGpmuTRyKNqNlyWtCZD++5VaKzjYzCyfjAcOcpwRci1wcPr8YOCaWis42Mwst0YFm6TpwJ3AKEnzJR0OnAZ8WtLjwB7p66q8j83MclODLqqKiP27eauuu+Y52MwsF080aWalVLBcc7CZWX6NGoo2ioOtiZ54/FEmHXrgP14/8/RTHH/iZCYefUwLq7LOzj1uPOO325BXF73F2COmArDVhmvw829+huWX68+y997nm2fezJxHX6zRUt8koGA3gm/eUdGuLmbtazbaeBQzZs1hxqw53HTbXaywwmDGf3ZCq8uyTi668QEmnHD5h5b98Ihx/PDCO9j+yGmcfMEsfjhxXGuKawvK/E9vaebpHtP46MWsfdbtt97CeutvwMh11m11KdbJHQ/MZ+Hfl3xoWQBDhwwEYOUhy/Pia2+2oLI20fzz2OrWtKFoRMyUtF6z2m83v73qMj6/z36tLsMy+vbZM7jutH05deKu9Osndj3m4laXVFhFPCra8hN0JU2UNEfSnIWvLWh1OU2xdOlSbrr+d3zu8//a6lIso4l7j+H4c2aw8QHncPw5t3DOceNbXVKhNWo+tkZpebBFxJSIGBsRY1ddbXiry2mKW26+gS1Hf4LV16g5KYEVxIF7bslvb38MgCtve4Sxo9ZscUUFV7Bka3mw9QVXX3Gph6Ft5sUFb7Lz6JEAjPvEujzx/OstrqjYinbwwKd7NNnixYuZ+acZnP7Ts1tdinXjghP3ZufR6zB85RV4YvrRnHzBLL56xh84/eg9GNC/H+8sXcbXzrih1WUWWsF2sTUv2NKLWceRTCw3H5gcEec3a3tFNWTIEB5++qVWl2FVHHzKdV0u3+noC7pcbh9VsFxr6lHR7i5mNbMSEcW7S5WHomaWTy+fo5aFg83McitYrjnYzKwBCpZsDjYzy6l3T+XIwsFmZrkUcXYPB5uZ5edgM7Oy8VDUzErHp3uYWekULNccbGaWU2/PSZSBg83MckmOihYr2RxsZpZbsWLN87GZWSM0aKJJScdKelDSPEnTJQ3qSTkONjPLrRETTUpaGzgGGBsRWwD9gS/3pB4PRc0stwbuYhsArCDpXWAw8EJPGnGPzcxyq2MkOrzj5k3pY2JHGxHxPPBj4FngReCNiLipJ/W4x2ZmudQ50eSCiBjbZTvSKsAEYH1gEXC5pIMiou57H7rHZmb5NO6GyXsAT0XEqxHxLnAVsGNPSnKwmVluDToo+iywvaTBSrqAuwMP96QeB5uZ5deAZIuIu4ArgPuAB0jyaUpPyvE+NjPLqXETTUbEZGBy3nYcbGaWiyeaNLNycrCZWdl4okkzK52CTe7hYDOz/AqWaw42M8vJd4I3s7Kp85KqXuFgM7PcihVrDjYza4CCddgcbGaWn0/3MLPyKVauOdjMLL+C5ZqDzczykXz7PTMro2LlmoPNzPIrWK452Mwsv4KNRB1sZpZX4yaabBQHm5nlklxS1eoqPszBZma5OdjMrHQ8FDWzcvG0RWZWNhnvGdqrHGxmll/Bks3BZma5Fe2SKt8J3sxya8CN4D9oSxom6QpJj0h6WNIO9dbjHpuZ5dfYDtvPgBsiYh9JA4HB9TbgYDOz3Bp1uoeklYFdgEMAImIpsLTudiKiIQU1gqRXgWdaXUcTDAcWtLoIq0tZf2frRsTqjWxQ0g0kP68sBgFvV7yeEhFTKtoaA0wBHgJGA/cC34iIxXXVVKRgKytJcyJibKvrsOz8O2sNSWOB2cBOEXGXpJ8Bf4uI79bTjg8emFmRzAfmR8Rd6esrgK3rbcTBZmaFEREvAc9JGpUu2p1kWFoXHzzoHVNqf8QKxr+z1vk6cEl6RPRJ4NB6G/A+NjMrHQ9Fzax0HGxmVjoONjMrHQdbE0kaJWkHSctJ6t/qeiwb/67anw8eNImkLwKnAM+njznAtIj4W0sLs25J2iQiHkuf94+I91pdk/WMe2xNIGk5YD/g8IjYHbgGGAn8u6ShLS3OuiTps8BcSb8GiIj33HNrXw625hkKbJw+vxr4HbAccIBUsMmr+jhJQ4CvAd8Elkq6GBxu7czB1gQR8S7wX8AXJe0cEe8Ds4C5wD+3tDj7iPQC68OAXwPHAYMqw62VtVnPONia53bgJuB/S9olIt6LiF8Da5HMWmAFEhEvRMSbEbEAmASs0BFukraWtGlrK7R6+JKqJomItyVdAgRwQvqH8Q7wMeDFlhZnVUXEa5ImAadLegToD+za4rKsDg62JoqI1yX9iuQi3kkk81AdFBEvt7YyqyUiFkj6KzAe+HREzG91TZadT/foJelO6Ej3t1nBSVoFuAz4VkT8tdX1WH0cbGbdkDQoIt6u/UkrGgebmZWOj4qaWek42MysdBxsZlY6DjYzKx0HWxuR9J6kuZLmSbpcUt13yK5oa5qkfdLn50navMpnx0nasQfbeFrSR+432d3yTp95s85tnSTpuHprtHJysLWXJRExJiK2ILk79pGVb0rq0QnXEfGViKh2J6BxQN3BZtYqDrb2dTuwUdqbul3StcBDkvpLOl3SPZL+ml4ahBK/kPSopD8Ca3Q0JOnW9Ea1SNpL0n2S7pc0Q9J6JAF6bNpb3FnS6pKuTLdxj6Sd0nVXk3STpAclnQfUnMVE0m8l3ZuuM7HTe2eky2dIWj1dtqGkG9J1bvc1nNYVX1LVhtKe2XjghnTR1sAWEfFUGg5vRMQ2kpYH7pB0E/AJYBSwOcn1qg8BUzu1uzrwK2CXtK1VI2KhpHOBNyPix+nnfg2cERGzJK0D3AhsBkwGZkXE9yX9L+DwDF/nsHQbKwD3SLoyIl4DhgBzIuJYSd9L2/4ayW3xjoyIxyVtB5wN7NaDH6OVmIOtvawgaW76/HbgfJIh4t0R8VS6fE9gq479Z8DKJPPC7QJMT6fheUHSLV20vz0ws6OtiFjYTR17AJtXTCs3VNKK6Ta+mK77e0mvZ/hOx0j6Qvp8ZFrra8D7wKXp8ouBq9Jt7AhcXrHt5TNsw/oYB1t7WRIRYyoXpH/giysXAV+PiBs7fe5fGlhHP2D7zpcb1Tt/pqRxJCG5Q0S8JelWYFA3H490u4s6/wzMOvM+tvK5ETgqnZ4cSZukM8TOBPZL98GtSdfT8MwGdpG0frruqunyvwMrVXzuJpK7dZN+riNoZgIHpMvGA6vUqHVl4PU01DYl6TF26Ad09DoPIBni/g14StKX0m1Ikue2s49wsJXPeST7z+6TNA/4JUnP/Grg8fS9C4E7O68YEa8CE0mGfffzwVDwOuALHQcPgGOAsenBiYf44Ojsf5IE44MkQ9Jna9R6AzBA0sPAaSTB2mExsG36HXYDvp8uPxA4PK3vQWBChp+J9TG+CN7MSsc9NjMrHQebmZWOg83MSsfBZmal42Azs9JxsJlZ6TjYzKx0/j+9T5T8sT9gGwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vDDFjpOX2WaN"
      },
      "source": [
        "# now doing same thing but for captions"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p2hVJDMw2Wix",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8667fdc8-39be-48c8-a50d-5e3fe4928c82"
      },
      "source": [
        "import inflect\n",
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "\n",
        "p = inflect.engine()\n",
        "stop_words=stopwords.words('english')\n",
        "\n",
        "caption_replace = ig_post_labeled.loc[:,['post_caption','engagement_binary_class']]\n",
        "replace_df = pd.read_csv('replacements.csv', header=None) # manually assigned word replacements\n",
        "\n",
        "for i in range((replace_df.shape[0])-1):\n",
        "    replacee=replace_df.loc[i,0]\n",
        "    if isinstance(replace_df.loc[i,1],str) == True:\n",
        "        replacedby=replace_df.loc[i,1]\n",
        "        caption_replace[\"post_caption\"]= caption_replace[\"post_caption\"].str.replace(rf\"\\b{replacee}\\b\", replacedby, case = False)\n",
        "        \n",
        "documents = []\n",
        "for i in range((caption_replace.shape[0])):\n",
        "    content = caption_replace.loc[i,'post_caption'].lower()\n",
        "    cleaned_content = [word for word in content.split() if not word in stop_words] # removed stopwords\n",
        "    final_content = []\n",
        "    for i in range (len(cleaned_content)):\n",
        "        if p.singular_noun(cleaned_content[i]) is not False:\n",
        "            cleaned_content[i] = p.singular_noun(cleaned_content[i]) # setting plurals to singular for accuracy\n",
        "        word = nltk.pos_tag([cleaned_content[i]])[0]\n",
        "        if word[1] in ['NN','VBG','VB','VBD','NNS','RB','POS','JJ','VBP']: # if word is a noun, verb, jerund\n",
        "            final_content.append(word[0]) # add it in there\n",
        "    documents += [' '.join(final_content)] ### this allows us to (hopefully) only include words that add value (like topics/labels)\n",
        "    \n",
        "vectorizer = TfidfVectorizer()\n",
        "vectors = vectorizer.fit_transform(documents)\n",
        "feature_names = vectorizer.get_feature_names()\n",
        "dense = vectors.todense()\n",
        "denselist = dense.tolist()\n",
        "cap_tfidf_df = pd.DataFrame(denselist, columns=feature_names).sum(axis=0).sort_values(ascending=False)\n",
        "cap_tfidf_df.to_csv(\"cap_tfidf.csv\")\n",
        "cap_tfidf_df.head(5)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
            "[nltk_data]       date!\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "person      18.193696\n",
              "water       16.430825\n",
              "photo       14.668865\n",
              "landform    14.155695\n",
              "flora       13.856893\n",
              "dtype: float64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 131
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KZwVbY1w2Y41",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 479
        },
        "outputId": "0860a234-3a57-425a-d5c0-d4092e884926"
      },
      "source": [
        "df = pd.read_csv(\"cap_tfidf.csv\", header=0, names=['capword','tfidf'])\n",
        "df = df[df.tfidf > 5] # using 5 as a threshold for 'significant' tfidf\n",
        "\n",
        "capword_list = []\n",
        "for index, rows in df.iterrows():  \n",
        "    capword_list.append(str(rows.values[0]))\n",
        "\n",
        "captions_df = caption_replace.loc[:,['post_caption','engagement_binary_class']]\n",
        "captions_df = caption_replace\n",
        "\n",
        "for word in capword_list:\n",
        "    captions_df[word] = 0 # initializing the columns with all 0s\n",
        "captions_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>post_caption</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "      <th>person</th>\n",
              "      <th>water</th>\n",
              "      <th>photo</th>\n",
              "      <th>landform</th>\n",
              "      <th>flora</th>\n",
              "      <th>color</th>\n",
              "      <th>fauna</th>\n",
              "      <th>sky</th>\n",
              "      <th>object</th>\n",
              "      <th>architecture</th>\n",
              "      <th>story</th>\n",
              "      <th>follow</th>\n",
              "      <th>world</th>\n",
              "      <th>year</th>\n",
              "      <th>new</th>\n",
              "      <th>see</th>\n",
              "      <th>image</th>\n",
              "      <th>nat</th>\n",
              "      <th>geo</th>\n",
              "      <th>check</th>\n",
              "      <th>day</th>\n",
              "      <th>link</th>\n",
              "      <th>bio</th>\n",
              "      <th>time</th>\n",
              "      <th>home</th>\n",
              "      <th>life</th>\n",
              "      <th>many</th>\n",
              "      <th>part</th>\n",
              "      <th>species</th>\n",
              "      <th>place</th>\n",
              "      <th>it</th>\n",
              "      <th>technology</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Photo by @erintrieb / On Election Day I set up...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Photo by @tasneemalsultan / A Saudi woman sits...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Photo by Keith Ladzinski @ladzinski / A kiss o...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Photo by @dina_litovsky / Most people don’t as...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Photo by @paoloverzone / Fluorescing a ghostly...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                        post_caption  ...  technology\n",
              "0  Photo by @erintrieb / On Election Day I set up...  ...           0\n",
              "1  Photo by @tasneemalsultan / A Saudi woman sits...  ...           0\n",
              "2  Photo by Keith Ladzinski @ladzinski / A kiss o...  ...           0\n",
              "3  Photo by @dina_litovsky / Most people don’t as...  ...           0\n",
              "4  Photo by @paoloverzone / Fluorescing a ghostly...  ...           0\n",
              "\n",
              "[5 rows x 34 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 132
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XJ1IQzkv2hH-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 564
        },
        "outputId": "e1a64a41-ba25-46d9-80cb-0e7fb5a37fbf"
      },
      "source": [
        "index = 0\n",
        "for capword in captions_df['post_caption']: # for each row in the 'captions' column\n",
        "    caption = capword.split(sep=None) # split the label string up into words\n",
        "    for word in caption: # for each of those words\n",
        "        for char in replace_chars:\n",
        "            word = word.replace(char,'') # remove the excess characters\n",
        "        word = word.lower() # make it lowercase\n",
        "        if word in capword_list:\n",
        "            captions_df[word][index] = 1 # set the value of that item to 1, in the corresponding row/column\n",
        "    index = index + 1\n",
        "captions_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:9: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  if __name__ == '__main__':\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>post_caption</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "      <th>person</th>\n",
              "      <th>water</th>\n",
              "      <th>photo</th>\n",
              "      <th>landform</th>\n",
              "      <th>flora</th>\n",
              "      <th>color</th>\n",
              "      <th>fauna</th>\n",
              "      <th>sky</th>\n",
              "      <th>object</th>\n",
              "      <th>architecture</th>\n",
              "      <th>story</th>\n",
              "      <th>follow</th>\n",
              "      <th>world</th>\n",
              "      <th>year</th>\n",
              "      <th>new</th>\n",
              "      <th>see</th>\n",
              "      <th>image</th>\n",
              "      <th>nat</th>\n",
              "      <th>geo</th>\n",
              "      <th>check</th>\n",
              "      <th>day</th>\n",
              "      <th>link</th>\n",
              "      <th>bio</th>\n",
              "      <th>time</th>\n",
              "      <th>home</th>\n",
              "      <th>life</th>\n",
              "      <th>many</th>\n",
              "      <th>part</th>\n",
              "      <th>species</th>\n",
              "      <th>place</th>\n",
              "      <th>it</th>\n",
              "      <th>technology</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Photo by @erintrieb / On Election Day I set up...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Photo by @tasneemalsultan / A Saudi woman sits...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Photo by Keith Ladzinski @ladzinski / A kiss o...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Photo by @dina_litovsky / Most people don’t as...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Photo by @paoloverzone / Fluorescing a ghostly...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                        post_caption  ...  technology\n",
              "0  Photo by @erintrieb / On Election Day I set up...  ...           0\n",
              "1  Photo by @tasneemalsultan / A Saudi woman sits...  ...           0\n",
              "2  Photo by Keith Ladzinski @ladzinski / A kiss o...  ...           0\n",
              "3  Photo by @dina_litovsky / Most people don’t as...  ...           0\n",
              "4  Photo by @paoloverzone / Fluorescing a ghostly...  ...           0\n",
              "\n",
              "[5 rows x 34 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 133
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PcQA2dxE2h9l",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3e77fcb3-6807-4ba6-93ff-2ddfe19095b2"
      },
      "source": [
        "# setting up logistic regression\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn import linear_model\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "X = captions_df.iloc[:, 2:].values\n",
        "y = captions_df.iloc[:, 1].values\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=0)\n",
        "\n",
        "lrmodel = LogisticRegression()\n",
        "lrmodel.fit(X_train,y_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
              "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
              "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
              "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
              "                   warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 134
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6mO2Ip222kJ7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "299f740b-f8b3-4138-ddbd-1da6f356a95f"
      },
      "source": [
        "# using the model to predict the test data\n",
        "y_pred = lrmodel.predict(X_test)\n",
        "\n",
        "# printing misclassifications\n",
        "print('Misclassified samples: %d' %(y_test != y_pred).sum())\n",
        "\n",
        "# printing accuracy score\n",
        "print('Accuracy: %.2f' % accuracy_score(y_test,y_pred))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Misclassified samples: 17\n",
            "Accuracy: 0.66\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331
        },
        "id": "jhz_qAf6C4tL",
        "outputId": "d68c6abc-7c3f-49e1-b40c-9960e302fe8d"
      },
      "source": [
        "plot_confusion_matrix(y_true = y_test, \n",
        "                      y_pred = y_pred, classes = np.unique(y_test),\n",
        "                      title = 'Captions: Confusion matrix')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion matrix, without normalization\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f731c1ee358>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 136
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATYAAAEYCAYAAADWGtrvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAc7klEQVR4nO3de/xc853H8dc7iSQiVxK3RFzqvkoQt1qqqLLblnaVhrZuldCqrt62tW3Zdher7eqNaopFaSiqaNWlVEkrNDSIRrGuQUhEiEhE4rN/nPNj8vP7zZz5nZnfnDm/99NjHmbOnPM9n5lffp/f93a+RxGBmVmZ9Gt1AGZmjebEZmal48RmZqXjxGZmpePEZmal48RmZqXjxNaLJJ0r6RutjqNZJG0haZakxZJOzFFOab4nSSdLOq/VcfQ16uvz2CQdBnwB2BJYDMwC/isipucs90jg0xHxj7mDbCBJA4GTgcOB9YH5wK3AtyLiiZxlnw+8EhEn5Y2z6CTtBVwSEeNaHYu9U5+usUn6AvB94DRgHWA8cA5wYCvjarIrgQ8DhwEjgO2Ae4B9GlD2hsCDDSinFCQNaHUMfVZE9MkHyS/1q8DHquyzM3AnsAh4DvgxMLDi/QBOBB4DFgDfIfljsRWwDFiZnmNRuv+FwH9WHH8s8CiwELgWWL9T2ccBj6TnP5u3a9ibAn8EXk7Pe3nGz7wvsBTYoMo+66exLExjO7bivVOBXwIXk9RuHwQmpu/dmn7eZeln3hy4jaTW2nH8kcD09LmAs4AXgFeAB4BtGvk9dfHZTgWuAC5J438gjfNraRxPA/tV7H8UMCfd9zFgSrp9jfR7fDP9rK+m39upJH84Lkk/06fTbZekxx0KPA4MT18fAMwDxrT696Fsj75cY9sNGAxcXWWflcBJwOh0/32Az3Ta5yPARGAHkpre0RExh+SX7c6IGBoRIzsXLGlv4HTgEGA94Engsk67fRDYCdg23e8D6fZvAzcBo4BxwI8qyv2NpK9283n2Be6OiKerfObLgLkkv6gHA6elsXb4cLrPSJIk82OAiNgbuAM4If3MD1c5B8B+wJ4kiWVE+vle7LxTzu+pKx8Cfk7y3f0VuJHkj9FY4FvATyv2fSEtezhJkjtL0g4RsYQkKT2bftahEfFsesyBJMltJHBp5Ykj4nLgz8APJa0FnE+S+OdXidd6oC8ntrWABRGxorsdIuKeiJgRESsi6X/6KfDeTrv9d0QsjIinSJq1kzKe/3Dggoi4NyJeJ6k17CZpo4p9zoiIRWnZfwAmpNvfIGn2rR8Ry6KiPzAiPhgRZ1T5zM91F5CkDYDdgX9Ly50FnAd8qmK36RFxfUSsJEkQ22X8vJ29AQwj6dtURMyJiK5iy/M9deWOiLgx/blfAYxJj3+DJGFuJGkkQET8NiL+LxJ/JPljskeNz3VnRPw6It6MiKVdvP9ZYG+S2ux1EfGbGuVZD/TlxPYiMLpaP4ikzdMa0DxJr5D0xY3utFtl7edJkppOFuun+wMQEa+mMY2t2GdexfPXgKHp86+QNOXulvSgpKMznvNFklpPtZgWRsTiim1P1ohpcE/6kiLiVpLa3tnAC5KmShreTUw9/Z668nzF86Ukf9xWVrym43hJB0iaIWmhpEXAP/HOn39n1WrDRMQikoS6DfC9GmVZD/XlxHYn8DpwUJV9fgI8BGwWEcNJRhPVaZ8NKp6PBzqaJLWGm58lqXUBIGkNkhrVM7UCj4h5EXFsRKwPTAHOkbRpreOA3wM7S+puJO9ZYE1Jwyq2jc8SUzeWAEMqXq9b+WZE/DAidgS2JmmSfrmbmHr0PeUhaRBwFfBdYJ20O+F63v75d/fzrfpzlzQBOBqYBvywMdFaZ302sUXEy8A3gbMlHSRpiKTV0r/SZ6a7DSPpBH5V0pbA8V0U9WVJo9Jm3OeBy9PtzwPj0ukVXZkGHCVpQvpLdBpwV2SYciHpYxXJ6SWSX6Y3M3zm3wM3A1dL2lHSAEnDJB0n6ei07+3PwOmSBkvaFjiGpDO8J2YBH02/203Tsjo+w06SdpG0GkkCXNbNZ+jx95TTQGAQyXSYFZIOIOkX7PA8sJakEVkLlDSY5Ls8maTPbqykzn221gB9NrEBRMT3SOawfZ3kH/DTwAnAr9NdvkQyLWIx8DPeTlqVriGZLjEL+C1JhzAko4QPAvMkLeji3L8HvkFSK3gOeBfw8Yyh7wTcJelVkg78z0fEYwCSfifp5CrHHkxS87icZFR1Nsngx+/T9ycBG5HUlK4GTklj7YmzgOUkSeAiVu1MH07ynb5E0tR8kWRUeRU5v6ceS5vjJ5KMAr9E8u/g2or3HyJJuo9JWiQpSxfE6cDTEfGTtL/wE8B/Stqs4R+gj+vzE3TzkBQkzdRHWx2Lmb2tT9fYzKycnNjMrHTcFDWz0nGNzcxKp1AX6WrA6qGBw2rvaIWx/VbjWx2C1eHJJ59gwYIFnedi5tJ/+IYRK7q6yOKdYun8GyNi/0aevyvFSmwDhzFoi0NaHYbV4U93/bjVIVgddt9lYsPLjBVLM//eLpt1dq0rNxqiUInNzNqRQMXq1XJiM7N8BPTr3+ooVuHEZmb5qaHddrk5sZlZTm6KmlkZucZmZqUiXGMzs7KRa2xmVkIeFTWzcvHggZmVjXBT1MxKyDU2MyuX4jVFixWNmbWnfsr2qEHSBZJekDS7YtuE9DaIsyTNlLRzzXByfhwz6+s6rhXN8qjtQqDzskZnAv8RERNI7ix3ZueDOnNT1MxyalxTNCJul7RR580kdzUDGMHb9+7tlhObmeWXfVR0tKSZFa+nRsTUGsf8K3CjpO+StDLfU+skTmxmll/2GtuCiKh3tcvjgZMi4ipJh5Dcu3ffage4j83M8pGyP3rmCOBX6fMrAA8emFkvaNzgQVeeBd6bPt8beKTWAW6KmllOjRs8kDQN2IukL24ucApwLPADSQOAZcDkWuU4sZlZfg26pCoiJnXz1o71lOPEZmb5eD02Myuf4l1S5cRmZvl5dQ8zKx0vNGlmpSI3Rc2sjNwUNbOykRObmZVJsjK4E5uZlYnSR4E4sZlZTqJfPw8emFnJuClqZqXjxGZm5eI+NjMrGyHX2MysfDx4YGal4xqbmZWL+9jMrIxcYzOzUvHggZmVkhObmZWLQP2c2MysZFxjM7PScWIzs1Ip4uBBsaYLm1l7UsZHrWKkCyS9IGl2p+2fk/SQpAclnVmrHNfYzCwfNbQpeiHwY+Dit4qX3gccCGwXEa9LWrtWIU5sZpZbo64VjYjbJW3UafPxwBkR8Xq6zws142lINGbWtzWoKdqNzYE9JN0l6Y+Sdqp1gBNbg517yuE8ecvpzLzi5Le2vXvzsdx20Rf5yy9P5srvT2HYGoNbGKHVsmjRIiYdejDbbbMlE969FTPuvLPVIRWepEwPYLSkmRWPyRmKHwCsCewKfBn4pWq0fZua2CTtL+nvkh6V9NVmnqsofn7dDA787NmrbPvJNw/j6z+8hp0OOY1r/3AfJx2xT4uisyy+dNLn2W+//blv9kPcfc99bLnVVq0OqdCyJrU0Fy2IiIkVj6kZTjEX+FUk7gbeBEZXO6BpiU1Sf+Bs4ABga2CSpK2bdb6i+NO9/8fCl19bZdum49dm+j2PAnDrjIc4aJ8JrQjNMnj55ZeZPv12jjz6GAAGDhzIyJEjWxxV8dWR2Hri18D70vNsDgwEFlQ7oJk1tp2BRyPisYhYDlxGMrLR58x57Dk+tNe2AHz0/Tswbp1RLY7IuvPE448zevQYJh9zFLtO3J7jJ3+aJUuWtDqswmtUYpM0DbgT2ELSXEnHABcAm6RTQC4DjoiIqFZOMxPbWODpitdz022rkDS5o70dK5Y2MZzWmXLqpUw+ZA/+dOlXGDpkEMvfWNnqkKwbK1asYNZf7+XYKcczY+ZfGbLGGnz3zDNaHVbhqZ8yPWqJiEkRsV5ErBYR4yLi/IhYHhGfiIhtImKHiLi1Vjktn+6RtrGnAvQbsnbVLNyuHn7ieT70maTfbdPxa3PAHv/Q4oisO2PHjWPsuHHsvMsuAHzkXw7me05s1TV2HltDNLPG9gywQcXrcem2PmfMqKFA8sP/6rEf4GdXTm9xRNadddddl3HjNuDhv/8dgNtuvYUttyp913AuAqRsj97SzBrbX4DNJG1MktA+DhzWxPMVwkWnH8keO27G6JFDefSGb/Ptc69n6OqDmHLongBcc+ssLr5mRoujtGr+5/s/4qhPHc7y5cvZaJNNmHre/7Y6pIIr3rWiTUtsEbFC0gnAjUB/4IKIeLBZ5yuKI752YZfbz552W6/GYT233YQJ/Omuma0Oo60ULK81t48tIq4Hrm/mOcysxQT9vNCkmZWJcGIzsxLqU01RM+sb+szggZn1Eb08lSMLJzYzyyWZx1aszObEZmY5yYMHZlY+rrGZWbm4j83MysZ9bGZWSgXLa05sZpafa2xmVi6+VtTMyqZjPbYicWIzs5z60HpsZtZ3FCyvObGZWX6usZlZqciDB2ZWRq6xmVnpFCyvObGZWX5Fq7E1876iZtYXZLynaJbcJ+kCSS9Imt3Fe1+UFJJG1yrHic3MclE6jy3LI4MLgf3fcQ5pA2A/4KkshTixmVlu/fsp06OWiLgdWNjFW2cBXwEiSzzuYzOz3OroYhstqfJu1FMjYmr1snUg8ExE3Je1L8+JzcxySfrPMme2BRExMXvZGgKcTNIMzcyJzcxya+L83HcBGwMdtbVxwL2Sdo6Ied0d1G1ik/QjqrRnI+LEnsdqZmXSrOkeEfEAsHbFeZ4AJkbEgmrHVauxzazynpnZWxqV1yRNA/Yi6YubC5wSEefXW063iS0iLup0wiER8Vq9JzCzchPQv0GZLSIm1Xh/oyzl1JzuIWk3SX8DHkpfbyfpnCyFm1kfkHEOW29enZBlHtv3gQ8ALwJExH3Ans0MyszaS6OuPGiUTKOiEfF0p2y7sjnhmFm7EdCvYNeKZklsT0t6DxCSVgM+D8xpblhm1k4KltcyJbbjgB8AY4FngRuBzzYzKDNrH2250GQ6X+TwXojFzNpU0ZqiWUZFN5F0naT56XIi10japDeCM7P2oIyP3pJlVPQXwC+B9YD1gSuAac0MyszaSztO9xgSET+PiBXp4xJgcLMDM7P2kIyKZnv0lmrXiq6ZPv2dpK8Cl5FcO3oocH0vxGZm7aCXa2NZVBs8uIckkXVEPKXivQC+1qygzKy9tM2oaERs3JuBmFl76miKFkmmKw8kbQNsTUXfWkRc3KygzKy9tFNTFABJp5AsI7I1Sd/aAcB0wInNzIDencqRRZZR0YOBfYB5EXEUsB0woqlRmVnbkJIJulkevSVLU3RpRLwpaYWk4cALwAZNjsvM2kjBWqKZEttMSSOBn5GMlL4K3NnUqMysrbTNqGiHiPhM+vRcSTcAwyPi/uaGZWbtQvRuMzOLahN0d6j2XkTc25yQzKyt9PIikllUq7F9r8p7Aezd4FgYv+G6/PtPvtzoYq2JPvzTGa0OwerwyPwlTSm3baZ7RMT7ejMQM2tfWaZX9CbfMNnMchFtVGMzM8tqQMGqbE5sZpZLcgeqYtXYsqygK0mfkPTN9PV4STs3PzQzaxeNWo9N0gXpSt2zK7Z9R9JDku6XdHU6r7Z6PBliPgfYDei4Q/Ni4OwMx5lZH9HA+4peCOzfadvNwDYRsS3wMBmWTMuS2HaJiM8CywAi4iVgYKYQzaz0Ou4r2ohrRSPidmBhp203RcSK9OUMYFytcrL0sb0hqT/J3DUkjQHezHCcmfUR/Xuvi+1o4PJaO2VJbD8ErgbWlvRfJKt9fD1fbGZWFqpv5Y7RkmZWvJ4aEVMznuffgRXApbX2zXKt6KWS7iFZukjAQRHhO8Gb2VvqGBRdEBET6y9fRwIfBPaJiKi1f5aFJscDrwHXVW6LiKfqDc7MyqmZi3tI2h/4CvDeiHgtyzFZmqK/5e2bugwGNgb+DvxDD+M0sxLpGDxoSFnSNJIVu0dLmgucQjIKOgi4OZ0vNyMijqtWTpam6Ls7nXgH4DPd7G5mfVCj5udGxKQuNp9fbzl1X3kQEfdK2qXe48yspAT9C3blQZY+ti9UvOwH7AA827SIzKyttOvt94ZVPF9B0ud2VXPCMbN21FaJLZ2YOywivtRL8ZhZGyraRfDVlgYfEBErJO3emwGZWXtpt6bo3ST9abMkXQtcAby1rnBE/KrJsZlZO2izex50GAy8SHKPg475bAE4sZkZAgYUrMpWLbGtnY6IzubthNah5iUNZtZ3tFONrT8wlFUTWgcnNjNLiX5dponWqZbYnouIb/VaJGbWlpKbubQ6ilVVS2wFC9XMCinjst+9qVpi26fXojCztiWgf8EyW7UbJi/s7j0zs0qNWt2jUXz7PTPLrWB5zYnNzPIR2e4K1Zuc2MwsnwLeMNmJzcxyK1Zac2Izs5xEGy40aWZWS8HymhObmeUl97GZWbl4VNTMSsk1NjMrnWKlNSc2M8tJ7Xj7PTOzWorWFC1an5+ZtSFlfNQsR7pA0guSZldsW1PSzZIeSf8/qlY5TmxmlpuU7ZHBhcD+nbZ9FbglIjYDbklfV+XEZma5JNM9lOlRS0TcDnReMu1A4KL0+UXAQbXKcR+bmeVWRxfbaEkzK15PjYipNY5ZJyKeS5/PA9apdRInNjPLSfUsNLkgIib29EwREZJq3kzKTVEzy6WRTdFuPC9pPYD0/y/UOsCJzczyyThwkGNGyLXAEenzI4Brah3gxGZmuTUqsUmaBtwJbCFprqRjgDOA90t6BNg3fV2V+9jMLDc16KKqiJjUzVt13TXPic3McvFCk2ZWSgXLa05sZpZfo5qijeLE1mB7brIW40etztI3VnLV/cmcwo3XHMKO40YwcvXV+PXseSxYsrzFUVqlL+y9CbtuOIpFS99g8mX3r/Lev0xYjym7b8jB58/klWUrWhRhsQko2I3gmzcq2tXFrH3Bw/Nf5XdzVp1m89Jry7n54fk8t/j1FkVl1dw8Zz4nXzfnHdvHDB3IjhuM4Hn/3GpQ5v96SzOne1zIOy9mLb15i1/n9ZUrV9m2aNkKXvZf+8J64LnFLH595Tu2H7f7hpz356eImvPc+7jmz2OrW9OaohFxu6SNmlW+WTPttvEoFixZzmMvvtbqUAqviKOiLZ+gK2mypJmSZi5e1PmifrPeN2hAPybtOJaL7p7b6lDaRqPWY2uUlie2iJgaERMjYuKwkWu2Ohwz1hs+iHWHDeLcQ7fl4k9uz5ihAznnkHczashqrQ6tuAqW2TwqatbJEwuXcsj/3vPW64s/uT0nXPGAR0Wr8HSPknvfpqNZf/ggBg/oz6Ttx3Lv3Jd5fcVKdttoTVZfrT8f2GJtFr62nN89VHOBAuslX3v/pmw7djgjBg/g0iO25+d3z+WGOfNbHVZbKVgXW/MSW3ox614kC8vNBU6JiPObdb6i+MOjC7rc/sRLz/RyJJbV6Tc/WvX9T/38r70USfsqWF5r6qhodxezmlmJiOLdpcpNUTPLp5fnqGXhxGZmuRUsrzmxmVkDFCyzObGZWU69ex1oFk5sZpZLEVf3cGIzs/yc2MysbNwUNbPS8XQPMyudguU1JzYzy6m31yTKwInNzHJJRkWLldmc2Mwst2KltQIsNGlmJdCghSYlnSTpQUmzJU2TNLgn4TixmVlujbhLlaSxwInAxIjYBugPfLwn8bgpama5NbCLbQCwuqQ3gCHAsz0pxDU2M8utjpbo6I6bN6WPyR1lRMQzwHeBp4DngJcj4qaexOMam5nlUudCkwsiYmKX5UijgAOBjYFFwBWSPhERl9Qbk2tsZpZP426YvC/weETMj4g3gF8B7+lJSE5sZpZbgwZFnwJ2lTRESRVwH2BOT+JxYjOz/BqQ2SLiLuBK4F7gAZL8NLUn4biPzcxyatxCkxFxCnBK3nKc2MwsFy80aWbl5MRmZmXjhSbNrHQKtriHE5uZ5VewvObEZmY5+U7wZlY2dV5S1Suc2Mwst2KlNSc2M2uAglXYnNjMLD9P9zCz8ilWXnNiM7P8CpbXnNjMLB/Jt98zszIqVl5zYjOz/AqW15zYzCy/grVEndjMLK/GLTTZKE5sZpZLcklVq6NYlRObmeXmxGZmpeOmqJmVi5ctMrOyyXjP0F7lxGZm+RUsszmxmVluRbukyneCN7PcGnAj+LfLkkZKulLSQ5LmSNqt3nhcYzOz/BpbYfsBcENEHCxpIDCk3gKc2Mwst0ZN95A0AtgTOBIgIpYDy+suJyIaElAjSJoPPNnqOJpgNLCg1UFYXcr6M9swIsY0skBJN5B8X1kMBpZVvJ4aEVMrypoATAX+BmwH3AN8PiKW1BVTkRJbWUmaGRETWx2HZeefWWtImgjMAHaPiLsk/QB4JSK+UU85HjwwsyKZC8yNiLvS11cCO9RbiBObmRVGRMwDnpa0RbppH5JmaV08eNA7ptbexQrGP7PW+RxwaToi+hhwVL0FuI/NzErHTVEzKx0nNjMrHSc2MysdJ7YmkrSFpN0krSapf6vjsWz8s2p/HjxoEkkfBU4DnkkfM4ELI+KVlgZm3ZK0eUQ8nD7vHxErWx2T9YxrbE0gaTXgUOCYiNgHuAbYAPg3ScNbGpx1SdIHgVmSfgEQEStdc2tfTmzNMxzYLH1+NfAbYDXgMKlgi1f1cZLWAE4A/hVYLukScHJrZ05sTRARbwD/A3xU0h4R8SYwHZgF/GNLg7N3SC+wPhr4BfAlYHBlcmtlbNYzTmzNcwdwE/BJSXtGxMqI+AWwPsmqBVYgEfFsRLwaEQuAKcDqHclN0g6StmxthFYPX1LVJBGxTNKlQABfS38xXgfWAZ5raXBWVUS8KGkK8B1JDwH9gfe1OCyrgxNbE0XES5J+RnIR7xSSdag+ERHPtzYyqyUiFki6HzgAeH9EzG11TJadp3v0krQTOtL+Nis4SaOAXwJfjIj7Wx2P1ceJzawbkgZHxLLae1rROLGZWel4VNTMSseJzcxKx4nNzErHic3MSseJrY1IWilplqTZkq6QVPcdsivKulDSwenz8yRtXWXfvSS9pwfneELSO+432d32Tvu8Wue5TpX0pXpjtHJyYmsvSyNiQkRsQ3J37OMq35TUownXEfHpiKh2J6C9gLoTm1mrOLG1rzuATdPa1B2SrgX+Jqm/pO9I+ouk+9NLg1Dix5L+Lun3wNodBUm6Lb1RLZL2l3SvpPsk3SJpI5IEelJaW9xD0hhJV6Xn+Iuk3dNj15J0k6QHJZ0H1FzFRNKvJd2THjO503tnpdtvkTQm3fYuSTekx9zhazitK76kqg2lNbMDgBvSTTsA20TE42lyeDkidpI0CPiTpJuA7YEtgK1Jrlf9G3BBp3LHAD8D9kzLWjMiFko6F3g1Ir6b7vcL4KyImC5pPHAjsBVwCjA9Ir4l6Z+BYzJ8nKPTc6wO/EXSVRHxIrAGMDMiTpL0zbTsE0hui3dcRDwiaRfgHGDvHnyNVmJObO1ldUmz0ud3AOeTNBHvjojH0+37Adt29J8BI0jWhdsTmJYuw/OspFu7KH9X4PaOsiJiYTdx7AtsXbGs3HBJQ9NzfDQ99reSXsrwmU6U9JH0+QZprC8CbwKXp9svAX6VnuM9wBUV5x6U4RzWxzixtZelETGhckP6C76kchPwuYi4sdN+/9TAOPoBu3a+3Kje9TMl7UWSJHeLiNck3QYM7mb3SM+7qPN3YNaZ+9jK50bg+HR5ciRtnq4QeztwaNoHtx5dL8MzA9hT0sbpsWum2xcDwyr2u4nkbt2k+3UkmtuBw9JtBwCjasQ6AngpTWpbktQYO/QDOmqdh5E0cV8BHpf0sfQckuS17ewdnNjK5zyS/rN7Jc0GfkpSM78aeCR972Lgzs4HRsR8YDJJs+8+3m4KXgd8pGPwADgRmJgOTvyNt0dn/4MkMT5I0iR9qkasNwADJM0BziBJrB2WADunn2Fv4Fvp9sOBY9L4HgQOzPCdWB/ji+DNrHRcYzOz0nFiM7PScWIzs9JxYjOz0nFiM7PScWIzs9JxYjOz0vl/VowCZjdwa9YAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8YBtNKrf2mKS"
      },
      "source": [
        "### now the same thing but for captions + labels"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RRxjHMLm2mYt",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "outputId": "47936abb-8640-4a3f-89c9-ad2c88c46c0a"
      },
      "source": [
        "predictions_df = ig_post_labeled.loc[:,['post_caption','labels','engagement_binary_class']]\n",
        "predictions_df['post_caption'] = caption_replace['post_caption']\n",
        "\n",
        "word_list = []\n",
        "for label in label_list:\n",
        "    word_list.append(label)\n",
        "    ''' maybe revise to:\n",
        "        if label not in word_list:\n",
        "          word_list.append(label)'''\n",
        "for capword in capword_list:\n",
        "    if capword not in word_list:\n",
        "        word_list.append(capword)\n",
        "\n",
        "for word in word_list:\n",
        "    predictions_df[word] = 0 # initializing the columns with all 0s\n",
        "predictions_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>post_caption</th>\n",
              "      <th>labels</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "      <th>sky</th>\n",
              "      <th>photography</th>\n",
              "      <th>tree</th>\n",
              "      <th>plant</th>\n",
              "      <th>water</th>\n",
              "      <th>phenomenon</th>\n",
              "      <th>landscape</th>\n",
              "      <th>natural</th>\n",
              "      <th>mountain</th>\n",
              "      <th>wildlife</th>\n",
              "      <th>cloud</th>\n",
              "      <th>nature</th>\n",
              "      <th>event</th>\n",
              "      <th>people</th>\n",
              "      <th>vehicle</th>\n",
              "      <th>animal</th>\n",
              "      <th>night</th>\n",
              "      <th>sea</th>\n",
              "      <th>adaptation</th>\n",
              "      <th>geological</th>\n",
              "      <th>monochrome</th>\n",
              "      <th>architecture</th>\n",
              "      <th>rock</th>\n",
              "      <th>black</th>\n",
              "      <th>terrestrial</th>\n",
              "      <th>mammal</th>\n",
              "      <th>recreation</th>\n",
              "      <th>white</th>\n",
              "      <th>and</th>\n",
              "      <th>area</th>\n",
              "      <th>environment</th>\n",
              "      <th>grass</th>\n",
              "      <th>fun</th>\n",
              "      <th>street</th>\n",
              "      <th>bird</th>\n",
              "      <th>landforms</th>\n",
              "      <th>human</th>\n",
              "      <th>...</th>\n",
              "      <th>metropolitan</th>\n",
              "      <th>lion</th>\n",
              "      <th>standing</th>\n",
              "      <th>herd</th>\n",
              "      <th>festival</th>\n",
              "      <th>pasture</th>\n",
              "      <th>portrait</th>\n",
              "      <th>party</th>\n",
              "      <th>world</th>\n",
              "      <th>wing</th>\n",
              "      <th>hairstyle</th>\n",
              "      <th>motor</th>\n",
              "      <th>protest</th>\n",
              "      <th>sculpture</th>\n",
              "      <th>trail</th>\n",
              "      <th>person</th>\n",
              "      <th>photo</th>\n",
              "      <th>flora</th>\n",
              "      <th>color</th>\n",
              "      <th>fauna</th>\n",
              "      <th>story</th>\n",
              "      <th>follow</th>\n",
              "      <th>year</th>\n",
              "      <th>new</th>\n",
              "      <th>see</th>\n",
              "      <th>image</th>\n",
              "      <th>nat</th>\n",
              "      <th>geo</th>\n",
              "      <th>check</th>\n",
              "      <th>day</th>\n",
              "      <th>link</th>\n",
              "      <th>bio</th>\n",
              "      <th>time</th>\n",
              "      <th>life</th>\n",
              "      <th>many</th>\n",
              "      <th>part</th>\n",
              "      <th>species</th>\n",
              "      <th>place</th>\n",
              "      <th>it</th>\n",
              "      <th>technology</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Photo by @erintrieb / On Election Day I set up...</td>\n",
              "      <td>['Hair', 'Face', 'White', 'Black', 'Hairstyle'...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Photo by @tasneemalsultan / A Saudi woman sits...</td>\n",
              "      <td>['Room', 'Textile', 'Tradition', 'Bazaar', 'Fl...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Photo by Keith Ladzinski @ladzinski / A kiss o...</td>\n",
              "      <td>['Mountainous landforms', 'Mountain', 'Sky', '...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Photo by @dina_litovsky / Most people don’t as...</td>\n",
              "      <td>['Motor vehicle', 'Vehicle', 'Mode of transpor...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Photo by @paoloverzone / Fluorescing a ghostly...</td>\n",
              "      <td>['Water', 'Blue', 'Liquid', 'Aqua', 'Transpare...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 249 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                        post_caption  ... technology\n",
              "0  Photo by @erintrieb / On Election Day I set up...  ...          0\n",
              "1  Photo by @tasneemalsultan / A Saudi woman sits...  ...          0\n",
              "2  Photo by Keith Ladzinski @ladzinski / A kiss o...  ...          0\n",
              "3  Photo by @dina_litovsky / Most people don’t as...  ...          0\n",
              "4  Photo by @paoloverzone / Fluorescing a ghostly...  ...          0\n",
              "\n",
              "[5 rows x 249 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 138
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vb_2cCab2bDs",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 695
        },
        "outputId": "eb5fb2c9-9ef6-42c0-b7d9-d9d48362e7af"
      },
      "source": [
        "index = 0\n",
        "for capword in predictions_df['post_caption']: # for each row in the 'labels' column\n",
        "    caption = capword.split(sep=None) # split the label string up into words\n",
        "    for word in caption: # for each of those words\n",
        "        for char in replace_chars:\n",
        "            word = word.replace(char,'') # remove the excess characters\n",
        "        word = word.lower() # make it lowercase\n",
        "        if word in word_list:\n",
        "            predictions_df[word][index] = 1 # set the value of that item to 1, in the corresponding row/column\n",
        "    index = index + 1\n",
        "    \n",
        "index = 0\n",
        "for label in predictions_df['labels']: # for each row in the 'labels' column\n",
        "    label = label.split(sep=None) # split the label string up into words\n",
        "    for item in label: # for each of those words\n",
        "        for char in replace_chars:\n",
        "            item = item.replace(char,'') # remove the excess characters\n",
        "        item = item.lower() # make it lowercase\n",
        "        if item in label_list:\n",
        "            predictions_df[item][index] = 1 # set the value of that item to 1, in the corresponding row/column\n",
        "    index = index + 1\n",
        "\n",
        "predictions_df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:9: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
            "  if __name__ == '__main__':\n",
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:20: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>post_caption</th>\n",
              "      <th>labels</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "      <th>sky</th>\n",
              "      <th>photography</th>\n",
              "      <th>tree</th>\n",
              "      <th>plant</th>\n",
              "      <th>water</th>\n",
              "      <th>phenomenon</th>\n",
              "      <th>landscape</th>\n",
              "      <th>natural</th>\n",
              "      <th>mountain</th>\n",
              "      <th>wildlife</th>\n",
              "      <th>cloud</th>\n",
              "      <th>nature</th>\n",
              "      <th>event</th>\n",
              "      <th>people</th>\n",
              "      <th>vehicle</th>\n",
              "      <th>animal</th>\n",
              "      <th>night</th>\n",
              "      <th>sea</th>\n",
              "      <th>adaptation</th>\n",
              "      <th>geological</th>\n",
              "      <th>monochrome</th>\n",
              "      <th>architecture</th>\n",
              "      <th>rock</th>\n",
              "      <th>black</th>\n",
              "      <th>terrestrial</th>\n",
              "      <th>mammal</th>\n",
              "      <th>recreation</th>\n",
              "      <th>white</th>\n",
              "      <th>and</th>\n",
              "      <th>area</th>\n",
              "      <th>environment</th>\n",
              "      <th>grass</th>\n",
              "      <th>fun</th>\n",
              "      <th>street</th>\n",
              "      <th>bird</th>\n",
              "      <th>landforms</th>\n",
              "      <th>human</th>\n",
              "      <th>...</th>\n",
              "      <th>metropolitan</th>\n",
              "      <th>lion</th>\n",
              "      <th>standing</th>\n",
              "      <th>herd</th>\n",
              "      <th>festival</th>\n",
              "      <th>pasture</th>\n",
              "      <th>portrait</th>\n",
              "      <th>party</th>\n",
              "      <th>world</th>\n",
              "      <th>wing</th>\n",
              "      <th>hairstyle</th>\n",
              "      <th>motor</th>\n",
              "      <th>protest</th>\n",
              "      <th>sculpture</th>\n",
              "      <th>trail</th>\n",
              "      <th>person</th>\n",
              "      <th>photo</th>\n",
              "      <th>flora</th>\n",
              "      <th>color</th>\n",
              "      <th>fauna</th>\n",
              "      <th>story</th>\n",
              "      <th>follow</th>\n",
              "      <th>year</th>\n",
              "      <th>new</th>\n",
              "      <th>see</th>\n",
              "      <th>image</th>\n",
              "      <th>nat</th>\n",
              "      <th>geo</th>\n",
              "      <th>check</th>\n",
              "      <th>day</th>\n",
              "      <th>link</th>\n",
              "      <th>bio</th>\n",
              "      <th>time</th>\n",
              "      <th>life</th>\n",
              "      <th>many</th>\n",
              "      <th>part</th>\n",
              "      <th>species</th>\n",
              "      <th>place</th>\n",
              "      <th>it</th>\n",
              "      <th>technology</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Photo by @erintrieb / On Election Day I set up...</td>\n",
              "      <td>['Hair', 'Face', 'White', 'Black', 'Hairstyle'...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Photo by @tasneemalsultan / A Saudi woman sits...</td>\n",
              "      <td>['Room', 'Textile', 'Tradition', 'Bazaar', 'Fl...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Photo by Keith Ladzinski @ladzinski / A kiss o...</td>\n",
              "      <td>['Mountainous landforms', 'Mountain', 'Sky', '...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Photo by @dina_litovsky / Most people don’t as...</td>\n",
              "      <td>['Motor vehicle', 'Vehicle', 'Mode of transpor...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Photo by @paoloverzone / Fluorescing a ghostly...</td>\n",
              "      <td>['Water', 'Blue', 'Liquid', 'Aqua', 'Transpare...</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 249 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                        post_caption  ... technology\n",
              "0  Photo by @erintrieb / On Election Day I set up...  ...          0\n",
              "1  Photo by @tasneemalsultan / A Saudi woman sits...  ...          0\n",
              "2  Photo by Keith Ladzinski @ladzinski / A kiss o...  ...          0\n",
              "3  Photo by @dina_litovsky / Most people don’t as...  ...          0\n",
              "4  Photo by @paoloverzone / Fluorescing a ghostly...  ...          0\n",
              "\n",
              "[5 rows x 249 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 139
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5-SHyHA_2uLc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d55c5565-9c63-42a5-fe94-13311bc6c83e"
      },
      "source": [
        "# setting up logistic regression\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn import linear_model\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "X = predictions_df.iloc[:, 3:].values\n",
        "y = predictions_df.iloc[:, 2].values\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=0)\n",
        "\n",
        "lrmodel = LogisticRegression()\n",
        "lrmodel.fit(X_train,y_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
              "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
              "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
              "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
              "                   warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 140
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nNhvdFdj2xDL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dcaa4fee-a184-4aee-df4c-5626c6a27c45"
      },
      "source": [
        "# using the model to predict the test data\n",
        "y_pred = lrmodel.predict(X_test)\n",
        "\n",
        "# printing misclassifications\n",
        "print('Misclassified samples: %d' %(y_test != y_pred).sum())\n",
        "\n",
        "# printing accuracy score\n",
        "print('Accuracy: %.2f' % accuracy_score(y_test,y_pred))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Misclassified samples: 30\n",
            "Accuracy: 0.70\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331
        },
        "id": "nxbPMPo1C9DZ",
        "outputId": "b74e64c3-d043-4b67-dde3-cc9b8ef9bee5"
      },
      "source": [
        "plot_confusion_matrix(y_true = y_test, \n",
        "                      y_pred = y_pred, classes = np.unique(y_test),\n",
        "                      title = 'Labels & Captions: Confusion matrix')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Confusion matrix, without normalization\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f731c2eed68>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 142
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATYAAAEYCAYAAADWGtrvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAfaElEQVR4nO3deZwcVbn/8c93ZrKTDRIgIBBkNaIEDDvyQ9bAdYOfKCAoAgJXAbmil+UqBFwIouIVQQzLZRUQMKxh3/HKEjDEEFYhYU0gIQGyZ5Ln/lE10Aw9PT3p7unqyvedV73SXV196qnq6afPqVN1ShGBmVmeNNU7ADOzanNiM7PccWIzs9xxYjOz3HFiM7PccWIzs9zJRWKTdL+kw7v7vVkn6XxJP613HLUiaRNJkyS9L+nYCsrJzX6SdLKkC+sdR71lKrFJmiZpt3rHUQ5JvSRdJmmOpLck/XeZ7ztQ0kRJ8yS9Kek2STtWIZ5DJD1cOC8ijoqIn1VadiUk9ZQ0RtILkuann/HFkoZXofj/BO6LiP4R8fsVLSQL+6kzknaW9Fpny0XELyMilz/UXZGpxNZgDgG2BD4JrA/c0NkbJP0Q+B3wS2ANYF3gPOArNYuy/q4DvgwcCAwENgeeAHatQtnrAU9XoZxckNRS7xgyIyIyMwHTgN2KzB8M3AK8DcxJH3+i4PX7gTOAx4D3gBuBVQte3xb4X2Au8BSwc7v3Hp4+3hB4AHgXmAVcUyLWQ4G/dWHbBgLzgP1KLLM18Pc0zjeBPwA9C14P4FjgpTS+s0h+nD4FLAKWpeuYmy5/CfDzgvd/F3gReAe4CVirXdlHAS+k6z8XUFf3S7vt2Q1YCKxTYpm10ljeSWP7bsFrY4C/AJcB75MksVHpa/em27so3eaNCz/LdJlDgIfTxwLOBt5K/0b+CWxWzf1UZNvGANcCV6Tx/zON86Q0jleBPQqW/w7wTLrsS8CR6fx+6X5cnm7rvHS/jSH54bgi3abD03lXpO/7BvAyMCB9vhcwAxha7+96rae6B9DuD2EaxRPbasD/B/oC/dM/lhsKXr8feB3YLP0juL7gw10bmA3sTZIEdk+fDy14b1tiuwr4r3S53sCOJWLdPP1inV7mto0GWoGWEst8jiQJtwDD0z/y4wpeD+A+YFWS2t7zBbEfQvolLlj+gy8ssAtJUtoS6AWcAzzYruxbgEFp2W8DozvbL+l7Tuxge8YCD3SyXx4kqbX2Bkam690lfW0MSeLaG2gm+fF6pN3nfniJ5x/sE2BPkpriIJIk9ylgWDX3U5Fta4t/z/QzvYwk0fwX0IMkgb5csPy/ARuk8f0/YAGwZfrazsBrRcpfCnw1/Wz6UJDY0mWuTLdvNeAN4Iv1/p53x9QQTdGImB0R10fEgoh4H/gFyQdf6PKImBIR84GfAl+X1AwcBEyIiAkRsTwi7gImknxZ2ltK0rxZKyIWRcTDRZZB0qrAzSR/iHtKGlPw2muSPlPkbasBsyKitcR2PhERj0REa0RMA/5UZDvPjIh3IuIVkmbtAR2V1843gYsj4smIWExSa9iu3bGusRExNy37PpJEAyX2S0R8MSLGdrDO1UhqnkVJWgfYATghLXcScCHwrYLFHk4/u2XA5SQ/KCtiKcmP4qYkNaxnIqJYbJXsp2Ieiog70s/9WmBo+v6lwNXAcEmDACLi1oj4VyQeAO4EPt/Jdv09Im5I/7YXFnn9+yTJ+n7g5oi4pZPycqEhEpukvpL+JGm6pPdIfuUHpYmrzasFj6eT/CIOIflC7idpbtsE7AgMK7Kq/yT5tXxM0tOSDu0gpP2AZyLidpIEuV96gHw4yS/zlCLvmQ0MKXUcRNLGkm6RNCPdzl+m21Co/Xau1VF57ayVLg9ARMxLY1q7YJkZBY8XAKukj8vdL+3Npvh+LozpnfTHqs30TmLqvSLHkiLiXpKm/bnAW5LGSRrQQUwrup+KmVnweCHJj9uygue0vV/SXpIekfRO+ne6Nx///Nt7tdSLETGXJKFuBvymk7JyoyESG3A8sAmwTUQMAHZK56tgmXUKHq9L8gs9i+SDvzwiBhVM/YrVMiJiRkR8NyLWAo4EzpO0YZF4WkgSJxExm6R5+23gDuDXkbYB2vk7sJik2dCRPwLPAhul23lyu20stp1vtIVfolzS5dZreyKpH0mN6vVO3teV/dLe3cDWkj5RIqZVJfUvmLduOTF1YD7J4Yo2axa+GBG/j4jPASNIjnX9uIOYVmg/VUJSL5JDKL8G1oiIQcAEPvz8O/p8S37ukkaSHA++CljhnuNGk8XE1kNS74KphaQJsRCYmzYDTy3yvoMkjZDUFzgduC79ZbwC+JKkPSU1p2XuXOzLJmm/gvlzSP5olhdZ1wRgK0lHSupBkkT/l+TLsqDYRkXEu8ApwLmSvprWQnukv9K/ShfrT3IQeJ6kTYF/L1LUjyUNTptxPwCuSefPBD4hqWex9ZP8YX9H0sj0S/RL4NG0yVtSF/ZL+22+G7gLGC/pc5JaJPWXdJSkQyPiVZL9dkb6uXwWOIzkM1sRk4B90327YVpW2zZsJWmb9POaT3Lsq9g2rPB+qlBPkmN6bwOtkvYC9ih4fSawmqSB5RYoqTfJvjyZpGNibUnfq17I2ZXFxDaBJIm1TWNIjiX1IamBPQLcXuR9l5McJJ1BciD6WID0y/MVkg/3bZIa3I8pvu1bAY9KmkfSG/aDiHip/UIR8TJJD9O3SJopT5H84X0BOFPS6GIbFhG/AX4I/KQglqP58FSRH5GcFvE+cAEfJq1CN5IcBJ8E3ApclM6/l6TXcIakWUXWfTfJscfrSY57bQDsXyzOIjrcL+l5eCeXeO/XSD7Ta0h6VacAo0hqc5AcIxxOUlMaD5yaxroizgaWkHwWl5IcOG8zgGSfziFpas4m6VX+iAr30wpLm+PHkvQCzyH5O7ip4PVnSZLuS+khlXIOQZwBvBoRf0yPFx4E/FzSRlXfgIxR8VaTZZGkIGmmvljvWMyyLIs1NjOzijixmVlmpMdaH5P0VNoDf1o6/xJJLyu5NnhS2inSIV+C0UAion0PqVneLCY5QXte2tHzsKTb0td+HBHXlVOIE5uZZUZ6qtS89GmPdOpyR0CmOg/U0ifUs3/nC1pmbPGpdesdgnXB9OnTmDVrVlVr/s0D1otoLXbRw8fFwrefJjnVps24iBhXuEx64v0TJNconxsRJ0i6BNiOpEZ3D8llfIs7Wk+mamzq2Z9em3y93mFYF/zt0T/UOwTrgh22GVX1MqN1Ydnf20WTzl0UESWDSM8/HZleajZe0mYkl7bNIDnfbxxwAsn5qkW588DMKiRQU3lTF6SXg91HMsjAm+k1tIuB/yEZCadDTmxmVhkBTc3lTZ0VJQ1tGxRAUh+SyxWflTQsnSeSyxKLXY/9gUw1Rc2sQalqh+2GAZemx9magL9ExC2S7pU0lCSNTiIZE69DTmxmViF1uZnZkYiYDGxRZP4uXSnHic3MKle9GltVOLGZWWVE1Wps1eLEZmYVkmtsZpZDZfR4dicnNjOrUPU6D6rFic3MKiPcFDWzHHKNzczyxU1RM8ujJjdFzSxP2q4VzRAnNjOrkJuiZpZH7hU1s9xxjc3MckW+pMrM8sidB2aWL+48MLM8clPUzHLF47GZWf64KWpmeeSmqJnljntFzSxX5KaomeWRm6JmljdyYjOzPElGBndiM7M8UTpliBObmVVINDW588DMcsZNUTPLHSc2M8sXH2Mzs7wRco3NzPLHnQdmljuusZlZvvgYm5nlkWtsZpYr7jwws1xyYjOzfBGoyYnNzHImazW2bJ18YmYNSVJZUxnl9Jb0mKSnJD0t6bR0/vqSHpX0oqRrJPUsVY4Tm5lVpK3zoBqJDVgM7BIRmwMjgdGStgXOBM6OiA2BOcBhpQpxYjOzyqnMqRORmJc+7ZFOAewCXJfOvxT4aqlynNjMrDLqUlN0iKSJBdMRHytOapY0CXgLuAv4FzA3IlrTRV4D1i4VkjsPzKxiXbhWdFZEjCq1QEQsA0ZKGgSMBzbtcjxdfYOZ2cdUqSlaKCLmAvcB2wGDJLVVxD4BvF7qva6xVVGvni3cfdFx9OzZQktzM+Pv/gc/P38C4047iM9/bkPenbcIgCNOuZzJz5f8XKwbHXn4odw24RaGrr46T0yaAsBJJ/yYCbfeTM8ePVl/gw0Yd+H/MGjQoDpHml3VOt1D0lBgaUTMldQH2J2k4+A+4GvA1cC3gRtLlVPTGpuk0ZKeS7toT6zlurJg8ZJWRh/xe7b5xli22f8M9th+BFt/ZjgAJ//uBrbdfyzb7j/WSS1jDv72Idx4y+0fmbfrbrvzxKQpPP6PyWy00cacdeYZdYou+8o9vlZm8hsG3CdpMvA4cFdE3AKcAPxQ0ovAasBFpQqpWY1NUjNwLknGfQ14XNJNETG1VuvMgvkLlwDQo6WZlpZmIqLOEVlndvz8TkyfNu0j83bbfY8PHm+9zbaMv/46rGPVqrFFxGRgiyLzXwK2LrecWtbYtgZejIiXImIJSRXyKzVcXyY0NYlHrj6RV+4Zy72PPMvjU6YDMOb7X+Kxa07iV8fvS88ePgLQSC675GL2HL1XvcPItCrW2KqiloltbeDVgudFu2glHdHW9RutC2sYTvdYvjzYdv+xbLjnTxi12XqM2GAYp5xzE5vv8zN2POgsBg/sx/Hf2a3eYVqZzjzjFzS3tLD/gd+sdyiZpiaVNXWXuveKRsS4iBgVEaPU0qfe4VTNu/MW8sDE59lj+xHMmPUeAEuWtnLZjY8w6tPD6xucleXySy9hwq23cMllV2buWshM6dp5bN2ilontdWCdgueddtE2uiGDV2HgKkly7t2rB7tusynPTZvJmkMGfLDMl7/wWab+6416hWhluvOO2/ntb37FdeNvom/fvvUOJ9MESOVN3aWWB3seBzaStD5JQtsfOLCG66u7NYcM4ILTD6a5qYmmJnH9XU9y20NTuO1PxzBkcH8kmPzcaxzzi6vrHaoV+NZBB/DQA/cza9YsNhj+CX56ymmc9aszWLx4MV8cvTuQdCCcc975dY40q1aigSYjolXS0cAdQDNwcUQ8Xav1ZcGUF95guwPO/Nj8vY48pw7RWLkuu+Kqj8075NCS11hbOxnLa7U9QTciJgATarkOM6szJWcDZInPOzCziggnNjPLoZWqKWpmK4eVpvPAzFYS3XwqRzmc2MysIsl5bNnKbE5sZlYhufPAzPLHNTYzyxcfYzOzvPExNjPLpYzlNSc2M6uca2xmli++VtTM8qZtPLYscWIzswqtROOxmdnKI2N5zYnNzCrnGpuZ5YrceWBmeeQam5nlTsbymhObmVXONTYzyxdfBG9meSOfx2ZmedTsXlEzy5uMVdic2MysMpI7D8wshzLWEu04sUk6B4iOXo+IY2sSkZk1nEaqsU3stijMrKFlLK91nNgi4tLC55L6RsSC2odkZo1EQHPGMltTZwtI2k7SVODZ9Pnmks6reWRm1hiUnMdWztRdOk1swO+APYHZABHxFLBTLYMys8YilTd1l3ISGxHxartZy2oQi5k1IAFNUllTp2VJ60i6T9JUSU9L+kE6f4yk1yVNSqe9S5VTzuker0raHghJPYAfAM+U8T4zW0lUsTbWChwfEU9K6g88Iemu9LWzI+LX5RRSTmI7CvhvYG3gDeAO4PsrELCZ5VA1B5qMiDeBN9PH70t6hiT3dEmniS0iZgHf7HKEZrbSKKeZmRoiqfBUsnERMa7YgpKGA1sAjwI7AEdL+hbJqWjHR8ScDuPpLApJn5R0s6S3Jb0l6UZJnyx3K8ws/1TmBMyKiFEFU0dJbRXgeuC4iHgP+COwATCSpEb3m1LxlNN58GfgL8AwYC3gWuCqMt5nZiuJap7ukR7Lvx64MiL+ChARMyNiWUQsBy4Ati5VRjmJrW9EXB4Rrel0BdC7rAjNLPeSXtHypk7LSrLfRcAzEfHbgvnDChbbB5hSqpxS14qumj68TdKJwNUk145+A5jQeYhmtlKo7sm3OwAHA/+UNCmddzJwgKSRJDloGnBkqUJKdR48kRbSFnFhQQGc1PWYzSyPqtgr+jAf5pxCXapMlbpWdP2uBmVmK5+2pmiWlDUem6TNgBEUHFuLiMtqFZSZNZZGGrYIAEmnAjuTJLYJwF7Aw4ATm5kBxduO9VROr+jXgF2BGRHxHWBzYGBNozKzhiFV71rRaimnKbowIpZLapU0AHgLWKfGcZlZA8lYS7SsxDZR0iCSk+KeAOYBf69pVGbWUKrVK1ot5Vwr+r304fmSbgcGRMTk2oZlZo1CdG8zsxylTtDdstRrEfFkbUIys4bSzYNIlqNUja3URaYB7FLlWNj4k2tz4TU/q3axVkODtzq63iFYFyx+7pWalNswp3tExBe6MxAza1xlDcXdjXzDZDOriGigGpuZWblaMlZlc2Izs4okd6DKVo2tnBF0JekgSaekz9eVVHKQNzNbuVRrPLaqxVPGMucB2wEHpM/fB86tWURm1nCydl/Rcpqi20TElpL+ARARcyT1rHFcZtYg2u4rmiXlJLalkppJzl1D0lBgeU2jMrOG0pytvFZWYvs9MB5YXdIvSEb7+ElNozKzhqFuHrmjHOVcK3qlpCdIhi4S8NWI8J3gzewDGctrZQ00uS6wALi5cF5E1ObaDDNrOBkb3KOspuitfHhTl97A+sBzwKdrGJeZNYiG7DyIiM8UPk9H/fheB4ub2UooY3mt61ceRMSTkrapRTBm1oAEzRnLbOUcY/thwdMmYEvgjZpFZGYNpVFvv9e/4HEryTG362sTjpk1ooZKbOmJuf0j4kfdFI+ZNaCsXQRfamjwloholbRDdwZkZo2l0Zqij5EcT5sk6SbgWmB+24sR8dcax2ZmjaDB7nnQpjcwm+QeB23nswXgxGZmCGjJWJWtVGJbPe0RncKHCa1N1DQqM2sojVRjawZW4aMJrY0Tm5mlRFPRNFE/pRLbmxFxerdFYmYNKbmZS72j+KhSiS1joZpZJnXzsN/lKJXYdu22KMysYQlozlhmK3XD5He6MxAza1wNN7qHmVlnMpbXnNjMrDKivNvddScnNjOrTCPeMNnMrDMqc+q0HGkdSfdJmirpaUk/SOevKukuSS+k/w8uVY4Tm5lVRCQDTZYzlaEVOD4iRgDbAt+XNAI4EbgnIjYC7kmfd8iJzcwqVq07wUfEmxHxZPr4feAZYG3gK8Cl6WKXAl8tVY6PsZlZhdSVY2xDJE0seD4uIsYVLVUaDmwBPAqsERFvpi/NANYotRInNjOrSBd7RWdFxKhOy5RWIRmp+7iIeK8wcUZESCp5vbqbomZWMUllTWWW1YMkqV1ZMO7jTEnD0teHAW+VKsOJzcwqVsVeUQEXAc9ExG8LXroJ+Hb6+NvAjaXKcVPUzCqi6t5+bwfgYOCfkial804GxgJ/kXQYMB34eqlCnNjMrGLVOkE3Ih6m48pd2QNzOLGZWcWydd2BE5uZVUHGrqhyYjOzyiSne2QrszmxmVnFXGMzs5yRB5o0s3xxU9TM8qdB7wRvZlaSE5uZ5Y7cFDWzPGkbaDJLnNjMrGIZy2tObGZWOTdFc26jNfoyuF8Pli4L/jH9PQD69Wpmg9X70iQI4F9vLWDeomX1DdQA6NWzhbsvOo6ePVtoaW5m/N3/4OfnT2DcaQfx+c9tyLvzFgFwxCmXM/n51+scbTYJyNiN4GuX2CRdDHwReCsiNqvVerJm5ntLeGPuYjZes98H84YP6cOrsxcyZ0Erg/u1sP6QPvzztXl1jNLaLF7Syugjfs/8hUtoaWni3ot/yJ1/mwrAyb+7gfF3T+qkBEvqa9nKbLUcaPISYHQNy8+k9xa20rrs46MWN6c/ac1NYnFryVGNrZvNX7gEgB4tzbS0NBPhz6dLyryRS3ceh6tZYouIB4F3alV+I3np7QUMH9qXrdYfyPpD+zJ91sJ6h2QFmprEI1efyCv3jOXeR57l8SnTARjz/S/x2DUn8avj96VnDx+16UiVb79XFXUfGlzSEZImSpo4d86seodTE8MG9uLltxfw+Mvv8vJbC9hojb71DskKLF8ebLv/WDbc8yeM2mw9RmwwjFPOuYnN9/kZOx50FoMH9uP47+xW7zAzrVpDg1dL3RNbRIyLiFERMWrQ4CH1DqcmVh/Qi9nzlgIwa95SVuntX/8senfeQh6Y+Dx7bD+CGbOSjp8lS1u57MZHGPXp4fUNLusyltnqnthWBktalzOwT5LMBvZpYdFS94hmxZDBqzBwlT4A9O7Vg1232ZTnps1kzSEDPljmy1/4LFP/9Ua9QmwIKvNfd3HVoco2WbMfA/u20NIstlp/IK/MXsiLM+fzydX7IsHy5fDCzAX1DtNSaw4ZwAWnH0xzUxNNTeL6u57ktoemcNufjmHI4P5IMPm51zjmF1fXO9RMW2lO0JV0FbAzyZ2fXwNOjYiLarW+rHhuxvyi8ye98n43R2LlmPLCG2x3wJkfm7/XkefUIZrGlbG8VrvEFhEH1KpsM8sOUb27VFWLm6JmVhmPx2ZmeZSxvObEZmZVkLHM5sRmZhXK3rWiTmxmVpGVanQPM1uJOLGZWd64KWpmuePTPcwsdzKW15zYzKxC3T0mURmc2MysIkmvaLYymxObmVUsW2nNic3MqiFjmc2Jzcwq5tM9zCx3MnaIzYnNzCqXsbzmxGZmlcniQJO+mYuZVaaKN0yWdLGktyRNKZg3RtLrkial096dlePEZmYVq+Ld9y4BRheZf3ZEjEynCZ0V4sRmZpWrUmaLiAeBdyoNx4nNzCpU7l1FBcld6yYWTEeUuZKjJU1Om6qDO1vYnQdmVpEuDjQ5KyJGdXEVfwR+BkT6/2+AQ0u9wTU2M6tcFQ+ytRcRMyNiWUQsBy4Atu7sPU5sZlaxLjRFu162NKzg6T7AlI6WbeOmqJlVrFqnsUm6CtiZ5Fjca8CpwM6SRpI0RacBR3ZWjhObmVWsWqfnRsQBRWZf1NVynNjMrDK+E7yZ5U0WL6lyYjOzimUrrTmxmVkVZKzC5sRmZpXzQJNmlj/ZymtObGZWuYzlNSc2M6uM5NvvmVkeZSuvObGZWeUyltec2MyschlriTqxmVmlVnzkjlpxYjOziiSXVNU7io9yYjOzijmxmVnuuClqZvniYYvMLG8quJ1BzTixmVnlMpbZnNjMrGK+pMrMcidbac2JzcyqIWOZzYnNzCqWtdM9FBH1juEDkt4Gptc7jhoYAsyqdxDWJXn9zNaLiKHVLFDS7ST7qxyzImJ0NddfTKYSW15JmhgRo+odh5XPn1lja6p3AGZm1ebEZma548TWPcbVOwDrMn9mDczH2Mwsd1xjM7PccWIzs9xxYjOz3HFiqyFJm0jaTlIPSc31jsfK48+q8bnzoEYk7Qv8Eng9nSYCl0TEe3UNzDokaeOIeD593BwRy+odk60Y19hqQFIP4BvAYRGxK3AjsA5wgqQBdQ3OipL0RWCSpD8DRMQy19walxNb7QwANkofjwduAXoAB0oZG7xqJSepH3A0cBywRNIV4OTWyJzYaiAilgK/BfaV9PmIWA48DEwCdqxrcPYxETEfOBT4M/AjoHdhcqtnbLZinNhq5yHgTuBgSTtFxLKI+DOwFrB5fUOz9iLijYiYFxGzgCOBPm3JTdKWkjatb4TWFR6PrUYiYpGkK4EATkq/GIuBNYA36xqclRQRsyUdCZwl6VmgGfhCncOyLnBiq6GImCPpAmAqSS1gEXBQRMysb2TWmYiYJWkysBewe0S8Vu+YrHw+3aObpAehIz3eZhknaTDwF+D4iJhc73isa5zYzDogqXdELKp3HNZ1TmxmljvuFTWz3HFiM7PccWIzs9xxYjOz3HFiayCSlkmaJGmKpGsl9a2grEskfS19fKGkESWW3VnS9iuwjmmSPna/yY7mt1tmXhfXNUbSj7oao+WTE1tjWRgRIyNiM2AJcFThi5JW6ITriDg8IqaWWGRnoMuJzaxenNga10PAhmlt6iFJNwFTJTVLOkvS45Imp5cGocQfJD0n6W5g9baCJN0vaVT6eLSkJyU9JekeScNJEuh/pLXFz0saKun6dB2PS9ohfe9qku6U9LSkC4FORzGRdIOkJ9L3HNHutbPT+fdIGprO20DS7el7HvI1nFaML6lqQGnNbC/g9nTWlsBmEfFymhzejYitJPUC/ibpTmALYBNgBMn1qlOBi9uVOxS4ANgpLWvViHhH0vnAvIj4dbrcn4GzI+JhSesCdwCfAk4FHo6I0yX9G3BYGZtzaLqOPsDjkq6PiNlAP2BiRPyHpFPSso8muS3eURHxgqRtgPOAXVZgN1qOObE1lj6SJqWPHwIuImkiPhYRL6fz9wA+23b8DBhIMi7cTsBV6TA8b0i6t0j52wIPtpUVEe90EMduwIiCYeUGSFolXce+6XtvlTSnjG06VtI+6eN10lhnA8uBa9L5VwB/TdexPXBtwbp7lbEOW8k4sTWWhRExsnBG+gWfXzgLOCYi7mi33N5VjKMJ2Lb95UZdHT9T0s4kSXK7iFgg6X6gdweLR7reue33gVl7PsaWP3cA/54OT46kjdMRYh8EvpEegxtG8WF4HgF2krR++t5V0/nvA/0LlrsTOKbtiaS2RPMgcGA6by9gcCexDgTmpEltU5IaY5smoK3WeSBJE/c94GVJ+6XrkCSPbWcf48SWPxeSHD97UtIU4E8kNfPxwAvpa5cBf2//xoh4GziCpNn3FB82BW8G9mnrPACOBUalnRNT+bB39jSSxPg0SZP0lU5ivR1okfQMMJYksbaZD2ydbsMuwOnp/G8Ch6XxPQ18pYx9YisZXwRvZrnjGpuZ5Y4Tm5nljhObmeWOE5uZ5Y4Tm5nljhObmeWOE5uZ5c7/AYH2umT08pVKAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ivk6Kgy7Epe6"
      },
      "source": [
        "## Task C: Topic Modeling (LDA)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xd4szTfPEtIz"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.decomposition import LatentDirichletAllocation as LDA\n",
        "import re\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 513
        },
        "id": "k3GDaBn6oYwN",
        "outputId": "4ce32e88-ee77-4751-c7f6-6800e7fc02f4"
      },
      "source": [
        "ig_post_labeled.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>post_url</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>like_num</th>\n",
              "      <th>post_caption</th>\n",
              "      <th>image_url</th>\n",
              "      <th>labels</th>\n",
              "      <th>like_percent</th>\n",
              "      <th>comment_percent</th>\n",
              "      <th>engagement_score</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "      <th>labels_processed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://www.instagram.com/p/CHYOgoNLjwD/</td>\n",
              "      <td>157</td>\n",
              "      <td>24098</td>\n",
              "      <td>Photo by @erintrieb / On Election Day I set up...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Hair', 'Face', 'White', 'Black', 'Hairstyle'...</td>\n",
              "      <td>0.014033</td>\n",
              "      <td>0.003340</td>\n",
              "      <td>0.007617</td>\n",
              "      <td>0</td>\n",
              "      <td>hair face white black hairstyle shoulder head ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>https://www.instagram.com/p/CHXl8P8gT0f/</td>\n",
              "      <td>455</td>\n",
              "      <td>83621</td>\n",
              "      <td>Photo by @tasneemalsultan / A Saudi woman sits...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Room', 'Textile', 'Tradition', 'Bazaar', 'Fl...</td>\n",
              "      <td>0.048695</td>\n",
              "      <td>0.009678</td>\n",
              "      <td>0.025285</td>\n",
              "      <td>0</td>\n",
              "      <td>room textile tradition bazaar flooring interio...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>https://www.instagram.com/p/CHXXmNpAhHV/</td>\n",
              "      <td>431</td>\n",
              "      <td>184808</td>\n",
              "      <td>Photo by Keith Ladzinski @ladzinski / A kiss o...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Mountainous landforms', 'Mountain', 'Sky', '...</td>\n",
              "      <td>0.107620</td>\n",
              "      <td>0.009168</td>\n",
              "      <td>0.048549</td>\n",
              "      <td>0</td>\n",
              "      <td>mountainous landforms mountain sky mountain ra...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://www.instagram.com/p/CHXKiRnAshz/</td>\n",
              "      <td>511</td>\n",
              "      <td>106213</td>\n",
              "      <td>Photo by @dina_litovsky / Most people don’t as...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Motor vehicle', 'Vehicle', 'Mode of transpor...</td>\n",
              "      <td>0.061851</td>\n",
              "      <td>0.010870</td>\n",
              "      <td>0.031262</td>\n",
              "      <td>0</td>\n",
              "      <td>motor vehicle vehicle mode of transport transp...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>https://www.instagram.com/p/CHWudbnoMhb/</td>\n",
              "      <td>554</td>\n",
              "      <td>162146</td>\n",
              "      <td>Photo by @paoloverzone / Fluorescing a ghostly...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Water', 'Blue', 'Liquid', 'Aqua', 'Transpare...</td>\n",
              "      <td>0.094423</td>\n",
              "      <td>0.011784</td>\n",
              "      <td>0.044840</td>\n",
              "      <td>0</td>\n",
              "      <td>water blue liquid aqua transparent material tu...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                   post_url  ...                                   labels_processed\n",
              "0  https://www.instagram.com/p/CHYOgoNLjwD/  ...  hair face white black hairstyle shoulder head ...\n",
              "1  https://www.instagram.com/p/CHXl8P8gT0f/  ...  room textile tradition bazaar flooring interio...\n",
              "2  https://www.instagram.com/p/CHXXmNpAhHV/  ...  mountainous landforms mountain sky mountain ra...\n",
              "3  https://www.instagram.com/p/CHXKiRnAshz/  ...  motor vehicle vehicle mode of transport transp...\n",
              "4  https://www.instagram.com/p/CHWudbnoMhb/  ...  water blue liquid aqua transparent material tu...\n",
              "\n",
              "[5 rows x 11 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v2pn0BsfoJC-",
        "outputId": "f5aaa7eb-8360-4152-a969-ed33ca9b3c49"
      },
      "source": [
        "# Remove punctuation\n",
        "\n",
        "ig_post_labeled['labels_processed'] = ig_post_labeled['labels'].map(lambda x: x.replace(\"[\", \"\").replace(\"]\", \"\").replace(\",\", \"\").replace(\"'\", \"\").lower())\n",
        "ig_post_labeled['labels_processed'].head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    hair face white black hairstyle shoulder head ...\n",
              "1    room textile tradition bazaar flooring interio...\n",
              "2    mountainous landforms mountain sky mountain ra...\n",
              "3    motor vehicle vehicle mode of transport transp...\n",
              "4    water blue liquid aqua transparent material tu...\n",
              "Name: labels_processed, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1bcTNV9Qoozw",
        "outputId": "7ac8dc82-fdb5-4bc5-b508-a36805407406"
      },
      "source": [
        "num_topics = 4\n",
        "\n",
        "image_labels = ig_post_labeled['labels_processed']\n",
        "count_vectorizer = CountVectorizer(stop_words='english')\n",
        "lda_data = count_vectorizer.fit_transform(image_labels)\n",
        "# Create and fit the LDA model\n",
        "lda = LDA(n_components=num_topics, n_jobs=-1)\n",
        "lda.fit(lda_data)\n",
        "#lda.components_[0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LatentDirichletAllocation(batch_size=128, doc_topic_prior=None,\n",
              "                          evaluate_every=-1, learning_decay=0.7,\n",
              "                          learning_method='batch', learning_offset=10.0,\n",
              "                          max_doc_update_iter=100, max_iter=10,\n",
              "                          mean_change_tol=0.001, n_components=4, n_jobs=-1,\n",
              "                          perp_tol=0.1, random_state=None,\n",
              "                          topic_word_prior=None, total_samples=1000000.0,\n",
              "                          verbose=0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JtoP7-Q-xOd9"
      },
      "source": [
        "#lda.components_[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "94qV6QBNyBt1"
      },
      "source": [
        "#count_vectorizer.get_feature_names()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lS9qDiayqEet",
        "outputId": "a68e20ed-44e0-44d4-d43e-62fbc01ae464"
      },
      "source": [
        "def print_topics(model, count_vectorizer, n_top_words):\n",
        "    words = count_vectorizer.get_feature_names()\n",
        "    for topic_idx, topic in enumerate(model.components_):\n",
        "        print(\"\\nTopic #%d:\" % topic_idx)\n",
        "        print(\" \".join([words[i]\n",
        "                        for i in topic.argsort()[:-n_top_words - 1:-1]]))\n",
        "        \n",
        "print(\"Topics found via LDA:\")\n",
        "print_topics(lda, count_vectorizer, 10)\n",
        "#'Mountain','Citylife','Nature','Wildlife'"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Topics found via LDA:\n",
            "\n",
            "Topic #0:\n",
            "mountain mountainous landforms sky rock range landscape dog military wilderness\n",
            "\n",
            "Topic #1:\n",
            "photography vehicle white monochrome black people event street fun art\n",
            "\n",
            "Topic #2:\n",
            "sky phenomenon plant tree natural water landscape cloud nature sea\n",
            "\n",
            "Topic #3:\n",
            "wildlife animal terrestrial mammal vertebrate bear snout ice grass bird\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "MZMKNnbzzHSB",
        "outputId": "6085db88-ca0a-441f-ca17-ca4011c41df0"
      },
      "source": [
        "#Output1 - A file showing which words load on which topics, \n",
        "words = count_vectorizer.get_feature_names()\n",
        "topic0 = lda.components_[0]\n",
        "topic1 = lda.components_[1]\n",
        "topic2 = lda.components_[2]\n",
        "topic3 = lda.components_[3]\n",
        "d = {'words': words,\n",
        "     'Mountain': topic0,\n",
        "     'Citylife': topic1,\n",
        "     'Nature': topic2,\n",
        "     'Wildlife': topic3}\n",
        "output1 = pd.DataFrame(d)\n",
        "output1.head(10)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>words</th>\n",
              "      <th>Mountain</th>\n",
              "      <th>Citylife</th>\n",
              "      <th>Nature</th>\n",
              "      <th>Wildlife</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>abaya</td>\n",
              "      <td>2.246536</td>\n",
              "      <td>0.253391</td>\n",
              "      <td>0.250027</td>\n",
              "      <td>0.250045</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>abdomen</td>\n",
              "      <td>1.249827</td>\n",
              "      <td>0.250051</td>\n",
              "      <td>0.250049</td>\n",
              "      <td>0.250073</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>abseiling</td>\n",
              "      <td>0.250090</td>\n",
              "      <td>0.273834</td>\n",
              "      <td>1.226006</td>\n",
              "      <td>0.250070</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>academic</td>\n",
              "      <td>1.234543</td>\n",
              "      <td>0.265376</td>\n",
              "      <td>0.250031</td>\n",
              "      <td>0.250050</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>accessory</td>\n",
              "      <td>0.311647</td>\n",
              "      <td>4.255936</td>\n",
              "      <td>0.250048</td>\n",
              "      <td>2.182369</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>accipitridae</td>\n",
              "      <td>0.250031</td>\n",
              "      <td>0.250016</td>\n",
              "      <td>0.250015</td>\n",
              "      <td>1.249938</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>accipitriformes</td>\n",
              "      <td>0.255403</td>\n",
              "      <td>0.250018</td>\n",
              "      <td>0.250017</td>\n",
              "      <td>3.244562</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>acrobatic</td>\n",
              "      <td>0.351668</td>\n",
              "      <td>1.148223</td>\n",
              "      <td>0.250050</td>\n",
              "      <td>0.250059</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>adaptation</td>\n",
              "      <td>2.185888</td>\n",
              "      <td>15.169724</td>\n",
              "      <td>10.398897</td>\n",
              "      <td>12.245491</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>adventure</td>\n",
              "      <td>1.030553</td>\n",
              "      <td>2.635481</td>\n",
              "      <td>4.266906</td>\n",
              "      <td>1.067059</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             words  Mountain   Citylife     Nature   Wildlife\n",
              "0            abaya  2.246536   0.253391   0.250027   0.250045\n",
              "1          abdomen  1.249827   0.250051   0.250049   0.250073\n",
              "2        abseiling  0.250090   0.273834   1.226006   0.250070\n",
              "3         academic  1.234543   0.265376   0.250031   0.250050\n",
              "4        accessory  0.311647   4.255936   0.250048   2.182369\n",
              "5     accipitridae  0.250031   0.250016   0.250015   1.249938\n",
              "6  accipitriformes  0.255403   0.250018   0.250017   3.244562\n",
              "7        acrobatic  0.351668   1.148223   0.250050   0.250059\n",
              "8       adaptation  2.185888  15.169724  10.398897  12.245491\n",
              "9        adventure  1.030553   2.635481   4.266906   1.067059"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CFISpHCD0iI-"
      },
      "source": [
        "output1.to_csv('TaskC_file1.csv', index=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S_bYKHmk7qtl"
      },
      "source": [
        "weights=lda.transform(lda_data)\n",
        "topic_weights=pd.DataFrame(weights).rename(columns={0:'mountain', 1: 'city life', 2: 'nature', 3:'wildlife'})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 359
        },
        "id": "uNViM4nZ-KJG",
        "outputId": "bfdaa4da-1db5-4ebe-80af-38bc83517de4"
      },
      "source": [
        "topic_weights.head(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>mountain</th>\n",
              "      <th>city life</th>\n",
              "      <th>nature</th>\n",
              "      <th>wildlife</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.020842</td>\n",
              "      <td>0.269554</td>\n",
              "      <td>0.020838</td>\n",
              "      <td>0.688767</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.025029</td>\n",
              "      <td>0.538959</td>\n",
              "      <td>0.025097</td>\n",
              "      <td>0.410916</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.941860</td>\n",
              "      <td>0.019288</td>\n",
              "      <td>0.019500</td>\n",
              "      <td>0.019352</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.019253</td>\n",
              "      <td>0.942270</td>\n",
              "      <td>0.019241</td>\n",
              "      <td>0.019236</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.017966</td>\n",
              "      <td>0.018183</td>\n",
              "      <td>0.945898</td>\n",
              "      <td>0.017952</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.019244</td>\n",
              "      <td>0.021024</td>\n",
              "      <td>0.019271</td>\n",
              "      <td>0.940461</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0.023086</td>\n",
              "      <td>0.931337</td>\n",
              "      <td>0.022781</td>\n",
              "      <td>0.022795</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>0.581655</td>\n",
              "      <td>0.019533</td>\n",
              "      <td>0.020050</td>\n",
              "      <td>0.378762</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>0.021638</td>\n",
              "      <td>0.024305</td>\n",
              "      <td>0.932491</td>\n",
              "      <td>0.021567</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>0.031932</td>\n",
              "      <td>0.031263</td>\n",
              "      <td>0.031927</td>\n",
              "      <td>0.904877</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   mountain  city life    nature  wildlife\n",
              "0  0.020842   0.269554  0.020838  0.688767\n",
              "1  0.025029   0.538959  0.025097  0.410916\n",
              "2  0.941860   0.019288  0.019500  0.019352\n",
              "3  0.019253   0.942270  0.019241  0.019236\n",
              "4  0.017966   0.018183  0.945898  0.017952\n",
              "5  0.019244   0.021024  0.019271  0.940461\n",
              "6  0.023086   0.931337  0.022781  0.022795\n",
              "7  0.581655   0.019533  0.020050  0.378762\n",
              "8  0.021638   0.024305  0.932491  0.021567\n",
              "9  0.031932   0.031263  0.031927  0.904877"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a-8E1qB7_T8Q",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 530
        },
        "outputId": "90c6dbbf-cdf4-4bbc-894b-9754dd6432b4"
      },
      "source": [
        "test_ig = ig_post_labeled.reset_index(drop=False)\n",
        "test_topics = topic_weights.reset_index(drop=False)\n",
        "post_weights = test_ig.merge(test_topics, on='index', how='inner')\n",
        "post_weights.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>index</th>\n",
              "      <th>post_url</th>\n",
              "      <th>comment_num</th>\n",
              "      <th>like_num</th>\n",
              "      <th>post_caption</th>\n",
              "      <th>image_url</th>\n",
              "      <th>labels</th>\n",
              "      <th>like_percent</th>\n",
              "      <th>comment_percent</th>\n",
              "      <th>engagement_score</th>\n",
              "      <th>engagement_binary_class</th>\n",
              "      <th>labels_processed</th>\n",
              "      <th>mountain</th>\n",
              "      <th>city life</th>\n",
              "      <th>nature</th>\n",
              "      <th>wildlife</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>https://www.instagram.com/p/CHYOgoNLjwD/</td>\n",
              "      <td>157</td>\n",
              "      <td>24098</td>\n",
              "      <td>Photo by @erintrieb / On Election Day I set up...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Hair', 'Face', 'White', 'Black', 'Hairstyle'...</td>\n",
              "      <td>0.014033</td>\n",
              "      <td>0.003340</td>\n",
              "      <td>0.007617</td>\n",
              "      <td>0</td>\n",
              "      <td>hair face white black hairstyle shoulder head ...</td>\n",
              "      <td>0.020842</td>\n",
              "      <td>0.269554</td>\n",
              "      <td>0.020838</td>\n",
              "      <td>0.688767</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>https://www.instagram.com/p/CHXl8P8gT0f/</td>\n",
              "      <td>455</td>\n",
              "      <td>83621</td>\n",
              "      <td>Photo by @tasneemalsultan / A Saudi woman sits...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Room', 'Textile', 'Tradition', 'Bazaar', 'Fl...</td>\n",
              "      <td>0.048695</td>\n",
              "      <td>0.009678</td>\n",
              "      <td>0.025285</td>\n",
              "      <td>0</td>\n",
              "      <td>room textile tradition bazaar flooring interio...</td>\n",
              "      <td>0.025029</td>\n",
              "      <td>0.538959</td>\n",
              "      <td>0.025097</td>\n",
              "      <td>0.410916</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>https://www.instagram.com/p/CHXXmNpAhHV/</td>\n",
              "      <td>431</td>\n",
              "      <td>184808</td>\n",
              "      <td>Photo by Keith Ladzinski @ladzinski / A kiss o...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Mountainous landforms', 'Mountain', 'Sky', '...</td>\n",
              "      <td>0.107620</td>\n",
              "      <td>0.009168</td>\n",
              "      <td>0.048549</td>\n",
              "      <td>0</td>\n",
              "      <td>mountainous landforms mountain sky mountain ra...</td>\n",
              "      <td>0.941860</td>\n",
              "      <td>0.019288</td>\n",
              "      <td>0.019500</td>\n",
              "      <td>0.019352</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>https://www.instagram.com/p/CHXKiRnAshz/</td>\n",
              "      <td>511</td>\n",
              "      <td>106213</td>\n",
              "      <td>Photo by @dina_litovsky / Most people don’t as...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Motor vehicle', 'Vehicle', 'Mode of transpor...</td>\n",
              "      <td>0.061851</td>\n",
              "      <td>0.010870</td>\n",
              "      <td>0.031262</td>\n",
              "      <td>0</td>\n",
              "      <td>motor vehicle vehicle mode of transport transp...</td>\n",
              "      <td>0.019253</td>\n",
              "      <td>0.942270</td>\n",
              "      <td>0.019241</td>\n",
              "      <td>0.019236</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>https://www.instagram.com/p/CHWudbnoMhb/</td>\n",
              "      <td>554</td>\n",
              "      <td>162146</td>\n",
              "      <td>Photo by @paoloverzone / Fluorescing a ghostly...</td>\n",
              "      <td>https://scontent-iad3-1.cdninstagram.com/v/t51...</td>\n",
              "      <td>['Water', 'Blue', 'Liquid', 'Aqua', 'Transpare...</td>\n",
              "      <td>0.094423</td>\n",
              "      <td>0.011784</td>\n",
              "      <td>0.044840</td>\n",
              "      <td>0</td>\n",
              "      <td>water blue liquid aqua transparent material tu...</td>\n",
              "      <td>0.017966</td>\n",
              "      <td>0.018183</td>\n",
              "      <td>0.945898</td>\n",
              "      <td>0.017952</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   index                                  post_url  ...    nature  wildlife\n",
              "0      0  https://www.instagram.com/p/CHYOgoNLjwD/  ...  0.020838  0.688767\n",
              "1      1  https://www.instagram.com/p/CHXl8P8gT0f/  ...  0.025097  0.410916\n",
              "2      2  https://www.instagram.com/p/CHXXmNpAhHV/  ...  0.019500  0.019352\n",
              "3      3  https://www.instagram.com/p/CHXKiRnAshz/  ...  0.019241  0.019236\n",
              "4      4  https://www.instagram.com/p/CHWudbnoMhb/  ...  0.945898  0.017952\n",
              "\n",
              "[5 rows x 16 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "id": "chVGxz4B_jsv",
        "outputId": "3638318a-6f78-4726-c1b2-55038d2d0f87"
      },
      "source": [
        "post_weights=post_weights.sort_values(by=['engagement_score'])\n",
        "q1=np.percentile(post_weights['engagement_score'], 25)\n",
        "q3=np.percentile(post_weights ['engagement_score'], 75)\n",
        "#average topic scores for low and high engagement quartile\n",
        "pd.DataFrame({'Topic':['Mountain','Citylife','Nature', 'Wildlife'],\n",
        "              'Low_engagement':post_weights[post_weights['engagement_score']<=q1]\\\n",
        "              [post_weights.columns[-num_topics:]].mean(),\\\n",
        "              'High_engagement':post_weights[post_weights['engagement_score']>=q3][post_weights.columns[-num_topics:]].mean()})\n",
        "#rows = topic name"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Topic</th>\n",
              "      <th>Low_engagement</th>\n",
              "      <th>High_engagement</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>mountain</th>\n",
              "      <td>Mountain</td>\n",
              "      <td>0.074132</td>\n",
              "      <td>0.177033</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>city life</th>\n",
              "      <td>Citylife</td>\n",
              "      <td>0.520110</td>\n",
              "      <td>0.185169</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>nature</th>\n",
              "      <td>Nature</td>\n",
              "      <td>0.237117</td>\n",
              "      <td>0.373017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>wildlife</th>\n",
              "      <td>Wildlife</td>\n",
              "      <td>0.168641</td>\n",
              "      <td>0.264781</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Topic  Low_engagement  High_engagement\n",
              "mountain   Mountain        0.074132         0.177033\n",
              "city life  Citylife        0.520110         0.185169\n",
              "nature       Nature        0.237117         0.373017\n",
              "wildlife   Wildlife        0.168641         0.264781"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8HV0VoJFEtpa"
      },
      "source": [
        "## Task D: Advice or Insight"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OlOdu5v1wFhk"
      },
      "source": [
        "Based on the Logistic Regression analysis, we found that the labels produced by Google Vision had the highest prediction power with an accuracy of 74%. Meanwhile when we ran the logistic regression with the labels and captions included, the prediction power lowered to an accuracy of 70%. The accuracy score for captions alone is 66%, which means that captions is not a good indicator of engagament compared to the labels we found using Google Vision. When the captions were included, they added extra words that induced more noise and caused prediction accuracy to lower. We used these labels to implement topic modeling below.\n",
        "\n",
        "Based on LDA, we found that the four topics representing the themes of the posts are : 'Wildlife', 'Mountain','Citylife', and 'Nature'. For example, 'Nature' posts included labels such as: sky, phenomenon, tree, water, landscape, cloud, and sea. We found that the high engagement posts were composed of 37.30% 'Nature', 26.47% 'Wildlife', 18.51% 'Citylife', and 17.07% 'Mountain'. Low engagement posts were composed of 52.01% 'Citylife', 23.71% 'Nature', 16.86% 'Wildlife' and 7.41% 'Mountain'. So that means National Geographic should focus on 'Nature' posts and decrease the number of posts related to 'Citylife'. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D0rf99CaExdb"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s8GCYdd316hN"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}