EDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EDA\n",
    "\n",
    "- Length of texts\n",
    "- Cleaning train.csv\n",
    "- Regular Expression Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Length of texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import pandas as pd\n",
    "import glob\n",
    "\n",
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4c8eb166f9724d75b60ed0994b40515a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/15594 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_df = pd.read_csv('../feedback-prize-2021/train.csv')\n",
    "train_df.loc[train_df.discourse_id==1623258656795.0, 'discourse_text'] =  train_df.loc[train_df.discourse_id==1623258656795.0, \n",
    "                                                                           'discourse_text'].map(lambda x: x.replace('florida', 'LOCATION_NAME')).values\n",
    "\n",
    "texts = []\n",
    "ids = []\n",
    "for fname in tqdm(glob.glob('../feedback-prize-2021/train/*.txt')):\n",
    "    with open(fname, encoding='utf-8') as f:\n",
    "        texts.append(f.read())\n",
    "    ids.append(fname.split('/')[-1].split('.')[0])\n",
    "\n",
    "df = pd.DataFrame({'id': ids, 'text': texts})\n",
    "\n",
    "\n",
    "df['text_len'] = df.text.apply(lambda x: len(re.split(r'\\n| ', x)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>text_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14605</th>\n",
       "      <td>73D6F19E24BD</td>\n",
       "      <td>The students are very mad that the principal i...</td>\n",
       "      <td>151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15090</th>\n",
       "      <td>4CB458757785</td>\n",
       "      <td>The authors claim to keep studing venus is a v...</td>\n",
       "      <td>151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7903</th>\n",
       "      <td>5503B0FB1886</td>\n",
       "      <td>I dont think the use of the technology to read...</td>\n",
       "      <td>152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6631</th>\n",
       "      <td>C0897AC15921</td>\n",
       "      <td>The challenge of exploring venus was being abl...</td>\n",
       "      <td>152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15202</th>\n",
       "      <td>8CE968DB99AB</td>\n",
       "      <td>Dear Principal,\\n\\nI am addressing the new pol...</td>\n",
       "      <td>152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2588</th>\n",
       "      <td>DC59B3A9D4A2</td>\n",
       "      <td>Ther are 3 main supportive reassons why this \"...</td>\n",
       "      <td>7493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11071</th>\n",
       "      <td>803FA7997692</td>\n",
       "      <td>I dont think that FACS should be able to be us...</td>\n",
       "      <td>8546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7284</th>\n",
       "      <td>BFF07046CF7A</td>\n",
       "      <td>The author shows us the many benifical factors...</td>\n",
       "      <td>8815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4129</th>\n",
       "      <td>E4559E81E304</td>\n",
       "      <td>This passage is about a germany mom from the s...</td>\n",
       "      <td>8944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2525</th>\n",
       "      <td>9B23715DFB32</td>\n",
       "      <td>if we look back at time in the united states y...</td>\n",
       "      <td>14718</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>15594 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 id                                               text  \\\n",
       "14605  73D6F19E24BD  The students are very mad that the principal i...   \n",
       "15090  4CB458757785  The authors claim to keep studing venus is a v...   \n",
       "7903   5503B0FB1886  I dont think the use of the technology to read...   \n",
       "6631   C0897AC15921  The challenge of exploring venus was being abl...   \n",
       "15202  8CE968DB99AB  Dear Principal,\\n\\nI am addressing the new pol...   \n",
       "...             ...                                                ...   \n",
       "2588   DC59B3A9D4A2  Ther are 3 main supportive reassons why this \"...   \n",
       "11071  803FA7997692  I dont think that FACS should be able to be us...   \n",
       "7284   BFF07046CF7A  The author shows us the many benifical factors...   \n",
       "4129   E4559E81E304  This passage is about a germany mom from the s...   \n",
       "2525   9B23715DFB32  if we look back at time in the united states y...   \n",
       "\n",
       "       text_len  \n",
       "14605       151  \n",
       "15090       151  \n",
       "7903        152  \n",
       "6631        152  \n",
       "15202       152  \n",
       "...         ...  \n",
       "2588       7493  \n",
       "11071      8546  \n",
       "7284       8815  \n",
       "4129       8944  \n",
       "2525      14718  \n",
       "\n",
       "[15594 rows x 3 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values('text_len')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<AxesSubplot:title={'center':'text_len'}>]], dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAWP0lEQVR4nO3df5BdZ33f8fcHC2NiEUuOiepaLjLBk4yBAWzVNiVtJUj8C4jJTKBm3CA7TjXTuC000CBDqQmYiSG0JJ4mgBo7iB9BuAZq1yalqrAmZTLY4AD+BY6FLcAaBwGWnQoIg8i3f9xH8vWyq72rvftLz/s1c2fPec5zzvmeZ3c/9+45595NVSFJ6sOTFroASdL8MfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9aYySrElSSZYtdC3SZAx9HdGS7EryS4tlO9JCM/QlqSOGvo5YST4E/CPgfybZl+R3kpyd5C+TPJrky0nWtb7/JMl3kpzc5p+XZG+SX5hsOzOo4bgk1yZ5OMnuJFclOaotuyTJZ5O8u+3rwSTnj3scpGGGvo5YVfXrwDeAl1fVcuAjwC3AVcDxwBuAjyd5elX9JfB+YEuSpwIfBt5SVV+duJ2qetcMyvgAsB94FvAC4BzgN4eWnwXcB5wAvAu4NkkO95il6Rj66sm/BD5VVZ+qqr+vqm3AF4AL2vK3AscBtwO7gT+azc6SrGrbfl1Vfa+q9gDvAS4a6vb1qvpvVfVjYAtwIrBqNvuVDsU7DNSTZwCvTPLyobYnA7cCVNWPknwAuAb47Zr9R9A+o23/4aEX708CvjnU528OTFTV91u/5bPcrzQlQ19HuuHg/ibwoar6V5N1THIScCXwp8B/TvKPq+qHk2xnVN8EfgicUFX7D2N9aew8vaMj3beAZ7bpDwMvT3JukqOSHJNkXZLV7Tz6B4BrgcuAh4G3T7GdkVTVw8D/ZvAE8tNJnpTk55L881kek3TYDH0d6X4P+I9JHgX+BXAh8Cbg2wxeif8HBr8H/w74WQYXbwu4FLg0yT+duJ0kb5jB/l8DHA3cC+wFbmBw3l5aEPE/Z0lSP3ylL0kdMfSlGUpycXuT1sTHPQtdmzQdT+9IUkcW9S2bJ5xwQq1Zs2bk/t/73vc49thj566gMbHO8bLO8VkKNYJ1TueOO+74TlU9fdKFVbVoH2eccUbNxK233jqj/gvFOsfLOsdnKdRYZZ3TAb5QU+Sq5/QlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0JakjI4V+kl1J7krypSRfaG3HJ9mW5P72dWVrT5JrkuxMcmeS04e2s6H1vz/Jhrk5pMmt2XTLwYck9Womr/TXV9Xzq2ptm98EbK+qU4HtbR7gfODU9tgIvBcGTxIM/hXdWcCZwJUHnigkSfNjNqd3LgS2tOktwCuG2j/YPgLic8CKJCcC5wLbquqRqtoLbAPOm8X+JUkzNNJHKyd5kMG/eivg/VW1OcmjVbWiLQ+wt6pWJLkZuLqqPtuWbQfeCKwDjqmqq1r7W4AfVNW7J+xrI4O/EFi1atUZW7duHflg9u3bx/Llyydddtfuxw5OP/ek40be5lw4VJ2LiXWO11KocynUCNY5nfXr198xdFbmCUb9aOVfrKrdSX4W2Jbkq8MLq6qSjOWD+atqM7AZYO3atbVu3bqR192xYwdT9b9k6Fz+rotH3+ZcOFSdi4l1jtdSqHMp1AjWORsjnd6pqt3t6x7gkwzOyX+rnbahfd3Tuu8GTh5afXVrm6pdkjRPpg39JMcmedqBaeAc4G7gJuDAHTgbgBvb9E3Aa9pdPGcDj1XVw8CngXOSrGwXcM9pbZKkeTLK6Z1VwCcHp+1ZBvxZVf2vJJ8Hrk9yGfB14FWt/6eAC4CdwPeBSwGq6pEkbwc+3/q9raoeGduRSJKmNW3oV9UDwPMmaf8u8JJJ2gu4fIptXQdcN/MyJUnj4DtyJakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjyxa6gLm0ZtMtC12CJC0qvtKXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHRk59JMcleSLSW5u86ckuS3JziQfS3J0a39Km9/Zlq8Z2sYVrf2+JOeO/WgkSYc0k1f6rwW+MjT/TuA9VfUsYC9wWWu/DNjb2t/T+pHkNOAi4NnAecAfJzlqduVLkmZipNBPshp4KfAnbT7Ai4EbWpctwCva9IVtnrb8Ja3/hcDWqvphVT0I7ATOHMMxSJJGlKqavlNyA/B7wNOANwCXAJ9rr+ZJcjLw51X1nCR3A+dV1UNt2deAs4C3tnU+3NqvbevcMGFfG4GNAKtWrTpj69atIx/Mvn37WL58+cH5u3Y/Nmm/55503MjbnAsT61ysrHO8lkKdS6FGsM7prF+//o6qWjvZsmk/hiHJy4A9VXVHknVjru0nVNVmYDPA2rVra9260Xe5Y8cOhvtfMsXHMOy6ePRtzoWJdS5W1jleS6HOpVAjWOdsjPLZOy8CfiXJBcAxwE8DfwisSLKsqvYDq4Hdrf9u4GTgoSTLgOOA7w61HzC8jiRpHkx7Tr+qrqiq1VW1hsGF2M9U1cXArcCvtW4bgBvb9E1tnrb8MzU4h3QTcFG7u+cU4FTg9rEdiSRpWrP5lM03AluTXAV8Ebi2tV8LfCjJTuARBk8UVNU9Sa4H7gX2A5dX1Y9nsX9J0gzNKPSragewo00/wCR331TV3wGvnGL9dwDvmGmRkqTx8B25ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6Mpv/kbtkrdl0y8HpXVe/dAErkaT55St9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHVk2tBPckyS25N8Ock9SX63tZ+S5LYkO5N8LMnRrf0pbX5nW75maFtXtPb7kpw7Z0clSZrUKK/0fwi8uKqeBzwfOC/J2cA7gfdU1bOAvcBlrf9lwN7W/p7WjySnARcBzwbOA/44yVFjPBZJ0jSmDf0a2Ndmn9weBbwYuKG1bwFe0aYvbPO05S9Jkta+tap+WFUPAjuBM8dxEJKk0Yx0Tj/JUUm+BOwBtgFfAx6tqv2ty0PASW36JOCbAG35Y8DPDLdPso4kaR6M9E9UqurHwPOTrAA+CfzCXBWUZCOwEWDVqlXs2LFj5HX37dv3hP6vf+7+qTs3M9n+uEysc7GyzvFaCnUuhRrBOmdjRv85q6oeTXIr8EJgRZJl7dX8amB367YbOBl4KMky4Djgu0PtBwyvM7yPzcBmgLVr19a6detGrm/Hjh0M979k6D9kTWXXxaNvf1wm1rlYWed4LYU6l0KNYJ2zMcrdO09vr/BJ8lTgl4GvALcCv9a6bQBubNM3tXna8s9UVbX2i9rdPacApwK3j+k4JEkjGOWV/onAlnanzZOA66vq5iT3AluTXAV8Ebi29b8W+FCSncAjDO7YoaruSXI9cC+wH7i8nTaSJM2TaUO/qu4EXjBJ+wNMcvdNVf0d8MoptvUO4B0zL1OSNA6+I1eSOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUkWlDP8nJSW5Ncm+Se5K8trUfn2Rbkvvb15WtPUmuSbIzyZ1JTh/a1obW//4kG+busCRJkxnllf5+4PVVdRpwNnB5ktOATcD2qjoV2N7mAc4HTm2PjcB7YfAkAVwJnAWcCVx54IlCkjQ/pg39qnq4qv6qTf8/4CvAScCFwJbWbQvwijZ9IfDBGvgcsCLJicC5wLaqeqSq9gLbgPPGeTCSpENLVY3eOVkD/AXwHOAbVbWitQfYW1UrktwMXF1Vn23LtgNvBNYBx1TVVa39LcAPqurdE/axkcFfCKxateqMrVu3jlzfvn37WL58+cH5u3Y/Nu06zz3puJG3Py4T61ysrHO8lkKdS6FGsM7prF+//o6qWjvZsmWjbiTJcuDjwOuq6m8HOT9QVZVk9GePQ6iqzcBmgLVr19a6detGXnfHjh0M979k0y3TrrPr4tG3Py4T61ysrHO8lkKdS6FGsM7ZGOnunSRPZhD4H6mqT7Tmb7XTNrSve1r7buDkodVXt7ap2iVJ82SUu3cCXAt8par+y9Cim4ADd+BsAG4can9Nu4vnbOCxqnoY+DRwTpKV7QLuOa1NkjRPRjm98yLg14G7knyptb0JuBq4PsllwNeBV7VlnwIuAHYC3wcuBaiqR5K8Hfh86/e2qnpkHAchSRrNtKHfLshmisUvmaR/AZdPsa3rgOtmUqAkaXx8R64kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOjLyP1E5Uq0Z+kcru65+6QJWIklzz1f6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkemDf0k1yXZk+Tuobbjk2xLcn/7urK1J8k1SXYmuTPJ6UPrbGj970+yYW4OR5J0KKO80v8AcN6Etk3A9qo6Fdje5gHOB05tj43Ae2HwJAFcCZwFnAlceeCJQpI0f6YN/ar6C+CRCc0XAlva9BbgFUPtH6yBzwErkpwInAtsq6pHqmovsI2ffCKRJM2xVNX0nZI1wM1V9Zw2/2hVrWjTAfZW1YokNwNXV9Vn27LtwBuBdcAxVXVVa38L8IOqevck+9rI4K8EVq1adcbWrVtHPph9+/axfPnyg/N37X5s5HUBnnvScTPqf7gm1rlYWed4LYU6l0KNYJ3TWb9+/R1VtXayZctmu/GqqiTTP3OMvr3NwGaAtWvX1rp160Zed8eOHQz3v2TTLTPa966LR9/XbEysc7GyzvFaCnUuhRrBOmfjcO/e+VY7bUP7uqe17wZOHuq3urVN1S5JmkeHG/o3AQfuwNkA3DjU/pp2F8/ZwGNV9TDwaeCcJCvbBdxzWpskaR5Ne3onyUcZnJM/IclDDO7CuRq4PsllwNeBV7XunwIuAHYC3wcuBaiqR5K8Hfh86/e2qpp4cViSNMemDf2qevUUi14ySd8CLp9iO9cB182oOknSWPmOXEnqiKEvSR0x9CWpI4a+JHXE0Jekjsz6HblHkjVD7+DddfVLF7ASSZobvtKXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR3xo5Wn4McsSzoS+Upfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BHfnDUC36gl6UjhK31J6oihL0kdmffQT3JekvuS7Eyyab73L0k9m9fQT3IU8EfA+cBpwKuTnDafNUhSz+b7Qu6ZwM6qegAgyVbgQuDeea7jsHlRV9JSNt+hfxLwzaH5h4Czhjsk2QhsbLP7ktw3g+2fAHxnVhXOQN552KvOa52zYJ3jtRTqXAo1gnVO5xlTLVh0t2xW1WZg8+Gsm+QLVbV2zCWNnXWOl3WOz1KoEaxzNub7Qu5u4OSh+dWtTZI0D+Y79D8PnJrklCRHAxcBN81zDZLUrXk9vVNV+5P8G+DTwFHAdVV1zxh3cVinhRaAdY6XdY7PUqgRrPOwpaoWugZJ0jzxHbmS1BFDX5I6csSE/kJ+vEOSk5PcmuTeJPckeW1rPz7JtiT3t68rW3uSXNNqvTPJ6UPb2tD6359kwxzVe1SSLya5uc2fkuS2Vs/H2kV2kjylze9sy9cMbeOK1n5fknPnoMYVSW5I8tUkX0nywsU4nkn+ffue353ko0mOWQzjmeS6JHuS3D3UNrbxS3JGkrvaOtckyZhq/P32Pb8zySeTrBhaNukYTfW7P9X3YRx1Di17fZJKckKbX5CxnJGqWvIPBheFvwY8Ezga+DJw2jzu/0Tg9Db9NOCvGXzMxLuATa19E/DONn0B8OdAgLOB21r78cAD7evKNr1yDur9beDPgJvb/PXARW36fcC/btO/BbyvTV8EfKxNn9bG+CnAKW3sjxpzjVuA32zTRwMrFtt4Mniz4YPAU4fG8ZLFMJ7APwNOB+4eahvb+AG3t75p654/phrPAZa16XcO1TjpGHGI3/2pvg/jqLO1n8zgppSvAycs5FjO6HjmcuPz9QBeCHx6aP4K4IoFrOdG4JeB+4ATW9uJwH1t+v3Aq4f639eWvxp4/1D7E/qNqbbVwHbgxcDN7QftO0O/aAfHsv1Av7BNL2v9MnF8h/uNqcbjGIRpJrQvqvHk8XeYH9/G52bg3MUynsAanhioYxm/tuyrQ+1P6DebGics+1XgI2160jFiit/9Q/1cj6tO4AbgecAuHg/9BRvLUR9HyumdyT7e4aSFKKT9yf4C4DZgVVU93Bb9DbCqTU9V73wcxx8AvwP8fZv/GeDRqto/yT4P1tOWP9b6z3WdpwDfBv40g9NQf5LkWBbZeFbVbuDdwDeAhxmMzx0svvE8YFzjd1Kbnut6f4PBK9/DqfFQP9ezluRCYHdVfXnCosU6lgcdKaG/KCRZDnwceF1V/e3wsho8jS/o/bFJXgbsqao7FrKOESxj8Of0e6vqBcD3GJyOOGiRjOdKBh8YeArwD4FjgfMWsqZRLYbxO5Qkbwb2Ax9Z6FomSvJTwJuA/7TQtRyOIyX0F/zjHZI8mUHgf6SqPtGav5XkxLb8RGBPa5+q3rk+jhcBv5JkF7CVwSmePwRWJDnwRr3hfR6spy0/DvjuPNT5EPBQVd3W5m9g8CSw2Mbzl4AHq+rbVfUj4BMMxnixjecB4xq/3W16TupNcgnwMuDi9uR0ODV+l6m/D7P1cwye6L/cfpdWA3+V5B8cRp1zOpaTmstzR/P1YPDK8AEG34gDF3OePY/7D/BB4A8mtP8+T7xw9q42/VKeeLHn9tZ+PINz2Svb40Hg+DmqeR2PX8j97zzxgtdvtenLeeKFx+vb9LN54kW1Bxj/hdz/C/x8m35rG8tFNZ4MPiH2HuCn2r63AP92sYwnP3lOf2zjx09efLxgTDWex+Cj1p8+od+kY8Qhfven+j6Mo84Jy3bx+Dn9BRvLkY9lLjc+nw8GV83/msGV/DfP875/kcGfyncCX2qPCxicV9wO3A/8n6Fvchj8M5mvAXcBa4e29RvAzva4dA5rXsfjof/M9oO3s/2iPKW1H9Pmd7blzxxa/82t/vuYg7sNgOcDX2hj+j/aL8qiG0/gd4GvAncDH2qhtODjCXyUwXWGHzH4y+mycY4fsLYd89eA/8qEi+6zqHEng3PfB36P3jfdGDHF7/5U34dx1Dlh+S4eD/0FGcuZPPwYBknqyJFyTl+SNAJDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXk/wM2bn5Hced89wAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.sort_values('text_len').hist(column='text_len', bins=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning train.csv\n",
    "> Just to share, I tried to use this file and the LB score improved by 0.002. This was on distilbert based small model only so there is a chance that this file can provide improved results on larger models.\n",
    "\n",
    "- [corrected train.csv (feedback prize)](https://www.kaggle.com/nbroad/corrected-train-csv-feedback-prize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regular Expression Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = re.compile('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = p.finditer(\"life is too short life\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(result).start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe"
  },
  "kernelspec": {
   "display_name": "Python 3.7.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}