Extracting other features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other features\n",
    "\n",
    "As mentioned before, the POS tags are not usually enough to provide our algorithm with the necessary information to make accurate inferences. Luckyly for us, we can provide our algorithm with many more features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vuelax.tokenisation import index_emoji_tokenize\n",
    "import pandas as pd\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting from our already labelled dataset (remember I have a file called `data/to_label.csv`). The following are just a few helper functions to read and augment our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labelled_data = pd.read_csv(\"data/to_label-done.csv\")\n",
    "labelled_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Little helper to read from our labelled dataset\n",
    "def read_whole_offers(dataset):\n",
    "    current_offer = 0\n",
    "    rows = []\n",
    "    for _, row in dataset.iterrows():\n",
    "        if row['offer_id'] != current_offer:\n",
    "            yield rows\n",
    "            current_offer = row['offer_id']\n",
    "            rows = []\n",
    "        rows.append(list(row.values))\n",
    "    yield rows\n",
    "            \n",
    "offers = read_whole_offers(labelled_data)\n",
    "offer_ids, tokens, positions, pos_tags, labels = zip(*next(offers))\n",
    "print(offer_ids)\n",
    "print(tokens)\n",
    "print(positions)\n",
    "print(pos_tags)\n",
    "print(labels)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building our training set  \n",
    "\n",
    "The features I decided to augment the data with are the following:  \n",
    "\n",
    " - Lengths of each individual tokens\n",
    " - Length of the whole offer (counted in tokens)\n",
    " - The POS tag of the token to the left\n",
    " - The POS tag of the token to the right\n",
    " - Whether the token is uppercase or not\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_more_features(tokens, pos_tags):\n",
    "    lengths =  [len(l) for l in tokens]\n",
    "    n_tokens =  [len(tokens) for l in tokens]\n",
    "    augmented = ['<p>'] + list(pos_tags) + ['</p>']\n",
    "    uppercase = [all([l.isupper() for l in token]) for token in tokens]\n",
    "    return lengths, n_tokens, augmented[:len(tokens)], augmented[2:], uppercase\n",
    "\n",
    "generate_more_features(tokens, pos_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "offers = read_whole_offers(labelled_data)\n",
    "\n",
    "extended_headers = [\n",
    "    \"offer_id\", \n",
    "    \"token\", \n",
    "    \"position\", \n",
    "    \"pos_tag\", \n",
    "    \"pos_left\", \n",
    "    \"pos_right\", \n",
    "    \"token_length\", \n",
    "    \"token_count\",\n",
    "    \"uppercase\",\n",
    "    \"label\"\n",
    "]\n",
    "\n",
    "with open(\"data/features-labels.csv\", \"w\") as w:\n",
    "    writer = csv.writer(w)\n",
    "    writer.writerow(extended_headers)\n",
    "    for offer in offers:\n",
    "        offer_ids, tokens, positions, pos_tags, labels = zip(*offer)\n",
    "        lenghts, n_tokens, lefts, rights, uppercase = generate_more_features(tokens, pos_tags)\n",
    "        data = zip(offer_ids, tokens, positions, pos_tags, lefts, rights, lenghts, n_tokens, uppercase, labels)\n",
    "        for row in data:\n",
    "            writer.writerow(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then in the file `data/features-labels.csv` our dataset will be ready to use to train our algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}