notebooks/Getting Started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with iQual\n",
    "\n",
    "This notebook provides a simple introduction to using iQual for text classification tasks. It demonstrates the core functionality of the package through a practical example of binary classification using the Stanford Politeness Corpus.\n",
    "\n",
    "**Key Features:**\n",
    "- This notebook can be run as-is without modifications\n",
    "- All necessary code and explanations are included\n",
    "\n",
    "Follow along to learn how to:\n",
    "1. Prepare text data with question and answer pairs\n",
    "2. Create and configure an iQual model\n",
    "3. Train a classifier on labeled examples\n",
    "4. Make predictions on new data\n",
    "5. Evaluate model performance\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install the package from PyPI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install iqual --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from iqual import iqualnlp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Stanford Politeness Corpus](https://convokit.cornell.edu/documentation/stack_politeness.html)\n",
    "\n",
    "In this tutorial, we'll use the Stanford Politeness Corpus, a dataset of requests from Stack Exchange annotated for politeness. This dataset provides a good example for binary classification:\n",
    "\n",
    "- **Task**: Determining whether a request is polite or impolite\n",
    "- **Features**: We'll use both the context and the request text to make predictions\n",
    "- **Labels**: Binary classification (1 = polite, 0 = impolite)\n",
    "\n",
    "This represents a common text classification task in computational linguistics and social science research. The ability to automatically detect politeness has applications in communication studies, workplace interaction analysis, and online community moderation.\n",
    "\n",
    "The dataset is available through [ConvoKit](https://convokit.cornell.edu/), a toolkit for conversational analysis. Let's prepare the data for iQual:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Downloading the Stack Exchange Politeness Corpus...\n",
      "Dataset already exists at /Users/adityachhabra/.convokit/saved-corpora/stack-exchange-politeness-corpus\n",
      "Download complete. Corpus contains 6603 utterances.\n"
     ]
    }
   ],
   "source": [
    "# Install the ConvoKit library and download the Stanford Politeness Corpus\n",
    "!pip install convokit --quiet\n",
    "\n",
    "# Import the necessary functions\n",
    "from convokit import Corpus, download\n",
    "\n",
    "# Download the Stack Exchange Politeness Corpus\n",
    "print(\"Downloading the Stack Exchange Politeness Corpus...\")\n",
    "corpus = Corpus(filename=download(\"stack-exchange-politeness-corpus\"))\n",
    "print(f\"Download complete. Corpus contains {len(corpus.get_utterance_ids())} utterances.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation\n",
    "\n",
    "The following code is specific to the Stanford Politeness Corpus structure. When working with your own datasets, you would adapt this preprocessing step to fit your data format.\n",
    "\n",
    "Key steps in this preparation:\n",
    "1. Extract the request text and politeness label from each utterance\n",
    "2. Create a context field by splitting the text (in a real dataset, you might have distinct context)\n",
    "3. Convert the politeness scores to binary values (1 = polite, 0 = impolite)\n",
    "4. Format the data to work with iQual's expected input structure\n",
    "\n",
    "Note that for demonstration purposes, we're artificially creating a \"context\" column by taking the first sentence of each request.\n",
    "\n",
    "In your own work with interview transcripts or conversational data, you would typically have separate interviewer questions and respondent answers, or distinct turns in a conversation that provide natural context for each response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created DataFrame with 6596 records\n",
      "Polite requests: 1649\n",
      "Impolite requests: 4947\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context</th>\n",
       "      <th>answer</th>\n",
       "      <th>is_polite</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2452</th>\n",
       "      <td>I'm not sure your question makes sense.</td>\n",
       "      <td>Do you mean, how do you measure the length of ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>492</th>\n",
       "      <td>Stack Exchange discussion:</td>\n",
       "      <td>What about `DefaultListBoxItemStyle`? Are you ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4233</th>\n",
       "      <td>Hi johnny8888, I'm on a Solaris server and the...</td>\n",
       "      <td>Can you please elaborate your answer?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3632</th>\n",
       "      <td>Stack Exchange discussion:</td>\n",
       "      <td>Do you mean you need real world use cases for ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>443</th>\n",
       "      <td>Stack Exchange discussion:</td>\n",
       "      <td>Do the URLs you want to add also come from the...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1165</th>\n",
       "      <td>Now, I am looking at the source for `whine.</td>\n",
       "      <td>pl` and it seems to be getting these from an S...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2039</th>\n",
       "      <td>This sounds a lot like a machine learning algo...</td>\n",
       "      <td>I'm no machine learning expert, but perhaps so...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2481</th>\n",
       "      <td>This is not really enough to tell what can be ...</td>\n",
       "      <td>Have you tried debug you code, for example tra...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4182</th>\n",
       "      <td>Yes, I can change the font.</td>\n",
       "      <td>Could you please provide me with an example?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3777</th>\n",
       "      <td>I'd suggest you get an iPod Touch to develop o...</td>\n",
       "      <td>Don't tell me you need something iPhone specif...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                context  \\\n",
       "2452            I'm not sure your question makes sense.   \n",
       "492                          Stack Exchange discussion:   \n",
       "4233  Hi johnny8888, I'm on a Solaris server and the...   \n",
       "3632                         Stack Exchange discussion:   \n",
       "443                          Stack Exchange discussion:   \n",
       "1165        Now, I am looking at the source for `whine.   \n",
       "2039  This sounds a lot like a machine learning algo...   \n",
       "2481  This is not really enough to tell what can be ...   \n",
       "4182                        Yes, I can change the font.   \n",
       "3777  I'd suggest you get an iPod Touch to develop o...   \n",
       "\n",
       "                                                 answer  is_polite  \n",
       "2452  Do you mean, how do you measure the length of ...          0  \n",
       "492   What about `DefaultListBoxItemStyle`? Are you ...          0  \n",
       "4233              Can you please elaborate your answer?          0  \n",
       "3632  Do you mean you need real world use cases for ...          0  \n",
       "443   Do the URLs you want to add also come from the...          0  \n",
       "1165  pl` and it seems to be getting these from an S...          1  \n",
       "2039  I'm no machine learning expert, but perhaps so...          1  \n",
       "2481  Have you tried debug you code, for example tra...          0  \n",
       "4182       Could you please provide me with an example?          0  \n",
       "3777  Don't tell me you need something iPhone specif...          0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Process the corpus into a DataFrame\n",
    "rows = []\n",
    "\n",
    "# Iterate through the utterances\n",
    "for utt in corpus.iter_utterances():\n",
    "    # Extract request text\n",
    "    request_text = utt.text\n",
    "    \n",
    "    # Get metadata\n",
    "    meta = utt.meta\n",
    "    \n",
    "    # Extract the binary politeness label (1=\"polite\", 0=\"neutral\", -1=\"impolite\")\n",
    "    if 'Binary' in meta:\n",
    "        binary_label = meta['Binary']\n",
    "        # Convert to our binary format (1=polite, 0=impolite/neutral)\n",
    "        is_polite = 1 if binary_label == 1 else 0\n",
    "        \n",
    "        # For Stack Exchange, create a simplified context\n",
    "        if len(request_text.split('.')) > 1:\n",
    "            # Split at the first period\n",
    "            parts = request_text.split('.', 1)\n",
    "            context_text = parts[0] + '.'\n",
    "            request_text = parts[1].strip()\n",
    "        else:\n",
    "            # For short texts, use a generic context\n",
    "            context_text = \"Stack Exchange discussion:\"\n",
    "        \n",
    "        # Add to our list of data points\n",
    "        rows.append({\n",
    "            'context': context_text,\n",
    "            'answer': request_text,\n",
    "            'is_polite': is_polite\n",
    "        })\n",
    "\n",
    "# Convert to DataFrame\n",
    "data = pd.DataFrame(rows)\n",
    "\n",
    "# Remove any rows where question or answer is empty\n",
    "data = data[data['context'].str.len() > 0]\n",
    "data = data[data['answer'].str.len() > 0]\n",
    "\n",
    "# Print stats\n",
    "print(f\"Created DataFrame with {len(data)} records\")\n",
    "print(f\"Polite requests: {sum(data['is_polite'] == 1)}\")\n",
    "print(f\"Impolite requests: {sum(data['is_polite'] == 0)}\")\n",
    "\n",
    "data.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split the data into training and testing sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data size: 4617\n",
      "Testing data size: 1979\n"
     ]
    }
   ],
   "source": [
    "train_data, test_data = train_test_split(data, test_size=0.3)\n",
    "\n",
    "# Create feature matrix (X) and target vector (y) for training\n",
    "X_train = train_data[['context', 'answer']]\n",
    "y_train = train_data['is_polite']\n",
    "\n",
    "# Create feature matrix for testing\n",
    "X_test = test_data[['context', 'answer']]\n",
    "y_test = test_data['is_polite']\n",
    "\n",
    "print(f\"Training data size: {len(X_train)}\")\n",
    "print(f\"Testing data size: {len(X_test)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model fit complete!\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: black;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: block;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 1ex;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;Input&#x27;,\n",
       "                 FeatureUnion(transformer_list=[(&#x27;question&#x27;,\n",
       "                                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                                  FunctionTransformer(func=&lt;function column_selector at 0x1122391c0&gt;,\n",
       "                                                                                      kw_args={&#x27;column_name&#x27;: &#x27;context&#x27;})),\n",
       "                                                                 (&#x27;vectorizer&#x27;,\n",
       "                                                                  Vectorizer(analyzer=&#x27;word&#x27;,\n",
       "                                                                             binary=False,\n",
       "                                                                             decode_error=&#x27;strict&#x27;,\n",
       "                                                                             dtype=&lt;class &#x27;numpy.float64&#x27;&gt;,\n",
       "                                                                             encoding=&#x27;utf-8&#x27;,\n",
       "                                                                             env=&#x27;scikit-learn&#x27;,\n",
       "                                                                             input=&#x27;conten...\n",
       "                            fit_intercept=True, intercept_scaling=1,\n",
       "                            l1_ratio=None, max_iter=100,\n",
       "                            model=&#x27;LogisticRegression&#x27;,\n",
       "                            multi_class=&#x27;deprecated&#x27;, n_jobs=None, penalty=&#x27;l2&#x27;,\n",
       "                            random_state=None, solver=&#x27;lbfgs&#x27;, tol=0.0001,\n",
       "                            verbose=0, warm_start=False)),\n",
       "                (&#x27;Threshold&#x27;,\n",
       "                 BinaryThresholder(threshold=np.float64(0.3052255080473171),\n",
       "                                   threshold_range=(np.float64(0.015664853903903313),\n",
       "                                                    np.float64(0.858797346850902))))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;&nbsp;Pipeline<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.pipeline.Pipeline.html\">?<span>Documentation for Pipeline</span></a><span class=\"sk-estimator-doc-link \">i<span>Not fitted</span></span></label><div class=\"sk-toggleable__content \"><pre>Pipeline(steps=[(&#x27;Input&#x27;,\n",
       "                 FeatureUnion(transformer_list=[(&#x27;question&#x27;,\n",
       "                                                 Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                                  FunctionTransformer(func=&lt;function column_selector at 0x1122391c0&gt;,\n",
       "                                                                                      kw_args={&#x27;column_name&#x27;: &#x27;context&#x27;})),\n",
       "                                                                 (&#x27;vectorizer&#x27;,\n",
       "                                                                  Vectorizer(analyzer=&#x27;word&#x27;,\n",
       "                                                                             binary=False,\n",
       "                                                                             decode_error=&#x27;strict&#x27;,\n",
       "                                                                             dtype=&lt;class &#x27;numpy.float64&#x27;&gt;,\n",
       "                                                                             encoding=&#x27;utf-8&#x27;,\n",
       "                                                                             env=&#x27;scikit-learn&#x27;,\n",
       "                                                                             input=&#x27;conten...\n",
       "                            fit_intercept=True, intercept_scaling=1,\n",
       "                            l1_ratio=None, max_iter=100,\n",
       "                            model=&#x27;LogisticRegression&#x27;,\n",
       "                            multi_class=&#x27;deprecated&#x27;, n_jobs=None, penalty=&#x27;l2&#x27;,\n",
       "                            random_state=None, solver=&#x27;lbfgs&#x27;, tol=0.0001,\n",
       "                            verbose=0, warm_start=False)),\n",
       "                (&#x27;Threshold&#x27;,\n",
       "                 BinaryThresholder(threshold=np.float64(0.3052255080473171),\n",
       "                                   threshold_range=(np.float64(0.015664853903903313),\n",
       "                                                    np.float64(0.858797346850902))))])</pre></div> </div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;Input: FeatureUnion<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.pipeline.FeatureUnion.html\">?<span>Documentation for Input: FeatureUnion</span></a></label><div class=\"sk-toggleable__content \"><pre>FeatureUnion(transformer_list=[(&#x27;question&#x27;,\n",
       "                                Pipeline(steps=[(&#x27;selector&#x27;,\n",
       "                                                 FunctionTransformer(func=&lt;function column_selector at 0x1122391c0&gt;,\n",
       "                                                                     kw_args={&#x27;column_name&#x27;: &#x27;context&#x27;})),\n",
       "                                                (&#x27;vectorizer&#x27;,\n",
       "                                                 Vectorizer(analyzer=&#x27;word&#x27;,\n",
       "                                                            binary=False,\n",
       "                                                            decode_error=&#x27;strict&#x27;,\n",
       "                                                            dtype=&lt;class &#x27;numpy.float64&#x27;&gt;,\n",
       "                                                            encoding=&#x27;utf-8&#x27;,\n",
       "                                                            env=&#x27;scikit-learn&#x27;,\n",
       "                                                            input=&#x27;content&#x27;,\n",
       "                                                            lowercase=True,\n",
       "                                                            max_df=...\n",
       "                                                            dtype=&lt;class &#x27;numpy.float64&#x27;&gt;,\n",
       "                                                            encoding=&#x27;utf-8&#x27;,\n",
       "                                                            env=&#x27;scikit-learn&#x27;,\n",
       "                                                            input=&#x27;content&#x27;,\n",
       "                                                            lowercase=True,\n",
       "                                                            max_df=1.0,\n",
       "                                                            max_features=None,\n",
       "                                                            min_df=1,\n",
       "                                                            model=&#x27;TfidfVectorizer&#x27;,\n",
       "                                                            ngram_range=(1, 1),\n",
       "                                                            norm=&#x27;l2&#x27;,\n",
       "                                                            preprocessor=None,\n",
       "                                                            smooth_idf=True,\n",
       "                                                            stop_words=None,\n",
       "                                                            strip_accents=None,\n",
       "                                                            sublinear_tf=False,\n",
       "                                                            token_pattern=&#x27;(?u)\\\\b\\\\w\\\\w+\\\\b&#x27;,\n",
       "                                                            tokenizer=None,\n",
       "                                                            use_idf=True,\n",
       "                                                            vocabulary=None))]))])</pre></div> </div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><label>question</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;FunctionTransformer<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.preprocessing.FunctionTransformer.html\">?<span>Documentation for FunctionTransformer</span></a></label><div class=\"sk-toggleable__content \"><pre>FunctionTransformer(func=&lt;function column_selector at 0x1122391c0&gt;,\n",
       "                    kw_args={&#x27;column_name&#x27;: &#x27;context&#x27;})</pre></div> </div></div><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">Vectorizer</label><div class=\"sk-toggleable__content \"><pre>Vectorizer(analyzer=&#x27;word&#x27;, binary=False, decode_error=&#x27;strict&#x27;,\n",
       "           dtype=&lt;class &#x27;numpy.float64&#x27;&gt;, encoding=&#x27;utf-8&#x27;, env=&#x27;scikit-learn&#x27;,\n",
       "           input=&#x27;content&#x27;, lowercase=True, max_df=1.0, max_features=None,\n",
       "           min_df=1, model=&#x27;TfidfVectorizer&#x27;, ngram_range=(1, 1), norm=&#x27;l2&#x27;,\n",
       "           preprocessor=None, smooth_idf=True, stop_words=None,\n",
       "           strip_accents=None, sublinear_tf=False,\n",
       "           token_pattern=&#x27;(?u)\\\\b\\\\w\\\\w+\\\\b&#x27;, tokenizer=None, use_idf=True,\n",
       "           vocabulary=None)</pre></div> </div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label  sk-toggleable\"><label>answer</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">&nbsp;FunctionTransformer<a class=\"sk-estimator-doc-link \" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.preprocessing.FunctionTransformer.html\">?<span>Documentation for FunctionTransformer</span></a></label><div class=\"sk-toggleable__content \"><pre>FunctionTransformer(func=&lt;function column_selector at 0x1122391c0&gt;,\n",
       "                    kw_args={&#x27;column_name&#x27;: &#x27;answer&#x27;})</pre></div> </div></div><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">Vectorizer</label><div class=\"sk-toggleable__content \"><pre>Vectorizer(analyzer=&#x27;word&#x27;, binary=False, decode_error=&#x27;strict&#x27;,\n",
       "           dtype=&lt;class &#x27;numpy.float64&#x27;&gt;, encoding=&#x27;utf-8&#x27;, env=&#x27;scikit-learn&#x27;,\n",
       "           input=&#x27;content&#x27;, lowercase=True, max_df=1.0, max_features=None,\n",
       "           min_df=1, model=&#x27;TfidfVectorizer&#x27;, ngram_range=(1, 1), norm=&#x27;l2&#x27;,\n",
       "           preprocessor=None, smooth_idf=True, stop_words=None,\n",
       "           strip_accents=None, sublinear_tf=False,\n",
       "           token_pattern=&#x27;(?u)\\\\b\\\\w\\\\w+\\\\b&#x27;, tokenizer=None, use_idf=True,\n",
       "           vocabulary=None)</pre></div> </div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">Classifier</label><div class=\"sk-toggleable__content \"><pre>Classifier(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "           intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "           model=&#x27;LogisticRegression&#x27;, multi_class=&#x27;deprecated&#x27;, n_jobs=None,\n",
       "           penalty=&#x27;l2&#x27;, random_state=None, solver=&#x27;lbfgs&#x27;, tol=0.0001,\n",
       "           verbose=0, warm_start=False)</pre></div> </div></div><div class=\"sk-item\"><div class=\"sk-estimator  sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label  sk-toggleable__label-arrow \">BinaryThresholder</label><div class=\"sk-toggleable__content \"><pre>BinaryThresholder(threshold=np.float64(0.3052255080473171),\n",
       "                  threshold_range=(np.float64(0.015664853903903313),\n",
       "                                   np.float64(0.858797346850902)))</pre></div> </div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('Input',\n",
       "                 FeatureUnion(transformer_list=[('question',\n",
       "                                                 Pipeline(steps=[('selector',\n",
       "                                                                  FunctionTransformer(func=<function column_selector at 0x1122391c0>,\n",
       "                                                                                      kw_args={'column_name': 'context'})),\n",
       "                                                                 ('vectorizer',\n",
       "                                                                  Vectorizer(analyzer='word',\n",
       "                                                                             binary=False,\n",
       "                                                                             decode_error='strict',\n",
       "                                                                             dtype=<class 'numpy.float64'>,\n",
       "                                                                             encoding='utf-8',\n",
       "                                                                             env='scikit-learn',\n",
       "                                                                             input='conten...\n",
       "                            fit_intercept=True, intercept_scaling=1,\n",
       "                            l1_ratio=None, max_iter=100,\n",
       "                            model='LogisticRegression',\n",
       "                            multi_class='deprecated', n_jobs=None, penalty='l2',\n",
       "                            random_state=None, solver='lbfgs', tol=0.0001,\n",
       "                            verbose=0, warm_start=False)),\n",
       "                ('Threshold',\n",
       "                 BinaryThresholder(threshold=np.float64(0.3052255080473171),\n",
       "                                   threshold_range=(np.float64(0.015664853903903313),\n",
       "                                                    np.float64(0.858797346850902))))])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a model\n",
    "model = iqualnlp.Model()\n",
    "\n",
    "# Add text features. Define upto 2 text columns, usually a context/question and another for response/answer.\n",
    "model.add_text_features(q_col='context', \n",
    "                        a_col='answer',\n",
    "                        model='TfidfVectorizer',\n",
    "                        env='scikit-learn',\n",
    "                       )\n",
    "\n",
    "# Add a classifier (Defaults to Logistic Regression)\n",
    "model.add_classifier(name='LogisticRegression')\n",
    "\n",
    "# Add a threshold component that will optimize for F1 score\n",
    "model.add_threshold(scoring_metric='f1')\n",
    "\n",
    "# Compile the model to finalize the pipeline structure\n",
    "model.compile()\n",
    "\n",
    "# Train the model on our training data\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "print(\"Model fit complete!\")\n",
    "\n",
    "model.model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make predictions on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)\n",
    "y_prob = model.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate performance metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance Metrics (Out-Sample):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1 Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.688226</td>\n",
       "      <td>0.38809</td>\n",
       "      <td>0.372047</td>\n",
       "      <td>0.379899</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Accuracy  Precision    Recall  F1 Score\n",
       "0  0.688226    0.38809  0.372047  0.379899"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "from sklearn.metrics import (\n",
    "            f1_score, \n",
    "            accuracy_score,\n",
    "            precision_score, \n",
    "            recall_score,\n",
    ")\n",
    "\n",
    "\n",
    "# Individual metrics\n",
    "metrics = {\n",
    "    'Accuracy': accuracy_score(y_test, y_pred),\n",
    "    'Precision': precision_score(y_test,y_pred),\n",
    "    'Recall': recall_score(y_test, y_pred),\n",
    "    'F1 Score': f1_score(y_test, y_pred)\n",
    "}\n",
    "\n",
    "# Display metrics as a dataframe\n",
    "metric_df = pd.DataFrame([metrics])\n",
    "print(\"Performance Metrics (Out-Sample):\")\n",
    "metric_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## See results dataframe with examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correct Predictions (True Positives):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Context</th>\n",
       "      <th>Answer</th>\n",
       "      <th>True Label</th>\n",
       "      <th>Predicted Label</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3131</th>\n",
       "      <td>Stack Exchange discussion:</td>\n",
       "      <td>Do you have any code we can look at? What have...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.441729</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6440</th>\n",
       "      <td>Stack Exchange discussion:</td>\n",
       "      <td>Christian, could you please provide a referenc...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.352674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>621</th>\n",
       "      <td>Can you please post some code.</td>\n",
       "      <td>How is data received at the client side?</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.500905</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Context  \\\n",
       "3131      Stack Exchange discussion:   \n",
       "6440      Stack Exchange discussion:   \n",
       "621   Can you please post some code.   \n",
       "\n",
       "                                                 Answer  True Label  \\\n",
       "3131  Do you have any code we can look at? What have...           1   \n",
       "6440  Christian, could you please provide a referenc...           1   \n",
       "621            How is data received at the client side?           1   \n",
       "\n",
       "      Predicted Label  Probability  \n",
       "3131                1     0.441729  \n",
       "6440                1     0.352674  \n",
       "621                 1     0.500905  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = pd.DataFrame({\n",
    "    'Context': X_test['context'],\n",
    "    'Answer': X_test['answer'],\n",
    "    'True Label': y_test,\n",
    "    'Predicted Label': y_pred,\n",
    "    'Probability': y_prob\n",
    "})\n",
    "\n",
    "# Display some examples of correct and incorrect predictions\n",
    "print(\"Correct Predictions (True Positives):\")\n",
    "results[(results['True Label'] == 1) & (results['Predicted Label'] == 1)].head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Incorrect Predictions (False Negatives):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Context</th>\n",
       "      <th>Answer</th>\n",
       "      <th>True Label</th>\n",
       "      <th>Predicted Label</th>\n",
       "      <th>Probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3113</th>\n",
       "      <td>Stack Exchange discussion:</td>\n",
       "      <td>Is it always the same for every file, or does ...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.239120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>570</th>\n",
       "      <td>What would you like in the case of, e.</td>\n",
       "      <td>g., `http://example.com/test.php?key=val` ? Or...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.223762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3029</th>\n",
       "      <td>Stack Exchange discussion:</td>\n",
       "      <td>Do the repeats have to be contiguous? is 5,1,4...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.273251</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     Context  \\\n",
       "3113              Stack Exchange discussion:   \n",
       "570   What would you like in the case of, e.   \n",
       "3029              Stack Exchange discussion:   \n",
       "\n",
       "                                                 Answer  True Label  \\\n",
       "3113  Is it always the same for every file, or does ...           1   \n",
       "570   g., `http://example.com/test.php?key=val` ? Or...           1   \n",
       "3029  Do the repeats have to be contiguous? is 5,1,4...           1   \n",
       "\n",
       "      Predicted Label  Probability  \n",
       "3113                0     0.239120  \n",
       "570                 0     0.223762  \n",
       "3029                0     0.273251  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Incorrect Predictions (False Negatives):\")\n",
    "results[(results['True Label'] == 1) & (results['Predicted Label'] == 0)].head(3)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}