fanfiction_data_parsing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a045ff60",
   "metadata": {},
   "source": [
    "# Data Processing and Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc83c83b",
   "metadata": {},
   "source": [
    "I have already downloaded slightly over 10000 fanfiction works from https://archiveofourown.org/media using a separate python script (this script is included separately in the upload and is linked to in my markdown file), and I will process that data here using the python library BeautifulSoup. Data collection with scrapy takes time, and it took me over a week to process my full dataset, but it's all here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac84f648",
   "metadata": {},
   "source": [
    "### Table of Contents\n",
    "- [Reading data into a DataFrame](#Reading-data-into-a-DataFrame)\n",
    "- [Looking at Statistics](#Looking-at-Statistics)\n",
    "- [Analysis](#Analysis)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b7d3025",
   "metadata": {},
   "source": [
    "## Reading data into a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b3a152d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import os\n",
    "import glob\n",
    "import lxml\n",
    "\n",
    "file_location = os.path.join('C:/Users/rsbru/OneDrive/Documents/data_science/Fanfiction-Classification-Analysis/data/', '*.html')\n",
    "filenames = glob.glob(file_location)\n",
    "files = {}\n",
    "for f in filenames:\n",
    "    with open(f, encoding = 'utf8') as fp:\n",
    "        soup = BeautifulSoup(fp, 'lxml')\n",
    "        file_name = f[87:]\n",
    "        files[file_name] = soup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28a2616f",
   "metadata": {},
   "source": [
    "Now that I have my files read into python as searchable html files, I can use BeautifulSoup's search operations to pull out the parts of the document that are relevant to my project (namely the text itself and the tags attached to the work)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "50210e05",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "file_names = [f[87:] for f in filenames]\n",
    "ratings = {}\n",
    "rat_dict = {}\n",
    "warnings = {}\n",
    "warn_dict = {}\n",
    "warns_list = []\n",
    "categories = {}\n",
    "cat_dict = {}\n",
    "cats_list = []\n",
    "fandoms = {}\n",
    "fan_dict = {}\n",
    "fans_list = []\n",
    "relationships = {}\n",
    "rel_dict = {}\n",
    "rels_list = []\n",
    "characters = {}\n",
    "char_dict = {}\n",
    "chars_list = []\n",
    "additional = {}\n",
    "extra_dict = {}\n",
    "extra_list = []\n",
    "texts = {}\n",
    "for fn in file_names:\n",
    "    if(files[fn].find('dd', class_=\"rating tags\") is not None):\n",
    "        rats = []\n",
    "        for child in files[fn].find('dd', class_=\"rating tags\").find_all('a'):\n",
    "            for string in child.stripped_strings:\n",
    "                rats.append(string)\n",
    "        ratings[fn] = ', '.join(rats)\n",
    "        rat_dict[fn] = rats\n",
    "    if(files[fn].find('dd', class_=\"warning tags\") is not None):\n",
    "        warns = []\n",
    "        for child in files[fn].find('dd', class_=\"warning tags\").find_all('a'):\n",
    "            for string in child.stripped_strings:\n",
    "                warns.append(string)\n",
    "        warnings[fn] = ', '.join(warns)\n",
    "        warn_dict[fn] = warns\n",
    "        for item in warn_dict[fn]:\n",
    "            warns_list.append(item)\n",
    "    if(files[fn].find('dd', class_=\"category tags\") is not None):\n",
    "        cats = []\n",
    "        for child in files[fn].find('dd', class_=\"category tags\").find_all('a'):\n",
    "            for string in child.stripped_strings:\n",
    "                cats.append(string)\n",
    "        categories[fn] = ', '.join(cats)\n",
    "        cat_dict[fn] = cats\n",
    "        for item in cat_dict[fn]:\n",
    "            cats_list.append(item)\n",
    "    if(files[fn].find('dd', class_=\"fandom tags\") is not None):\n",
    "        fans = []\n",
    "        for child in files[fn].find('dd', class_=\"fandom tags\").find_all('a'):\n",
    "            for string in child.stripped_strings:\n",
    "                fans.append(string)\n",
    "        fandoms[fn] = ', '.join(fans)\n",
    "        fan_dict[fn] = fans\n",
    "        for item in fan_dict[fn]:\n",
    "            fans_list.append(item)\n",
    "    if(files[fn].find('dd', class_=\"relationship tags\") is not None):\n",
    "        rels = []\n",
    "        for child in files[fn].find('dd', class_=\"relationship tags\").find_all('a'):\n",
    "            for string in child.stripped_strings:\n",
    "                rels.append(string)\n",
    "        relationships[fn] = ', '.join(rels)\n",
    "        rel_dict[fn] = rels\n",
    "        for item in rel_dict[fn]:\n",
    "            rels_list.append(item)\n",
    "    if(files[fn].find('dd', class_=\"character tags\") is not None):\n",
    "        chars = []\n",
    "        for child in files[fn].find('dd', class_=\"character tags\").find_all('a'):\n",
    "            for string in child.stripped_strings:\n",
    "                chars.append(string)\n",
    "        characters[fn] = ', '.join(chars)\n",
    "        char_dict[fn] = chars\n",
    "        for item in char_dict[fn]:\n",
    "            chars_list.append(item)\n",
    "    if(files[fn].find('dd', class_=\"freeform tags\") is not None):\n",
    "        adds = []\n",
    "        for child in files[fn].find('dd', class_=\"freeform tags\").find_all('a'):\n",
    "            for string in child.stripped_strings:\n",
    "                adds.append(string)\n",
    "        additional[fn] = ', '.join(adds)\n",
    "        extra_dict[fn] = adds\n",
    "        for item in extra_dict[fn]:\n",
    "            extra_list.append(item)\n",
    "    tex = files[fn].find_all('p')\n",
    "    pars = []\n",
    "    for text in tex:\n",
    "        for string in text.stripped_strings:\n",
    "            pars.append(string)\n",
    "    texts[fn] = '\\n'.join(pars)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6c6d63b",
   "metadata": {},
   "source": [
    "I will now organize the data I've pulled out of my raw files into a DataFrame using pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ed770669",
   "metadata": {},
   "outputs": [],
   "source": [
    "fanfics = pd.DataFrame({'Filename': file_names})\n",
    "fanfics['Rating'] = fanfics['Filename'].map(ratings)\n",
    "fanfics['Warning'] = fanfics['Filename'].map(warnings)\n",
    "fanfics['Category'] = fanfics['Filename'].map(categories)\n",
    "fanfics['Fandom'] = fanfics['Filename'].map(fandoms)\n",
    "fanfics['Relationships'] = fanfics['Filename'].map(relationships)\n",
    "fanfics['Characters'] = fanfics['Filename'].map(characters)\n",
    "fanfics['Additional'] = fanfics['Filename'].map(additional)\n",
    "fanfics['Text'] = fanfics['Filename'].map(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f40c5ec4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Filename</th>\n",
       "      <th>Rating</th>\n",
       "      <th>Warning</th>\n",
       "      <th>Category</th>\n",
       "      <th>Fandom</th>\n",
       "      <th>Relationships</th>\n",
       "      <th>Characters</th>\n",
       "      <th>Additional</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1005380.html</td>\n",
       "      <td>Explicit</td>\n",
       "      <td>Creator Chose Not To Use Archive Warnings</td>\n",
       "      <td>M/M</td>\n",
       "      <td>Harry Potter - J. K. Rowling</td>\n",
       "      <td>Sirius Black/Remus Lupin, James/Lily, Peter Pe...</td>\n",
       "      <td>Remus Lupin, Sirius Black, James Potter, Lily ...</td>\n",
       "      <td>Humor, Angst, First War with Voldemort</td>\n",
       "      <td>While we've done our best to make the core fun...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10057010.html</td>\n",
       "      <td>Mature</td>\n",
       "      <td>No Archive Warnings Apply, Major Character Dea...</td>\n",
       "      <td>M/M</td>\n",
       "      <td>Harry Potter - J. K. Rowling</td>\n",
       "      <td>Sirius Black/Remus Lupin, Sirius Black &amp; Remus...</td>\n",
       "      <td>Remus Lupin, Sirius Black, James Potter, Lily ...</td>\n",
       "      <td>Marauders' Era, Marauders, Marauders Friendshi...</td>\n",
       "      <td>While we've done our best to make the core fun...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10074443.html</td>\n",
       "      <td>Explicit</td>\n",
       "      <td>No Archive Warnings Apply</td>\n",
       "      <td>M/M</td>\n",
       "      <td>Batman (Movies - Nolan), Dark Knight Rises - F...</td>\n",
       "      <td>Bane (DCU)/John Blake</td>\n",
       "      <td>Bane (DCU), John Blake</td>\n",
       "      <td>nightwing!blake, dub-con, Fanart, molesting ag...</td>\n",
       "      <td>While we've done our best to make the core fun...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1008747.html</td>\n",
       "      <td>Explicit</td>\n",
       "      <td>Underage</td>\n",
       "      <td>M/M</td>\n",
       "      <td>Harry Potter - J. K. Rowling</td>\n",
       "      <td>Harry Potter/Voldemort, Harry Potter/Tom Riddl...</td>\n",
       "      <td>Harry Potter, Voldemort, Tom Riddle | Voldemor...</td>\n",
       "      <td>Angst, First Time, Work In Progress, Alternate...</td>\n",
       "      <td>While we've done our best to make the core fun...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10159223.html</td>\n",
       "      <td>Explicit</td>\n",
       "      <td>Rape/Non-Con</td>\n",
       "      <td>F/M, Multi, F/F</td>\n",
       "      <td>Spider-Man (Comicverse)</td>\n",
       "      <td>Peter Parker/Calypso, Betty Brant/Peter Parker...</td>\n",
       "      <td>Peter Parker, Calypso, Betty Brant, Felicia Ha...</td>\n",
       "      <td>Adultery, Femdom, Seduction, Office Sex, Three...</td>\n",
       "      <td>While we've done our best to make the core fun...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Filename    Rating                                            Warning  \\\n",
       "0   1005380.html  Explicit          Creator Chose Not To Use Archive Warnings   \n",
       "1  10057010.html    Mature  No Archive Warnings Apply, Major Character Dea...   \n",
       "2  10074443.html  Explicit                          No Archive Warnings Apply   \n",
       "3   1008747.html  Explicit                                           Underage   \n",
       "4  10159223.html  Explicit                                       Rape/Non-Con   \n",
       "\n",
       "          Category                                             Fandom  \\\n",
       "0              M/M                       Harry Potter - J. K. Rowling   \n",
       "1              M/M                       Harry Potter - J. K. Rowling   \n",
       "2              M/M  Batman (Movies - Nolan), Dark Knight Rises - F...   \n",
       "3              M/M                       Harry Potter - J. K. Rowling   \n",
       "4  F/M, Multi, F/F                            Spider-Man (Comicverse)   \n",
       "\n",
       "                                       Relationships  \\\n",
       "0  Sirius Black/Remus Lupin, James/Lily, Peter Pe...   \n",
       "1  Sirius Black/Remus Lupin, Sirius Black & Remus...   \n",
       "2                              Bane (DCU)/John Blake   \n",
       "3  Harry Potter/Voldemort, Harry Potter/Tom Riddl...   \n",
       "4  Peter Parker/Calypso, Betty Brant/Peter Parker...   \n",
       "\n",
       "                                          Characters  \\\n",
       "0  Remus Lupin, Sirius Black, James Potter, Lily ...   \n",
       "1  Remus Lupin, Sirius Black, James Potter, Lily ...   \n",
       "2                             Bane (DCU), John Blake   \n",
       "3  Harry Potter, Voldemort, Tom Riddle | Voldemor...   \n",
       "4  Peter Parker, Calypso, Betty Brant, Felicia Ha...   \n",
       "\n",
       "                                          Additional  \\\n",
       "0             Humor, Angst, First War with Voldemort   \n",
       "1  Marauders' Era, Marauders, Marauders Friendshi...   \n",
       "2  nightwing!blake, dub-con, Fanart, molesting ag...   \n",
       "3  Angst, First Time, Work In Progress, Alternate...   \n",
       "4  Adultery, Femdom, Seduction, Office Sex, Three...   \n",
       "\n",
       "                                                Text  \n",
       "0  While we've done our best to make the core fun...  \n",
       "1  While we've done our best to make the core fun...  \n",
       "2  While we've done our best to make the core fun...  \n",
       "3  While we've done our best to make the core fun...  \n",
       "4  While we've done our best to make the core fun...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fanfics.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ce37a8b",
   "metadata": {},
   "source": [
    "## Looking at Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44653964",
   "metadata": {},
   "source": [
    "Just for fun here (and to make sure my DataFrame is functional), let's look at the most common fandoms in my (limited) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f8fb6cde",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pretty printing has been turned OFF\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Harry Potter - J. K. Rowling                                                                                                                                                                                                                                                              617\n",
       "僕のヒーローアカデミア | Boku no Hero Academia | My Hero Academia                                                                                                                                                                                                                                    459\n",
       "原神 | Genshin Impact (Video Game)                                                                                                                                                                                                                                                          197\n",
       "Naruto                                                                                                                                                                                                                                                                                    193\n",
       "Encanto (2021)                                                                                                                                                                                                                                                                            179\n",
       "                                                                                                                                                                                                                                                                                         ... \n",
       "Percy Jackson and the Olympians - Rick Riordan, Percy Jackson and the Olympians & Related Fandoms - All Media Types, The Heroes of Olympus - Rick Riordan, The Kane Chronicles - Rick Riordan, Magnus Chase and the Gods of Asgard - Rick Riordan, The Trials of Apollo - Rick Riordan      1\n",
       "SuperM (Korea Band), SHINee, EXO (Band)                                                                                                                                                                                                                                                     1\n",
       "Rookie Blue, Rizzoli & Isles                                                                                                                                                                                                                                                                1\n",
       "Monster (Manga)                                                                                                                                                                                                                                                                             1\n",
       "The Silmarillion and other histories of Middle-Earth - J. R. R. Tolkien, Silmarillion, Tolkien - Fandom, jrr tolkien - Fandom                                                                                                                                                               1\n",
       "Name: Fandom, Length: 2696, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%pprint\n",
    "fanfics.Fandom.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "282792bb",
   "metadata": {},
   "source": [
    "I did all of the above work for my first progress report. Everything below this is my finalized data cleaning and organization for my second progress report. First, I'm going to look at the value counts for every label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2af1eb73",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Explicit                 3234\n",
       "Teen And Up Audiences    2327\n",
       "Mature                   2313\n",
       "Not Rated                1051\n",
       "General Audiences         837\n",
       "Name: Rating, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fanfics.Rating.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "96c91c73",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Creator Chose Not To Use Archive Warnings                                                                                                              3187\n",
       "No Archive Warnings Apply                                                                                                                              3065\n",
       "Graphic Depictions Of Violence                                                                                                                         1127\n",
       "Graphic Depictions Of Violence, Major Character Death                                                                                                   394\n",
       "Underage                                                                                                                                                222\n",
       "                                                                                                                                                       ... \n",
       "Graphic Depictions Of Violence, Major Character Death, Underage, No Archive Warnings Apply                                                                1\n",
       "Creator Chose Not To Use Archive Warnings, Major Character Death, No Archive Warnings Apply                                                               1\n",
       "Graphic Depictions Of Violence, Major Character Death, Underage, Rape/Non-Con                                                                             1\n",
       "Creator Chose Not To Use Archive Warnings, Graphic Depictions Of Violence, No Archive Warnings Apply, Rape/Non-Con, Underage                              1\n",
       "Creator Chose Not To Use Archive Warnings, Rape/Non-Con, Graphic Depictions Of Violence, Major Character Death, No Archive Warnings Apply, Underage       1\n",
       "Name: Warning, Length: 96, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fanfics.Warning.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8acf3992",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "M/M                            2532\n",
       "F/M                            1840\n",
       "Gen                             862\n",
       "F/F                             628\n",
       "Multi                           345\n",
       "                               ... \n",
       "F/F, Gen, Multi, Other            1\n",
       "M/M, F/F, Multi                   1\n",
       "F/F, Gen, M/M, Other              1\n",
       "F/F, F/M, Multi, Gen              1\n",
       "M/M, Multi, Other, F/F, F/M       1\n",
       "Name: Category, Length: 178, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fanfics.Category.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "667e018e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Hermione Granger/Draco Malfoy                                                                                                                                                                                                                                                                                                                           33\n",
       "Fezco/Lexi Howard                                                                                                                                                                                                                                                                                                                                       24\n",
       "Draco Malfoy/Harry Potter                                                                                                                                                                                                                                                                                                                               22\n",
       "Lán Zhàn | Lán Wàngjī/Wèi Yīng | Wèi Wúxiàn                                                                                                                                                                                                                                                                                                             21\n",
       "Derek Hale/Stiles Stilinski                                                                                                                                                                                                                                                                                                                             21\n",
       "                                                                                                                                                                                                                                                                                                                                                        ..\n",
       "Aizawa Shouta | Eraserhead & Midoriya Izuku, Akaguro Chizome | Stain & Midoriya Izuku                                                                                                                                                                                                                                                                    1\n",
       "James Potter/Lily Evans Potter, Arthur Weasley/Molly Weasley, Dudley Dursley & Harry Potter, Luna Lovegood/Pansy Parkinson, Petunia Evans Dursley/Vernon Dursley, Vernon Dursley & Harry Potter, Petunia Evans Dursley & Minerva McGonagall, Harry Potter/Percy Weasley, Marcus Flint/Harry Potter, Percy Weasley/Harry Potter/Marcus Flint              1\n",
       "Blaine Anderson/Kurt Hummel, Blaine Anderson/Sebastian Smythe, Other Relationship Tags to Be Added                                                                                                                                                                                                                                                       1\n",
       "Kotaki Nozomu/Shigeoka Daiki, Fujii Ryusei/Shigeoka Daiki, Fujii Ryusei/Kotaki Nozomu                                                                                                                                                                                                                                                                    1\n",
       "Ranboo/Toby Smith | Tubbo, Ranboo/TommyInnit (Video Blogging RPF), Ranboo/Wilbur Soot, Cuptoast | Crumb/Ranboo (Video Blogging RPF), Aimee | Aimsey/Ranboo (Video Blogging RPF), Wilbur Soot/TommyInnit, Toby Smith | Tubbo/TommyInnit, Toby Smith | Tubbo/Technoblade, Grayson | Purpled/Luke | Punz, Grayson | Purpled/Ranboo (Video Blogging RPF)     1\n",
       "Name: Relationships, Length: 8150, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fanfics.Relationships.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "934299eb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Caitlyn (League of Legends), Vi (League of Legends)                                                                                                                                                                                                                                                                                            8\n",
       "Tartaglia | Childe (Genshin Impact), Zhongli (Genshin Impact)                                                                                                                                                                                                                                                                                  8\n",
       "Original Characters                                                                                                                                                                                                                                                                                                                            7\n",
       "Clay | Dream (Video Blogging RPF), GeorgeNotFound (Video Blogging RPF), Sapnap (Video Blogging RPF)                                                                                                                                                                                                                                            6\n",
       "Sun (Five Nights at Freddy's), Moon (Five Nights at Freddy's)                                                                                                                                                                                                                                                                                  6\n",
       "                                                                                                                                                                                                                                                                                                                                              ..\n",
       "Jason Todd, Selina Kyle, Damian Wayne, Willis Todd, Barbara Gordon, Dick Grayson, Bruce Wayne, Tim Drake, Mentioned Clark Kent - Character, Catherine Elizabeth Todd, Alfred Pennyworth, Kate Kane, Cassandra Cain, Stephanie Brown, Joker (DCU), Sheila Haywood, Duke Thomas, Lucius Fox, Harvey Bullock, Clark Kent, Diana (Wonder Woman)    1\n",
       "Percy Jackson, Nico di Angelo, Jason Grace, Harry Potter, Hermione Granger, Ron Weasley, Neville Longbottom, Luna Lovegood, Ginny Weasley, Rubeus Hagrid, Albus Dumbledore                                                                                                                                                                     1\n",
       "Clint Barton, James \"Bucky\" Barnes, Natasha Romanov (Marvel), Steve Rogers, Tony Stark, Bruce Banner                                                                                                                                                                                                                                           1\n",
       "Megamind (Megamind), Minion (Megamind), Metro Man (Megamind), Roxanne Ritchi, Lord Scott (Megamind), Lady Scott (Megamind), Random Citizens, Psycho Delic (Megamind), Hot Flash (Megamind), Lady Doppler, Destruction Worker, Senior Stiletto, The Warden (Megamind), The Mayor, Megamind's Prison Uncles                                      1\n",
       "Kido Shinji, Ryuga | Kamen Rider Ryuki, Akiyama Ren, Kitaoka Shuuichi, Yura Gorou, Tezuka Miyuki, Shibaura Jun                                                                                                                                                                                                                                 1\n",
       "Name: Characters, Length: 9232, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fanfics.Characters.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "95da8e0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Alternate Universe - Canon Divergence                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          9\n",
       "Smut                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           4\n",
       "Alternate Universe                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             3\n",
       "Drabble Collection                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             3\n",
       "Time Travel                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    2\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              ..\n",
       "SI-OC, Self-Insert, pansy-centric, Pairings undecided, if at all, Deviates From Canon, Slow Build, Hogwarts, Feminism, No - I will not stay in the kitchen, Pure-blood traditions and rituals, Pansy's just trying to live - okay?, imperfect characters                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       1\n",
       "First Dates, Awkwardness, First Kiss, Self Loathing, Fluff, Protective Langris Vaude, langris is bad at feelings, Oblivious Asta (Black Clover), they both are awkward                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         1\n",
       "Non-Traditional Alpha/Beta/Omega Dynamics, Alpha/Beta/Omega Dynamics, Alpha/Omega, Alpha/Alpha, Omega/Omega, Class & Dynamic Exploration, Alpha Dallas Winston, Omega Ponyboy Curtis, omega sodapop curtis, Alpha Darry Curtis, Alpha Paul Holden, Alpha Cherry Valance, Beta Johnny Cade, Soc!Darry Curtis, Soc!Sodapop Curtis, Soc!Johnny Cade, Soc!Ponyboy Curtis, Pastoral, Pastoral Themes, Religious Themes                                                                                                                                                                                                                                                                                                                                                                                                                                              1\n",
       "Angst, Crime Fighting, Ninja, Hurt/Comfort, Amnesia                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            1\n",
       "Dominant Reader, Strap-Ons, Dildos, Anal Sex, Anal Fingering, Riding, Kissing, Dom/sub, Bottom Rengoku Kyoujurou, Finger Sucking, Blow Jobs, Dildo sucking, Gender-Neutral Pronouns, Vampire Sex, Blood Drinking, Knifeplay, Blood Kink, Vibrators, Human/Vampire Relationship, Tattoos, Power Bottom, Femdom, Vaginal Sex, Vaginal Fingering, Breast Sucking, Frottage, Coming on Face, Humiliation, Alternate Universe - College/University, Alternate Universe - Modern Setting, Lesbian Sex, Lesbian Character, Semi-Public Sex, Accidental Voyeurism, Cunnilingus, Friends With Benefits, Spanking, Coming Untouched, Submissive Himejima Gyomei, Tribadism, Clothed Sex, Sharing Clothes, Marking, Biting, Nipple Play, Nipple Licking, Hand Jobs, First Time, Iguro Obanai is Submissive and Breedable, Bottom Uzui Tengen, Phone Sex, Light Bondage    1\n",
       "Name: Additional, Length: 9410, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fanfics.Additional.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3b63f48",
   "metadata": {},
   "source": [
    "What I'm seeing here is that my choice to join tags using commas was a good choice to get them all into a readable DataFrame, but not to see what fics share tags. I don't think I can assign multiple tags to each fic within the DataFrame without making it really ugly and inefficient, so while I will use my DataFrame as a pretty thing that I can print to csv, I will work with regular dictionaries from here on out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "380aba05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Creator Chose Not To Use Archive Warnings', 3734), ('No Archive Warnings Apply', 3422), ('Graphic Depictions Of Violence', 2431), ('Rape/Non-Con', 946), ('Major Character Death', 866), ('Underage', 714)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "warn_freq = nltk.FreqDist(warns_list)\n",
    "warn_freq.most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d2b1c077",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('M/M', 4388), ('F/M', 3888), ('F/F', 1856), ('Multi', 1677), ('Gen', 1659), ('Other', 600)]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cat_freq = nltk.FreqDist(cats_list)\n",
    "cat_freq.most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d3d26d17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Harry Potter - J. K. Rowling', 753), ('僕のヒーローアカデミア | Boku no Hero Academia | My Hero Academia', 575), ('Marvel Cinematic Universe', 304), ('Minecraft (Video Game)', 300), ('Naruto', 241), ('Video Blogging RPF', 232), ('原神 | Genshin Impact (Video Game)', 205), ('Star Wars - All Media Types', 203), ('Encanto (2021)', 194), ('The Avengers (Marvel Movies)', 181), ('방탄소년단 | Bangtan Boys | BTS', 167), (\"Five Nights at Freddy's\", 167), ('Batman - All Media Types', 165), ('Shingeki no Kyojin | Attack on Titan', 132), ('鬼滅の刃 | Demon Slayer: Kimetsu no Yaiba (Anime)', 130), ('Biohazard | Resident Evil (Gameverse)', 129), ('Game of Thrones (TV)', 126), ('A Song of Ice and Fire - George R. R. Martin', 118), ('Miraculous Ladybug', 116), ('Haikyuu!!', 115)]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fan_freq = nltk.FreqDist(fans_list)\n",
    "fan_freq.most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "07e1b961",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Minor or Background Relationship(s)', 150), ('Sirius Black/Remus Lupin', 119), ('Other Relationship Tags to Be Added', 108), ('Aizawa Shouta | Eraserhead/Yamada Hizashi | Present Mic', 100), ('Draco Malfoy/Harry Potter', 95), ('Wilbur Soot & Technoblade & TommyInnit & Phil Watson', 93), ('Aizawa Shouta | Eraserhead & Midoriya Izuku', 89), ('Bakugou Katsuki/Midoriya Izuku', 87), ('Félix Madrigal/Pepa Madrigal', 82), ('Hermione Granger/Draco Malfoy', 75), ('Midoriya Izuku/Todoroki Shouto', 70), ('Lán Zhàn | Lán Wàngjī/Wèi Yīng | Wèi Wúxiàn', 70), ('Wilbur Soot & TommyInnit', 70), ('James Potter/Lily Evans Potter', 69), ('Peter Parker & Tony Stark', 69), ('Bakugou Katsuki & Midoriya Izuku', 66), ('Agustín Madrigal/Julieta Madrigal', 66), ('Technoblade & TommyInnit (Video Blogging RPF)', 63), ('Fezco/Lexi Howard', 63), ('Harry Potter/Ginny Weasley', 59)]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rel_freq = nltk.FreqDist(rels_list)\n",
    "rel_freq.most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4bb0122b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Original Characters', 615), ('Harry Potter', 536), ('Original Female Character(s)', 449), ('Reader', 423), ('Midoriya Izuku', 422), ('Hermione Granger', 365), ('Bakugou Katsuki', 324), ('Original Male Character(s)', 295), ('Draco Malfoy', 294), ('Aizawa Shouta | Eraserhead', 287), ('Ron Weasley', 285), ('Sirius Black', 260), ('Severus Snape', 249), ('Remus Lupin', 237), ('Tony Stark', 227), ('TommyInnit (Video Blogging RPF)', 227), ('Wilbur Soot', 227), ('Todoroki Shouto', 221), ('Technoblade (Video Blogging RPF)', 221), ('Clay | Dream (Video Blogging RPF)', 210)]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_freq = nltk.FreqDist(chars_list)\n",
    "char_freq.most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "af97a63d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Angst', 1700), ('Fluff', 1445), ('Hurt/Comfort', 1349), ('Slow Burn', 1142), ('Alternate Universe - Canon Divergence', 1139), ('Other Additional Tags to Be Added', 934), ('Smut', 929), ('Fluff and Angst', 836), ('Anal Sex', 770), ('Oral Sex', 675), ('Romance', 613), ('Angst with a Happy Ending', 603), ('Eventual Smut', 563), ('Canon-Typical Violence', 538), ('Alpha/Beta/Omega Dynamics', 514), ('Alternate Universe', 498), ('Rough Sex', 484), ('Emotional Hurt/Comfort', 471), ('Explicit Sexual Content', 465), ('Fluff and Smut', 460)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extra_freq = nltk.FreqDist(extra_list)\n",
    "extra_freq.most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e217543",
   "metadata": {},
   "source": [
    "I will continue to update this script as I move on to analysis with the (many) data structures I now have at my disposal. For now, I have a much better idea of the shape of my data, and I'm much more comfortable moving forward to analysis and classification."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4188c0b",
   "metadata": {},
   "source": [
    "# Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38009caf",
   "metadata": {},
   "source": [
    "Getting into this analysis, this is going to be more complicated than the analysis I've done before on other datasets because if I've learned anything from delving into my data, it's that these fanfictions are not easily slotted into one category. Because of that, I'm going to move straight to some unsupervised machine learning on the CRC. It's possible that this dataset just isn't very well-suited to traditional classification, but even if that's the case, I'd like to explore clustering. For now, I'm going to pickle my DataFrame using pandas so that I can work with it in other locations. This .pkl file will NOT be uploaded to my remote repository, as I've already placed my data in its final form for sharing. This file is solely for my personal use, and anyone attempting to replicate my work can create their own file with their own DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f88db982",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.to_pickle(fanfics, \"fanfiction_df.pkl\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}