Spit_Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split dataset of politician's photos\n",
    "\n",
    "**Note: all folders are empty in order to respect the privacy of the politicians! There is no public dataset or final model. I used izquierda (left) for PSOE and derecha (right) for PP.**\n",
    "\n",
    "Use `izquierda` and `derecha` from `cropped` to copy files into `train` and `validation` from `data_politics`. This folder should already exists and have both subfolders (or ajust the script to create them). The current **data_politics** folder will be the dataset folder for the next deep learning classifications.\n",
    "\n",
    "Therefore, we shall use a dataset split percentage such as **80% train** and **20% test**.\n",
    "\n",
    "We need `os` and `shutil` to manage the files, `random` to randomly split the dataset in train and validation subsets. You should have a folder structure such as:\n",
    "\n",
    "* `./data_politics/train/izquierda`\n",
    "* `./data_politics/train/derecha`\n",
    "* `./data_politics/validation/izquierda`\n",
    "* `./data_politics/validation/derecha`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These lines are the locations for the source files (the entire dataset) and the future locations of the splitted dataset in train and validation subsets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Source dataset: from where to copy the files\n",
    "sourceFolderClass1 = 'cropped/izquierda'\n",
    "sourceFolderClass2 = 'cropped/derecha'\n",
    "# Destination folders: splitted dataset in train and validation for polyps and non-polyps\n",
    "destFolderClass1_tr  = 'data_politics/train/izquierda'\n",
    "destFolderClass2_tr  = 'data_politics/train/derecha'\n",
    "destFolderClass1_val = 'data_politics/validation/izquierda'\n",
    "destFolderClass2_val = 'data_politics/validation/derecha'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the list with all the files in the source folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class 1 - izquierda: 50\n",
      "Class 2 - derecha: 50\n"
     ]
    }
   ],
   "source": [
    "sourceFiles1 = os.listdir(sourceFolderClass1)\n",
    "sourceFiles2 = os.listdir(sourceFolderClass2)\n",
    "print(\"Class 1 - izquierda:\", len(sourceFiles1))\n",
    "print(\"Class 2 - derecha:\", len(sourceFiles2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's suffle the lists with the source files using a random seed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(1)\n",
    "random.shuffle(sourceFiles1)\n",
    "random.shuffle(sourceFiles2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We shall define a number of files to copy in the `validation` subfolder for each class. If you want a different split, you should modify `val_files`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Validation split ...\n",
      "--> Done!\n"
     ]
    }
   ],
   "source": [
    "# No of file to copy in VALIDATION folder for each class\n",
    "val_files = 10\n",
    "\n",
    "# Copy validation files\n",
    "print('--> Validation split ...')\n",
    "for i in range(val_files):\n",
    "    # copy validation files for PSOE\n",
    "    File1 = os.path.join(sourceFolderClass1,sourceFiles1[i])\n",
    "    File2 = os.path.join(destFolderClass1_val,  sourceFiles1[i])\n",
    "    shutil.copy(File1,File2)\n",
    "    # copy validation files for PP\n",
    "    File1 = os.path.join(sourceFolderClass2, sourceFiles2[i])\n",
    "    File2 = os.path.join(destFolderClass2_val,   sourceFiles2[i])\n",
    "    shutil.copy(File1, File2)\n",
    "\n",
    "print('--> Done!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Train split ...\n",
      "--> Done!\n"
     ]
    }
   ],
   "source": [
    "# Copy training images for PSOE\n",
    "print('--> Train split ...')\n",
    "for i in range(val_files,len(sourceFiles1)):\n",
    "    File1 = os.path.join(sourceFolderClass1,  sourceFiles1[i])\n",
    "    File2 = os.path.join(destFolderClass1_tr, sourceFiles1[i])\n",
    "    shutil.copy(File1,File2)\n",
    "# copy training images for PP\n",
    "for i in range(val_files,len(sourceFiles2)):    \n",
    "    File1 = os.path.join(sourceFolderClass2,  sourceFiles2[i])\n",
    "    File2 = os.path.join(destFolderClass2_tr, sourceFiles2[i])\n",
    "    shutil.copy(File1, File2)\n",
    "\n",
    "print('--> Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a splitted dataset into train and validation subfolder with each class inside:\n",
    "* **100** images in the entire dataset;\n",
    "* **80** images for training: 40 PSOE + 40 PP;\n",
    "* **20** images for validation: 10 PSOE + 10 PP.\n",
    "\n",
    "Let's check the composition of the subsets for the future classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Dataset: data_politics\n",
      "> Train - izquierda: 40\n",
      "> Train - derecha: 40\n",
      "> Validation - izquierda: 10\n",
      "> Validation - derecha: 10\n"
     ]
    }
   ],
   "source": [
    "print('--> Dataset: data_politics')\n",
    "print('> Train - izquierda:', len(os.listdir(destFolderClass1_tr)))\n",
    "print('> Train - derecha:', len(os.listdir(destFolderClass2_tr)))\n",
    "print('> Validation - izquierda:', len(os.listdir(destFolderClass1_val)))\n",
    "print('> Validation - derecha:', len(os.listdir(destFolderClass2_val)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to use Deep Learning to find a classifier for PSOE - PP political affinity. Remember you could modify the dataset splitting, remove manually a specific list of files, use different names for the folders, etc.\n",
    "\n",
    "Let's create some classifiers with the next script [Small_CNNs.ipynb](./Small_CNNs.ipynb).\n",
    "\n",
    "Have fun with DL! @muntisa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acknowledgements\n",
    "\n",
    "I gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research ([https://developer.nvidia.com/academic_gpu_seeding](https://developer.nvidia.com/academic_gpu_seeding))."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}