04_prep_imported_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Author\n",
    "[Koen Aerts](https://koenaerts.ca/) @ [Mobia Technology Innovations](https://mobia.io)\n",
    "\n",
    "[myOpenHealth](https://mobia.io/healthcare/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "This notebook reads all the CSV files that were generated by notebook \"02_import_mitdb_data\" and combines them all in 3 new CSV files that can be used for training your model. More specifically the following is done:\n",
    "* Read all CSV files generated by the \"02_import_mitdb_data\" notebook.\n",
    "* Shuffle all records randomly.\n",
    "* Save 60% of the records in a training CSV file.\n",
    "* Save 20% of the records in a validation CSV file.\n",
    "* Save 20% of the records in a test CSV file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initialize\n",
    "Import dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glob import glob\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load data\n",
    "Read all the CSV files into memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alldata = np.empty(shape=[0, 188])\n",
    "print(alldata.shape)\n",
    "paths = glob('data_ecg/*.csv')\n",
    "for path in paths:\n",
    "    print('Loading ', path)\n",
    "    csvrows = np.loadtxt(path, delimiter=',')\n",
    "    alldata = np.append(alldata, csvrows, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shuffle and separate the data\n",
    "All records are shuffled randomly, then split into training, validation, and testing groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomly mix rows\n",
    "np.random.shuffle(alldata)\n",
    "totrows = len(alldata)\n",
    "trainrows = int((totrows * 3 / 5) + 0.5) # 60%\n",
    "testrows = int((totrows * 1 / 5) + 0.5) # 20%\n",
    "validaterows = totrows - trainrows - testrows # 20%\n",
    "mark1 = trainrows\n",
    "mark2 = mark1 + testrows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Save data\n",
    "Data is saved in 3 separate CSV files: training, validation, testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('train.csv', \"wb\") as fin:\n",
    "    np.savetxt(fin, alldata[:mark1], delimiter=\",\", fmt='%f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('test.csv', \"wb\") as fin:\n",
    "    np.savetxt(fin, alldata[mark1:mark2], delimiter=\",\", fmt='%f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('validate.csv', \"wb\") as fin:\n",
    "    np.savetxt(fin, alldata[mark2:], delimiter=\",\", fmt='%f')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}