kaggle-planet-amazon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kaggle Planet competition with Fast.ai\n",
    "[The fast.ai course](https://course.fast.ai) is a great way to get started with Deep Learning if you have only some coding experience, but no machine learning experience. Recently, fast.ai decided to switch from Keras to their self-rolled library which is built on top of PyTorch. I can get that decision I think, it's just that so far, the documentation isn't quite equivalent (in the sense that Keras' documentation is excellent, and fast.ai's is... non-existent?)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Fast.ai by examples\n",
    "Hence, I figured it would be cool to walk through a Kaggle competition challenge, using the fast.ai library, and dissecting every step. They're mostly notes for myself, but maybe a lost reader might find it useful too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/envs/fastai/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.\n",
      "  from numpy.core.umath_tests import inner1d\n"
     ]
    }
   ],
   "source": [
    "from fastai.model import resnet34\n",
    "from fastai.dataset import get_cv_idxs, ImageClassifierData\n",
    "from fastai.transforms import tfms_from_model, transforms_top_down\n",
    "from fastai.conv_learner import ConvLearner\n",
    "import torch\n",
    "\n",
    "# Custom metrics stuff\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.metrics import fbeta_score\n",
    "import warnings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Setting up data ingestion\n",
    "Fast.ai models expect a data stream to feed them the training samples during training, whatever these samples look like. The fast.ai library offers some pretty handy shortcuts for that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "PATH = \"data/amazon/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The Planet competition provides us with a bunch of images in one folder, and a mapping `.csv` that maps a filename to its classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "label_csv = f\"{PATH}mapping.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Validation set\n",
    "If I have to explain what a validation set is or why it is important, it's likely you're a bit in over your head. Probably best if you Google that.\n",
    "At any rate, the fast.ai library offers a nice shortcut to generate a train/test split called `get_cv_idxs()`, which I'm guessing is short form for *Get Cross-Validation Indexes*. It only requires one argument: the number of total samples available to you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# Figure out amount of training samples by reading the lines in the mapping\n",
    "with open(label_csv, 'r') as csv_file:\n",
    "    number_of_training_samples = sum(1 for row in csv_file)\n",
    "    # Subtract 1 as this counted the header row of the .csv\n",
    "    number_of_training_samples -= 1\n",
    "    \n",
    "val_idxs = get_cv_idxs(number_of_training_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Choosing the model\n",
    "This is chosen to be `resnet34` here, as it's a bit lighter. Of course, any of the other models, like the `resnext` ones etc can be chosen too. What's beautiful about fast.ai is that  it is extremely modular: if you define the model properly, all other functions in the library will automatically adapt to deal with its architecture, input sizes etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "f_model = resnet34"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data stream\n",
    "With the data ingestion set up and the model chosen, we need to create some kind of data stream that will feed the model during training. Effectively, during training, we want to feed the model batches of images. This section is about three things:  \n",
    "1. Resizing the images correctly\n",
    "1. Augmenting them if we choose to\n",
    "1. Creating an object that correctly feeds the model\n",
    "\n",
    "For those first two, we'll use the `tfms_from_model()` function. For the second part, we'll use the `ImageClassifierData` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Image resizing\n",
    "A good practice when training a convolutional network is to start out with small images (like 64x64). This allows the convnet to get a first feeling for the images rather quickly and not get sidetracked by details at first. As you progress, you can bump up the image sizes to feed it more details and squeeze out those last few percentages. That's the first two arguments of `tfms_from_model()`: the model that we have chosen, so that we correctly size, and the `sz` argument, which is the image size that we want.\n",
    "\n",
    "### Data augmentation\n",
    "For images, data augmentation is a neat trick where you \"create\" more images that can be used in training by slightly modifying the original ones. The later arguments in `tfms_from_model()` are all about image augmentation. `aug_tfms` is about what kind of transformation you want to do (maybe mirror it, flip it etc). Read the code for more options. max_zoom puts an upper limit on the zooming function used to augment. For more details, I suggest you watch the first class from Fast.ai DL1.  \n",
    "Note that you can choose to only resize and not augment by not passing along any arguments here. I'm not entirely clear on what kind of images you wouln't want to augment though.\n",
    "\n",
    "### The output data stream\n",
    "`ImageClassifierData` is a stream that will output something that `model.fit()` can take in. We need to tell it where to find the training data (first two arguments), where to find the labels (`label_csv` argument), the suffix (because the filenames have `.jpg` while the mapping does not), what transforms to apply (`tfms`), the validation indexes to use, and where the test set is (when we call inference)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(sz, bs=64):\n",
    "    tfms = tfms_from_model(f_model, sz, aug_tfms=transforms_top_down, max_zoom=1.05)\n",
    "    return ImageClassifierData.from_csv(PATH, \"train\", label_csv, bs, suffix=\".jpg\", tfms=tfms,\n",
    "                                       val_idxs=val_idxs, test_name=\"test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional: understanding what comes out of a data loader\n",
    "Looking at the dimensions probably helps. For `x`: `[torch.FloatTensor of size 64x3x256x256]`  \n",
    "`256`: the image size that we indicated  \n",
    "`3`: RGB channels of each image  \n",
    "`64`: batch size  \n",
    "\n",
    "`y`: `[torch.FloatTensor of size 64x17]`  \n",
    "`17`: the amount of distinct classes it found in the `mapping.csv` file  \n",
    "`64`: batch size  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#(x,y) = next(iter(data.val_dl))\n",
    "#x\n",
    "#y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Custom metrics\n",
    "The Kaggle planet competition will measured based on F2, so we should do our training based on F2 too. It's a bit more advanced to look at custom metrics, so I'll leave that as an exercise to the reader to go dig into the code. I'll just include the effective lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def f2(preds, targs, start=0.17, end=0.24, step=0.01):\n",
    "    with warnings.catch_warnings():\n",
    "        warnings.simplefilter(\"ignore\")\n",
    "        return max([fbeta_score(targs, (preds>th), 2, average='samples')\n",
    "                    for th in np.arange(start,end,step)])\n",
    "\n",
    "metrics=[f2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining a learner\n",
    "Training the model involves setting up a so-called \"learner\". In this case, given we used a fast.ai-supported network, `resnet34`, we can use the `ConvLearner` class. To avoid having to retrain `resnet34` from scratch, we'll also use the `pretrained` property, which returns the network with its pre-trained weights. If this is the first time you run it, it'll first download the pretrained weights and store them in a place it can easily find them again (in the `tmp` directory in your `PATH`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "sz = 256\n",
    "bs = 64\n",
    "data = get_data(sz, bs)\n",
    "learn = ConvLearner.pretrained(f_model, data, metrics=metrics, ps=0.6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Congratulations! You've fired the jupyter_done event"
     ]
    }
   ],
   "source": [
    "!curl -X POST https://maker.ifttt.com/trigger/jupyter_done/with/key/bC7PnY5XAPTf8fgjJPqjl0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deciding on the learning rate\n",
    "Again some stuff that's useful with fast.ai: it has a built-in function that helps you in finding the right learning rate through reading a plot. Again, for more details, refer to Lecture 1 of DL1 v2 of the fast.ai classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "473c60477abc44b0bf6932d78a792549",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   f2                          \n",
      "    0      0.228463   0.357444   0.795254  \n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEOCAYAAABmVAtTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VOXd///XZ7KSlSVhDwRkEwUXArjVpW5oLWq1Ira2WpVqa7X2rm399n70a/W+f71rv7W2auvaTUsVtbVUcalad1kCgiwiskpYE9YQsufz+2OG3DGGZAIzOcnk/Xw8zmPOOXOdM58rB+Yz17nOOZe5OyIiIgChoAMQEZHOQ0lBREQaKSmIiEgjJQUREWmkpCAiIo2UFEREpJGSgoiINFJSEBGRRkoKIiLSSElBREQaJQcdQHvl5eV5YWFh0GGIiHQpCxcuLHP3/LbKdbmkUFhYSHFxcdBhiIh0KWa2IZpyOn0kIiKNlBRERKSRkoKIiDRSUhARkUZKCiIi0khJQUREGnW5S1LjYU3pPpLMGNyrB2tKK9hRUc0R+Vn0zU7DzIIOT0Skw3SrpNDQ4GzYuZ/lm/dwzysfM7BnD1Zs3kvZvmoAkkNGXcP/jlndOzOVvKxUPjcynwZ3+uekc/Hxg2hogJweyWSkfvrPV1ffQHKSGl8i0nV1m6Tg7nzt9/N5e3VZ4zoD+uemcfrofLLSkkkOGUcNyiEvK4012/excms5q7fv47G5G6itb8AdfvbCSiCcMCYV9qa2voGBPXuwvbyKf63YRmFeJnsra0lJClHYJ5O87DQaGpwGd848sh+j+mUxILcHeVmprbZC6huckPGpMlW19WzZU8WKzXvZX1PHsLxMemWm0jsjlV6ZqZ+q646KGhoanJweKaSnJMX+DyoiCanbJIWXV2zj7dVlTJ9UwNiBuVwwbsCnvkib+9zI/70bvKHBqXdnw479PP/BFpKTjAXrd7Jqezn1Dc68dTvJSU/mvHEDqKypp09mKjX1DXyycz+LNuwiLSXE7v21vLBs66c+wyzcOslJT6FXZir7quqoqW8gLTnEnspakkJGZmoySSFjX3UdeyprDxrviL5ZpERaKZt3VzaWDRnk9EhhYG4PCvMyGJjbg537a6ipa2DF5r3kZqQwsbA3+6rrKC2v5sgBOQzu1YOSXZU0NDj52WmEQsYxg3PpnZnKzooayqvqOGpgDukpSdQ1OLV1DfTMSNGpNpEE0G2Sgrtz8og+3Hnh0e0+xRMKGSGMEX2zuPmskYf0+bX1DSzfvJflm/ewt7KOqtp6Gtypa3DKyqvZWVFDWkqI3B4p1Dc4yUkhQgY1dQ3UN0BGahL9ctLolZnKsQU9yUhNZv2OCvZW1rJ5dxXz1u0gKfKlPGFoT4bnZREy2Lm/lp0V1WzaVcmHW8r598pSzCA7PZnCPpnU1Dfw6NvryElPpndmKq9+uI0GDycsA5qcTWtVz4wUUpNCpKWEyM9KY0jvDPKy0hien8Xw/Exye6SQkZrE4F4ZJIWUPEQ6K3OP8n99J1FUVOR69tHhcfdP/aqvb/DGL+ryqnCSGZ6fSX2DU1Fdx/6aet7fuJv91XX0ykwlIzWJpZv2UF/vpCSHSA4Za8sqqKlroLa+gS27q9i0u5KyfdVU1zV86rNTk0P0ibTQBvXswaRhvZk4rDfHDO5J71ZabiJyeMxsobsXtVlOSUHipaHB2bynkjWlFeyvrqO8qo6Pt5eza3/41Nbq7fvCySXSHOmZkcKwvEyG5WUyIDedvKw0+uWkc9TAHIb2yQyyKiJdXrRJoducPpKOFwoZg3tlMLhXxkHL7K+p4/1PdvPhlr2sK6tgXVkF763ZwZY9VZ8qV9C7BwNzezC4VwaDevXg/HH9GdM/J95VEOl24tpSMLMpwK+BJOARd/+fZu8PAf4E9IyU+ZG7z2ltn2opdA/uzvbyasr2VfPemh18ULKHzbsr2bS7km17q2hwGJaXycXHDWLaxAL65aQHHbJIpxb46SMzSwJWAWcDJcACYLq7r2hS5iHgfXf/nZmNBea4e2Fr+1VSkLJ91bywdAsvLt/KO6t3ADCmfzZTjx3I1ScNo0eqLsEVaa4znD6aBKx297WRgJ4ALgRWNCnjwIFzALnA5jjGIwkiLyuNK08s5MoTC1m9fR+vfLiN11Zu564XP+KeVz7mrCP78pXJQznpiD66TFakneKZFAYBG5sslwCTm5W5HXjZzL4DZAJnxTEeSUAj+mYxom8W1592BPPX7eT5DzYze8lm5izdysi+WRw5IIexA3O4+uRC0pLVghBpSzxPH30ZONfdr40sXwlMcvfvNCnzvUgMvzSzE4FHgaPdvaHZvmYAMwCGDBkyYcOGqEaVk26qqraev7+/iec/2MLKreWU7asmLyuVKyYP5auTh9BX/Q/SDXWGPoUTgdvd/dzI8m0A7v6zJmWWA1PcfWNkeS1wgrtvP9h+1acg7fXu6jIeeXsdr63cTkqScc7Y/nznzBG6ekm6lc7Qp7AAGGlmw4BNwOXAFc3KfAKcCfzRzI4E0oHSOMYk3dBJI/I4aUQe68oqeHzuBmYt2MjzS7fw+TF9+dF5YxjVLzvoEEU6jXhfkno+cA/hy01/7+7/bWZ3AMXuPjtyxdHDQBbhTucfuPvLre1TLQU5XGX7qnlywUYeeGMN+6rrOH1UPreeO4axA9VykMQV+OmjeFFSkFjZVVHDH99dz5/eW8/u/bVcMH4A3/n8SEb3V8tBEo+SgkiU9uyv5ZG31/LAG2uorXcuOnYgP5gyhoE9ewQdmkjMRJsUNCKMdHu5GSn8xzmjee+2M/n2GUcwZ9lWzvnVmzy9sISu9qNJ5HApKYhE5GWlceu5Y3j1e6cxdkAO339qCTfOfJ/d+2uCDk2kwygpiDRT0DuDv844gR9MGc1Ly7cy5Z63eP2jg14lLZJQlBREWpAUMr51+gj+/q2TyUxL4qo/LOCbjxVTsmt/0KGJxJWSgkgrxg3OZc7Nn+PWc0fzxqpSzvnVm8xZuiXosETiRklBpA1pyUl8+4wRvPK90xjdP5tv/WURP3z6A6pq64MOTSTmlBREojS4VwZPzDiBb542nFkLN/LlB95jy57KoMMSiSklBZF2SEtO4rbzjuThK4tYW7qP8379Fm9/XBZ0WCIxo6QgcgjOGtuPf9x4Cv1z0rn2zwtYsH5n0CGJxISSgsghGtE3i8evnczAnj34xh8WsLRkT9AhiRw2JQWRw5CXlcbj10wmp0cKX/v9PFZtKw86JJHDoqQgcpgG9uzBzOsmk5IU4oqH57J6uxKDdF1KCiIxMLRPJjOvOwEwpj88jzWl+4IOSeSQKCmIxMiIvlk8MWMy7s70h+ayVolBuiAlBZEYGtE3m5nXnUB9gzP94bmsK6sIOiSRdlFSEImxUf3CiaG2Ptxi2LBDiUG6DiUFkTgY3T+bv1w7meq6eqY/NJfte6uCDkkkKkoKInFy5IAcHrtmMrv213L94wvZX1MXdEgibVJSEImjowflcvdlx7B4425ueHwR9Q0ayU06NyUFkTg7b9wA7rzoaN5YVcp9r60OOhyRVikpiHSAKyYN4eLjBvHrV1cxb+2OoMMROai4JgUzm2JmH5nZajP7UQvv/8rMFkemVWa2O57xiATFzLjzoqMZ0juDm59YzK4KjfssnVPckoKZJQH3A+cBY4HpZja2aRl3v8Xdj3X3Y4F7gb/FKx6RoGWlJXPfFcezo6KaW5/+gAb1L0gnFM+WwiRgtbuvdfca4AngwlbKTwf+Gsd4RAJ39KBcbjvvSF75cBv3vPpx0OGIfEY8k8IgYGOT5ZLIus8ws6HAMOC1OMYj0ilcfXIhl04YzL2vfcx7a9S/IJ1LPJOCtbDuYO3ly4Gn3b3FQW/NbIaZFZtZcWlpacwCFAmCmXHHhUdR2CeT/5i1mD37a4MOSaRRPJNCCVDQZHkwsPkgZS+nlVNH7v6Quxe5e1F+fn4MQxQJRkZqMvdMO5bt5dX85z+W4a7+Bekc4pkUFgAjzWyYmaUS/uKf3byQmY0GegHvxTEWkU7nmIKefPeskfxzyWaeXlgSdDgiQByTgrvXATcCLwEfArPcfbmZ3WFmU5sUnQ484fqpJN3QDaeP4IThvfnpP1ewU5epSidgXe27uKioyIuLi4MOQyRmVm8v59x73uKC8QO4Z9qxmLXUHSdyeMxsobsXtVVOdzSLBGxE32xu+vxI/rF4M7OKN7a9gUgcKSmIdAI3fn4EJx3Rhzv+uYKNO/cHHY50Y0oKIp1AUsi469LxmBk/fOYDXY0kgVFSEOkkBvfK4IdTRvPumh38a8W2oMORbkpJQaQTmT5pCMPzM/n5iyupq28IOhzphpQURDqR5KQQP5wyhjWlFcwq1r0L0vGUFEQ6mXPG9qNoaC9+9coqDeEpHU5JQaSTMTNuO38MpeXVPPLWuqDDkW5GSUGkE5owtDdTjurPg2+soWxfddDhSDeipCDSSd06ZTRVdQ38RuMuSAdSUhDppI7Iz2L6pAJmzvuEdWUVQYcj3YSSgkgndvOZo0hNDvGLl1YGHYp0E0oKIp1YfnYaM04dzpylW3n/k11BhyPdgJKCSCd33eeGk5eVxs/mrNTjLyTulBREOrnMtGRuPmsk89fv5I1VGo5W4ktJQaQLmFZUwMDcdH777zVBhyIJTklBpAtITQ4x49ThzF+/k/nrdgYdjiSwNpOCmWWaWSgyP8rMpppZSvxDE5GmLp80hLysNN23IHEVTUvhTSDdzAYBrwJXA3+MZ1Ai8lnpKUl889ThvL26jIUb1FqQ+IgmKZi77we+BNzr7hcDY+Mbloi05CsnDKFPZir3vKLWgsRHVEnBzE4EvgI8H1mXHL+QRORgMlKTmXHqcN76uEz3LUhcRJMUvgvcBvzd3Zeb2XDg3/ENS0QO5qsnDCU7PZnfv7M+6FAkAbWZFNz9DXef6u4/j3Q4l7n7TdHs3MymmNlHZrbazH50kDKXmdkKM1tuZjPbGb9It5OZlsy0ogLmLN3Clj2VQYcjCSaaq49mmlmOmWUCK4CPzOzWKLZLAu4HziPcBzHdzMY2KzOScCvkZHc/inCrRETa8PWTCnF3HntvQ9ChSIKJ5vTRWHffC1wEzAGGAFdGsd0kYLW7r3X3GuAJ4MJmZa4D7nf3XQDuvj3qyEW6sYLeGZw9th9/nf8JVbX1QYcjCSSapJASuS/hIuAf7l4LRPMAlkHAxibLJZF1TY0CRpnZO2Y218ymRBO0iMDVJw9j1/5aZi/eHHQokkCiSQoPAuuBTOBNMxsK7I1iO2thXfNkkgyMBE4HpgOPmFnPz+zIbIaZFZtZcWmpnv0iAjB5WG9G9cviT++t14PyJGai6Wj+jbsPcvfzPWwDcEYU+y4BCposDwaa/6QpIdL6cPd1wEeEk0TzGB5y9yJ3L8rPz4/io0USn5nxtRMLWb55L4t0earESDQdzblmdveBX+pm9kvCrYa2LABGmtkwM0sFLgdmNyvzLJEEY2Z5hE8nrW1XDUS6sYuPG0R2WrI6nCVmojl99HugHLgsMu0F/tDWRu5eB9wIvAR8CMyK3Odwh5lNjRR7CdhhZisI3/twq7vvaH81RLqnzLRkLpkwmDlLt1K2rzrocCQBWFvnIs1ssbsf29a6jlJUVOTFxcVBfLRIp7R6ezln3f0mt547mm+fMSLocKSTMrOF7l7UVrloWgqVZnZKkx2fDOiOGZFOYkTfbE46og8z531CfYM6nOXwRJMUbgDuN7P1ZrYBuA+4Pr5hiUh7XHnCUDbtruS1lbrVRw5Pmw+2c/fFwDFmlhNZjuZyVBHpQGeP7Ue/nDQem7uBs8f2Czoc6cIOmhTM7HsHWQ+Au98dp5hEpJ2Sk0JcMWkov3plFevLKijMi+YCQZHPau30UXYbk4h0ItMnFZAcMh6bq8tT5dAdtKXg7j/tyEBE5PD0zUnn/HEDmLVgI7ecPYqsNA17Iu0XTUeziHQRV59cSHl1Hc8sLAk6FOmilBREEshxQ3pxbEFP/vjuehp0eaocAiUFkQRz9cmFrCur4I1VeniktF+bJx3NLA24BChsWt7d74hfWCJyqM4fN4A7n1vBX+d/whlj+gYdjnQx0bQU/kF4cJw6oKLJJCKdUEpSiEsmDObVldvZXl4VdDjSxURzecJgd9fgNyJdyGVFBTz4xlqeWbiJG04/IuhwpAuJpqXwrpmNi3skIhIzR+RnMamwN7OKN2oAHmmXaJLCKcBCM/vIzD4ws6Vm9kG8AxORwzNtYgHryiqYv25n0KFIFxLN6aPz4h6FiMTc+eMGcPvs5Ty5YCOTh/cJOhzpIqIZjnMD0BP4YmTqGVknIp1Yj9QkLjxuIM8v3cKeytqgw5EuIprhOG8G/gL0jUyPm9l34h2YiBy+yycOobqugdmLNwUdinQR0fQpXANMdvefuPtPgBOA6+IblojEwtGDchk7IIcnizcGHYp0EdEkBQPqmyzXR9aJSBdw+aQClm3ay7JNe4IORbqAaJLCH4B5Zna7md0OzAUejWtUIhIzFx4ziLTkEE8uUGtB2hZNR/PdwNXATmAXcLW73xPvwEQkNnIzUjh/3ACeXbyJypr6tjeQbu2gSeHA8Jtm1htYDzwOPAZsiKwTkS5i2sQCyqvqeGHZlqBDkU6utZbCzMjrQqC4yXRguU1mNiVy09tqM/tRC+9fZWalZrY4Ml3bzvhFJAqTh/WmsE8GT+gUkrShtZHXLoi8DjuUHZtZEnA/cDZQAiwws9nuvqJZ0Sfd/cZD+QwRiY6ZcdnEAu568SPWlu5jeH5W0CFJJxXNfQqvRrOuBZOA1e6+1t1rgCcIP21VRAJw6fGDSQoZs4o1KpscXGt9CumRvoM8M+tlZr0jUyEwMIp9DwKatlVLIuuauyTyTKWnzaygHbGLSDv0zUnn82P68vTCEmrrG4IORzqp1loK3yTcfzAm8npg+gfh00JtaelehuaPa/wnUOju44FXgD+1uCOzGWZWbGbFpaUaTUrkUF0+sYCyfdW8tnJ70KFIJ3XQpODuv470J3zf3Ye7+7DIdIy73xfFvkuApr/8BwObm33GDnevjiw+DEw4SCwPuXuRuxfl5+dH8dEi0pLTRuXTLydN9yzIQbX5lFR3v9fMjgbGAulN1v+5jU0XACPNbBiwCbgcuKJpATMb4O4HrpGbCnzYjthFpJ2Sk0JcOmEwv3t9DVv3VNE/N73tjaRbiaaj+f8C90amM4C7CH+Bt8rd64AbgZcIf9nPcvflZnaHmR3Y/iYzW25mS4CbgKsOqRYiErXLigpocHh6oVoL8lnW1qhMZrYUOAZ4392PMbN+wCPu/sWOCLC5oqIiLy6O6jYJETmIKx6ey8Zd+3nj+2cQCulRZt2BmS1096K2ykXz7KNKd28A6iJ3OW8Hhh9ugCISnGkTC9i4s5L31u4IOhTpZKJJCsVm1pNwR/BCYBEwP65RiUhcnXtUf3J7pOgOZ/mMaDqavxWZfcDMXgRy3F1jNIt0YekpSVx83CBmzvuEXRU19MpMDTok6SRau3nt+OYT0BtIjsyLSBc2bWIBNfUNPKtR2aSJ1loKv4y8pgNFwBLCN6SNB+YBp8Q3NBGJpyMH5HDM4FyemL+Rq04qxEwdztL6zWtnuPsZwAbg+MjNYxOA44DVHRWgiMTPtIlD+GhbOUtKNCqbhEXT0TzG3ZceWHD3ZcCx8QtJRDrKF48ZQI+UJJ5c8EnQoUgnEU1S+NDMHjGz083sNDN7GN15LJIQstNTuGD8AGYv3kxFdV3Q4UgnEE1SuBpYDtwMfBdYEVknIglg2sQCKmrqef4Djcom0V2SWgX8KjKJSIKZMLQXR+Rn8mTxRi6bqKfXd3etXZI6K/K6NDLewaemjgtRROLJzLh84hAWbtjFx9vKgw5HAtba6aObI68XAF9sYRKRBHHx8YNISTI9UltavSR1S+R1Q0tTx4UoIvGWl5XG2WP78bf3N1FdVx90OBKg1k4flZvZ3hamcjPb25FBikj8TZs4hJ0VNbyyQqOydWettRSy3T2nhSnb3XM6MkgRib9TRuQxqGcPZs7XiYDuLJpLUgEws75mNuTAFM+gRKTjJYWMKyYP4Z3VO1ilDuduK5qR16aa2cfAOuANYD3wQpzjEpEAXDFpCGnJIf7wzvqgQ5GARNNSuBM4AVjl7sOAM4F34hqViASiV2YqXzp+EH9bVMKuipqgw5EARJMUat19BxAys5C7/xs9+0gkYV110jCq6xr4q56H1C1FkxR2m1kW8CbwFzP7NaCHpIgkqNH9szl5RB8ee28DtfUNQYcjHSyapHAhsB+4BXgRWINuXhNJaFefNIwte6p4cdnWoEORDhZNUpgBDHT3Onf/k7v/JnI6SUQS1OfH9GVonwz+8M66oEORDhZNUsgBXjKzt8zs22bWL9qdm9kUM/vIzFab2Y9aKXepmbmZFUW7bxGJn1DIuOqkQhZ9spvFG3cHHY50oDaTgrv/1N2PAr4NDATeMLNX2trOzJKA+4HzgLHAdDMb20K5bOAmwkN8ikgncemEwWSlJau10M1EffMasB3YCuwA+kZRfhKw2t3XunsN8ATh/onm7gTuAqraEYuIxFl2egqXFRXw/Adb2LZX/z27i2huXrvBzF4HXgXygOvcfXwU+x4ENH3kYklkXdN9HwcUuPtzUUcsIh3mqpMKqXfn8bl69EV3EU1LYSjwXXc/yt3/r7uviHLf1sI6b3zTLER44J7/aHNHZjPMrNjMiktLS6P8eBE5XEP6ZHDmmH78Zd4nVNXq6andQTR9Cj9y98WHsO8SoOkwToOBzU2Ws4GjgdfNbD3hu6Znt9TZ7O4PuXuRuxfl5+cfQigicqi+cUohOytqeGK+bmbrDtrTp9BeC4CRZjbMzFKBy4HZB9509z3unufuhe5eCMwFprp7cRxjEpF2OnF4H04c3od7X1ut1kI3ELek4O51wI3AS8CHwCx3X25md5jZ1Hh9rojElplx05kj2VFRwzOLSoIOR+IsOZ47d/c5wJxm635ykLKnxzMWETl0JwzvzbhBuTz61jqmTxxCKNRSl6EkgniePhKRBGFmXHfqcNaWVfD80i1BhyNxpKQgIlH5wrgBjO6Xzd3/WqUH5SUwJQURiUpSyPj+uaNZV1bB0wvVt5ColBREJGpnHdmX44f05J5XVulKpASlpCAiUTMzfjBlDNv2VvPn99YHHY7EgZKCiLTLCcP7cNqofH77+hr2VtUGHY7EmJKCiLTbreeOZvf+Wh5+c23QoUiMKSmISLsdPSiXC8YP4JG31lFaXh10OBJDSgoicki+d/YoauobuP/fq4MORWJISUFEDsnw/Cy+PGEwM+d9wsad+4MOR2JESUFEDtnNZ40kFIL/en4F7t72BtLpKSmIyCEbkNuDm88cxUvLt/G3RZuCDkdiQElBRA7LN08dznFDevKzF1bqEtUEoKQgIoclFDJ+OvUodlRUc++rHwcdjhwmJQUROWzjB/dkWlEBf3hnPau3lwcdjhwGJQURiYlbzx1Nj9Qkbp+tTueuTElBRGKiT1YaP5gyhrdXl/HEgo1BhyOHSElBRGLmK5OGcPKIPvzXcyt070IXpaQgIjETChl3XXoMZsb3n1pCQ4NOI3U1SgoiElODevbgJ18cy7x1O3ls7oagw5F2UlIQkZj78oTBnDoqn7teXMnm3ZVBhyPtoKQgIjFnZvz3RUfT4PCfzy7T1UhdSFyTgplNMbOPzGy1mf2ohfevN7OlZrbYzN42s7HxjEdEOk5B7wy+f+5oXlu5nb+/r0dgdBVxSwpmlgTcD5wHjAWmt/ClP9Pdx7n7scBdwN3xikdEOt5VJxVSNLQXP/3nCrbvrQo6HIlCPFsKk4DV7r7W3WuAJ4ALmxZw971NFjMBtTFFEkhSyLjr0vFU1dbzY51G6hLimRQGAU3vYCmJrPsUM/u2ma0h3FK4KY7xiEgAhudn8R/njOJfK7Yxe8nmoMORNsQzKVgL6z7zM8Hd73f3I4AfAv/Z4o7MZphZsZkVl5aWxjhMEYm3a04ZzvFDevLjvy9jTem+oMORVsQzKZQABU2WBwOt/Ux4AriopTfc/SF3L3L3ovz8/BiGKCIdISlk3HvF8aQmh/jmYwvZV10XdEhyEPFMCguAkWY2zMxSgcuB2U0LmNnIJotfAPTcXZEENahnD+6bfhxrS/dx61NL1L/QScUtKbh7HXAj8BLwITDL3Zeb2R1mNjVS7EYzW25mi4HvAV+PVzwiEryTRuRx23lH8sKyrTzwxtqgw5EWJMdz5+4+B5jTbN1PmszfHM/PF5HO59rPDWNxyW5+8dJKRvTN4uyx/YIOSZrQHc0i0qHMjLsuGc/Rg3K54fGFvLumLOiQpAklBRHpcJlpyTx+7WQK8zL51l8WsWFHRdAhSYSSgogEIic9hUe+VgTAN/64gJ0VNQFHJKCkICIBKszL5IGvTqBkVyXf+esijb/QCSgpiEigThjeh9unHsU7q3fw/aeWUFffEHRI3Vpcrz4SEYnG5RML2FlRwy9e+oh91XXce8VxpCUnBR1Wt6SWgogEzsz49hkj+OnUo3h5xTau+/NCKmvqgw6rW1JSEJFO4+snFXLXJeN5c1UpF9z7FtvL9bjtjqakICKdymUTC3j8msls2VPF1x6dz579tUGH1K0oKYhIp3PKyDweurKItaUVfPXReZTtqw46pG5DSUFEOqVTRubxwJXH8/H2cq58dD57KtVi6AhKCiLSaX1+TD8evLKI1dvLufZPC9T53AGUFESkUzttVD73TDuO4g27+ObjC6nQWAxxpaQgIp3eF8YP4OdfGs/bH5dy5aPzKK/SqaR4UVIQkS7hsokF/PYrx/NByR6+/MB7rC/TQ/TiQUlBRLqMKUcP4JGvF7FtbxXTH57Lko27gw4p4SgpiEiXcvrovsy87gQqa+u58P53+NZfFrJ1j25yixUlBRHpco4ckMPLt5zKLWeN4tUPt3PmL1/n0bfX6WF6MaCkICJdUt/sdG4+ayT/uuU0Jg7rzZ3PrWDqfe+w6JNdQYfWpSkpiEiXNqRPBn+4aiK/+8rx7Kyo4Uu/fZcvP/Auz32wWeMzHAJz71p/tKLV7IG5AAAM0UlEQVSiIi8uLg46DBHphPZV1/Hn99bzVHEJ68oqGNong8uKCrh0wmD65aQHHd5hqaypp0fqoT9O3MwWuntRm+WUFEQk0dQ3OC8s28Jj721g3rqdhAzOGN2Xb51xBBOG9g46vHZxd95ds4Nbn1rCj78wli+MH3BI+4k2KcR1kB0zmwL8GkgCHnH3/2n2/veAa4E6oBT4hrtviGdMIpL4kkLGBeMHcsH4gawvq2BW8UZmFZdwye/e48wxfTnnqH70z+3B5GG9SU/pvIP5uDv//fyHPPL2Ogb17EFhXkbcPzNuLQUzSwJWAWcDJcACYLq7r2hS5gxgnrvvN7MbgNPdfVpr+1VLQUQORUV1HQ+8sYZnFpawOXIJa15WKpOH9aFvThr52Wn0yUzlzCP7kZeVFnC08MaqUu5++SOWlOzhqycM4f+cfyQZqYf+Oz7w00dmdiJwu7ufG1m+DcDdf3aQ8scB97n7ya3tV0lBRA6Hu7Nyazmf7NzPs+9vYuXWcrbsqaSqNnw5a3LIGJ6fybhBPRnTP5shfTLon5NOz4wUMtOSGxNGXX0DlbX1ZKenxDS2pZv28PMXV/LO6h0U9sngqycM5ZpThmFmh7XvznD6aBCwsclyCTC5lfLXAC/EMR4REcyMIwfkcOSAHM49qj8Q/jKurmtg/Y4KnluyhZVb9/Laym08s6jkM9sPy8tkSO8MVmzZS2l5NRmpSaSnJDGmfzZ9s9Mo6J3BurIKjhvSi/PH9advdjpJIcPdP/XF7u5U1TawfPMeyvbV8EHJbv61Yhsfb99Hr4wUvnvWSK4/7YgOP70Vz6TQUlprsVliZl8FioDTDvL+DGAGwJAhQ2IVn4gIEE4U4S/2HMb0z2lcv2d/Let2VLBtbxWl5dXsrKhh/rqdLNqwi5NG9KFvdjpmsKeylgXrdrK2tIJnF28mOWQ898EW7nwufLa8Z0YKeytryUpLZnT/bLLSkllSsoedFTWNn5UUMiYV9uZrJw5l6jGDyM2IXQukPeKZFEqAgibLg4HNzQuZ2VnAj4HT3L3F4ZXc/SHgIQifPop9qCIin5WbkcKxGT3btc3eqloyU5PZsKOCOUu3sGt/Lftr6umTmcqeylqWbd7Dtr3VnDoyj9H9cyjo3YPCPpkU9M4gt0cwiaCpeCaFBcBIMxsGbAIuB65oWiDSj/AgMMXdt8cxFhGRDpET6WMYnp/FjZ8fGXA07Re3O5rdvQ64EXgJ+BCY5e7LzewOM5saKfYLIAt4yswWm9nseMUjIiJti+t9Cu4+B5jTbN1PmsyfFc/PFxGR9tGzj0REpJGSgoiINFJSEBGRRkoKIiLSSElBREQaKSmIiEijLjeegpmVAruBPZFVuS3M5wFlh/ExTfd5KGVaeq/5upbibm3+cOoUTX1aK9fe+jRf1jFqm45RdO/pGH16vj31Geru+W2WcvcuNwEPtTYPFMdq/4dSpqX3mq9rqw4tzB9ynaKpT2vl2lsfHSMdIx2jrnGMWpq66umjf0YxH6v9H0qZlt5rvi6aOnRkfVor1976NF/WMWqbjlF07+kYtS+Wdutyp4+iYWbFHsVzw7uSRKtTotUHEq9OiVYfSLw6xaM+XbWl0JaHgg4gDhKtTolWH0i8OiVafSDx6hTz+iRkS0FERA5NorYURETkECgpiIhIIyUFERFp1O2SgpmdbmZvmdkDZnZ60PHEgpllmtlCM7sg6FhiwcyOjByfp83shqDjOVxmdpGZPWxm/zCzc4KOJxbMbLiZPWpmTwcdy6GK/L/5U+TYfCXoeGIhFselSyUFM/u9mW03s2XN1k8xs4/MbLWZ/aiN3TiwD0gnPI50YGJUH4AfArPiE2X7xKJO7v6hu18PXAYEevlgjOrzrLtfB1wFTItjuFGJUZ3Wuvs18Y20/dpZty8BT0eOzdTP7KyTaE+dYnJcYn03XDwn4FTgeGBZk3VJwBpgOJAKLAHGAuOA55pNfYFQZLt+wF8SoD5nER7/+irggkQ4RpFtpgLvAlckQn0i2/0SOD5RjlFku6eDrs9h1O024NhImZlBxx6LOsXiuMR1OM5Yc/c3zayw2epJwGp3XwtgZk8AF7r7z4DWTqfsAtLiEWe0YlEfMzsDyCT8j7zSzOa4e0NcA29FrI6Ru88GZpvZ88DM+EXcuhgdIwP+B3jB3RfFN+K2xfj/UafSnroRPlMwGFhMJz5r0s46rTjcz+u0f4h2GARsbLJcElnXIjP7kpk9CDwG3Bfn2A5Fu+rj7j929+8S/uJ8OMiE0Ir2HqPTzew3keM052DlAtSu+gDfIdyiu9TMro9nYIehvceoj5k9ABxnZrfFO7jDdLC6/Q24xMx+RxwfGxEnLdYpFselS7UUDsJaWHfQO/Lc/W+E/zF0Vu2qT2MB9z/GPpSYae8xeh14PV7BxEB76/Mb4DfxCycm2lunHUBnTXDNtVg3d68Aru7oYGLkYHU67OOSCC2FEqCgyfJgYHNAscRCotUHEq9OiVYfSMw6HZCIdYtbnRIhKSwARprZMDNLJdzpOjvgmA5HotUHEq9OiVYfSMw6HZCIdYtfnYLuWW9nL/xfgS1ALeFMeU1k/fnAKsK98T8OOs7uWp9ErFOi1SdR65TIdevoOumBeCIi0igRTh+JiEiMKCmIiEgjJQUREWmkpCAiIo2UFEREpJGSgoiINFJSkLgzs30d8BlTo3zMeCw/83QzO+kQtjvOzB6JzF9lZp3iGVxmVtj88cwtlMk3sxc7KibpeEoK0mWYWdLB3nP32e7+P3H4zNaeD3Y60O6kAPwf4N5DCihg7l4KbDGzk4OOReJDSUE6lJndamYLzOwDM/tpk/XPWnj0uOVmNqPJ+n1mdoeZzQNONLP1ZvZTM1tkZkvNbEykXOMvbjP7Y+Qpq++a2VozuzSyPmRmv418xnNmNufAe81ifN3M/j8zewO42cy+aGbzzOx9M3vFzPpFHmV8PXCLmS02s89FfkU/E6nfgpa+OM0sGxjv7ktaeG+omb0a+du8amZDIuuPMLO5kX3e0VLLy8KjiD1vZkvMbJmZTYusnxj5Oywxs/lmlh1pEbwV+Rsuaqm1Y2ZJZvaLJsfqm03efhZIiJHKpAVB38KtKfEnYF/k9RzgIcJPeAwRHrDl1Mh7vSOvPYBlQJ/IsgOXNdnXeuA7kflvAY9E5q8C7ovM/xF4KvIZYwk/dx7gUsKP4g4B/QmPqXFpC/G+Dvy2yXIvaLz7/1rgl5H524HvNyk3EzglMj8E+LCFfZ8BPNNkuWnc/wS+Hpn/BvBsZP45YHpk/voDf89m+72E8KPTDyznEh58ZS0wMbIuh/CTkTOA9Mi6kUBxZL6QyEAuwAzgPyPzaUAxMCyyPAhYGvS/K03xmRLh0dnSdZwTmd6PLGcR/lJ6E7jJzC6OrC+IrN8B1APPNNvPgUefLyQ8pGJLnvXw2BIrzKxfZN0pwFOR9VvN7N+txPpkk/nBwJNmNoDwF+26g2xzFjDWrPGpxjlmlu3u5U3KDABKD7L9iU3q8xhwV5P1F0XmZwL/r4VtlwL/z8x+Djzn7m+Z2Thgi7svAHD3vRBuVQD3mdmxhP++o1rY3znA+CYtqVzCx2QdsB0YeJA6SBenpCAdyYCfufuDn1ppdjrhL9QT3X2/mb1OeAxtgCp3r2+2n+rIaz0H/zdc3WTemr1Go6LJ/L3A3e4+OxLr7QfZJkS4DpWt7LeS/61bW6J+MJm7rzKzCYQfkvYzM3uZ8GmelvZxC7ANOCYSc1ULZYxwi+ylFt5LJ1wPSUDqU5CO9BLwDTPLAjCzQWbWl/Cv0F2RhDAGOCFOn/824ZG2QpHWw+lRbpcLbIrMf73J+nIgu8nyy8CNBxYiv8Sb+xAYcZDPeZfwI5AhfM7+7cj8XMKnh2jy/qeY2UBgv7s/TrglcTywEhhoZhMjZbIjHee5hFsQDcCVhMf7be4l4AYzS4lsOyrSwoBwy6LVq5Sk61JSkA7j7i8TPv3xnpktBZ4m/KX6IpBsZh8AdxL+EoyHZwg/engZ8CAwD9gTxXa3A0+Z2VtAWZP1/wQuPtDRDNwEFEU6ZlfQwghY7r4SyI10ODd3E3B15O9wJXBzZP13ge+Z2XzCp59ainkcMN/MFgM/Bv7L3WuAacC9ZrYE+BfhX/m/Bb5uZnMJf8FXtLC/RwiP97socpnqg/xvq+wM4PkWtpEEoEdnS7diZlnuvs/M+gDzgZPdfWsHx3ALUO7uj0RZPgOodHc3s8sJdzpfGNcgW4/nTeBCd98VVAwSP+pTkO7mOTPrSbjD+M6OTggRvwO+3I7yEwh3DBuwm/CVSYEws3zC/StKCAlKLQUREWmkPgUREWmkpCAiIo2UFEREpJGSgoiINFJSEBGRRkoKIiLS6P8Hzgw12I/UBNIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lrf = learn.lr_find()\n",
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=0.4\n",
    "lrs = np.array([lr/9, lr/3, lr])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)\n",
    "learn.unfreeze()\n",
    "learn.fit(lrs, 3, cycle_len=1, cycle_mult=2)\n",
    "learn.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save(f'{sz}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load(f'{sz}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sz = 128\n",
    "learn.set_data(get_data(sz))\n",
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)\n",
    "learn.unfreeze()\n",
    "learn.fit(lrs, 3, cycle_len=1, cycle_mult=2)\n",
    "learn.freeze()\n",
    "learn.save(f'{sz}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd579115db874c1c800cb1b8addc69d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   f2                          \n",
      "    0      0.133094   0.114973   0.902654  \n",
      "    1      0.128123   0.11152    0.904999                    \n",
      "    2      0.125143   0.108777   0.908218                    \n",
      "    3      0.12554    0.108833   0.908446                    \n",
      "    4      0.122794   0.106674   0.908239                    \n",
      "    5      0.120309   0.105499   0.91096                     \n",
      "    6      0.11715    0.105262   0.910802                    \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "281b6bf2cd944bc3a5304732247ebc08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   f2                          \n",
      "    0      0.104537   0.090692   0.925343  \n",
      "    1      0.100349   0.090464   0.922541                     \n",
      "    2      0.091029   0.085241   0.930068                     \n",
      "    3      0.099358   0.089241   0.926882                     \n",
      "    4      0.093725   0.085511   0.931084                     \n",
      "    5      0.087369   0.08333    0.931182                     \n",
      "    6      0.086666   0.082431   0.93223                      \n",
      "\n"
     ]
    }
   ],
   "source": [
    "sz = 256\n",
    "learn.set_data(get_data(sz))\n",
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)\n",
    "learn.unfreeze()\n",
    "learn.fit(lrs, 3, cycle_len=1, cycle_mult=2)\n",
    "learn.freeze()\n",
    "learn.save(f'{sz}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db5ffade1b8d47a59c8c09993363a2d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   f2                          \n",
      "    0      0.120431   0.100955   0.918047  \n",
      "    1      0.109882   0.09452    0.922982                    \n",
      "    2      0.100795   0.088636   0.927518                    \n",
      "    3      0.106012   0.091146   0.926916                    \n",
      "    4      0.099447   0.087868   0.927395                     \n",
      "    5      0.09422    0.08526    0.93036                      \n",
      "    6      0.093546   0.084552   0.929986                     \n",
      "\n"
     ]
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit(lrs, 3, cycle_len=1, cycle_mult=2)\n",
    "learn.freeze()\n",
    "learn.save(f'{sz}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step plan\n",
    "1. Set up path to train and test samples\n",
    "1. Set up validation set using `get_cv_idxs()`\n",
    "1. Choose model\n",
    "1. Decide on transforms if applicable\n",
    "1. Create an *ClassifierData object, either `from csv` or `from file` or whatever the case is\n",
    "1. Set up the learner, most likely one of the *Learner classes, probably pretrained\n",
    "1. Train, unfreeze, train more, freeze [optional: bump up image resolution]\n",
    "1. Set `val_idxs` to [0] in the `get_data()` to include them in training\n",
    "1. Submit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              \r"
     ]
    }
   ],
   "source": [
    "multi_pred, y = learn.TTA(is_test=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[False, False, False, False, False, False, False, False, False, False, False,  True,  True, False,\n",
       "        False, False, False],\n",
       "       [False, False, False, False, False,  True, False, False, False, False, False, False,  True, False,\n",
       "        False, False, False],\n",
       "       [ True, False, False, False, False,  True, False, False, False,  True, False, False,  True,  True,\n",
       "        False, False, False],\n",
       "       [False, False, False, False, False,  True, False, False, False, False, False, False,  True,  True,\n",
       "         True, False, False],\n",
       "       [ True, False, False, False, False,  True, False, False, False,  True, False, False,  True,  True,\n",
       "        False, False,  True]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = np.mean(multi_pred, 0)\n",
    "labels = preds>0.2\n",
    "labels[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "subm_df = pd.DataFrame({\n",
    "    'image_name': [fname[5:-4] for fname in data.test_ds.fnames],\n",
    "    'tags': [\" \".join([data.classes[idx] for idx in row.nonzero()[0].tolist()]) for row in labels]\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image_name</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>file_9235</td>\n",
       "      <td>partly_cloudy primary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>file_8058</td>\n",
       "      <td>clear primary</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>test_17915</td>\n",
       "      <td>agriculture clear habitation primary road</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>test_35695</td>\n",
       "      <td>clear primary road selective_logging</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>file_7850</td>\n",
       "      <td>agriculture clear habitation primary road water</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   image_name                                             tags\n",
       "0   file_9235                            partly_cloudy primary\n",
       "1   file_8058                                    clear primary\n",
       "2  test_17915        agriculture clear habitation primary road\n",
       "3  test_35695             clear primary road selective_logging\n",
       "4   file_7850  agriculture clear habitation primary road water"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subm_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "subm_df.to_csv(f'subm/amazon-01.gz', compression=\"gzip\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using competition: planet-understanding-the-amazon-from-space\n",
      "Successfully submitted to Planet: Understanding the Amazon from Space"
     ]
    }
   ],
   "source": [
    "!kaggle competitions submit -f subm/amazon-01.gz -m \"Maybe names got shuffled?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fastai",
   "language": "python",
   "name": "fastai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}