04_transfer_learning_in_tensorflow_part_1_feature_extraction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04_transfer_learning_in_tensorflow_part_1_feature_extraction.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPTOThku3umT+sZBpj2+yq/",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrdbourke/tensorflow-deep-learning/blob/main/04_transfer_learning_in_tensorflow_part_1_feature_extraction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ObwiuiGkZb87"
      },
      "source": [
        "# 04. Transfer Learning with TensorFlow Part 1: Feature Extraction\n",
        "\n",
        "We've built a bunch of convolutional neural networks from scratch and they all seem to be learning, however, there is still plenty of room for improvement.\n",
        "\n",
        "To improve our model(s), we could spend a while trying different configurations, adding more layers, changing the learning rate, adjusting the number of neurons per layer and more.\n",
        "\n",
        "However, doing this is very time consuming.\n",
        "\n",
        "Luckily, there's a technique we can use to save time.\n",
        "\n",
        "It's called **transfer learning**, in other words, taking the patterns (also called weights) another model has learned from another problem and using them for our own problem.\n",
        "\n",
        "There are two main benefits to using transfer learning:\n",
        "1. Can leverage an existing neural network architecture proven to work on problems similar to our own.\n",
        "2. Can leverage a working neural network architecture which has **already learned** patterns on similar data to our own. This often results in achieving great results with less custom data.\n",
        "\n",
        "What this means is, instead of hand-crafting our own neural network architectures or building them from scratch, we can utilise models which have worked for others.\n",
        "\n",
        "And instead of training our own models from scratch on our own datasets, we can take the patterns a model has learned from datasets such as [ImageNet](http://www.image-net.org/) (millions of images of different objects) and use them as the foundation of our own. Doing this often leads to getting great results with less data.\n",
        "\n",
        "Over the next few notebooks, we'll see the power of transfer learning in action.\n",
        "\n",
        "## What we're going to cover\n",
        "\n",
        "We're going to go through the following with TensorFlow:\n",
        "\n",
        "- Introduce transfer learning (a way to beat all of our old self-built models)\n",
        "- Using a smaller dataset to experiment faster (10% of training samples of 10 classes of food)\n",
        "- Build a transfer learning feature extraction model using TensorFlow Hub\n",
        "- Introduce the TensorBoard callback to track model training results\n",
        "- Compare model results using TensorBoard\n",
        "\n",
        "## How you can use this notebook\n",
        "\n",
        "You can read through the descriptions and the code (it should all run, except for the cells which error on purpose), but there's a better option.\n",
        "\n",
        "Write all of the code yourself.\n",
        "\n",
        "Yes. I'm serious. Create a new notebook, and rewrite each line by yourself. Investigate it, see if you can break it, why does it break?\n",
        "\n",
        "You don't have to write the text descriptions but writing the code yourself is a great way to get hands-on experience.\n",
        "\n",
        "Don't worry if you make mistakes, we all do. The way to get better and make less mistakes is to **write more code**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UTWetPM7AWfY"
      },
      "source": [
        "## Using a GPU\n",
        "\n",
        "To begin, let's check to see if we're using a GPU. Using a GPU will make sure our model trains faster than using just a CPU.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nq4kxIpQMpZT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5a53217a-7674-437c-df90-8e9ae37a54ed"
      },
      "source": [
        "# Are we using a GPU?\n",
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Fri Feb 12 03:39:41 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.39       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   37C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ol3NDTVlRLSv"
      },
      "source": [
        "If the cell above doesn't output something which looks like:\n",
        "\n",
        "```\n",
        "Fri Sep  4 03:35:21 2020       \n",
        "+-----------------------------------------------------------------------------+\n",
        "| NVIDIA-SMI 450.66       Driver Version: 418.67       CUDA Version: 10.1     |\n",
        "|-------------------------------+----------------------+----------------------+\n",
        "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
        "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
        "|                               |                      |               MIG M. |\n",
        "|===============================+======================+======================|\n",
        "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
        "| N/A   35C    P0    26W / 250W |      0MiB / 16280MiB |      0%      Default |\n",
        "|                               |                      |                 ERR! |\n",
        "+-------------------------------+----------------------+----------------------+\n",
        "                                                                               \n",
        "+-----------------------------------------------------------------------------+\n",
        "| Processes:                                                                  |\n",
        "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
        "|        ID   ID                                                   Usage      |\n",
        "|=============================================================================|\n",
        "|  No running processes found                                                 |\n",
        "+-----------------------------------------------------------------------------+\n",
        "```\n",
        "\n",
        "Go to Runtime -> Change Runtime Type -> Hardware Accelerator and select \"GPU\", then rerun the cell above."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7NY09457NKv4"
      },
      "source": [
        "## Transfer leanring with TensorFlow Hub: Getting great results with 10% of the data\n",
        "\n",
        "If you've been thinking, \"surely someone else has spent the time crafting the right model for the job...\" then you're in luck. \n",
        "\n",
        "For many of the problems you'll want to use deep learning for, chances are, a working model already exists.\n",
        "\n",
        "And the good news is, you can access many of them on TensorFlow Hub.\n",
        "\n",
        "[TensorFlow Hub](https://tfhub.dev/) is a repository for existing model components. It makes it so you can import and use a fully trained model with as little as a URL.\n",
        "\n",
        "Now, I really want to demonstrate the power of transfer learning to you.\n",
        "\n",
        "To do so, what if I told you we could get much of the same results (or better) than our best model has gotten so far with only 10% of the original data, in other words, 10x less data.\n",
        "\n",
        "This seems counterintuitive right?\n",
        "\n",
        "Wouldn't you think more examples of what a picture of food looked like led to better results?\n",
        "\n",
        "And you'd be right if you thought so, generally, more data leads to better results.\n",
        "\n",
        "However, what if you didn't have more data? What if instead of 750 images per class, you had 75 images per class?\n",
        "\n",
        "Collecting 675 more images of a certain class could take a long time.\n",
        "\n",
        "So this is where another major benefit of transfer learning comes in.\n",
        "\n",
        "**Transfer learning often allows you to get great results with less data.**\n",
        "\n",
        "But don't just take my word for it. Let's download a subset of the data we've been using, namely 10% of the training data from the `10_food_classes` dataset and use it to train a food image classifier on.\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/04-transfer-learning-feature-extraction.png)\n",
        "*What we're working towards building. Taking a pre-trained model and adding our own custom layers on top, extracting all of the underlying patterns learned on another dataset our own images.*\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UIwVrX6vXb4z"
      },
      "source": [
        "## Downloading and becoming one with the data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qwWwP657Szfv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "46b1d719-f47b-4918-92e8-92a18c1935b7"
      },
      "source": [
        "# Get data (10% of labels)\n",
        "import zipfile\n",
        "\n",
        "# Download data\n",
        "!wget https://storage.googleapis.com/ztm_tf_course/food_vision/10_food_classes_10_percent.zip\n",
        "\n",
        "# Unzip the downloaded file\n",
        "zip_ref = zipfile.ZipFile(\"10_food_classes_10_percent.zip\", \"r\")\n",
        "zip_ref.extractall()\n",
        "zip_ref.close()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-02-12 03:39:42--  https://storage.googleapis.com/ztm_tf_course/food_vision/10_food_classes_10_percent.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.15.112, 172.253.62.128, 142.250.31.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.15.112|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 168546183 (161M) [application/zip]\n",
            "Saving to: ‘10_food_classes_10_percent.zip’\n",
            "\n",
            "10_food_classes_10_ 100%[===================>] 160.74M   172MB/s    in 0.9s    \n",
            "\n",
            "2021-02-12 03:39:43 (172 MB/s) - ‘10_food_classes_10_percent.zip’ saved [168546183/168546183]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "agzJYtfFBl6I",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f347e60c-72ed-49c6-e706-c8be3439bea4"
      },
      "source": [
        "# How many images in each folder?\n",
        "import os\n",
        "\n",
        "# Walk through 10 percent data directory and list number of files\n",
        "for dirpath, dirnames, filenames in os.walk(\"10_food_classes_10_percent\"):\n",
        "  print(f\"There are {len(dirnames)} directories and {len(filenames)} images in '{dirpath}'.\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "There are 2 directories and 0 images in '10_food_classes_10_percent'.\n",
            "There are 10 directories and 0 images in '10_food_classes_10_percent/test'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/fried_rice'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/sushi'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/chicken_curry'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/pizza'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/ramen'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/steak'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/ice_cream'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/grilled_salmon'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/hamburger'.\n",
            "There are 0 directories and 250 images in '10_food_classes_10_percent/test/chicken_wings'.\n",
            "There are 10 directories and 0 images in '10_food_classes_10_percent/train'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/fried_rice'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/sushi'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/chicken_curry'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/pizza'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/ramen'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/steak'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/ice_cream'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/grilled_salmon'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/hamburger'.\n",
            "There are 0 directories and 75 images in '10_food_classes_10_percent/train/chicken_wings'.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F0r-zyagV7Qa"
      },
      "source": [
        "Notice how each of the training directories now has 75 images rather than 750 images. This is key to demonstrating how well transfer learning can perform with less labelled images.\n",
        "\n",
        "The test directories still have the same amount of images. This means we'll be training on less data but evaluating our models on the same amount of test data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EES-NoeaXfYT"
      },
      "source": [
        "## Creating data loaders (preparing the data)\n",
        "\n",
        "Now we've downloaded the data, let's use the [`ImageDataGenerator`](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator) class along with the `flow_from_directory` method to load in our images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YAp0GN60S-rK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb4a2dc3-fad9-416b-b81a-b2a0d3abcb45"
      },
      "source": [
        "# Setup data inputs\n",
        "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "IMAGE_SHAPE = (224, 224)\n",
        "BATCH_SIZE = 32\n",
        "\n",
        "train_dir = \"10_food_classes_10_percent/train/\"\n",
        "test_dir = \"10_food_classes_10_percent/test/\"\n",
        "\n",
        "train_datagen = ImageDataGenerator(rescale=1/255.)\n",
        "test_datagen = ImageDataGenerator(rescale=1/255.)\n",
        "\n",
        "print(\"Training images:\")\n",
        "train_data_10_percent = train_datagen.flow_from_directory(train_dir,\n",
        "                                               target_size=IMAGE_SHAPE,\n",
        "                                               batch_size=BATCH_SIZE,\n",
        "                                               class_mode=\"categorical\")\n",
        "\n",
        "print(\"Testing images:\")\n",
        "test_data = train_datagen.flow_from_directory(test_dir,\n",
        "                                              target_size=IMAGE_SHAPE,\n",
        "                                              batch_size=BATCH_SIZE,\n",
        "                                              class_mode=\"categorical\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training images:\n",
            "Found 750 images belonging to 10 classes.\n",
            "Testing images:\n",
            "Found 2500 images belonging to 10 classes.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6QWuVeSvQPoK"
      },
      "source": [
        "Excellent! Loading in the data we can see we've got 750 images in the training dataset belonging to 10 classes (75 per class) and 2500 images in the test set belonging to 10 classes (250 per class)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Qcwii2uYjOx"
      },
      "source": [
        "## Setting up callbacks (things to run whilst our model trains)\n",
        "\n",
        "Before we build a model, there's an important concept we're going to get familiar with because it's going to play a key role in our future model building experiments.\n",
        "\n",
        "And that concept is **callbacks**.\n",
        "\n",
        "[Callbacks](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks) are extra functionality you can add to your models to be performed during or after training. Some of the most popular callbacks include:\n",
        "* [**Experiment tracking with TensorBoard**](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard) - log the performance of multiple models and then view and compare these models in a visual way on [TensorBoard](https://www.tensorflow.org/tensorboard) (a dashboard for inspecting neural network parameters). Helpful to compare the results of different models on your data.\n",
        "* [**Model checkpointing**](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ModelCheckpoint) - save your model as it trains so you can stop training if needed and come back to continue off where you left. Helpful if training takes a long time and can't be done in one sitting.\n",
        "* [**Early stopping**](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) - leave your model training for an arbitrary amount of time and have it stop training automatically when it ceases to improve. Helpful when you've got a large dataset and don't know how long training will take.\n",
        "\n",
        "We'll explore each of these overtime but for this notebook, we'll see how the TensorBoard callback can be used.\n",
        "\n",
        "The TensorBoard callback can be accessed using [`tf.keras.callbacks.TensorBoard()`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard). \n",
        "\n",
        "Its main functionality is saving a model's training performance metrics to a specified `log_dir`.\n",
        "\n",
        "By default, logs are recorded every epoch using the `update_freq='epoch'` parameter. This is a good default since tracking model performance too often can slow down model training.\n",
        "\n",
        "To track our modelling experiments using TensorBoard, let's create a function which creates a TensorBoard callback for us.\n",
        "\n",
        "> 🔑 **Note:** We create a function for creating a TensorBoard callback because as we'll see later on, each model needs its own TensorBoard callback instance (so the function will create a new one each time it's run).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2yamhJ8xJA5x"
      },
      "source": [
        "# Create tensorboard callback (functionized because need to create a new one for each model)\n",
        "import datetime\n",
        "def create_tensorboard_callback(dir_name, experiment_name):\n",
        "  log_dir = dir_name + \"/\" + experiment_name + \"/\" + datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\")\n",
        "  tensorboard_callback = tf.keras.callbacks.TensorBoard(\n",
        "      log_dir=log_dir\n",
        "  )\n",
        "  print(f\"Saving TensorBoard log files to: {log_dir}\")\n",
        "  return tensorboard_callback"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "11TjBJQXdCyZ"
      },
      "source": [
        "Because you're likely to run multiple experiments, it's a good idea to be able to track them in some way.\n",
        "\n",
        "In our case, our function saves a model's performance logs to a directory named `[dir_name]/[experiment_name]/[current_timestamp]`, where:\n",
        "* `dir_name` is the overall logs directory\n",
        "* `experiment_name` is the particular experiment\n",
        "* `current_timestamp` is the time the experiment started based on Python's [`datetime.datetime().now()`](https://docs.python.org/3/library/datetime.html#datetime.datetime.now)\n",
        "\n",
        "> 🔑 **Note:** Depending on your use case, the above experimenting tracking naming method may work or you might require something more specific. The good news is, the TensorBoard callback makes it easy to track modelling logs as long as you specify where to track them. So you can get as creative as you like with how you name your experiments, just make sure you or your team can understand them.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8UP8vA_nYoI7"
      },
      "source": [
        "## Creating models using TensorFlow Hub\n",
        "\n",
        "In the past we've used TensorFlow to create our own models layer by layer from scratch.\n",
        "\n",
        "Now we're going to do a similar process, except the majority of our model's layers are going to come from [TensorFlow Hub](https://tfhub.dev/).\n",
        "\n",
        "In fact, we're going to use two models from TensorFlow Hub:\n",
        "1. [ResNetV2](https://arxiv.org/abs/1603.05027) -  a state of the art computer vision model architecture from 2016.\n",
        "2. [EfficientNet](https://arxiv.org/abs/1905.11946) - a state of the art computer vision architecture from 2019.\n",
        "\n",
        "State of the art means that at some point, both of these models have achieved the lowest error rate on [ImageNet (ILSVRC-2012-CLS)](http://www.image-net.org/), the gold standard of computer vision benchmarks.\n",
        "\n",
        "You might be wondering, how do you find these models on TensorFlow Hub?\n",
        "\n",
        "Here are the steps I took:\n",
        "\n",
        "1. Go to [tfhub.dev](https://tfhub.dev/).\n",
        "2. Choose your problem domain, e.g. \"Image\" (we're using food images).\n",
        "3. Select your TF version, which in our case is TF2.\n",
        "4. Remove all \"Problem domanin\" filters except for the problem you're working on. \n",
        "  * **Note:** \"Image feature vector\" can be used alongside almost any problem, we'll get to this soon.\n",
        "5. The models listed are all models which could potentially be used for your problem.\n",
        "\n",
        "> 🤔 **Question:** *I see many options for image classification models, how do I know which is best?*\n",
        "\n",
        "You can see a list of state of the art models on [paperswithcode.com](https://www.paperswithcode.com), a resource for collecting the latest in deep learning paper results which have code implementations for the findings they report.\n",
        "\n",
        "Since we're working with images, our target are the [models which perform best on ImageNet](https://paperswithcode.com/sota/image-classification-on-imagenet).\n",
        "\n",
        "You'll probably find not all of the model architectures listed on paperswithcode appear on TensorFlow Hub. And this is okay, we can still use what's available.\n",
        "\n",
        "To find our models, let's narrow down our search using the Architecture tab.\n",
        "\n",
        "6. Select the Architecture tab on TensorFlow Hub and you'll see a dropdown menu of architecture names appear. \n",
        "  * The rule of thumb here is generally, names with larger numbers means better performing models. For example, EfficientNetB4 performs better than EfficientNetB0.\n",
        "    * However, the tradeoff with larger numbers can mean they take longer to compute. \n",
        "7. Select EfficientNetB0 and you should see [something like the following](https://tfhub.dev/s?module-type=image-classification,image-feature-vector&network-architecture=efficientnet-b0&tf-version=tf2):\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/04-tensorflow-hub-efficientnetb0.png)\n",
        "8. Clicking the one titled \"[efficientnet/b0/feature-vector](https://tfhub.dev/tensorflow/efficientnet/b0/feature-vector/1)\" brings us to a page with a button that says \"Copy URL\". That URL is what we can use to harness the power of EfficientNetB0.\n",
        "  * Copying the URL should give you something like this: https://tfhub.dev/tensorflow/efficientnet/b0/feature-vector/1\n",
        "\n",
        "> 🤔 **Question:** *I thought we were doing image classification, why do we choose feature vector and not classification?*\n",
        "\n",
        "Great observation. This is where the differnet types of transfer learning come into play, as is, feature extraction and fine-tuning.\n",
        "\n",
        "1. **\"As is\" transfer learning** is when you take a pretrained model as it is and apply it to your task without any changes. \n",
        "\n",
        "  * For example, many computer vision models are pretrained on the ImageNet dataset which contains 1000 different classes of images. This means passing a single image to this model will produce 1000 different prediction probability values (1 for each class). \n",
        "\n",
        "    * This is helpful if you have 1000 classes of image you'd like to classify and they're all the same as the ImageNet classes, however, it's not helpful if you want to classify only a small subset of classes (such as 10 different kinds of food). Model's with `\"/classification\"` in their name on TensorFlow Hub provide this kind of functionality.\n",
        "\n",
        "2. **Feature extraction transfer learning** is when you take the underlying patterns (also called weights) a pretrained model has learned and adjust its outputs to be more suited to your problem. \n",
        "\n",
        "  * For example, say the pretrained model you were using had 236 different layers (EfficientNetB0 has 236 layers), but the top layer outputs 1000 classes because it was pretrained on ImageNet. To adjust this to your own problem, you might remove the original activation layer and replace it with your own but with the right number of output classes. The important part here is that **only the top few layers become trainable, the rest remain frozen**. \n",
        "\n",
        "    * This way all the underlying patterns remain in the rest of the layers and you can utilise them for your own problem. This kind of transfer learning is very helpful when your data is similar to the data a model has been pretrained on.\n",
        "\n",
        "3. **Fine-tuning transfer learning** is when you take the underlying patterns (also called weights) of a pretrained model and adjust (fine-tune) them to your own problem. \n",
        "\n",
        "    * This usually means training **some, many or all** of the layers in the pretrained model. This is useful when you've got a large dataset (e.g. 100+ images per class) where your data is slightly different to the data the original model was trained on.\n",
        "\n",
        "A common workflow is to \"freeze\" all of the learned patterns in the bottom layers of a pretrained model so they're untrainable. And then train the top 2-3 layers of so the pretrained model can adjust its outputs to your custom data (**feature extraction**).\n",
        "\n",
        "After you've trained the top 2-3 layers, you can then gradually \"unfreeze\" more and more layers and run the training process on your own data to further **fine-tune** the pretrained model.\n",
        "\n",
        "> 🤔 **Question:** *Why train only the top 2-3 layers in feature extraction?*\n",
        "\n",
        "The lower a layer is in a computer vision model as in, the closer it is to the input layer, the larger the features it learn. For example, a bottom layer in a computer vision model to identify images of cats or dogs might learn the outline of legs, where as, layers closer to the output might learn the shape of teeth. Often, you'll want the larger features (learned patterns are also called features) to remain, since these are similar for both animals, where as, the differences remain in the more fine-grained features.\n",
        "\n",
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/04-different-kinds-of-transfer-learning.png)\n",
        "*The different kinds of transfer learning. An original model, a feature extraction model (only top 2-3 layers change) and a fine-tuning model (many or all of original model get changed).*\n",
        "\n",
        "Okay, enough talk, let's see this in action. Once we do, we'll explain what's happening.\n",
        "\n",
        "First we'll import TensorFlow and TensorFlow Hub."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xsoE9nUJNN6s"
      },
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from tensorflow.keras import layers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nvGge7Xevt_F"
      },
      "source": [
        "Now we'll get the feature vector URLs of two common computer vision architectures, [EfficientNetB0 (2019)](https://tfhub.dev/tensorflow/efficientnet/b0/feature-vector/1) and [ResNetV250 (2016)](https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/4) from TensorFlow Hub using the steps above.\n",
        "\n",
        "We're getting both of these because we're going to compare them to see which performs better on our data.\n",
        "\n",
        "> 🔑 **Note:** Comparing different model architecture performance on the same data is a very common practice. The simple reason is because you want to know which model performs best for your problem.\n",
        "\n",
        "> **Update:** As of 14 August 2021, [EfficientNet V2 pretrained models are available on TensorFlow Hub](https://tfhub.dev/google/collections/efficientnet_v2/1). The original code in this notebook uses EfficientNet V1, it has been left unchanged. In [my experiments with this dataset](https://github.com/mrdbourke/tensorflow-deep-learning/discussions/166), V1 outperforms V2. Best to experiment with your own data and see what suits you."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LZfUivHxOCbP"
      },
      "source": [
        "# Resnet 50 V2 feature vector\n",
        "resnet_url = \"https://tfhub.dev/google/imagenet/resnet_v2_50/feature_vector/4\"\n",
        "\n",
        "# Original: EfficientNetB0 feature vector (version 1)\n",
        "efficientnet_url = \"https://tfhub.dev/tensorflow/efficientnet/b0/feature-vector/1\"\n",
        "\n",
        "# # New: EfficientNetB0 feature vector (version 2)\n",
        "# efficientnet_url = \"https://tfhub.dev/google/imagenet/efficientnet_v2_imagenet1k_b0/feature_vector/2\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdwjFaCRwdCX"
      },
      "source": [
        "These URLs link to a saved pretrained model on TensorFlow Hub.\n",
        "\n",
        "When we use them in our model, the model will automatically be downloaded for us to use.\n",
        "\n",
        "To do this, we can use the [`KerasLayer()`](https://www.tensorflow.org/hub/api_docs/python/hub/KerasLayer) model inside the TensorFlow hub library.\n",
        "\n",
        "Since we're going to be comparing two models, to save ourselves code, we'll create a function `create_model()`. This function will take a model's TensorFlow Hub URL, instatiate a Keras Sequential model with the appropriate number of output layers and return the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p7vXoqSjId0f"
      },
      "source": [
        "def create_model(model_url, num_classes=10):\n",
        "  \"\"\"Takes a TensorFlow Hub URL and creates a Keras Sequential model with it.\n",
        "  \n",
        "  Args:\n",
        "    model_url (str): A TensorFlow Hub feature extraction URL.\n",
        "    num_classes (int): Number of output neurons in output layer,\n",
        "      should be equal to number of target classes, default 10.\n",
        "\n",
        "  Returns:\n",
        "    An uncompiled Keras Sequential model with model_url as feature\n",
        "    extractor layer and Dense output layer with num_classes outputs.\n",
        "  \"\"\"\n",
        "  # Download the pretrained model and save it as a Keras layer\n",
        "  feature_extractor_layer = hub.KerasLayer(model_url,\n",
        "                                           trainable=False, # freeze the underlying patterns\n",
        "                                           name='feature_extraction_layer',\n",
        "                                           input_shape=IMAGE_SHAPE+(3,)) # define the input image shape\n",
        "  \n",
        "  # Create our own model\n",
        "  model = tf.keras.Sequential([\n",
        "    feature_extractor_layer, # use the feature extraction layer as the base\n",
        "    layers.Dense(num_classes, activation='softmax', name='output_layer') # create our own output layer      \n",
        "  ])\n",
        "\n",
        "  return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IirF2Ohlz-6i"
      },
      "source": [
        "Great! Now we've got a function for creating a model, we'll use it to first create a model using the ResNetV250 architecture as our feature extraction layer.\n",
        "\n",
        "Once the model is instantiated, we'll compile it using `categorical_crossentropy` as our loss function, the Adam optimizer and accuracy as our metric."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-KVRwwbDT-HL"
      },
      "source": [
        "# Create model\n",
        "resnet_model = create_model(resnet_url, num_classes=train_data_10_percent.num_classes)\n",
        "\n",
        "# Compile\n",
        "resnet_model.compile(loss='categorical_crossentropy',\n",
        "                     optimizer=tf.keras.optimizers.Adam(),\n",
        "                     metrics=['accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZinVcxBi0jsv"
      },
      "source": [
        "![](https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/images/04-resnet-feature-extractor.png)\n",
        "*What our current model looks like. A ResNet50V2 backbone with a custom dense layer on top (10 classes instead of 1000 ImageNet classes). **Note:** The Image shows ResNet34 instead of ResNet50. **Image source:** https://arxiv.org/abs/1512.03385.*\n",
        "\n",
        "Beautiful. Time to fit the model.\n",
        "\n",
        "We've got the training data ready in `train_data_10_percent` as well as the test data saved as `test_data`.\n",
        "\n",
        "But before we call the fit function, there's one more thing we're going to add, a callback. More specifically, a TensorBoard callback so we can track the performance of our model on TensorBoard.\n",
        "\n",
        "We can add a callback to our model by using the `callbacks` parameter in the fit function.\n",
        "\n",
        "In our case, we'll pass the `callbacks` parameter the `create_tensorboard_callback()` we created earlier with some specific inputs so we know what experiments we're running.\n",
        "\n",
        "Let's keep this experiment short and train for 5 epochs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2GTl0fwE0Hx6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4cf373f4-7143-45b0-ddfc-d1cb009e555a"
      },
      "source": [
        "# Fit the model\n",
        "resnet_history = resnet_model.fit(train_data_10_percent,\n",
        "                                  epochs=5,\n",
        "                                  steps_per_epoch=len(train_data_10_percent),\n",
        "                                  validation_data=test_data,\n",
        "                                  validation_steps=len(test_data),\n",
        "                                  # Add TensorBoard callback to model (callbacks parameter takes a list)\n",
        "                                  callbacks=[create_tensorboard_callback(dir_name=\"tensorflow_hub\", # save experiment logs here\n",
        "                                                                         experiment_name=\"resnet50V2\")]) # name of log files"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: tensorflow_hub/resnet50V2/20210115-011336\n",
            "Epoch 1/5\n",
            "24/24 [==============================] - 29s 814ms/step - loss: 2.6383 - accuracy: 0.1690 - val_loss: 1.3164 - val_accuracy: 0.5852\n",
            "Epoch 2/5\n",
            "24/24 [==============================] - 17s 737ms/step - loss: 1.0979 - accuracy: 0.6683 - val_loss: 0.9081 - val_accuracy: 0.7124\n",
            "Epoch 3/5\n",
            "24/24 [==============================] - 17s 732ms/step - loss: 0.7306 - accuracy: 0.8073 - val_loss: 0.7874 - val_accuracy: 0.7500\n",
            "Epoch 4/5\n",
            "24/24 [==============================] - 17s 735ms/step - loss: 0.5689 - accuracy: 0.8540 - val_loss: 0.7224 - val_accuracy: 0.7640\n",
            "Epoch 5/5\n",
            "24/24 [==============================] - 17s 740ms/step - loss: 0.4724 - accuracy: 0.8913 - val_loss: 0.6969 - val_accuracy: 0.7744\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i5SuOe672UJi"
      },
      "source": [
        "Wow! \n",
        "\n",
        "It seems that after only 5 epochs, the ResNetV250 feature extraction model was able to blow any of the architectures we made out of the water, achieving around 90% accuracy on the training set and nearly 80% accuracy on the test set...**with only 10 percent of the training images!**\n",
        "\n",
        "That goes to show the power of transfer learning. And it's one of the main reasons whenever you're trying to model your own datasets, you should look into what pretrained models already exist.\n",
        "\n",
        "Let's check out our model's training curves using our `plot_loss_curves` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ot2QPj41ODCQ"
      },
      "source": [
        "# If you wanted to, you could really turn this into a helper function to load in with a helper.py script...\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Plot the validation and training data separately\n",
        "def plot_loss_curves(history):\n",
        "  \"\"\"\n",
        "  Returns separate loss curves for training and validation metrics.\n",
        "  \"\"\" \n",
        "  loss = history.history['loss']\n",
        "  val_loss = history.history['val_loss']\n",
        "\n",
        "  accuracy = history.history['accuracy']\n",
        "  val_accuracy = history.history['val_accuracy']\n",
        "\n",
        "  epochs = range(len(history.history['loss']))\n",
        "\n",
        "  # Plot loss\n",
        "  plt.plot(epochs, loss, label='training_loss')\n",
        "  plt.plot(epochs, val_loss, label='val_loss')\n",
        "  plt.title('Loss')\n",
        "  plt.xlabel('Epochs')\n",
        "  plt.legend()\n",
        "\n",
        "  # Plot accuracy\n",
        "  plt.figure()\n",
        "  plt.plot(epochs, accuracy, label='training_accuracy')\n",
        "  plt.plot(epochs, val_accuracy, label='val_accuracy')\n",
        "  plt.title('Accuracy')\n",
        "  plt.xlabel('Epochs')\n",
        "  plt.legend();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ywQ9Wr9UODJ_",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 573
        },
        "outputId": "a225ebca-4308-4175-95e9-849c4bd21ab4"
      },
      "source": [
        "plot_loss_curves(resnet_history)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEWCAYAAABollyxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU5dnH8e+dnSxASEjYSSLIrqCRRQHFFXGhVVtcanFFca21VmutotXWautbqVVL1brUtS4VFRQQFK2AJBiWsBO2hC0k7Fu2+/3jnCRDyDIhk5lkcn+uay5mzjLnntH8zpnnnPM8oqoYY4wJXiGBLsAYY0zjsqA3xpggZ0FvjDFBzoLeGGOCnAW9McYEOQt6Y4wJchb0xhgT5CzoTYsmIhtE5NxA12FMY7KgN8aYIGdBb0wVIhIpIn8VkS3u468iEunOSxSRT0Vkt4gUisg3IhLizrtfRPJEZJ+IrBKRcwL7SYxxhAW6AGOaoN8CQ4GBgAIfAw8BvwPuBXKB9u6yQwEVkV7AHcBpqrpFRFKAUP+WbUz17IjemGNdAzymqjtUNR94FLjWnVcMdAS6q2qxqn6jTodRpUAk0FdEwlV1g6quC0j1xlRhQW/MsToBGz1eb3SnATwNrAVmiEiOiDwAoKprgV8Ak4AdIvKOiHTCmCbAgt6YY20Bunu87uZOQ1X3qeq9qpoGXAr8srwtXlXfUtXh7roK/Mm/ZRtTPQt6YyBcRKLKH8DbwEMi0l5EEoGHgX8DiMjFItJDRATYg9NkUyYivUTkbPek7WHgEFAWmI9jzNEs6I2BaTjBXP6IAjKAJcBSYBHwuLtsT2AWsB+YBzyvqnNw2uefBHYC24Ak4Df++wjG1Exs4BFjjAludkRvjDFBzoLeGGOCXJ1BLyJdRWSOiCwXkWwRubuaZa4RkSUislREvhORkz3mbXCnZ4lIhq8/gDHGmNp5c2dsCXCvqi4SkTggU0Rmqupyj2XWA2eq6i4RuRCYAgzxmD9KVXf6rmxjjDHeqjPoVXUrsNV9vk9EVgCdgeUey3znscp8oEtDikpMTNSUlJSGvIUxxrQomZmZO1W1fXXz6tXXjdt/xyBgQS2L3QhM93itOHcRKvAPVZ1Sw3tPACYAdOvWjYwMa+UxxhhvicjGmuZ5HfQiEgt8APxCVffWsMwonKAf7jF5uKrmiUgSMFNEVqrq3KrrujuAKQDp6el2zacxxviIV1fdiEg4Tsi/qaof1rDMScBLwFhVLSifrqp57r87gI+AwQ0t2hhjjPe8uepGgJeBFar6TA3LdAM+BK5V1dUe02PcE7iISAxwPrDMF4UbY4zxjjdNN2fgdNG6VESy3GkP4nT0hKq+iNMXSALwvLNfoERV04Fk4CN3Whjwlqp+7tNPYIzxi+LiYnJzczl8+HCgS2nRoqKi6NKlC+Hh4V6v481VN98CUscyNwE3VTM9Bzj52DWMMc1Nbm4ucXFxpKSk4B68GT9TVQoKCsjNzSU1NdXr9ezOWGOMVw4fPkxCQoKFfACJCAkJCfX+VWVBb4zxmoV84B3Pf4OgCfqikjL+8fU6MjcWBroUY4xpUoIm6EvKynj1uw08/HE2pWV2Gb4xxpQLmqCPjgjjwTF9yN6yl3cWbgp0OcYYH9u9ezfPP/98vdcbM2YMu3fvrnWZhx9+mFmzZh1vadWKjY316fs1RNAEPcDFJ3VkSGo7/vzFKnYfLAp0OcYYH6op6EtKSmpdb9q0abRt27bWZR577DHOPffcBtXXlNWrr5umTkSYdGk/Lpr8Dc/MXM1jY/sHuiRjgtKjn2SzfEu1PaEct76dWvPIJf1qnP/AAw+wbt06Bg4cSHh4OFFRUcTHx7Ny5UpWr17Nj370IzZv3szhw4e5++67mTBhAgApKSlkZGSwf/9+LrzwQoYPH853331H586d+fjjj2nVqhXXXXcdF198MVdccQUpKSmMHz+eTz75hOLiYv7zn//Qu3dv8vPzufrqq9myZQvDhg1j5syZZGZmkpiYWOvnUlV+/etfM336dESEhx56iHHjxrF161bGjRvH3r17KSkp4YUXXuD000/nxhtvJCMjAxHhhhtu4J577mnwdxtUR/QAfTq25tqh3fn3/I0+/x/RGBM4Tz75JCeccAJZWVk8/fTTLFq0iGeffZbVq52b8V955RUyMzPJyMhg8uTJFBQUHPMea9as4fbbbyc7O5u2bdvywQcfVLutxMREFi1axMSJE/nzn/8MwKOPPsrZZ59NdnY2V1xxBZs2eddE/OGHH5KVlcXixYuZNWsW9913H1u3buWtt97iggsuqJg3cOBAsrKyyMvLY9myZSxdupTrr7/+OL+towXVEX25X57Xi6mLtzBpajbv3jLULgkzxsdqO/L2l8GDBx9109DkyZP56KOPANi8eTNr1qwhISHhqHVSU1MZOHAgAKeeeiobNmyo9r0vu+yyimU+/NDp3uvbb7+teP/Ro0cTHx/vVZ3ffvstV111FaGhoSQnJ3PmmWeycOFCTjvtNG644QaKi4v50Y9+xMCBA0lLSyMnJ4c777yTiy66iPPPP9/7L6QWQXdED9AmOpz7LujN9xsK+WTJ1kCXY4xpBDExMRXPv/rqK2bNmsW8efNYvHgxgwYNqvamosjIyIrnoaGhNbbvly9X2zINNXLkSObOnUvnzp257rrreP3114mPj2fx4sWcddZZvPjii9x00zEdDhyXoAx6gHGndaV/59b84bMVHDjSOP+hjDH+ExcXx759+6qdt2fPHuLj44mOjmblypXMnz/f59s/44wzeO+99wCYMWMGu3bt8mq9ESNG8O6771JaWkp+fj5z585l8ODBbNy4keTkZG6++WZuuukmFi1axM6dOykrK+Pyyy/n8ccfZ9GiRT6pPSibbgBCQ4RHL+3H5S/M4+9z1vLr0b0DXZIxpgESEhI444wz6N+/P61atSI5Obli3ujRo3nxxRfp06cPvXr1YujQoT7f/iOPPMJVV13FG2+8wbBhw+jQoQNxcXF1rvfjH/+YefPmcfLJJyMiPPXUU3To0IHXXnuNp59+mvDwcGJjY3n99dfJy8vj+uuvp6ysDIA//vGPPqldVJvezUXp6enqqxGmfvluFp8u2cqMe0aSkhhT9wrGmGqtWLGCPn36BLqMgDly5AihoaGEhYUxb948Jk6cSFZWVt0rNoLq/luISKbba/AxgrbpptwDF/YmIiyE33+6vO6FjTGmBps2beK0007j5JNP5q677uKf//xnoEvyWtA23ZRLah3FXef04A/TVjJn5Q5G9U4KdEnGmGaoZ8+e/PDDD0dNKygo4Jxzzjlm2S+//PKYK34CKeiDHuC601N5Z+FmHv0km9N7JBAZFhrokowxQSAhISFgzTf1EfRNNwARYSE8ckk/NhQc5JVvNwS6HGOM8asWEfQAZ57YnvP6JvO32WvYtseGQjPGtBwtJugBfndRX0rKlD9OXxHoUowxxm/qDHoR6Soic0RkuYhki8jd1SwjIjJZRNaKyBIROcVj3ngRWeM+xvv6A9RHt4RobhmZxsdZW1i4wQYoMca0DN4c0ZcA96pqX2AocLuI9K2yzIVAT/cxAXgBQETaAY8AQ4DBwCMi4l0HEY3ktrN60KlNFI/YACXGBLXa+oPfsGED/fu3nN5t6wx6Vd2qqovc5/uAFUDnKouNBV5Xx3ygrYh0BC4AZqpqoaruAmYCo336CeqpVUQov72oL8u37uWt722AEmNM8KvX5ZUikgIMAhZUmdUZ2OzxOtedVtP0gBozoAPD0hL4y4xVXDygI/ExEYEuyZjmZfoDsG2pb9+zwwC48MkaZz/wwAN07dqV22+/HYBJkyYRFhbGnDlz2LVrF8XFxTz++OOMHTu2Xps9fPgwEydOJCMjg7CwMJ555hlGjRpFdnY2119/PUVFRZSVlfHBBx/QqVMnfvrTn5Kbm0tpaSm/+93vGDduXIM+tj94fTJWRGKBD4BfqKrPO3oXkQkikiEiGfn5+b5++6rbYtKl/dh3uIS/zFzVqNsyxvjGuHHjKjoVA3jvvfcYP348H330EYsWLWLOnDnce++91Ldbl7///e+ICEuXLuXtt99m/PjxHD58mBdffJG7776brKwsMjIy6NKlC59//jmdOnVi8eLFLFu2jNGjA9pA4TWvjuhFJBwn5N9U1Q+rWSQP6Orxuos7LQ84q8r0r6rbhqpOAaaA09eNN3U1RK8OcVw7tDuvz9vAVYO70a9Tm8bepDHBo5Yj78YyaNAgduzYwZYtW8jPzyc+Pp4OHTpwzz33MHfuXEJCQsjLy2P79u106NDB6/f99ttvufPOOwHo3bs33bt3Z/Xq1QwbNownnniC3NxcLrvsMnr27MmAAQO49957uf/++7n44osZMWJEY31cn/LmqhsBXgZWqOozNSw2Ffi5e/XNUGCPqm4FvgDOF5F49yTs+e60JuGe806kbXQEk6Zm1/sowBjjfz/5yU94//33effddxk3bhxvvvkm+fn5ZGZmkpWVRXJycrX90B+Pq6++mqlTp9KqVSvGjBnD7NmzOfHEE1m0aBEDBgzgoYce4rHHHvPJthqbN003ZwDXAmeLSJb7GCMit4rIre4y04AcYC3wT+A2AFUtBH4PLHQfj7nTmoQ2rcL59QW9WLhhF1MXbwl0OcaYOowbN4533nmH999/n5/85Cfs2bOHpKQkwsPDmTNnDhs3bqz3e44YMYI333wTgNWrV7Np0yZ69epFTk4OaWlp3HXXXYwdO5YlS5awZcsWoqOj+dnPfsZ9993ns/7iG1udTTeq+i1Q61h86hwO317DvFeAV46rOj/4aXpX3vp+E3+YtoJz+yQTE9kiuv8xplnq168f+/bto3PnznTs2JFrrrmGSy65hAEDBpCenk7v3vUfd+K2225j4sSJDBgwgLCwMF599VUiIyN57733eOONNwgPD6dDhw48+OCDLFy4kPvuu4+QkBDCw8N54YUXGuFT+l7Q90fvjUWbdnHZ899x65kn8MCFNkCJMdVp6f3RNyXWH/1xOKVbPJef0oWXv80hJ39/oMsxxhifsqB33X9hLyLDQm2AEmOCyNKlSxk4cOBRjyFDhgS6LL+zBmlXUlwUvzi3J49/toIvV2znnD7Jda9kTAujqjgX4jUPAwYMaBb9xdfH8TS32xG9h/Gnp9AjKZbHPl3O4eLSQJdjTJMSFRVFQUGBXYocQKpKQUEBUVFR9VrPjug9hIeG8Mglfbn25e95+dv13D6qR6BLMqbJ6NKlC7m5uTT2neumdlFRUXTp0qVe61jQVzGiZ3su6JfMc7PXctkpnenYplWgSzKmSQgPDyc1NTXQZZjjYE031Xjoor6UqfKHaSsDXYoxxjSYBX01uraL5pYzT+CTxVuYn1MQ6HKMMaZBLOhrMPHME+jcthWTpmZTUloW6HKMMea4WdDXoFVEKA9d1IeV2/bZACXGmGbNgr4Wo/t34IweCfxlxmoKDxQFuhxjjDkuFvS1EBEeuaQf+4+U8OcZNkCJMaZ5sqCvw4nJcYwflsLb329iWd6eQJdjjDH1ZkHvhV+c15OEmAgesQFKjDHNkAW9F1pHhfPrC3qTuXEX/83KC3Q5xhhTLxb0Xrri1C6c3LUtf5y2kv1HSgJdjjHGeM2C3kshIcKjl/Zjx74j/O3LNYEuxxhjvGZBXw8Du7blp+ldeOV/61lnA5QYY5oJC/p6uu+C3kSFhfLoJ8vtxKwxplmoM+hF5BUR2SEiy2qYf5+IZLmPZSJSKiLt3HkbRGSpO89/g8A2ovZxkfzivBOZuzqfWSt2BLocY4ypkzdH9K8Co2uaqapPq+pAVR0I/Ab4WlULPRYZ5c6vdtDa5ujnw7rTMymW39sAJcaYZqDOoFfVuUBhXcu5rgLeblBFzUB4aAiTLu3HpsKDvPRNTqDLMcaYWvmsjV5EonGO/D/wmKzADBHJFJEJdaw/QUQyRCSjOYxgc0aPRC7s34G/z1nHlt2HAl2OMcbUyJcnYy8B/lel2Wa4qp4CXAjcLiIja1pZVaeoarqqprdv396HZTWe317UB0V5YtqKQJdijDE18mXQX0mVZhtVzXP/3QF8BAz24fYCrkt8NBPP7MFnS7by3bqdgS7HGGOq5ZOgF5E2wJnAxx7TYkQkrvw5cD5Q7ZU7zdktZ6bRJb4Vj05dbgOUGGOaJG8ur3wbmAf0EpFcEblRRG4VkVs9FvsxMENVD3hMSwa+FZHFwPfAZ6r6uS+LbwqiwkN56KK+rNq+j3/P3xjocowx5hhhdS2gqld5scyrOJdhek7LAU4+3sKakwv6JTOiZyLPzFzNJSd3IiE2MtAlGWNMBbsz1gecAUr6crCo1AYoMcY0ORb0PtIjKY7rTk/hnYWbWZK7O9DlGGNMBQt6H7r73J4kxETyyNRsysqsHxxjTNNgQe9DcVHhPHBhb37YtJsPf7ABSowxTYMFvY9dNqgzg7q15cnpK9l3uDjQ5RhjjAW9r5UPUFJw4AiTbYASY0wTYEHfCE7q0pZx6V351/82sHbHvkCXY4xp4SzoG8mvLuhFqwgboMQYE3gW9I0kMTaSX553It+s2cmM5dsDXY4xpgWzoG9E1w7tzonJNkCJMSawLOgbUZg7QEnurkP842sboMQYExgW9I3s9BMSuWhAR57/ai25uw4GuhxjTAtkQe8HD17UBxH4gw1QYowJAAt6P+jcthW3n9WDaUu38b+1NkCJMca/LOj95OaRaXRt14pHP8mm2AYoMcb4kQW9n0SFh/K7i/qyevt+3phnA5QYY/zHgt6PzuubzMgT2/N/s1azc/+RQJdjjGkhLOj9SER4+OK+HCoq5anPVwa6HGNMC2FB72c9kmK5YXgq72XkkrXZBigxxjQ+bwYHf0VEdojIshrmnyUie0Qky3087DFvtIisEpG1IvKALwtvzu48uwft42yAEmOMf3hzRP8qMLqOZb5R1YHu4zEAEQkF/g5cCPQFrhKRvg0pNljERYXzmwt7s3jzbt5flBvocowxQa7OoFfVuUDhcbz3YGCtquaoahHwDjD2ON4nKP1oYGdO6daWpz5fyV4boMQY04h81UY/TEQWi8h0EennTusMbPZYJtedVi0RmSAiGSKSkZ+f76Oymq6QEOGxsf0pOFDEs7NsgBJjTOPxRdAvArqr6snA34D/Hs+bqOoUVU1X1fT27dv7oKymr3/nNlx5Wjde+24Da7bbACXGmMbR4KBX1b2qut99Pg0IF5FEIA/o6rFoF3ea8fCr808kOiKUSZ9k2wAlxphG0eCgF5EOIiLu88HuexYAC4GeIpIqIhHAlcDUhm4v2CTERnLv+b3439oCvsjeFuhyjDFByJvLK98G5gG9RCRXRG4UkVtF5FZ3kSuAZSKyGJgMXKmOEuAO4AtgBfCeqmY3zsdo3q4Z0o3eHeL4/acrOFRkA5QYY3xLmmJzQXp6umZkZAS6DL+an1PAlVPmc/c5PbnnvBMDXY4xppkRkUxVTa9unt0Z20QMTUvg4pM68uLX69hcaAOUGGN8x4K+CfntRX0IEeGJz2yAEmOM71jQNyEd27TijrN78Hn2Nr5dYwOUGGN8w4K+iblxeCrd2kUzyQYoMcb4iAV9ExMVHsrDF/dl7Y79vPbdhkCXY4wJAhb0TdA5fZI4q1d7np21hvx9NkCJMaZhLOiboPIBSg6XlPInG6DEGNNAFvRNVFp7Z4CS9zNz+WHTrkCXY4xpxizom7A7z+5JUlwkk2yAEmNMA1jQN2GxkWE8OKYPi3P38J/MzXWvYIwx1bCgb+LGDuxEevd4nvp8FXsO2QAlxpj6s6Bv4kSESZf2o/BgEX+dtTrQ5RhjmiEL+magf+c2XD24G6/P28iqbTZAiTGmfoIr6MuC907SX53fi9jIMCZNtQFKjDH1EzxBX1YK/7oQvnoSig8Huhqfi4+J4Ffnn8i8nAKmL7MBSowx3gueoC86AK07wVd/hOeHwppZga7I564e0p0+HVvzxGc2QIkxxnvBE/RRreEn/4Jr/wshofDm5fDutbAnN9CV+UxoiPDopf3I232IF75aG+hyjDHNRPAEfbkTRsHE7+Ds38GaGfDcYPjfs1AaHJcmDk5tx6Und+LFuTlsKrABSowxdQu+oAcIi4SRv4LbF0DqSJj5MLw4Ajb8L9CV+cSDY/oQFiI8/tnyQJdijGkGvBkc/BUR2SEiy2qYf42ILBGRpSLynYic7DFvgzs9S0T8PwhsfApc/Q5c+bbThv/qGPjwFti/w++l+FKHNlHccXYPZizfztzV+YEuxxjTxHlzRP8qMLqW+euBM1V1APB7YEqV+aNUdWBNg9b6Re8xztH9iF/Bsg/gb+nw/T+dK3WaqRuHp5KS4AxQUlQSvJeVGmMars6gV9W5QGEt879T1fLuFecDXXxUm29FRMM5v4Pb5kHnQTDtV/DPUZCbGejKjktkWCgPX9KXnPwDNkCJMaZWvm6jvxGY7vFagRkikikiE2pbUUQmiEiGiGTk5zdic0RiT+fKnCtegX3b4aVz4JO74WCN+7Im6+zeyZzdO4lnv1zDjr3Bd++AMcY3fBb0IjIKJ+jv95g8XFVPAS4EbheRkTWtr6pTVDVdVdPbt2/vq7JqKhb6Xw53LISht8GiN+C5dPjh383u7tqHL+5LUUkZT9oAJcaYGvgk6EXkJOAlYKyqFpRPV9U8998dwEfAYF9sz2eiWsPoP8AtcyGhJ3x8O/xrNGyr9rxzk5SSGMONI1L5cFEemRub368SY0zja3DQi0g34EPgWlVd7TE9RkTiyp8D5wNNM0E79Ifrp8PY56FgLfxjJHz+Gzi8N9CVeeWOUT3o0DqKSVOXU2oDlBhjqvDm8sq3gXlALxHJFZEbReRWEbnVXeRhIAF4vspllMnAtyKyGPge+ExVP2+Ez+AbISEw6Bq4IwNO+TnMfwGeOw2Wvg9NvBOxmMgwfjOmN0vz9vBehg1QYow5mjTFnhDT09M1I8P/l90fJS8TPv0lbM2C1DPhor84J3KbKFVl3D/mszZ/P3PuPYs20eGBLskY40ciklnTZezBeWesL3Q+FW6eDWP+DFuy4Plh8OVjUNQ0ux0oH6Bk98Ei/s8GKDHGeLCgr01IKAy+Ge7MgAFXwDd/gb8PgZXTAl1Ztfp2as01Q7rzxvyNrNzWPM4vGGManwW9N2KT4McvwnXTICIG3rkK3hoHuzYEurJj3Hv+icRFhfHIxzZAiTHGYUFfHylnwK3fwHm/h/XfOEf3c5+GkiOBrqxC2+gIfnV+LxasL+TTJVsDXY4xpgmwoK+v0HA44y7nZqsTL4DZj8MLp8O62YGurMJVg7vRr1Nr/jBtBQeLSgJdjjEmwCzoj1ebzvDT1+FnH4CWwRs/hv9cB3u3BLqyigFKtu45zPNz1gW6HGNMgFnQN1SPc2HiPDjrQeck7XOnwXfPBXygk/SUdvxoYCemzM1hY8GBgNZijAksC3pfCI+Cs+6H2+dD99Nhxm/hH2fCxnkBLes3Y/oQHir8/tMVAa3DGBNYFvS+1C4Nrn4Pxr0Jh/c4/eb89zbYH5jBQZJbR3HnOT2ZtWI7X61q3oOtGGOOnwW9r4lAn4vhju9h+D2w5F147lRY+HJABjq54YxU0hJjeOyT5TZAiTEtlAV9Y4mIgXMnOQOVdzgJPvslvHQubPnBv2WEhfC7S/qSs/MAr/xvvV+3bYxpGizoG1v7XjD+E7jsJdibB1NGwWf3wqFdda/rI6N6JXFunyT+9uUattsAJca0OBb0/iACJ/3EufZ+yC2Q8Yozbm3W237rGfN3F/eluFR5croNUGJMS2NB709RbeDCP8GEr6BdKvz3VvjXGNi+vNE33T0hhptHpvLRD3lkbLABSoxpSSzoA6HjyXDDDLj0b5C/Al4cDl/8Fo7sa9TN3j6qBx3bRPHI1GwboMSYFsSCPlBCQpwBTu7IdAY8mfccPDcYsj9qtOac6IgwHhzTh+wte3ln4aZG2YYxpumxoA+0mATnyP7GWc7z/1wH/74MChqn64KLT+rIkNR2/PmLVew+WNQo2zDGNC0W9E1F19Pg5q/gwqcgNwOeHwqzn4DiQz7dTPkAJXsOFfOXGTZAiTEtgQV9UxIa5lyVc8dC6PsjmPuU0xXy6i98upk+HVtz7dDuvLlgI9lb9vj0vY0xTY8FfVMU1wEu/6dz/X1YFLz1U3jnGtjtu3b1X57Xi/joCK54YR5/nL6Cgv1Np099Y4xveRX0IvKKiOwQkWU1zBcRmSwia0VkiYic4jFvvIiscR/jfVV4i5A6Em791rnDdt1s52TtN89AScPb1ttEh/PBxNO5oF8yU+bmMPxPcyzwjQlS4s1wcyIyEtgPvK6q/auZPwa4ExgDDAGeVdUhItIOyADSAQUygVNVtdbbQtPT0zUjI6O+nyW47d4Mnz8AKz+FxBOdQcvTzvTJW6/dsZ+/zV7D1MVbaBUeys+HpXDziFQSYiN98v7GmMYnIpmqml7dPK+O6FV1LlDbXTZjcXYCqqrzgbYi0hG4AJipqoVuuM8ERtevfANA265w5Ztw9X+gtAhevxQ+uAn2bWvwW/dIiuXZKwcx856RnNc3mX/MXceIp+bw5PSVFB6wK3OMae581UbfGdjs8TrXnVbT9GOIyAQRyRCRjPz8wHTr2yyceD7cNh/OvB+Wf+wMdDL/BSht+JCBPZLiKgL/3D5O4A//02z+9LkFvjHNWZM5GauqU1Q1XVXT27dvH+hymrbwVjDqQSfwu5zmNOlMOQs2f++Tt++RFMfkqyoD/8Wv1zHiT7N5ygLfmGbJV0GfB3T1eN3FnVbTdOMLCSc4Y9b+9HU4WAAvnwcf3wEHCnzy9uWBP+MXIzm7TzIveAT+Lgt8Y5oNXwX9VODn7tU3Q4E9qroV+AI4X0TiRSQeON+dZnxFBPqOda69P/0uWPy2M9BJ5qtQ5puBRnomx/G3qwbxxS9GMqp3Ei987TTpPP2FBb4xzYG3V928DZwFJALbgUeAcABVfVFEBHgO50TrQeB6Vc1w170BeNB9qydU9V91bc+uummAHSuc/u43/g86p8PFzzidqPnQ6u37mPzlGj5bupXo8FCuOyOFm4anER8T4QXX7foAABTDSURBVNPtGGO8V9tVN14Fvb9Z0DeQqjOE4YyHnCad026Gs3/rdJPsQ6u27WPy7DVMW7qVmIgwrjs9hZtGpNI22gLfGH+zoG+pDu2G2Y9DxssQnQgXPAEDfuI09/hQeeB/tmQrsZFhXH9GCjcOt8A3xp8s6Fu6LT84zTl5mZAywrnZKqm3zzezaltlk44FvjH+ZUFvnBOzi16DWZOgaD8MuwPO/LUziLmPrdy2l8lfrmHa0m3EVQR+Gm2iw32+LWOMw4LeVDqwE2Y+Aln/htZd4LQboMNJkNwP4jr6tFnHAt8Y/7GgN8faNB+m3w9bsyqntYqH5P5O6Jc/2veBiOgGbWrFVifwpy9zA394KjeekWqBb4wPWdCbmh3aDTuWw/Zs2L7M/Xc5FB9wFxDnxqzkfkfvBNp0c4ZDrIejAj8qjBvOSOWG4am0aWWBb0xDWdCb+ikrg90b3ND32AEUrsfphBSIiIPkvh5H//0hqS9Eta7z7ZdvcQL/82wLfGN8xYLe+MaR/ZC/0uPI390JHPYYpapttyrNP/2hXRqEhB7zdtlb9jD5yzV8kb2duKgwbhyeyvVnWOAbczws6E3jUYW9eVWafrJh5xrQUmeZsChI6nN0809SP2cwdKoP/BuGp9I6ygLfGG9Z0Bv/Kz4MO1cdfeS/bRkc3Fm5TFzHo47810h3nllUxvQVhbSOCuPG4WlcPzzFAt8YL1jQm6Zj/45jm37yVzmDqQCEhHOobQ8WF3Vm9q72bApPJX3ISH561qm0bmU3XhlTEwt607SVFkPB2mObf/ZW9mhdSBwH2vQmqecpRHZ2r/tv39vpm98YU2vQh/m7GGOOERrutOEn9YEBV1ROP1gIO5azZVUGa5cuoPWuVSRm/Asy3KN/CYGEHkef+E3uB226+rw/H2OaMwt603RFt4OU4XRKGU6nC2Bp7h7umrWSNauWckrkFq7supdBkXmEb/kBsj+qXC+y9dE3fSX3d3YikXGB+yzGBJA13ZhmZ2nuHp79cjWzVuygTatwbh6RyvhTE4jbu7ZK+382HNlbuWJ8yrGXfsanVHvppzHNjbXRm6DkGfhto8O5eUQaPx/Wnbjyq3RUYc/mY9v+C9aCuqNvhUdXc+lnX+fXhDHNiAW9CWpLcnfz7Kw1fLmyMvDHn55CbGQNLZPFh9wbv7KdSz7LdwKHCiuXad352Lb/hB7O+QRjmiALetMiLN68m2e/XMNsbwPfkyrs21bNjV+roKzEWSY0wrnzN7YDxCU7/8YmQVwHiE2u/LdVvJ0MNn5nQW9alAYFflUlRVCwxj36Xwq7N8H+7c5OYf92KD547DqhEU7glz9q2inEtLdfCMZnGhz0IjIaeBYIBV5S1SerzP8/YJT7MhpIUtW27rxSYKk7b5OqXlrX9izojS9kbd7Ns7NWM2dVPvHR4dw8Mo2fDzvOwK+OqjOIy77tsN8N/vLn+7Y7r8t3Cp7NQhUEohMqw7+2nUIjDBBjgkuDgl5EQoHVwHlALrAQuEpVl9ew/J3AIFW9wX29X1Vj61OwBb3xpaqBP2HkCfx8WHdifBX43igpggM7qt8p7N9R+Qth//bKpiJPEbEezUNJNTQfdXBOIluzUYvU0KAfBkxS1Qvc178BUNU/1rD8d8AjqjrTfW1Bb5qEHzbt4tkv1/BVIAO/LmVlcGiX+6vA3QlU/EKoslMo2n/s+iHh7o6gpp2C+6shJgnCrEuJYNLQoL8CGK2qN7mvrwWGqOod1SzbHZgPdFF1ui4UkRIgCygBnlTV/9awnQnABIBu3bqdunHjRi8/njH1s2jTLp6dtYavV+fTLiaCCSPTuHZoEwt8bxzZf3TzUE07Bc+O5DxFJ3g0GdWyU7AbzZoFfwb9/Tghf6fHtM6qmiciacBs4BxVXVfbNu2I3vhD1cC/ZWQa1w7rTnREMwv8upQWw4H8yl8Cx+wUPB7lnct5Co9xw7+OnUJ0Qr1HHTO+09C+bvKArh6vu7jTqnMlcLvnBFXNc//NEZGvgEFArUFvjD+c0i2e124YTOZGp0nnj9NXMmVujnOEH0yBHxoOrTs5j9qous1G248+Z+B5gnl7NqybffQdx+VCwpwmIc8TydHtnJvSwls54xKUP694lM+rMi0s0s41+JA3R/RhOCdjz8EJ+IXA1aqaXWW53sDnQKq6byoi8cBBVT0iIonAPGBsTSdyy9kRvQmEzI27+Ous1XyzZicJMRHccmYaPxsaRIHvS0UHqzQbVdkplD8/tAvKio9jA1IZ/FV3AuFRNe8gPOeFR7s7lzrmBUkXGL64vHIM8FecyytfUdUnROQxIENVp7rLTAKiVPUBj/VOB/4BlAEhwF9V9eW6tmdBbwIpc2Mhf521xgLfV0pLoOSQc0dyxeMglBx2/i2uZl7xoSrrHHQGs6ltXsmh46svNPLYXxlH7SDqMe+YnY/HvEa+Z8JumDLmOGRsKOTZL53AT4yN4JaRJ3D1kG7N76RtS1FW5u48PHcEVXckByuXqW5e8aHad0Dl88r7SqqPkLA6fp1EOTfRXfLX4/r4FvTGNIBn4IeHCid3acuQtHYMTUvg1O7xdqTf0qg6J62r/so4ZgdT27wqO5HyeRGxcNPM4yrLgt4YH8jcuIsZy7exIKeQpXl7KC1TwkKEk7q0YWhaQkXw2xG/CQQLemN8bP+REjI2FLJgfSHzcwpYklsZ/APc4B+S2o70lHa+63LBmFpY0BvTyA4cKSFz4y7m5xRUBH9JmRIaIgzo3KaiqSe9e3xlf/nG+JAFvTF+drCoMvgX5BSyOHc3xaVO8Pfv1No54k9zjvhbW/AbH7CgNybADhaVsGjjbhasd474szY7wR8i0L9zZVPPaakW/Ob4WNAb08QcKirlh01uU8/6QrI27aaotIwQgX6d2jAk1WnqOS21HW1aWfCbulnQG9PEHS4uZdGmXSzIcU7u/rB5N0UlZYhA346tK474h6Qm0Cbagt8cy4LemGbmcHEpWZt3V5zcXbSpMvj7dGhdcXJ3SGo72kZbd8PGgt6YZu9wcSmLN++uuJwzc+MujrjB3ys5ruI6/iGp7YiPseBviSzojQkyR0pKWZK7h/nrCpi/3gn+w8XObfm9O8RVhP7g1HYkxEYGuFrjDxb0xgS5opIyluQ6TT0L1heSsWEXh4pLAeeIv7ypZ3BqOxIt+IOSBb0xLUxRSRlL83YzP6eyqedgkRP8PZNiK5p6Bqe2o32cBX8wsKA3poUrLi1jad6eihu4MjYUcsAN/h5JsRWXcw5Ja0dSXFSAqzXHw4LeGHOU4tIyluXtqTi5u3B9ZfCntY+pOOIfmtqOpNYW/M2BBb0xplYlpWVkb9lbcTnnwg272H+kBIC0xBiGpCUwNM25jr9DGwv+psiC3hhTLyWlZSzfureiqef79YXsc4M/NTHmqKaejm1aBbhaAxb0xpgGKi1TVmytPOJfsL6QfYed4O+eEM3Q1ASGnuAc8Xdqa8EfCBb0xhif8gz+BesLWZBTwF43+Lu1i6444h/UrS1d20UTHhoS4IqDny8GBx8NPIszOPhLqvpklfnXAU8Dee6k51T1JXfeeOAhd/rjqvpaXduzoDemeSktU1Zu21vRV8+C9YXsOVQMQFiI0K1dNKmJMaQmxpDWPtb9N4akuEhEJMDVB4cGBb2IhAKrgfOAXGAhcJWqLvdY5jogXVXvqLJuOyADSAcUyAROVdVdtW3Tgt6Y5q2sTFm5bR/Lt+4lJ38/63ceqHgcKakcWDsmIpTU9jGkJrrh7+4AUhJjrLvmeqot6L0Z42wwsFZVc9w3ewcYCyyvdS3HBcBMVS10150JjAbe9qZwY0zzFBIi9O3Umr6dWh81vaxM2bLnUEXo5+QfIGfnAbI27+LTJVvwPO5MjI2sCP7KXwMxdGsXQ0SYNQXVhzdB3xnY7PE6FxhSzXKXi8hInKP/e1R1cw3rdq5uIyIyAZgA0K1bNy/KMsY0NyEhQpf4aLrERzOiZ/uj5h0uLmVz4UHW5Zcf/Tu/BGYu307BgaLK9xDoWqUpKM193qF1FCEh1hRUla9GLf4EeFtVj4jILcBrwNn1eQNVnQJMAafpxkd1GWOaiajwUHomx9EzOe6YeXsOFrO+wA3//AOs23mA9fkHWJBTWNGnD0Cr8FBS3CYgz18BaYmxLboff2+CPg/o6vG6C5UnXQFQ1QKPly8BT3mse1aVdb+qb5HGmJatTXQ4A6PbMrBr26Omqyrb9x4hJ38/OR7nAbK37OHz7G2UllUeM7aLiag4D5DavnxnEEv3hGiiwkP9/ZH8ypugXwj0FJFUnOC+ErjacwER6aiqW92XlwIr3OdfAH8QkXj39fnAbxpctTHGACJChzZRdGgTxek9Eo+aV1RSxuZdB1mff4ActxkoJ/8AX6/O5z+ZuR7vAZ3btqrcCXhcGdSpbStCg6ApqM6gV9USEbkDJ7RDgVdUNVtEHgMyVHUqcJeIXAqUAIXAde66hSLye5ydBcBj5SdmjTGmMUWEhXBC+1hOaB8LJB81b9/hYjbsPHjUDmD9zgN8sCivouuH8vdISYgmLTHWvTqo/MqgWOKjw5vNpaF2w5QxxrhUlfz9R9xfAQc8dgL72VR4kOLSyrxs0yr8qEtCyy8RTU2MoVWE/5uCGnp5pTHGtAgiQlJcFElxUQxJSzhqXklpGbm7nEtDc3YeqLg/YF5OAR/+cNRpSzq1iar4BZCaGOueEI6hc9tWhAXgLmELemOM8UJYaAgpic7NXKOqzDtYVFJ5U5jbDLRu5wE+ztpS0ScQQHioc5ew5yWh5ecEEmMjGq0pyILeGGMaKDoijH6d2tCvU5ujpqsqhQeKPH4FVN4f8PWqfIpKK+8SjosMo3fHON67ZZjPA9+C3hhjGomIkBAbSUJsJOkp7Y6aV1qmbNl9yDkX4DYDHSkpa5Sjegt6Y4wJgNAQoWu7aLq2i+bME9vXvUIDWIcRxhgT5CzojTEmyFnQG2NMkLOgN8aYIGdBb4wxQc6C3hhjgpwFvTHGBDkLemOMCXJNsvdKEckHNh7n6onATh+W4ytWV/1YXfVjddVPMNbVXVWrvfOqSQZ9Q4hIRk1ddQaS1VU/Vlf9WF3109LqsqYbY4wJchb0xhgT5IIx6KcEuoAaWF31Y3XVj9VVPy2qrqBrozfGGHO0YDyiN8YY48GC3hhjglyzDXoRGS0iq0RkrYg8UM38SBF5152/QERSmkhd14lIvohkuY+b/FDTKyKyQ0SW1TBfRGSyW/MSETmlsWvysq6zRGSPx3f1sJ/q6ioic0RkuYhki8jd1Szj9+/My7r8/p2JSJSIfC8ii926Hq1mGb//PXpZl9//Hj22HSoiP4jIp9XM8+33parN7gGEAuuANCACWAz0rbLMbcCL7vMrgXebSF3XAc/5+fsaCZwCLKth/hhgOiDAUGBBE6nrLODTAPz/1RE4xX0eB6yu5r+j378zL+vy+3fmfgex7vNwYAEwtMoygfh79KYuv/89emz7l8Bb1f338vX31VyP6AcDa1U1R1WLgHeAsVWWGQu85j5/HzhHGmuI9frV5XeqOhcorGWRscDr6pgPtBWRjk2groBQ1a2qush9vg9YAXSuspjfvzMv6/I79zvY774Mdx9Vr/Lw+9+jl3UFhIh0AS4CXqphEZ9+X8016DsDmz1e53Ls//AVy6hqCbAHSGgCdQFc7v7cf19EujZyTd7wtu5AGOb+9J4uIv38vXH3J/MgnKNBTwH9zmqpCwLwnbnNEFnADmCmqtb4ffnx79GbuiAwf49/BX4NlNUw36ffV3MN+ubsEyBFVU8CZlK51zbHWoTTf8fJwN+A//pz4yISC3wA/EJV9/pz27Wpo66AfGeqWqqqA4EuwGAR6e+P7dbFi7r8/vcoIhcDO1Q1s7G3Va65Bn0e4Lnn7eJOq3YZEQkD2gAFga5LVQtU9Yj78iXg1EauyRvefJ9+p6p7y396q+o0IFxEEv2xbREJxwnTN1X1w2oWCch3VlddgfzO3G3uBuYAo6vMCsTfY511Bejv8QzgUhHZgNO8e7aI/LvKMj79vppr0C8EeopIqohE4JysmFplmanAePf5FcBsdc9sBLKuKu24l+K0swbaVODn7pUkQ4E9qro10EWJSIfydkkRGYzz/2ujh4O7zZeBFar6TA2L+f0786auQHxnItJeRNq6z1sB5wErqyzm979Hb+oKxN+jqv5GVbuoagpORsxW1Z9VWcyn31fY8a4YSKpaIiJ3AF/gXOnyiqpmi8hjQIaqTsX5g3hDRNbinPC7sonUdZeIXAqUuHVd19h1icjbOFdjJIpILvAIzokpVPVFYBrOVSRrgYPA9Y1dk5d1XQFMFJES4BBwpR921uAccV0LLHXbdwEeBLp51BaI78ybugLxnXUEXhORUJwdy3uq+mmg/x69rMvvf481aczvy7pAMMaYINdcm26MMcZ4yYLeGGOCnAW9McYEOQt6Y4wJchb0xhgT5CzoTYshIqUevRRmSTW9izbgvVOkhl44jQm0ZnkdvTHH6ZB7O7wxLYod0ZsWT0Q2iMhTIrLU7b+8hzs9RURmux1efSki3dzpySLykdtx2GIROd19q1AR+ac4fZ/PcO/GRETuEqcP+SUi8k6APqZpwSzoTUvSqkrTzTiPeXtUdQDwHE7PguB0Cvaa2+HVm8Bkd/pk4Gu347BTgGx3ek/g76raD9gNXO5OfwAY5L7PrY314Yypid0Za1oMEdmvqrHVTN8AnK2qOW6nYdtUNUFEdgIdVbXYnb5VVRNFJB/o4tEZVnm3wTNVtaf7+n4gXFUfF5HPgf04PUn+16OPdGP8wo7ojXFoDc/r44jH81Iqz4FdBPwd5+h/odsboTF+Y0FvjGOcx7/z3OffUdmZ1DXAN+7zL4GJUDGwRZua3lREQoCuqjoHuB+nu9ljflUY05jsyMK0JK08en0E+FxVyy+xjBeRJThH5Ve50+4E/iUi9wH5VPZQeTcwRURuxDlynwjU1EVxKPBvd2cgwGS3b3Rj/Mba6E2L57bRp6vqzkDXYkxjsKYbY4wJcnZEb4wxQc6O6I0xJshZ0BtjTJCzoDfGmCBnQW+MMUHOgt4YY4Lc/wNkJ7PGUXHLPgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEWCAYAAAB2X2wCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3wUdf7H8dcnm14IqZSEkAihCoJEiggiiIeCcFawoqdynmI5r9hBEe/8eXpnw4IK9vM82yEiShUUUECxECAJoYWaAiEB0r+/P2YJISRkgU1md/N5Ph55MDszO/PJJPtm8p3vfEeMMSillPJ+fnYXoJRSyj000JVSykdooCullI/QQFdKKR+hga6UUj5CA10ppXyEBrpSSvkIDXTldURksYjsFZEgu2tRypNooCuvIiLJwCDAAKObcL/+TbUvpU6WBrryNtcDK4A3gPGHZ4pIOxH5WERyRSRfRF6osewWEVknIkUiki4iZzrnGxHpWGO9N0RkqnN6iIjkiMi9IrILmCkiUSIy27mPvc7pxBrvjxaRmSKyw7n8U+f8X0Xk4hrrBYhInoj0brSjpJolDXTlba4H3nV+/UZEWomIA5gNbAGSgQTgfQARuQJ4xPm+Flhn9fku7qs1EA20ByZgfV5mOl8nAYeAF2qs/zYQCnQH4oF/Oee/BVxbY72LgJ3GmB9drEMpl4iO5aK8hYicAywC2hhj8kRkPfAK1hn7LOf8ilrv+RKYY4x5to7tGSDVGJPlfP0GkGOMeUhEhgBfAS2MMSX11NMLWGSMiRKRNsB2IMYYs7fWem2BDUCCMWa/iHwIfG+MefKkD4ZSddAzdOVNxgNfGWPynK/fc85rB2ypHeZO7YCNJ7m/3JphLiKhIvKKiGwRkf3AEqCl8y+EdkBB7TAHMMbsAL4FLhORlsCFWH9hKOVWeqFHeQURCQGuBBzONm2AIKAlsBtIEhH/OkJ9G9Chns0exGoiOaw1kFPjde0/X/8EdAb6GWN2Oc/QfwTEuZ9oEWlpjNlXx77eBG7G+swtN8Zsr/+7Verk6Bm68ha/BSqBbkAv51dXYKlz2U7gCREJE5FgERnofN9rwJ9FpI9YOopIe+eyNcDVIuIQkRHAuQ3UEIHVbr5PRKKByYcXGGN2Al8ALzovngaIyOAa7/0UOBO4C6tNXSm300BX3mI8MNMYs9UYs+vwF9ZFyauAi4GOwFass+yxAMaY/wKPYzXPFGEFa7Rzm3c537cPuMa57HieAUKAPKx2+7m1ll8HlAPrgT3A3YcXGGMOAR8BKcDHJ/i9K+USvSiqVBMRkUlAJ2PMtQ2urNRJ0DZ0pZqAs4nmJqyzeKUahTa5KNXIROQWrIumXxhjlthdj/Jd2uSilFI+Qs/QlVLKR9jWhh4bG2uSk5Pt2r1SSnml1atX5xlj4upaZlugJycns2rVKrt2r5RSXklEttS3TJtclFLKR2igK6WUj9BAV0opH+FSG7pznItnAQfwmjHmiVrL2wMzgDigALjWGJNzzIYaUF5eTk5ODiUldY5WqnxMcHAwiYmJBAQE2F2KUj6hwUB3Dg06DRiONUbGShGZZYxJr7HaU8Bbxpg3RWQo8HdO4o64nJwcIiIiSE5ORkRO9O3KixhjyM/PJycnh5SUFLvLUconuNLk0hfIMsZkG2PKsJ4EM6bWOt2Ahc7pRXUsd0lJSQkxMTEa5s2AiBATE6N/jSnlRq4EegLWbcuH5Tjn1fQTcKlz+hIgQkRiTqYgDfPmQ3/WSrmXu/qh/xl4QURuwHqKy3assauPIiITsJ7NSFJSkpt2rZRSnqm8soq84lL27C9lT1EpuUWl7CkqYWiXeHomtnT7/lwJ9O1Yj9c6LNE5r5rzEVuXAohIOHBZXU9tMcZMB6YDpKWl6SAySimvdLCswhnOh8O65KjpXGd4Fxwso67hsmLCg2wL9JVAqoikYAX5OODqmiuISCzW8xSrgPuxerx4nX379vHee+9x2223ndD7LrroIt577z1atqz/BzRp0iQGDx7M+eeff6plKqUagTGGwkPlx4T0keAuqZ4uLj328bX+fkJcRBDxEUEkRoVyZvso4iOCnPOCiY8IIr5FELHhQQQ4GqfHeIOBboypEJGJwJdY3RZnGGPWisgUYJUxZhYwBPi78ynqS4DbG6XaRrZv3z5efPHFYwK9oqICf//6D9WcOXMa3PaUKVNOuT67NPT9K+XJKiqryD9Qxp79peQWl1Q3f+wpKjmqKSS3qJSyyqpj3h8a6LDCOCKYrm1aMLiTFcw1QzouPIio0ED8/Oy9LuTSp9QYMweYU2vepBrTHwIfurOwRz9bS/qO/e7cJN3atmDyxd3rXX7fffexceNGevXqRUBAAMHBwURFRbF+/XoyMjL47W9/y7Zt2ygpKeGuu+5iwoQJwJFxaYqLi7nwwgs555xzWLZsGQkJCfzvf/8jJCSEG264gVGjRnH55ZeTnJzM+PHj+eyzzygvL+e///0vXbp0ITc3l6uvvpodO3YwYMAA5s2bx+rVq4mNja2z3vrqmTt3Lg888ACVlZXExsayYMECiouLueOOO1i1ahUiwuTJk7nssssIDw+nuLgYgA8//JDZs2fzxhtvcMMNNxAcHMyPP/7IwIEDGTduHHfddRclJSWEhIQwc+ZMOnfuTGVlJffeey9z587Fz8+PW265he7du/Pcc8/x6afWE93mzZvHiy++yCeffOLOH6dq5krKK6vbpGuHdG6NduuCA6VU1dHs0TI0oDqoT4sNI84Z0ofPsq2wDiY8yHtOZryn0ibwxBNP8Ouvv7JmzRoWL17MyJEj+fXXX6v7Sc+YMYPo6GgOHTrEWWedxWWXXUZMzNGdeTIzM/n3v//Nq6++ypVXXslHH33Etdce+8Sx2NhYfvjhB1588UWeeuopXnvtNR599FGGDh3K/fffz9y5c3n99dePW29d9VRVVXHLLbewZMkSUlJSKCgoAOCxxx4jMjKSX375BYC9e/c2eDxycnJYtmwZDoeD/fv3s3TpUvz9/Zk/fz4PPPAAH330EdOnT2fz5s2sWbMGf39/CgoKiIqK4rbbbiM3N5e4uDhmzpzJ7373O5d+Bqp5M8awv6SC3Fpnz7XbqPcUlVJUcmyzh8NPiA0PJD4imNaRwfRMjLSaPVoEHxXSseGBBPk7bPgOG5fHBvrxzqSbSt++fY+66eW5556rPsvctm0bmZmZxwR6SkoKvXr1AqBPnz5s3ry5zm1feuml1et8/LH1zOBvvvmmevsjRowgKirquPXVVU9ubi6DBw+urjs62noe8vz583n//fer39vQtgGuuOIKHA7rl76wsJDx48eTmZmJiFBeXl693VtvvbW6Sebw/q677jreeecdbrzxRpYvX85bb+mD7puzyipD/oEj7dG5dV1IdJ5Vl1Yc2+wR5O9X3czRqVUE53SMJb6FdTZ95Iw6mOiwQBw2N3vYyWMD3ROEhYVVTy9evJj58+ezfPlyQkNDGTJkSJ03xQQFBVVPOxwODh06VOe2D6/ncDioqDj2TKMhrtbTkJp9wWu/v+b3//DDD3PeeefxySefsHnzZoYMGXLc7d54441cfPHFBAcHc8UVV2gbvA/bVVjCjsJDVlNH7QuJzjPt/ANlVNbR7tEi2J9459nzmUlR1cEc36LGxcQWQUQE+et9Cy7QT1kNERERFBUV1bmssLCQqKgoQkNDWb9+PStWrHD7/gcOHMgHH3zAvffey1dffXXcZpH66unfvz+33XYbmzZtqm5yiY6OZvjw4UybNo1nnnkGsJpcoqKiaNWqFevWraNz58588sknRERE1Lu/hATrfrI33nijev7w4cN55ZVXOO+886qbXKKjo2nbti1t27Zl6tSpzJ8/301HSHmCopJylm3MZ2lmLksz89iSf/Co5SIQExZUfcGwW5sWR0I6/MgFxbiIIIIDfK/Zw04a6DXExMQwcOBATj/9dEJCQmjVqlX1shEjRvDyyy/TtWtXOnfuTP/+/d2+/8mTJ3PVVVfx9ttvM2DAAFq3bl1vwNZXT1xcHNOnT+fSSy+lqqqK+Ph45s2bx0MPPcTtt9/O6aefjsPhYPLkyVx66aU88cQTjBo1iri4ONLS0qovkNb217/+lfHjxzN16lRGjhxZPf/mm28mIyODnj17EhAQwC233MLEiRMBuOaaa8jNzaVr165uPlKqKVVWGX7O2cfSzDyWZubyw9Z9VFYZQgMdDDgthusHJJMSG1rd6yM6LBD/RuqWp47PtodEp6WlmdpPLFq3bl2z/vCXlpbicDjw9/dn+fLl/OEPf2DNmjV2l3XSJk6cSO/evbnpppvqXae5/8w9Vc7eg3yTmceSzFy+zcqn8FA5InB620gGd4plUGocZyZFEeivwd3URGS1MSatrmV6hu5Btm7dypVXXklVVRWBgYG8+uqrdpd00vr06UNYWBhPP/203aUoFxworWBFdj5LnSGenXsAgNYtgrmgWysGdYpjYIcYYsKDGtiSspMGugdJTU3lxx9/PGpefn4+w4YNO2bdBQsWHNPDxpOsXr3a7hLUcVRVGdbu2M+SzFyWZOTyw9a9lFcaggP86H9aDNf0a8/g1Fg6xofrxUgvooHu4WJiYry62UV5jp2Fh5zt4Hl8k5nL3oNW19NubVrwu3NSGJwaR5/2UXqh0otpoCvlow6VVfLdpvzqi5kZu60L3rHhQZzXOZ7BneIY2DGWuAhtRvEVGuhK+YiqKsP6XUUsycxlaWYuKzftpayyikB/P/qlRHN5n0QGpcbRpXWENqP4KA10pbzYnqISvnE2oyzNzCOvuBSAzq0iuH5AewZ1iqNfSrQ2ozQTGuhKeZGS8kpWbi6weqNk5LJ+l3UjXHRYIINSre6Eg1JjadUi2OZKlR000E9BzZEKlWoMxhgydhezNDOXJZl5fJedT2lFFQEOIa19NH8d0ZnBqXF0a9PC9qFblf000H2AjlfuW/KLS/kmK48lGdbFzD1FVjNKx/hwru6XxODUOPqdFk1ooP7M1dE89zfii/tg1y/u3WbrHnDhE/Uuvu+++2jXrh233249n+ORRx7B39+fRYsWsXfvXsrLy5k6dSpjxoxpcFfFxcWMGTOmzve99dZbPPXUU4gIPXv25O2332b37t3ceuutZGdnA/DSSy/Rtm1bRo0axa+//grAU089RXFxMY888ghDhgyhV69efPPNN1x11VV06tSJqVOnUlZWRkxMDO+++y6tWrWqcxz0wsJCfv755+pxXV599VXS09P517/+dUqHV52c0opKVm/ZW90b5dft1nMAWoYGMLBjLINTYzknNY6EliE2V6o8necGug3Gjh3L3XffXR3oH3zwAV9++SV33nknLVq0IC8vj/79+zN69OgGewkEBwfzySefHPO+9PR0pk6dyrJly4iNja0er/zOO+/k3HPP5ZNPPqGyspLi4uIGxywvKyvj8PAJe/fuZcWKFYgIr732Gk8++SRPP/10neOgBwQE8Pjjj/OPf/yDgIAAZs6cySuvvHKqh0+5yBjDxtwD1YNbrcjO52BZJf5+wplJUfz5gk4MSo3j9ITIZj0UrDpxnhvoxzmTbiy9e/dmz5497Nixg9zcXKKiomjdujV//OMfWbJkCX5+fmzfvp3du3fTunXr427LGMMDDzxwzPsWLlzIFVdcUf0UosPjhy9cuLB6zHCHw0FkZGSDgT527Njq6ZycHMaOHcvOnTspKyurHg+9vnHQhw4dyuzZs+natSvl5eX06NHjBI+WOhF7D5Tx7cY8ljqbUXYUWkMVp8SGVXcn7H9aNBHBATZXqryZ5wa6Ta644go+/PBDdu3axdixY3n33XfJzc1l9erVBAQEkJyc7NK44yf7vpr8/f2pqjoy2P/xxiu/4447uOeeexg9ejSLFy/mkUceOe62b775Zv72t7/RpUsXbrzxxhOqSzWsvLKKH7fuq76Y+XPOPoyBiGB/BnaI5fahsQxOjaNddKjdpSofooFey9ixY7nlllvIy8vj66+/5oMPPiA+Pp6AgAAWLVrEli1bXNpOYWFhne8bOnQol1xyCffccw8xMTHV44cPGzaMl156ibvvvru6yaVVq1bs2bOH/Px8wsPDmT17NiNGjKh3f4fHK3/zzTer59c3Dnq/fv3Ytm0bP/zwAz///POpHDKF9RfZlvyDzrFRrGaU4tIK/AR6J0Vx17BUBqXGcUZipA4tqxqNBnot3bt3p6ioiISEBNq0acM111zDxRdfTI8ePUhLS6NLly4ubae+93Xv3p0HH3yQc889F4fDQe/evXnjjTd49tlnmTBhAq+//joOh4OXXnqJAQMGMGnSJPr27UtCQsJx9/3II49wxRVXEBUVxdChQ9m0aRNAveOgA1x55ZWsWbPGpcfRqWMVHipn+cY8ljgvZm4rsJ5OlRgVwuhebRmcGsuADrFEhmgzimoaOh56MzZq1Cj++Mc/1jmaY1Pxpp95RWUVP+UUWs0oGbms2baPKgPhQf4M6BDDYOeNPe1jQvXWetVodDx0dZR9+/bRt29fzjjjDFvD3BtsK7CaUZZm5PHtxjyKSioQgZ6JLbn9vI4MSo2jd1JLArQZRXkADfRT9Msvv3DdddcdNS8oKIjvvvvOpooa1rJlSzIyMuwuwyNVVFaxeEMuX2dYA1xtdj4vs21kMBed3obBneI4u0MMUWGBNleq1LFcCnQRGQE8CziA14wxT9RangS8CbR0rnOfMWbOyRRkjPGqP1d79Oih45WfJLua++pSVlHFJz/mMG3RRrYWHCQkwMGADjGMPzuZQalxdIgL86rfS9U8NRjoIuIApgHDgRxgpYjMMsak11jtIeADY8xLItINmAMkn2gxwcHB5OfnExMTox8eH2eMIT8/n+BgeweRKq2o5L+rcnhp8Ua27ztEj4RIXr62D+d1iSPIX0coVN7FlTP0vkCWMSYbQETeB8YANQPdAC2c05HAjpMpJjExkZycHHJzc0/m7crLBAcHk5iYaMu+S8or+c/Kbby0eCO79pfQq11Lpl5yOkM6xenJhPJargR6ArCtxuscoF+tdR4BvhKRO4Aw4Py6NiQiE4AJAElJSccsDwgIqL7DUanGcKiskne/28IrS7LJLSrlrOQo/nFFT87pGKtBrryeuy6KXgW8YYx5WkQGAG+LyOnGmKqaKxljpgPTweq26KZ9K9WgA6UVvL1iC68tzSavuIwBp8Xw3Lje9D8tWoNc+QxXAn070K7G60TnvJpuAkYAGGOWi0gwEAvscUeRSp2sopJy3lpuBfneg+UMSo3lzmGpnJUcbXdpSrmdK4G+EkgVkRSsIB8HXF1rna3AMOANEekKBAPaEK5sU3iwnJnLNjHjm03sL6lgaJd47hjakd5Jeles8l0NBroxpkJEJgJfYnVJnGGMWSsiU4BVxphZwJ+AV0Xkj1gXSG8wntQnTTUbew+UMePbTbzx7WaKSisY3q0Vdw5NpUdipN2lKdXoXGpDd/Ypn1Nr3qQa0+nAQPeWppTr8otLeXXpJt5evpkDZZVc1KM1E89LpVvbFg2+VylfoXeKKq+2p6iEV5dk886KrZRUVDKqZ1smnteRzq0j7C5NqSanga680q7CEl7+eiP//n4r5ZVV/LZXAred15GO8eF2l6YUVFVB+QEoO+j8t9Z0q9Mh2v1dtDXQlVfZvu8QLy/eyH9WbqPKGC7pncDt53UkOTas4TcrVVtFGZQVQ/lBK3Crp53BW3O69uty5/plznmHA7zsAFQcOv5+R/4Tom9y+7ejga68wraCg7y4OIsPV+cAcHmfdtw2pIM+8ac5qKqyAtKVYD1eyNZ1plxV4Xod4gcBYRAYCoFhR6aDW0BEawgMt14HhB6Zrrle9XQYRDbOHdIa6Mqjbco7wLRFWXzy43YcIlzVN4nfn9uBhJYhdpemaqssP/lgPersuNaZcvnBE6vDEeQM0HBnuDpDNLy1czr0SLDWtV7NQK457R8MHn4Tmga68khZe4qZtiiL/63ZToDDj+sHtOf3gzvQOtLewbx8VnkJlBRC6X4o2Q8l+2pMFx5nutCaLjsAVeUnsEOpO0ADwyG8Va1ldU2H1R3OAWHgaL6x1ny/c+WRNuwq4vmFmXz+y06C/R3cPOg0bh6UQnyEBnm9KiucIVt45N+S/XVM7zt2/uH3VJYdfx/iB0EREBwJQZHWvy3bQVB3q8nhmKCtFcC1X3vB2a430kBXHmHtjkKeX5DF3LW7CAt0cOu5Hbj5nBRiwoPsLq1xVVVZzQt1ngHXdZZcR1iXH2h4PwFhVggHt4CgFhAaC9GnOQO6hTW/OqxbHDs/MFwD2AtooCtb/Zyzj+cWZDF/3W4igvy5c2hHfndOCi1DveCJQMZARUmtgC1soImiVnNFaREcPYbdsRyBxwZsRGvndMtagdzi6OA+PK8ZN0M0J/pTVrZYvWUvzy/MZPGGXCJDArhneCfGn51MZEiAfUUd2gu5GVCQ7WyeOHw2XFh/M4ZLTRXOwD2mqaJW8FZPtzw6oAO0uUm5RgNdNanvNxXw/MJMlmbmERUawF9+05nrB7QnIriJgtwYKN4NuRusr7zD/2ZY82vTpgrlRTTQVaMzxrA8O5/nFmSyIruA2PBAHrioC9f0a09YUCP9ClZVwb4tVlAfFd4Z1hn3YUEtILYTdDzf+jeuM8SkQmi0NlUor6O/rarRGGNYmpnH8wszWbl5L/ERQTw8qhtX900iJNBNz+usKIOCjUfOsg8Hd16m1b59WFgcxHWBHpdboR3byXod0VrPoJXP0EBXbmeMYfGGXJ5dkMmabftoExnMlDHduTKtHcEBJxnkZQdqnW07pwuywVQeWS8yCeI6Qcq5R864YztZZ9xK+TgNdOU2xhjmpe/m+YVZ/LK9kISWIfztkh5c1ieBIH8Xg/xgwdFt24fDu7DGY239/K127LjO0G20daYd2wliU60+zko1Uxro6pRVVRnmrt3F8wuzWLdzP+1jQnnysp5ccmYCAQ6/Y99gDOzfcaRN+/C/uevhYN6R9fxDrJBO6g+x460z77guEJUC/l7QrVGpJqaBrk5aZZXh81928sLCTDJ2F3NabBj/vPIMRp/RFn+HH1RVQv7Goy9IHv63rOjIhoIjIbYzdL7Q2UTS2QrvyCTwq+M/BKVUnTTQ1QmrqKzis5938PzCLLJzD9AtLogZF4YyJLoAv/x34OP1VmjnZ0Fl6ZE3hre2gvqMcVZwHw7v8Hi9MKmUG2igK5eVHyzk62+/5fuVy4k6uInHg/fQM2YXocXbkEWH73YUiGpvBXXHoc6z7S5W00lIS1vrV8rXaaCrYx3Ic16QXA95GVTt2cChnemElezmfOB8oCogAInqgMT1gtixR3qTxKZCgA5tq5QdNNCbK2OgMOfoC5KHuwIeKqherdwRQlZVW9ZVpHKgxQjO6N2XHmf0xS86BRw23qavlDqGBrqvM8bqq71n3bEXJmuO0hcSbZ1ld72YsuhUFuVH88IvfvxaHE6f9jHcOSyVS1JjEW3rVspjaaD7su2rYd5k2Lz0yLyItlZwn3ndkRtv4rpAWCwHyyp4Z8UWpi/eRF5xKf1Pi+bdq1IZcFqMBrlSXkAD3RcVZMOCx2Dtx9ZgUhdMhaSzrfbt4BbHrF5cWsFbi7N4bekmCg6UcU7HWO4Y2pt+p8XYULxS6mS5FOgiMgJ4FnAArxljnqi1/F/Aec6XoUC8MUa7NDS1A3mw5B+w8nWrfXvwX+DsO+sMcYDCQ+W8uWwzr3+zicJD5QzpHMcdQ1Pp0z6qiQtXSrlDg4EuIg5gGjAcyAFWisgsY0z64XWMMX+ssf4dQO9GqFXVp+wgrHgRvnnGahc/83oYcr818FQd9h0sY8a3m5n57SaKSio4v2sr7hjakTPa6f/BSnkzV87Q+wJZxphsABF5HxgDpNez/lXAZPeUp46rsgJ+eg8W/Q2KdkLnkXD+ZKtdvA4FB8p4bWk2by3fQnFpBSO6t2bi0I6cnhDZxIUrpRqDK4GeANQYGYkcoF9dK4pIeyAFWFjP8gnABICkpKQTKlTVYAxkfAnzJ1vdDRPPgstnQvsBda6eW1TKq0uzeWfFFg6VVzKyRxsmDu1Il9Z1N8UopbyTuy+KjgM+NKbmeKZHGGOmA9MB0tLSjJv33TzkrIJ5k2DLtxDdAa58C7qOrvfW+WVZefzuzZWUVVQx+oy2TBzakY7xEU1ctFKqKbgS6NuBdjVeJzrn1WUccPupFqXqkL8RFkyB9E+thzWMfBrOHH/cm3vKK6t46H+/0rpFMDNv7EtKrA4tq5QvcyXQVwKpIpKCFeTjgKtrryQiXYAoYLlbK2zuinNhyZOwagY4guDc++DsiRDU8Fn2W8u3kJ17gBk3pGmYK9UMNBjoxpgKEZkIfInVbXGGMWatiEwBVhljZjlXHQe8b4zRphR3KDsAy1+Eb5+B8kPQZ7wV5hGtXHp7fnEpz8zPYHCnOM7rHN/IxSqlPIFLbejGmDnAnFrzJtV6/Yj7ymrGKivgx7dh8RNQvAu6jILzH7FuCjoB/5yXwcGySiaN6qp3eSrVTOidop7CGNgwB+Y/Yg2S1a4fXPmm9bSeE7Ru537+/f1Wrh+QrBdAlWpGNNA9wbaVMO9h2LocYlJh7LvQZeRJPfTBGMOUz9KJDAngj+d3aoRilVKeSgPdTnlZsOBRWDcLwuJh1L+g9/XgOPkfy5drd7E8O5/HxnQnMlSHt1WqOdFAt0PxHvj6/2DVTPAPhiEPwIDbISj8lDZbUl7J43PW0blVBFf11Ru3lGpuNNCbUmkxLH8Blj0PFSWQdiOce6/1TE03eP2bTWwrOMS7N/ezHtKslGpWNNCbQmU5/PCW1XPlwB7rzs5hkyG2o9t2sXt/CdMWZXFBt1YM7Bjrtu0qpbyHBnpjMgbWz4b5j0J+JiQNgHHvQru+bt/Vk3M3UFFpeHBkV7dvWynlHTTQG8vW76yeK9u+s54MNO7f0PnCk+q50pCftu3jox9yuPXcDrSP0TtClWquNNDdLTfD6rmyfjaEt4KLn4Ve155Sz5XjMcbw6GdriQ0PYuJQ9zXhKKW8jwa6uxTthsV/t9rKA0LgvIdgwG0Q2LhnzLN+2sEPW/fx5BDPIPwAABZ3SURBVOU9CQ/SH6dSzZkmwKkqLbJ6rSx7ASpL4aybYPBfITyu0Xd9sKyCv89ZT4+ESC4/M7HR96eU8mwa6CershxWv2H1Jz+QC91+C8MmQUyHJivh5a+z2bW/hBeu7o2fn47XolRzp4F+ooyx7uyc/ygUbIT2A+Gq9yExrUnLyNl7kFe+3sjFZ7QlLTm6SfetlPJMGugnYssy62lBOSshrosV5J1GNErPlYY88cV6ROC+C7s0+b6VUp5JA90VuRusURA3zIGINjD6eTjj6kbrudKQ7zcVMPvnndw1LJWEliG21KCU8jwa6Mezf6fVc+XHtyEgDIY+DP1vg8BQ20qqrLK6KbaJDObWc5uuvV4p5fk00OtSsh+WPQfLp1kXP/tOgMF/gTD7b6n/cPU21u7Yz7PjehES6LC7HKWUB9FAr6mi7EjPlYN50P1SGPYwRJ9md2UAFJWU848vN9CnfRSjz2hrdzlKKQ+jgQ5Wz5X0T2HBFCjIhuRBMPxRSOhjd2VHeWFhFnnFZcy44Sx9rJxS6hga6Ju/tXqubF8FcV3h6v9C6nBbeq4cz6a8A8z4dhOX90mkZ2JLu8tRSnmg5hvoe9ZbPVcyvoCItjBmGpxxFfh5Zrv045+vI9Dhx19/09nuUpRSHqr5Bfr+nbD4b/DjOxAYbo1L3u9WW3uuNGRpZi7z1+3mryM6E98i2O5ylFIeqvkEekkhfPssLH8RqiqsEB/0ZwiLsbuy46qorOKx2ekkRYfyu4EpdpejlPJgLgW6iIwAngUcwGvGmCfqWOdK4BHAAD8ZY652Y50nr6IMVs2AJU/CwXw4/XIY+hBEe0c4vvf9VjJ2F/PytX0IDvDM5iCllGdoMNBFxAFMA4YDOcBKEZlljEmvsU4qcD8w0BizV0Tc85DMU2EMrP3Y6rmydzOkDIbhU6Btb7src9m+g2X8c14GZ3eI4TfdW9ldjlLKw7lyht4XyDLGZAOIyPvAGCC9xjq3ANOMMXsBjDF73F3oCdm01Oq5suMHiO8O13wEHYd5XM+VhjwzP5P9h8qZdHE37aaolGqQK4GeAGyr8ToH6FdrnU4AIvItVrPMI8aYubU3JCITgAkASUlJJ1Pv8e1Oh/mTIfMraJEAv30Jeo712J4rx5O5u4i3V2zh6n5JdGndwu5ylFJewF0XRf2BVGAIkAgsEZEexph9NVcyxkwHpgOkpaUZN+0bCrfDor/BT+9BYASc/yj0+7315CAvZIxhyux0wgId3DNcuykqpVzjSqBvB9rVeJ3onFdTDvCdMaYc2CQiGVgBv9ItVdanpBC++ReseAlMlTVw1qA/Qah3jw++cP0elmbmMWlUN6LDAu0uRynlJVwJ9JVAqoikYAX5OKB2D5ZPgauAmSISi9UEk+3OQo9SUQorX4cl/4BDBdDjSqvnSlT7RttlUymrqGLq5+voEBfGdQO8//tRSjWdBgPdGFMhIhOBL7Hax2cYY9aKyBRglTFmlnPZBSKSDlQCfzHG5DdKxRlfwpy/wL4tcNoQq3mlba9G2ZUd3ly2mU15B3jjxrMIcPjZXY5Syou41IZujJkDzKk1b1KNaQPc4/xqXGUHIKgFXPux1XPFh+QVl/LcgkzO6xzHkM729/xUSnkX77tTtPsl1gOZ/Xzv7PXprzZwqLySh0Z1s7sUpZQX8r5UFPHJMP91eyHvr9zG+LOT6RAXbnc5Sikv5HvJ6IUOd1OMCg3kzmGpdpejlPJSGugeYM4vu/h+UwF/uqATkSEBdpejlPJSGug2Kymv5G9z1tGldQTjzmqEu2eVUs2G910U9TGvLslm+75D/PuW/jj8dLwWpdTJ0zN0G+0qLOHFxRu58PTWDOjg2eOyK6U8nwa6jf5v7noqjeGBi7raXYpSygdooNvkh617+eTH7dwyKIV20Z77+DullPfQQLdBVZXh0c/SiY8I4rYhHe0uRynlIzTQbfDpmu38tG0f947oQliQXpdWSrmHBnoTO1BawRNfrOeMdi25pHeC3eUopXyIBnoTe2nxRvYUlTJpVDf8tJuiUsqNNNCb0LaCg0xfms1ve7WlT/sou8tRSvkYDfQm9Pcv1uEQ4d4Lu9hdilLKB2mgN5EV2fnM+WUXfxjSgTaR3vmsU6WUZ9NAbwKVzm6KCS1DmDD4NLvLUUr5KA30JvCfldtYt3M/91/UheAAh93lKKV8lAZ6Iys8VM7TX22gb3I0I3u0sbscpZQP07taGtnzCzIpOFjGmxd3Q0S7KSqlGo+eoTeijbnFvLFsM1f2acfpCZF2l6OU8nEa6I3o8c/XERzg4M+/6Wx3KUqpZkADvZEs3rCHhev3cMfQjsRFBNldjlKqGdBAbwTllVU8Njud5JhQbhiYbHc5SqlmwqVAF5ERIrJBRLJE5L46lt8gIrkissb5dbP7S/Ue76zYwsbcAzw4shtB/tpNUSnVNBrs5SIiDmAaMBzIAVaKyCxjTHqtVf9jjJnYCDV6lYIDZfxrXgaDUmM5v2u83eUopZoRV87Q+wJZxphsY0wZ8D4wpnHL8l7/mpfBgbJKHh6l3RSVUk3LlUBPALbVeJ3jnFfbZSLys4h8KCLt6tqQiEwQkVUisio3N/ckyvVs63ft593vtnBtvyQ6tYqwuxylVDPjrouinwHJxpiewDzgzbpWMsZMN8akGWPS4uLi3LRrz2CMYcpn6UQEB3D3+Z3sLkcp1Qy5EujbgZpn3InOedWMMfnGmFLny9eAPu4pz3t8lb6bZRvzuWd4J6LCAu0uRynVDLkS6CuBVBFJEZFAYBwwq+YKIlJzkJLRwDr3lej5SisqefzzdaTGh3NNvyS7y1FKNVMN9nIxxlSIyETgS8ABzDDGrBWRKcAqY8ws4E4RGQ1UAAXADY1Ys8eZ8c1mthYc5O2b+uLv0K79Sil7uDQ4lzFmDjCn1rxJNabvB+53b2neYU9RCS8szOT8rvEMSvWt6wJKKe+ip5On6B9zN1BWWcWDI7vZXYpSqpnTQD8FP+fs48MfcrhxYAopsWF2l6OUauY00E/S4W6KMWGBTBza0e5ylFJKA/1kffbzTlZt2cufL+hMi+AAu8tRSikN9JNxqKySJ+aso3vbFlyRVudNsUop1eQ00E/CK0s2sqOwhMkXd8fhp+O1KKU8gwb6Cdqx7xAvf72RkT3b0Dcl2u5ylFKqmgb6CXrii/UYA/df2MXuUpRS6iga6Cdg1eYCZv20g98PPo3EqFC7y1FKqaNooLuoqsrw6GfptG4RzK1DOthdjlJKHUMD3UUf/ZDDL9sLue/CLoQGujRiglJKNSkNdBcUl1bw5Jcb6J3UkjG92tpdjlJK1UkD3QXTFmWRW1TK5Iu762PllFIeSwO9AVvyD/D60k1cemYCvdq1tLscpZSqlwZ6A/42Zx3+DuHeEdpNUSnl2TTQj2NZVh5frt3N7ed1pFWLYLvLUUqp49JAr0dFZRVTZqeTGBXCTeek2F2OUko1SAO9Hv9euY31u4p48KKuBAc47C5HKaUapIFeh8KD5fzzqw30S4lmxOmt7S5HKaVcooFeh2cWZFB4qJxJF3fTbopKKa+hgV5L1p4i3l6+hbFnJdG9baTd5SillMs00Gt5bPY6QgId/PmCTnaXopRSJ0QDvYZF6/fwdUYudw1LJSY8yO5ylFLqhLgU6CIyQkQ2iEiWiNx3nPUuExEjImnuK7FplFVU8djsdE6LDeP6Acl2l6OUUieswUAXEQcwDbgQ6AZcJSLd6lgvArgL+M7dRTaFt5ZvJjvvAA+N6kqgv/7hopTyPq4kV18gyxiTbYwpA94HxtSx3mPA/wElbqyvSeQXl/LsgkzO7RTHeZ3j7S5HKaVOiiuBngBsq/E6xzmvmoicCbQzxnx+vA2JyAQRWSUiq3Jzc0+42Mby9LwMDpZV8vCortpNUSnltU65bUFE/IB/An9qaF1jzHRjTJoxJi0uLu5Ud+0W6Tv28/73W7l+QHs6xkfYXY5SSp00VwJ9O9CuxutE57zDIoDTgcUishnoD8zyhgujxhimzF5LZEgAdw/TbopKKe/mSqCvBFJFJEVEAoFxwKzDC40xhcaYWGNMsjEmGVgBjDbGrGqUit1o7q+7WJFdwD0XdCYyNMDucpRS6pQ0GOjGmApgIvAlsA74wBizVkSmiMjoxi6wsZSUV/L4nHV0bhXBVWe1a/gNSinl4Vx62rExZg4wp9a8SfWsO+TUy2p8r3+ziZy9h3j35n74O7SbolLK+zXLJNu9v4Rpi7K4oFsrBnaMtbscpZRyi2YZ6P83dz0VlYYHR3a1uxSllHKbZhfoa7bt4+MftvO7c1JoHxNmdzlKKeU2zSrQjTE8+tla4iKCmDi0o93lKKWUWzWrQP/fmh38uHUff/lNZ8KDXLoerJRSXqPZBPrBsgqe+GI9PRIiufzMRLvLUUopt2s2gf7y4o3s2l/C5Iu74een47UopXxPswj0nL0HeWVJNqPPaEtacrTd5SilVKNoFoH+9y/WIwL3XdjF7lKUUqrR+Hygf5edz+c/7+TWczvQtmWI3eUopVSj8elAr6wyTJmdTtvIYH4/uIPd5SilVKPy6UD/76ptrN2xn3sv7EJIoMPucpRSqlH5bKAXlZTz1FcbSGsfxegz2tpdjlJKNTqfDfQXFmaRV1zGpIu76WPllFLNgk8G+qa8A8z4dhNX9EmkZ2JLu8tRSqkm4ZOB/vjn6wh0+PGXEZ3tLkUppZqMzwX60sxc5q/bzcShqcRHBNtdjlJKNRmfCvSKyiqmfJZOUnQovzsn2e5ylFKqSflUoL/73VYy9xTz4MiuBPlrN0WlVPPiM4G+90AZ/5yXwdkdYrigWyu7y1FKqSbnM4H+zPwMikrKtZuiUqrZ8olAz9hdxDvfbeXqfkl0ad3C7nKUUsoWXh/oxhgem51OWKCDe4ZrN0WlVPPl9YG+YN0elmbmcff5nYgOC7S7HKWUso1LgS4iI0Rkg4hkich9dSy/VUR+EZE1IvKNiHRzf6nHKq2oZOrn6XSIC+O6Ae2bYpdKKeWxGgx0EXEA04ALgW7AVXUE9nvGmB7GmF7Ak8A/3V5pHd5ctpnN+Qd5eFQ3Ahxe/8eGUkqdEldSsC+QZYzJNsaUAe8DY2quYIzZX+NlGGDcV2LdcotKeW5BFkO7xDOkc3xj704ppTyevwvrJADbarzOAfrVXklEbgfuAQKBoXVtSEQmABMAkpKSTrTWozz91QZKyit5cGTXU9qOUkr5Cre1UxhjphljOgD3Ag/Vs850Y0yaMSYtLi7upPf16/ZC/rNqGzecnUyHuPCT3o5SSvkSVwJ9O9CuxutE57z6vA/89lSKOh5jDFM+SycqNJA7hqU21m6UUsrruBLoK4FUEUkRkUBgHDCr5goiUjNZRwKZ7ivxaJ//spPvNxfw5ws6ExkS0Fi7UUopr9NgG7oxpkJEJgJfAg5ghjFmrYhMAVYZY2YBE0XkfKAc2AuMb6yCw4L8Gd6tFWPPatfwykop1YyIMY3eIaVOaWlpZtWqVbbsWymlvJWIrDbGpNW1TDtvK6WUj9BAV0opH6GBrpRSPkIDXSmlfIQGulJK+QgNdKWU8hEa6Eop5SM00JVSykfYdmORiOQCW07y7bFAnhvLcRet68RoXSfOU2vTuk7MqdTV3hhT5+iGtgX6qRCRVfXdKWUnrevEaF0nzlNr07pOTGPVpU0uSinlIzTQlVLKR3hroE+3u4B6aF0nRus6cZ5am9Z1YhqlLq9sQ1dKKXUsbz1DV0opVYsGulJK+QiPDnQRGSEiG0QkS0Tuq2N5kIj8x7n8OxFJ9pC6bhCRXBFZ4/y6uYnqmiEie0Tk13qWi4g856z7ZxE500PqGiIihTWO16QmqKmdiCwSkXQRWSsid9WxTpMfLxfrsuN4BYvI9yLyk7OuR+tYp8k/jy7WZcvn0blvh4j8KCKz61jm/uNljPHIL6zH3W0ETgMCgZ+AbrXWuQ142Tk9DviPh9R1A/CCDcdsMHAm8Gs9yy8CvgAE6A985yF1DQFmN/GxagOc6ZyOADLq+Dk2+fFysS47jpcA4c7pAOA7oH+tdez4PLpSly2fR+e+7wHeq+vn1RjHy5PP0PsCWcaYbGNMGfA+MKbWOmOAN53THwLDREQ8oC5bGGOWAAXHWWUM8JaxrABaikgbD6iryRljdhpjfnBOFwHrgIRaqzX58XKxribnPAbFzpcBzq/aPSqa/PPoYl22EJFEYCTwWj2ruP14eXKgJwDbarzO4dhf7Op1jDEVQCEQ4wF1AVzm/DP9QxHxlCdau1q7HQY4/2z+QkS6N+WOnX/q9sY6u6vJ1uN1nLrAhuPlbD5YA+wB5hlj6j1eTfh5dKUusOfz+AzwV6CqnuVuP16eHOje7DMg2RjTE5jHkf+FVd1+wBqf4gzgeeDTptqxiIQDHwF3G2P2N9V+G9JAXbYcL2NMpTGmF5AI9BWR05tivw1xoa4m/zyKyChgjzFmdWPvqyZPDvTtQM3/SROd8+pcR0T8gUgg3+66jDH5xphS58vXgD6NXJOrXDmmTc4Ys//wn83GmDlAgIjENvZ+RSQAKzTfNcZ8XMcqthyvhuqy63jV2P8+YBEwotYiOz6PDdZl0+dxIDBaRDZjNcsOFZF3aq3j9uPlyYG+EkgVkRQRCcS6aDCr1jqzgPHO6cuBhcZ5hcHOumq1s47Gagf1BLOA6529N/oDhcaYnXYXJSKtD7cdikhfrN/LRg0C5/5eB9YZY/5Zz2pNfrxcqcum4xUnIi2d0yHAcGB9rdWa/PPoSl12fB6NMfcbYxKNMclYGbHQGHNtrdXcfrz8T+XNjckYUyEiE4EvsXqWzDDGrBWRKcAqY8wsrF/8t0UkC+ui2zgPqetOERkNVDjruqGx6wIQkX9j9YCIFZEcYDLWRSKMMS8Dc7B6bmQBB4EbPaSuy4E/iEgFcAgY1wT/MQ8ErgN+cba/AjwAJNWoy47j5UpddhyvNsCbIuLA+g/kA2PMbLs/jy7WZcvnsS6Nfbz01n+llPIRntzkopRS6gRooCullI/QQFdKKR+hga6UUj5CA10ppXyEBrryOSJSWWNkvTVSx4iYp7DtZKln1Eil7Oax/dCVOgWHnLeCK9Ws6Bm6ajZEZLOIPCkivzjH0O7onJ8sIgudgzctEJEk5/xWIvKJcxCsn0TkbOemHCLyqljjb3/lvEMREblTrHHMfxaR9236NlUzpoGufFFIrSaXsTWWFRpjegAvYI2GB9YAV286B296F3jOOf854GvnIFhnAmud81OBacaY7sA+4DLn/PuA3s7t3NpY35xS9dE7RZXPEZFiY0x4HfM3A0ONMdnOAbB2GWNiRCQPaGOMKXfO32mMiRWRXCCxxsBOh4e0nWeMSXW+vhcIMMZMFZG5QDHW6Ief1hinW6kmoWfoqrkx9UyfiNIa05UcuRY1EpiGdTa/0jmCnlJNRgNdNTdja/y73Dm9jCMDI10DLHVOLwD+ANUPUYisb6Mi4ge0M8YsAu7FGgr1mL8SlGpMegahfFFIjZEKAeYaYw53XYwSkZ+xzrKvcs67A5gpIn8BcjkyquJdwHQRuQnrTPwPQH3D5zqAd5yhL8BzzvG5lWoy2oaumg1nG3qaMSbP7lqUagza5KKUUj5Cz9CVUspH6Bm6Ukr5CA10pZTyERroSinlIzTQlVLKR2igK6WUj/h/MboILQDtMhkAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5BGVFEIi3_CT"
      },
      "source": [
        "And what about a summary of our model?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aps1FV4qWrZb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "306849a0-a3a2-4404-d046-eceb7186871f"
      },
      "source": [
        "# Resnet summary \n",
        "resnet_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "feature_extraction_layer (Ke (None, 2048)              23564800  \n",
            "_________________________________________________________________\n",
            "output_layer (Dense)         (None, 10)                20490     \n",
            "=================================================================\n",
            "Total params: 23,585,290\n",
            "Trainable params: 20,490\n",
            "Non-trainable params: 23,564,800\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "okdbmBA0SgCb"
      },
      "source": [
        "You can see the power of TensorFlow Hub here. The feature extraction layer has 23,564,800 parameters which are prelearned patterns the model has already learned on the ImageNet dataset. Since we set `trainable=False`, these patterns remain frozen (non-trainable) during training.\n",
        "\n",
        "This means during training the model updates the 20,490 parameters in the output layer to suit our dataset.\n",
        "\n",
        "Okay, we've trained a ResNetV250 model, time to do the same with EfficientNetB0 model.\n",
        "\n",
        "The setup will be the exact same as before, except for the `model_url` parameter in the `create_model()` function and the `experiment_name` parameter in the `create_tensorboard_callback()` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MrGi-CpMXHav",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "251ab535-7852-4e05-c356-dba75c37ba34"
      },
      "source": [
        "# Create model\n",
        "efficientnet_model = create_model(model_url=efficientnet_url, # use EfficientNetB0 TensorFlow Hub URL\n",
        "                                  num_classes=train_data_10_percent.num_classes)\n",
        "\n",
        "# Compile EfficientNet model\n",
        "efficientnet_model.compile(loss='categorical_crossentropy',\n",
        "                           optimizer=tf.keras.optimizers.Adam(),\n",
        "                           metrics=['accuracy'])\n",
        "\n",
        "# Fit EfficientNet model \n",
        "efficientnet_history = efficientnet_model.fit(train_data_10_percent, # only use 10% of training data\n",
        "                                              epochs=5, # train for 5 epochs\n",
        "                                              steps_per_epoch=len(train_data_10_percent),\n",
        "                                              validation_data=test_data,\n",
        "                                              validation_steps=len(test_data),\n",
        "                                              callbacks=[create_tensorboard_callback(dir_name=\"tensorflow_hub\", \n",
        "                                                                                     # Track logs under different experiment name\n",
        "                                                                                     experiment_name=\"efficientnetB0\")])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving TensorBoard log files to: tensorflow_hub/efficientnetB0/20210115-011549\n",
            "Epoch 1/5\n",
            "24/24 [==============================] - 27s 819ms/step - loss: 2.0594 - accuracy: 0.3025 - val_loss: 1.2756 - val_accuracy: 0.7524\n",
            "Epoch 2/5\n",
            "24/24 [==============================] - 16s 706ms/step - loss: 1.1388 - accuracy: 0.7528 - val_loss: 0.8559 - val_accuracy: 0.8328\n",
            "Epoch 3/5\n",
            "24/24 [==============================] - 16s 707ms/step - loss: 0.7618 - accuracy: 0.8380 - val_loss: 0.6862 - val_accuracy: 0.8496\n",
            "Epoch 4/5\n",
            "24/24 [==============================] - 16s 704ms/step - loss: 0.6424 - accuracy: 0.8662 - val_loss: 0.6034 - val_accuracy: 0.8616\n",
            "Epoch 5/5\n",
            "24/24 [==============================] - 16s 708ms/step - loss: 0.4856 - accuracy: 0.9102 - val_loss: 0.5512 - val_accuracy: 0.8668\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iMbtls4C57Dr"
      },
      "source": [
        "Holy smokes! The EfficientNetB0 model does even better than the ResNetV250 model! Achieving over 85% accuracy on the test set...again **with only 10% of the training data**.\n",
        "\n",
        "How cool is that?\n",
        "\n",
        "With a couple of lines of code we're able to leverage state of the art models and adjust them to our own use case.\n",
        "\n",
        "Let's check out the loss curves."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8UzgNYFdODRB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 573
        },
        "outputId": "24ffc097-9536-4ff7-f739-51bf2cf883a8"
      },
      "source": [
        "plot_loss_curves(efficientnet_history)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEWCAYAAAB2X2wCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUZfr/8fedThqEVJIAoYSEQBAlIqgg2EAsoKBY1oKFr911LeDaXf2JZXfVtWBZbKuiKyLgIirSRFEJSA2EElpIIIWSBAikPL8/ZhJCSINM5sxM7td15WJmzsk594zmk5PnnHM/YoxBKaWU+/OyugCllFKOoYGulFIeQgNdKaU8hAa6Ukp5CA10pZTyEBroSinlITTQlVLKQ2igq1ZBRLaKyPlW16FUS9JAV0opD6GBrlotEfEXkVdEJMf+9YqI+NuXRYjINyKyT0T2iMhPIuJlXzZBRHaKSLGIZIrIeda+E6VsfKwuQCkLPQoMAPoCBpgBPAY8DjwAZAOR9nUHAEZEkoC7gdONMTkikgB4O7dspeqmR+iqNbsOeMYYk2eMyQeeBq63LysDOgCdjTFlxpifjK3xUQXgD6SIiK8xZqsxZrMl1StViwa6as1igW01nm+zvwbwErAJ+F5EskRkIoAxZhPwZ+ApIE9EpopILEq5AA101ZrlAJ1rPO9kfw1jTLEx5gFjTFfgMuAvVWPlxphPjTFn27/XAC84t2yl6qaBrloTXxEJqPoCPgMeE5FIEYkAngD+AyAil4hIdxERYD+2oZZKEUkSkXPtJ09LgUNApTVvR6ljaaCr1mQ2tgCu+goA0oFVwGpgOfCsfd1EYC5QAiwB3jTGzMc2fj4JKAB2AVHAI857C0rVT3SCC6WU8gx6hK6UUh5CA10ppTyEBrpSSnkIDXSllPIQlt36HxERYRISEqzavVJKuaVly5YVGGMi61pmWaAnJCSQnp5u1e6VUsotici2+pbpkItSSnkIDXSllPIQGuhKKeUhtB+6UuoYZWVlZGdnU1paanUprVpAQADx8fH4+vo2+Xs00JVSx8jOziYkJISEhARsvcmUsxljKCwsJDs7my5dujT5+3TIRSl1jNLSUsLDwzXMLSQihIeHn/BfSRroSqnjaJhb72T+G7hdoOfsO8TTs9ZSVqEtqJVSqia3C/Q1O/fz/s9beWdRltWlKKWUS3G7QL+wVwwXp3bg1bkb2ZRXYnU5SikH27dvH2+++eYJf9+IESPYt29fg+s88cQTzJ0792RLq1NwcLBDt9ccjQa6iEwRkTwRWVPP8rYiMktEVorIWhEZ5/gyj/XUZb1o4+fNhGmrqKzUCTqU8iT1BXp5eXmD3zd79mzatWvX4DrPPPMM559/frPqc2VNuWzxA+B14KN6lt8FZBhjLhWRSCBTRD4xxhxxUI3HiQzx54lLUnjgvyv5+Ndt3HhmQkvtSqlW7elZa8nIKXLoNlNiQ3ny0l71Lp84cSKbN2+mb9+++Pr6EhAQQFhYGOvXr2fDhg2MGjWKHTt2UFpayn333cf48eOBo/2hSkpKuOiiizj77LP55ZdfiIuLY8aMGbRp04abbrqJSy65hDFjxpCQkMCNN97IrFmzKCsr47///S/Jycnk5+dz7bXXkpOTw8CBA/nhhx9YtmwZERERDb4vYwwPP/ww3377LSLCY489xtixY8nNzWXs2LEUFRVRXl7OW2+9xZlnnsktt9xCeno6IsLNN9/M/fff3+zPttEjdGPMImBPQ6sAIfbJdIPt6zb8q9QBrjgtjsE9Inlhznqy9x5s6d0ppZxk0qRJdOvWjRUrVvDSSy+xfPlyXn31VTZs2ADAlClTWLZsGenp6bz22msUFhYet42NGzdy1113sXbtWtq1a8e0adPq3FdERATLly/njjvu4OWXXwbg6aef5txzz2Xt2rWMGTOG7du3N6nur776ihUrVrBy5Urmzp3LQw89RG5uLp9++inDhg2rXta3b19WrFjBzp07WbNmDatXr2bcOMcMbDjixqLXgZlADhACjDXG1HkJioiMB8YDdOrUqVk7FRH+3+W9ufCfi3h0+ho+GHe6XmqllIM1dCTtLP379z/m5prXXnuN6dOnA7Bjxw42btxIeHj4Md/TpUsX+vbtC0C/fv3YunVrndu+4oorqtf56quvAFi8eHH19ocPH05YWFiT6ly8eDHXXHMN3t7eREdHc84557B06VJOP/10br75ZsrKyhg1ahR9+/ala9euZGVlcc8993DxxRdz4YUXNv0DaYAjTooOA1YAsUBf4HURCa1rRWPMO8aYNGNMWmRkne18T0h8WCAThiezcEM+0//Y2eztKaVcT1BQUPXjBQsWMHfuXJYsWcLKlSs59dRT67z5xt/fv/qxt7d3vePvVes1tE5zDR48mEWLFhEXF8dNN93ERx99RFhYGCtXrmTIkCFMnjyZW2+91SH7ckSgjwO+MjabgC1AsgO22yTXD+hMWucwnvkmg/ziw87arVKqhYSEhFBcXFznsv379xMWFkZgYCDr16/n119/dfj+zzrrLL744gsAvv/+e/bu3duk7xs0aBCff/45FRUV5Ofns2jRIvr378+2bduIjo7mtttu49Zbb2X58uUUFBRQWVnJ6NGjefbZZ1m+fLlDanfEkMt24DzgJxGJBpIAp10k7uUlTBrdhxGv/sRTs9byxrWnOWvXSqkWEB4ezllnnUXv3r1p06YN0dHR1cuGDx/O5MmT6dmzJ0lJSQwYMMDh+3/yySe55ppr+Pjjjxk4cCAxMTGEhIQ0+n2XX345S5Ys4ZRTTkFEePHFF4mJieHDDz/kpZdewtfXl+DgYD766CN27tzJuHHjqKy0jU4///zzDqldjGn4sj8R+QwYAkQAu4EnAV8AY8xkEYnFdiVMB0CAScaY/zS247S0NOPIGYvemL+Jl77L5O3r+zGsV4zDtqtUa7Nu3Tp69uxpdRmWOXz4MN7e3vj4+LBkyRLuuOMOVqxYYUktdf23EJFlxpi0utZv9AjdGHNNI8tzAMeM6DfD+MFd+WZVLo9/vYYBXcNp26bpLSeVUqrK9u3bueqqq6isrMTPz493333X6pKazGPa5/p6e/Hi6D6MfGMxz89ex6TRfawuSSnlhhITE/njjz+Oea2wsJDzzjvvuHV//PHH466wsZLHBDpAanxbbhvclbcXZnHpKbGc1b3hGwGUUqopwsPDLRt2ORFu18ulMfef34OE8EAe+Wo1h45UWF2OUko5jccFeoCvN5NG92H7noP844dMq8tRSimn8bhABxjQNZzrzujEvxdvYcWOhruvKaWUp/DIQAeYeFEy0aEBTPhyFUfKdTIMpZTn89hADwnw5bnLe5O5u5i3Fmy2uhylVAtpqB/51q1b6d27txOrsZbHBjrAucnRjOwby+vzN7Jhd923EiullKfwqMsW6/LEJSks2pDPw1+uYtodZ+LtpR0ZlWqybyfCrtWO3WZMKlw0qd7FEydOpGPHjtx1110APPXUU/j4+DB//nz27t1LWVkZzz77LCNHjjyh3ZaWlnLHHXeQnp6Oj48P//jHPxg6dChr165l3LhxHDlyhMrKSqZNm0ZsbCxXXXUV2dnZVFRU8PjjjzN27NhmvW1n8OgjdIDwYH+euqwXK3bs44NftlpdjlKqEWPHjq1ujgXwxRdfcOONNzJ9+nSWL1/O/PnzeeCBB2isbUltb7zxBiLC6tWr+eyzz7jxxhspLS1l8uTJ3HfffaxYsYL09HTi4+OZM2cOsbGxrFy5kjVr1jB8+HBHv80W4fFH6ACXnRLLjBU5vPxdJhf0jKZTeKDVJSnlHho4km4pp556Knl5eeTk5JCfn09YWBgxMTHcf//9LFq0CC8vL3bu3Mnu3buJiWl636bFixdzzz33AJCcnEznzp3ZsGEDAwcO5LnnniM7O5srrriCxMREUlNTeeCBB5gwYQKXXHIJgwYNaqm361Aef4QOtskwnh3VG28v4a/TV5/wb3allHNdeeWVfPnll3z++eeMHTuWTz75hPz8fJYtW8aKFSuIjo6usw/6ybj22muZOXMmbdq0YcSIEcybN48ePXqwfPlyUlNTeeyxx3jmmWccsq+W1ioCHSC2XRsmXpTM4k0F/HdZttXlKKUaMHbsWKZOncqXX37JlVdeyf79+4mKisLX15f58+ezbdu2E97moEGD+OSTTwDYsGED27dvJykpiaysLLp27cq9997LyJEjWbVqFTk5OQQGBvKnP/2Jhx56yGH9yltaqxhyqXJt/07MXJnDs99kMKRHJFGhAVaXpJSqQ69evSguLiYuLo4OHTpw3XXXcemll5KamkpaWhrJySc+h86dd97JHXfcQWpqKj4+PnzwwQf4+/vzxRdf8PHHH+Pr60tMTAx//etfWbp0KQ899BBeXl74+vry1ltvtcC7dLxG+6G3FEf3Q2+qrPwSLnr1J4YmRTH5+n5O379Srq6190N3JSfaD73VDLlU6RoZzP0X9GDO2l18uzrX6nKUUsphWtWQS5Vbz+7CN6tyeHzGWgZ2C6ddoJ/VJSmlmmH16tVcf/31x7zm7+/Pb7/9ZlFF1miVge7j7cULo/tw2es/8+z/1vHyladYXZJSLsUYg4j73ISXmprqFv3KT8TJDIe3uiGXKr1i23L7OV35clk2izbkW12OUi4jICCAwsJCvbzXQsYYCgsLCQg4sQs3WuURepV7zk3k2zW7eOSr1Xx//2CC/Fv1x6EUAPHx8WRnZ5Ofrwc6VgoICCA+Pv6EvqdVJ1iArzcvju7DlW8v4aXvMnnqsl5Wl6SU5Xx9fenSpYvVZaiT0OiQi4hMEZE8EVnTwDpDRGSFiKwVkYWOLbFlpSW054YBnflwyVaWbdtrdTlKKXXSmjKG/gFQb2caEWkHvAlcZozpBVzpmNKc56HhycS2bcOEaas4XK7zkCql3FOjgW6MWQTsaWCVa4GvjDHb7evnOag2pwn29+G5y3uzKa+EN+ZtsrocpZQ6KY64yqUHECYiC0RkmYjcUN+KIjJeRNJFJN3VTrgMSYriitPieHPBZtblFlldjlJKnTBHBLoP0A+4GBgGPC4iPepa0RjzjjEmzRiTFhkZ6YBdO9bjF6fQLtCXCdNWUV6h85AqpdyLIwI9G/jOGHPAGFMALALc8k6dsCA/nrqsF6uy9zPl5y1Wl6OUUifEEYE+AzhbRHxEJBA4A1jngO1a4uLUDlyQEs3fv9/A1oIDVpejlFJN1pTLFj8DlgBJIpItIreIyO0icjuAMWYdMAdYBfwOvGeMqfcSR1dXNRmGn48XE6atorJS75ZTSrmHRm8sMsZc04R1XgJeckhFLiA6NIBHR/Rk4lermbp0B9ee0cnqkpRSqlGttpdLY8ae3pGBXcN5fvY6du13zFRXSinVkjTQ6yEiTBqdSlllJY99rfOQKqVcnwZ6AzqHB/HghUnMXZfHN6t0MgyllGvTQG/EuLO6cErHdjw1cy17DhyxuhyllKqXBnojvL2EF0f3oai0jL99k2F1OUopVS8N9CZIignhziHdmf7HTuavd7tWNUqpVkIDvYnuHNqNxKhgHp2+muLSMqvLUUqp42igN5G/jzcvjOlDblEpL87JtLocpZQ6jgb6CTitUxjjzuzCx79u4/ctDXUUVkop59NAP0EPDutBfFgbJk5bRWmZToahlHIdGugnKNDPh0lX9CGr4ACv/bjR6nKUUqqaBvpJODsxgqvS4nl7URZrdu63uhyllAI00E/aoyNSaB/kx8NfrqJMJ8NQSrkADfST1DbQl7+N7E1GbhHv/pRldTlKKaWB3hzDe8cwIjWGV+ZuZHN+idXlKKVaOQ30Znrqsl608fVmok6GoZSymAZ6M0WFBPDYxT1ZunUvn/y2zepylFKtmAa6A4zpF8+gxAgmfbuenfsOWV2OUqqV0kB3ABHh/12eigEena6TYSilrKGB7iAd2wfy0LAkFmTmM2NFjtXlKKVaIQ10B7phYAKndWrH07PWUlBy2OpylFKtTKOBLiJTRCRPRNY0st7pIlIuImMcV5578fYSXhjdhwOHK3h6lk6GoZRyrqYcoX8ADG9oBRHxBl4AvndATW4tMTqEe87tzqyVOfyQsdvqcpRSrUijgW6MWQQ01iv2HmAaoNP5AP93TjeSY0J47OvVFOlkGEopJ2n2GLqIxAGXA281Yd3xIpIuIun5+fnN3bXL8vPx4sUxfcgvPszzs9dbXY5SqpVwxEnRV4AJxphGO1QZY94xxqQZY9IiIyMdsGvX1Se+HbcO6spnv2/nl80FVpejlGoFHBHoacBUEdkKjAHeFJFRDtiu27v//B50Dg/kka9Wc+iIToahlGpZzQ50Y0wXY0yCMSYB+BK40xjzdbMr8wBt/Lx5/opUthUe5JW5G6wuRynl4Zpy2eJnwBIgSUSyReQWEbldRG5v+fLc35ndIrimfyfe/SmLVdn7rC5HKeXBxKrb1NPS0kx6erol+3a2otIyLvjHQsIC/Zh599n4+ej9XEqpkyMiy4wxaXUt02RxgtAAX54blcr6XcW8vXCz1eUopTyUBrqTnJ8SzaWnxPKveZvYuLvY6nKUUh5IA92Jnrw0hSB/byZMW0WFToahlHIwDXQnigj254lLU1i+fR8fLdlqdTlKKQ+jge5ko/rGMSQpkhfnZLJjz0Gry1FKeRANdCcTEZ67PBUvgb/qZBhKKQfSQLdAXLs2TLwomZ82FjBt+U6ry1FKeQgNdItcd0ZnTk8I42/fZJBXXGp1OUopD6CBbhEvL2HS6D4cKqvgqZlrrS5HKeUBNNAt1C0ymD+fn8js1buYsybX6nKUUm5OA91itw3qSq/YUB6fsZb9B3UyDKXUyXO/QK8og7XTwUOuDvH19uKF0X3Yc+AIz83WeUiVUifP/QJ9xSfw35tg5j1QfsTqahyid1xb/m9wV75Iz2bxRp0MQyl1ctwv0E+9AQY9CH98DB9dBiWeMZXdvecl0jUiiIlfreLgkXKry1FKuSH3C3QvLzjvcRj9b8j5A94dCrtWW11VswX4ejNpdB+y9x7i5e90Mgyl1Ilzv0CvkjoGxn0LlRXw7wshY6bVFTVb/y7tuX5AZ97/ZQvLt++1uhyllJtx30AHiDsNxs+HqBT44npY8ILbnyx9eHgSHUIDmPDlKg6X6zykSqmmc+9ABwiJgZv+B32uhgX/z3bC9Ij7Nr0KCfDluctT2ZhXwpvzdTIMpVTTuX+gA/gGwOWT4YJnIGMGTBkG+7OtruqkDU2O4vJT43hzwSbW7yqyuhyllJvwjEAHEIGz7oNrP4c9W+CdobDjd6urOmmPX5JCaIAvE77UyTCUUk3jOYFepccwuHUu+AXBBxfDH59YXdFJaR/kx1OX9WJl9n7e/3mL1eUopdxAo4EuIlNEJE9E1tSz/DoRWSUiq0XkFxE5xfFlnqCoZLhtHnQaADPuhO8etV0N42Yu6dOB83tG8/L3mWwrPGB1OUopF9eUI/QPgOENLN8CnGOMSQX+BrzjgLqaL7A9/Okr6D8elrwOn14FpfutruqEiAjPjuqNr5cXE6fpZBhKqYY1GujGmEXAngaW/2KMqbpo+lcg3kG1NZ+3L4x4CS55BbIWwLvnQaF7XTkS0zaAR0b0ZElWIZ8v3WF1OUopF+boMfRbgG/rWygi40UkXUTS8/OdeMt+2ji4YQYcLLTdWbp5nvP27QBXn96RAV3b89zsdewu0skwlFJ1c1igi8hQbIE+ob51jDHvGGPSjDFpkZGRjtp10yScbbsJKTQO/jMGfp3sNjcheXkJk67ow5HySh77eo0OvSil6uSQQBeRPsB7wEhjTKEjttkiwhLglu+hx3CYM8GtOjYmRATxwIU9+CFjN7NX77K6HKWUC2p2oItIJ+Ar4HpjjOt3lfIPgbH/ccuOjTef1YU+8W15cuYa9h5wj19ESinnacpli58BS4AkEckWkVtE5HYRud2+yhNAOPCmiKwQkfQWrNcx3LRjo499Mox9B8v42/90Mgyl1LHEqvHYtLQ0k57uAtm/czlMvc52SeMVb0PPS62uqFH/+D6T1+Zt4oNxpzMkKcrqcpRSTiQiy4wxaXUt87w7RU9UdcfGnvD5n2Dhiy5/svSuc7vTPSqYR6evoeSwToahlLLRQIdjOzbOf87lOzb6+3jzwug+5Ow/xEtz1ltdjlLKRWigV6nq2Hj+027RsbFf5zBuHJjAR79uI31rvfd9KaVaEQ30mkTg7D+7TcfGh4YlEdu2DROmraK0zP161SilHEsDvS61Ozau+NTqiuoU5O/D81eksjn/AK/P22R1OUopi2mg16dmx8av73DZjo2De0Qypl88kxduZm2OezUfU0o5lgZ6Q6o6Np5+m0t3bHzs4p60C/RjwrRVlFdUWl2OUsoiGuiN8faFi1+GS/7psh0b2wX68beRvVizs4j3FutkGEq1VhroTZV2s0t3bLwotQPDe8Xwzx82kJVfYnU5SikLaKCfCBfv2PjMyF74+3gx8avVVOo8pEq1OhroJ6p2x8ZZ97pMx8ao0AAeuziF37fs4dPft1tdjlLKyTTQT0bNjo3LP4KPRsKBAqurAuDKtHjO6h7OpG/Xk7v/kNXlKKWcSAP9ZB3TsXG57SYkF+jYKCI8f3kfKioNj07XyTCUak000JsrdQyMmw2VZfDvYbBultUV0Sk8kAeHJTFvfR4zV+ZYXY5Sykk00B0hrh+MX2C7GclFOjbedGYCfTu24+lZGRSWHLa0FqWUc2igO0pIDNw0G/qMtXVs/HKcpR0bvb2EF8f0obi0jGe+0ckwlGoNNNAdyTcALn/b1rFx7dfw/nBLOzb2iA7h7qGJzFiRw18+X8HuolLLalFKtTwNdEer2bGxMMvyjo13Du3GnUO68c2qXIa+vIA3F2zSzoxKeSgN9JbiIh0bfb29eHh4Mj/8ZTBnd4/gxTmZXPjPRXy/dpdeAaOUh9FAb0lVHRs7nmF5x8bO4UG8c0Ma/7nlDAJ8vRj/8TKu//fvbNhdbEk9SinHazTQRWSKiOSJyJp6louIvCYim0RklYic5vgy3Vhge7h+eo2OjWMt7dh4dmIEs+8dxNOX9WL1zv1c9OpPPDljDfsOusbdrkqpk9eUI/QPgOENLL8ISLR/jQfean5ZHuaYjo3z4b3zLe3Y6OPtxY1nJrDgwSFc278TH/+6jSEvL+DjJVu1/a5SbqzRQDfGLAIamrRyJPCRsfkVaCciHRxVoEep6th4oADePRc2z7e0nLAgP/42qjez7xtEz5hQHp+xlkv+tZhfNrtGGwOl1IlxxBh6HLCjxvNs+2uqLtUdG2PhP6NdomNjckwon952BpP/dBolh8u59t3fuP3jZezYY9119EqpE+fUk6IiMl5E0kUkPT8/35m7di3VHRuHuUzHRhFheO8OzP3LOTw0LImFG/I57x8Lefm7TA4cLre0NqVU0zgi0HcCHWs8j7e/dhxjzDvGmDRjTFpkZKQDdu3G/ENg7Ccw6AGX6tgY4OvNXUO7M//BIYzoHcPr8zdx7t8XMP2PbL3MUSkX54hAnwncYL/aZQCw3xiT64Dtej4vLzjviVodG+u8mMjpYtoG8MrVpzLtjoFEhwZw/+crGf3WL6zcsc/q0pRS9ZDGjrpE5DNgCBAB7AaeBHwBjDGTRUSA17FdCXMQGGeMSW9sx2lpaSY9vdHVWo+dy2DqdVBaBFe8DT0vtbqiapWVhmnLs3lhTiYFJYe5sl88Dw1PIiokwOrSlGp1RGSZMSatzmVW/RmtgV6H4l0w9VpbuA99DAY/aGsl4CKKS8t4ff4mpizegr+PN3ef251xZyXg7+NtdWlKtRoa6O6krNR2knTV59Drchj5JvgFWl3VMbYUHOC5/2Uwd10eCeGBPHZxCuf1jEJc6JePUp6qoUDXW/9djYt1bKxLl4gg3rvxdD68uT8+3l7c+lE6N0z5nU152kZAKSvpEbory5wD024F3zZw9SfQsb/VFR2nrKKSj5ds459zN3DwSAU3DOzMn8/rQdtAX6tLU8oj6RG6u0oaDrf+YBtysbBjY0N8vb24+ewuLHhwCFef3pEPf9nK0L8v4JPftlFRqZc5KuVMGuiuLqon3Db/aMfG7x+zrGNjQ8KD/Xnu8lRm3XM23aOCeXT6Gi7512J+zSq0ujSlWg0NdHdQs2PjL/+yvGNjQ3rFtuXz8QN449rTKDpUxtXv/Mpdnywne6+2EVCqpekYurtJnwKzH4L2XeGaqRDezeqK6lVaVsE7i7J4c8EmjIH/G9yV24d0I9DPx+rSlHJbOobuSdJuhuu/dpmOjQ0J8PXm3vMSmffAEIb1iuG1eZs47+8LmbFip7YRUKoFaKC7oy6DbDMhVXVs/O1tyzs2NiS2XRteu+ZU/nv7QMKD/bhv6gqunLyE1dmuOWyklLvSQHdX7bsc7dj47cMw6z7LOzY25vSE9sy462xeGJ3K1sIDXPbGYiZ8uYr84sNWl6aUR9AxdHdXWQnzn4Wf/g6dzoSxH0NQhNVVNaqotIx//biR93/eShv70MyNZybg56PHGEo1RG/9bw1Wfwkz7oKgKLjmM4jpbXVFTbI5v4Tn/reOeevz6BoRxOOXpDA0OcrqspRyWXpStDVIHQPjZkNlGfz7Qlg3y+qKmqRbZDBTbjqd98edDgLjPljKTe//zqa8EqtLU8rtaKB7krh+tpuQopLh8z/Bwpdc+mRpTUOTophz32Aeu7gny7buZfgri3j2mwz2HyqzujSl3IYOuXiiskMw815Y/QX0ugJGvuFyHRsbUlBymL9/n8nUpTtoH+jHg8OSuCqtI95e2s1RKR1yaW1828AV79g7Nk6HKcMgY6Yt6N1ARLA/z1/Rh1l3n03XyCAe+Wo1l72+mN+37LG6NKVcmh6he7rMOTDzbjiQD75BtoZfKaMg8QJb8Ls4YwzfrMrl+dnryNlfyqWnxDLxomTi2rl+7Uq1BL3KpbWrKIetP0HG17aTpQcL3S7cDx2pYPLCzUxeuBkRuP2cbvzf4G608dPZklTrooGujqooh22LbUMxNcO9xzDbDEkuHu7Zew/y/Lfr+d+qXOLateGREclcnNpBZ0tSrYYGuqpbdbh/DZACbAgAABP8SURBVOtm1gr3UdD9Apc9mfpbViFPz8ogI7eI/gnteeLSFHrHtbW6LKVanAa6apwbhntFpeHzpTt4+ftM9h48wtWnd+LBC3sQHuxvdWlKtZhmB7qIDAdeBbyB94wxk2ot7wR8CLSzrzPRGDO7oW1qoLuwY8J9FhwscOlw33+ojFfnbuSjJVtp4+fNn8/vwQ0DO+PrrRdxKc/TrEAXEW9gA3ABkA0sBa4xxmTUWOcd4A9jzFsikgLMNsYkNLRdDXQ3UVEO236uMeZeFe4X2sbcXSjcN+UV88w361i0IZ9ukbY2AkOStI2A8iwNBXpTZhroD2wyxmTZNzYVGAlk1FjHAKH2x22BnJMvV7kUbx/oeo7ta8TLtnDP+Np2Xfva6eAbaDtyTxkFiRdaGu7do0L4cNzpzFufx9++yeCm95dybnIUj1+SQpeIIMvqUspZmnKEPgYYboy51f78euAMY8zdNdbpAHwPhAFBwPnGmGV1bGs8MB6gU6dO/bZt2+ao96GcrerIvSrcDxa4VLgfKa/kg1+28NqPmzhcXsHNZ3Xh7nO7ExLga1lNSjlCc4dcmhLof7Fv6+8iMhD4N9DbGFNZ33Z1yMWD1Az3dbPsNzG5RrjnFZfy8neZ/HdZNuFBfjw8LJkx/eLx0jYCyk01N9AHAk8ZY4bZnz8CYIx5vsY6a7GF/g778yxggDEmr77taqB7qMqKY8fcq8I90T7mblG4r8rex9OzMli2bS+pcW156rIU+nVu7/Q6lGqu5ga6D7aToucBO7GdFL3WGLO2xjrfAp8bYz4QkZ7Aj0CcaWDjGuitgIuFuzGGmStzeH72enYVlTKyr62NQIe2rnsjlVK1OeKyxRHAK9guSZxijHlORJ4B0o0xM+1XtrwLBGM7QfqwMeb7hrapgd7KVIe7/Tr3Y8K9aljGOScuDx4p560Fm3l7URbeItw5pBu3De5KgK+2EVCuT28sUq7FRcJ9x56DPP/tOmav3kV8WBseHdGT4b1jtI2Acmka6Mp1VVbAtl/swzI1w/2CGsMyLRvuv2wu4JlZGazfVcyAru158tJe9OwQ2vg3KmUBDXTlHuoKd582R29iasFwL6+oZOrSHfz9+0z2HSqjX6cwhiZHMSQpkpQOoXrUrlyGBrpyP1XhXnWd+4G8o+GeMsp2SWQLhPu+g0d4/+et/Lh+N2t2FgEQFeLPkKRIhiZFcVZiBKF6LbuykAa6cm8WhXtecSkLM/NZkJnPoo35FJeW4+Ml9OtsO3ofmhRFj+hgPXpXTqWBrjyHReFeVlHJH9v3MT8zj/nr81i/qxiA2LYBnJMUxdCkSM7qHkGQf1O6aSh18jTQlWeqrIDtS2xj7jXDveqEaguFO0Du/kMszMxnfmYeizcWcOBIBX7eXvTv0p4hSZEMSYqiW2SQHr0rh9NAV56vOty/howZtcJ9FCQOA//gFtn1kfJK0rfuYcGGfOavz2NjXgkAHdu3YWiSbWhmQNdwnS5POYQGumpdaob7uplQsttp4Q6269sXbMhnYWYeP28q5FBZBf4+XgzoGs5Q+9F7gnZ/VCdJA121XhaHe2lZBb9v2cP8zDwWZuaTVXAAgC4RQdVXzvTv0l7vUlVNpoGuFNjD/dej17lXh/v59uvcWzbcAbYWHGBBZh7zM/P5NauQw+WVtPH15sxu4QxJjmJIj0g6tneNCUOUa9JAV6q2qnDPsI+5l+wGnwDbkXvKKIjrB+06g1fLTWN36EgFv2YV2q6cycxjx55DACRGBVcfvacltMfPR6fSU0dpoCvVkLrCHWxH7xHdITIZIpMgIsn2uH0X8HbszUXGGLIKDjB/fR4LMvP5bUshZRWGID9vzk6MYEiS7a5V7QypNNCVaqrKCshZAXlrIT/z6Nf+7UfX8fKF8G41Qt7+FZ4IvgEOKePA4XJ+2Ww7el+wPo+c/aUAJMeE2FoS9IjktM5hOhF2K6SBrlRzHS6Bwo3Hhnz+eti7Baom5hIv2zBN1RF91VdED/APOeldG2PYsLvEPvaeR/rWvZRXGkICfBicGMmQpEjOSYokKsQxv0yUa9NAV6qllB+Gwk3HhnzBBijYCJVlR9cLja8V8vZ/A0981qSi0jJ+3ljAAvuNTXnFhwHoHRfK0KQohiRF0bdjO7x1mj2PpIGulLNVlMPerbaArwr5/PW2oC87eHS9oMhaY/T2r+BoaMJdpsYYMnKLWJCZz4LMPJZt20ulgXaBvgxOjGRociSDEyMJD/ZvufeqnEoDXSlXUVkJ+3ccDfj89ZC/wXZ0f3j/0fUC2tqCPqLHsUM4ofENXnmz7+ARftpYwPzMPBZtyKeg5Agi0Ce+HUPtV86kxrXVSbLdmAa6Uq7OGCjeBQWZx47TF2Ta+sJX8Q2CiER7yFeFfbJt7N772MZglZWGNTn7mb/eNjSzMnsfxkB4kB/n9IhkSHIUgxMjaBfo5+Q3q5pDA10pd3ag8Nigr3pctPPoOt5+tqtsInscO4QT3g18bMMthSWHq4/eF27IZ9/BMrwETusUVt1QrFesTubh6jTQlfJEpUW2Mfn89TUCfz3s3YZtrnZAvG3Xzdcavqlo352VeWUsWG+7a3X1TttwT9VkHkOSojhbJ/NwSRroSrUmZYdsQV97nH7PZqgsP7peu07VJ2KLQrqRfiCK2btC+W7zIZ3Mw4U1O9BFZDjwKuANvGeMmVTHOlcBT2E7NFhpjLm2oW1qoCvlZBVlsCfLHvA1hnAKN0J5afVqJjiGopBubKqMZUlxJD/tDWeTicUvNIohydE6mYfFmhXoIuINbAAuALKBpcA1xpiMGuskAl8A5xpj9opIlDEmr6HtaqAr5SIqK2DftuNPxuZnwpGS6tVKvELJrIglsyKWLInHJzqZTkmn0b9PL7pFhejRu5M0FOhN+RXbH9hkjMmyb2wqMBLIqLHObcAbxpi9AI2FuVLKhXh5Q/uutq+ki46+bgwU5VQf0QcXZHJq3nr67F6O75F5UAAUQPHiNmyWCA76R1IZ3AH/sDjaRXckokMCvu1iIaQDBEc5vP+NOl5TAj0O2FHjeTZwRq11egCIyM/YhmWeMsbMqb0hERkPjAfo1KnTydSrlHIWEWgbZ/vqfh4AXoCXMXCgAPLXs3f7GvKyVlG+L5uAA7sIKVhCZME+fDZVHrMpg1AWEI53aAe828VBSIwt6ENiICT26PPA8BbtcOnpHDUI5gMkAkOAeGCRiKQaY/bVXMkY8w7wDtiGXBy0b6WUM4lAcCQERxLWZRBh5xxdVFFp2JJXxOZt28jNzmLPru2U7skm8HA+0SV7iT6wl/j8TGK8fiW0Yt/x2/bygeAYCO1Qf+iHxNhuvNIhnuM0JdB3Ah1rPI+3v1ZTNvCbMaYM2CIiG7AF/FKHVKmUcgveXkL3mLZ0j+kDZ/Spfj2vuJR1ucVk5BQxPbeIjJz97CjYT4TZR7TspbN/Eamhh+jRppiOvvuJMHsIzN+AV9aiY++greIbWCvwO9R6bP/Xr3VNFtKUQF8KJIpIF2xBfjVQ+wqWr4FrgPdFJALbEEyWIwtVSrmvqJAAokICOKdHZPVrB4+Uk7mr2Bb0ufv5X04RL+cUc6isAgAfL6F7VDB9u/hyWthhUoIPkuBfTPCRPNtdtcW5tn93Lrf9W37o+B0HtK3/KD+kg+0vgeBojxnfbzTQjTHlInI38B228fEpxpi1IvIMkG6MmWlfdqGIZAAVwEPGmMKWLFwp5d4C/Xw4tVMYp3YKq36totKwrfAAGblFZOQUkZFbxLzNRUy1d5SEYGLbRpASO4CUDqGkJIeS0qEt8e0C8DpSVCPoc48GfnEuFOVCwSIo2XXstfgACARF1B36oTXH9yNcfnxfbyxSSrm8gpLDrKsR8hk5RWzOL6HSHl8h/j707BBKzw4hpMTaQj4xOvj4ybcrK+FgIRTnHHuUXxX6Vc8P5FN9t22VqvH9kJj6Qz8kBgLatej4vt4pqpTyOKVlFWTuKj7maH5dbhEHj9iGbLy9hO6RwfaADyUlNpSeHUJpH9SEZmQVZbapCGuGflEdvwRK6zix69Om4cCvGu8/yfF9DXSlVKtQWWnYvufgMSGfkVPErqKjd8LGhAYcE/IpHULp1D7w5FoKHzloG8ap6yi/eJftL4Gi3OPH9wfeDcOeO6n32Nwbi5RSyi14eQkJEUEkRAQxIrVD9euFJYerT75m5BSxLreYhRvyqbCP2QT5edOzRsCnxIbSIzrk+CGb2vwCj96UVR9j4HDRsaEf0cMRb/c4eoSulGqVSssq2Li7pDrkbUM2xZQctp009fYSukYEHXc0b/XsT3qErpRStQT4epMa35bU+LbVr1VWGnbsPXjMmPzSLXuYsSKnep3oUP9jxuRTOoSSEB7kErNAaaArpZSdl5fQOTyIzuFBXFRjyGbvgSO2q2xqjM0v2lhQPWQT6OdNcszRK2xSYkNJig6hjV8jQzYOpkMuSil1EkrLKtiUV3L05GtuEetyiii2D9l4CXSNDCalQ+gx4/ORIc0bstEhF6WUcrAAX296x7Wld9zRIRtjDNl7D7G2xhU2y7btZebKo0M2kSH+jB/UldsGN3Ai9SRpoCullIOICB3bB9KxfSDDe8dUv77v4BH7VTa2kI8KbZkTqxroSinVwtoF+jGwWzgDu4W36H5cuzGBUkqpJtNAV0opD6GBrpRSHkIDXSmlPIQGulJKeQgNdKWU8hAa6Eop5SE00JVSykNY1stFRPKBbSf57RFAgQPLcRRXrQtctzat68RoXSfGE+vqbIyJrGuBZYHeHCKSXl9zGiu5al3gurVpXSdG6zoxra0uHXJRSikPoYGulFIewl0D/R2rC6iHq9YFrlub1nVitK4T06rqcssxdKWUUsdz1yN0pZRStWigK6WUh3DpQBeR4SKSKSKbRGRiHcv9ReRz+/LfRCTBReq6SUTyRWSF/etWJ9U1RUTyRGRNPctFRF6z171KRE5zkbqGiMj+Gp/XE06oqaOIzBeRDBFZKyL31bGO0z+vJtbl9M/Lvt8AEfldRFbaa3u6jnWc/jPZxLqs+pn0FpE/ROSbOpY5/rMyxrjkF+ANbAa6An7ASiCl1jp3ApPtj68GPneRum4CXrfgMxsMnAasqWf5COBbQIABwG8uUtcQ4Bsnf1YdgNPsj0OADXX8d3T659XEupz+edn3K0Cw/bEv8BswoNY6VvxMNqUuq34m/wJ8Wtd/r5b4rFz5CL0/sMkYk2WMOQJMBUbWWmck8KH98ZfAeSIiLlCXJYwxi4A9DawyEvjI2PwKtBORDi5Ql9MZY3KNMcvtj4uBdUBcrdWc/nk1sS5L2D+HEvtTX/tX7asqnP4z2cS6nE5E4oGLgffqWcXhn5UrB3ocsKPG82yO/x+7eh1jTDmwH2jZSfuaVhfAaPuf6V+KSMcWrqmpmlq7FQba/2T+VkR6OXPH9j91T8V2ZFeTpZ9XA3WBRZ+XfQhhBZAH/GCMqfczc+LPZFPqAuf/TL4CPAxU1rPc4Z+VKwe6O5sFJBhj+gA/cPS3sKrbcmz9KU4B/gV87awdi0gwMA34szGmyFn7bUwjdVn2eRljKowxfYF4oL+I9HbWvhvShLqc+jMpIpcAecaYZS25n9pcOdB3AjV/i8bbX6tzHRHxAdoChVbXZYwpNMYctj99D+jXwjU1VVM+U6czxhRV/clsjJkN+IpIREvvV0R8sYXmJ8aYr+pYxZLPq7G6rPq8atWwD5gPDK+1yIqfyUbrsuBn8izgMhHZim1Y9lwR+U+tdRz+WblyoC8FEkWki4j4YTtpMLPWOjOBG+2PxwDzjP0Mg5V11RpnvQzbOKgrmAncYL96YwCw3xiTa3VRIhJTNXYoIv2x/X/ZoiFg39+/gXXGmH/Us5rTP6+m1GXF52XfV6SItLM/bgNcAKyvtZrTfyabUpezfyaNMY8YY+KNMQnYMmKeMeZPtVZz+Gfl05xvbknGmHIRuRv4DtuVJVOMMWtF5Bkg3RgzE9v/+B+LyCZsJ92udpG67hWRy4Bye103tXRdACLyGbYrICJEJBt4EtsJIowxk4HZ2K7c2AQcBMa5SF1jgDtEpBw4BFzthF/MZwHXA6vtY68AfwU61ajLis+rKXVZ8XmB7QqcD0XEG9svkS+MMd9Y/TPZxLos+ZmsraU/K731XymlPIQrD7kopZQ6ARroSinlITTQlVLKQ2igK6WUh9BAV0opD6GBrjyOiFTU6Kq3QuroiNmMbSdIPV0jlbKay16HrlQzHLLfBq5Uq6JH6KrVEJGtIvKiiKy298/ubn89QUTm2Rs3/SgineyvR4vIdHsTrJUicqZ9U94i8q7Yem9/b787ERG5V2x9zFeJyFSL3qZqxTTQlSdqU2vIZWyNZfuNManA69i64YGtwdWH9sZNnwCv2V9/DVhob4J1GrDW/noi8IYxphewDxhtf30icKp9O7e31JtTqj56p6jyOCJSYowJruP1rcC5xpgsewOsXcaYcBEpADoYY8rsr+caYyJEJB+Ir9HUqaql7Q/GmET78wmArzHmWRGZA5Rg6374dY0e3Uo5hR6hq9bG1PP4RByu8biCo+eiLgbewHY0v9TeQU8pp9FAV63N2Br/LrE//oWjjZGuA36yP/4RuAOqJ1BoW99GRcQL6GiMmQ9MwNYK9bi/EpRqSXoEoTxRmxqdCgHmGGOqLl0ME5FV2I6yr7G/dg/wvog8BORztKvifcA7InILtiPxO4D62ud6A/+xh74Ar9l7cyvlNDqGrloN+xh6mjGmwOpalGoJOuSilFIeQo/QlVLKQ+gRulJKeQgNdKWU8hAa6Eop5SE00JVSykNooCullIf4/6YhuwebisAnAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEWCAYAAAB2X2wCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV9bnv8c9D5oRMhJARCPMQBhlEAQcEUVtnLaK1FjkOV1snPK21Pa16vb29fZ1zWke0RY91rlVbe6xHZbaK4oCzCcg8JJB5hsx57h9rJ4SQkB3Yydp753m/XvvF3mvaTxbsL7/89m/9lqgqxhhjAt8AtwswxhjjGxboxhgTJCzQjTEmSFigG2NMkLBAN8aYIGGBbowxQcIC3RhjgoQFugk4IvKOiJSLSITbtRjjTyzQTUARkSzgdECBi/rwfUP76r2MOV4W6CbQ/BD4EHgaWNK6UESGisjfRKRYREpF5NF2624Qkc0iUi0iuSIy3bNcRWR0u+2eFpFfe57PE5E8EfmZiBQAfxKRRBF5w/Me5Z7nme32HyQifxKR/Z71f/cs/0ZELmy3XZiIlIjItF47S6ZfskA3geaHwAuex7kikiIiIcAbwB4gC8gAXgIQkUXAfZ794nBa9aVevlcqMAgYDtyI83n5k+f1MKAWeLTd9s8B0UA2MAR4wLP8WeAH7bb7LnBAVT/3sg5jvCI2l4sJFCJyGrAeSFPVEhHZAvwRp8X+umd5U4d9VgJvqupDnRxPgTGqut3z+mkgT1V/KSLzgFVAnKrWdVHPScB6VU0UkTQgH0hS1fIO26UD3wIZqlolIq8CH6vqvx/3yTCmE9ZCN4FkCbBKVUs8r1/0LBsK7OkY5h5DgR3H+X7F7cNcRKJF5I8iskdEqoB3gQTPbwhDgbKOYQ6gqvuB94HLRSQB+A7ObxjG+JR90WMCgohEAVcAIZ4+bYAIIAEoBIaJSGgnob4PGNXFYQ/hdJG0SgXy2r3u+OvrvwLjgFNUtcDTQv8cEM/7DBKRBFWt6OS9ngGux/nMbVTV/K5/WmOOj7XQTaC4BGgGJgIneR4TgPc86w4AvxWRGBGJFJG5nv2eBH4iIjPEMVpEhnvWfQF8X0RCROQ84MxuaojF6TevEJFBwL2tK1T1APAW8Jjny9MwETmj3b5/B6YDt+P0qRvjcxboJlAsAf6kqntVtaD1gfOl5FXAhcBoYC9OK3sxgKq+AvxfnO6ZapxgHeQ55u2e/SqAqz3rjuVBIAoowem3f7vD+muARmALUATc0bpCVWuBvwIjgL/18Gc3xiv2pagxfURE7gHGquoPut3YmONgfejG9AFPF811OK14Y3qFdbkY08tE5AacL03fUtV33a7HBC/rcjHGmCDhVQtdRM4TkW9FZLuI3N3J+uEislZEvvJMnJTZ2XGMMcb0nm5b6J6LJrYCC3FGD3wCXKWque22eQV4Q1WfEZH5wFJVPWZf4eDBgzUrK+sEyzfGmP7l008/LVHV5M7WefOl6Cxgu6ruBBCRl4CLgdx220wE7vQ8X0/3w7/Iyspi06ZNXry9McaYViKyp6t13nS5ZOB8odMqz7OsvS+ByzzPLwViRSSpk0JuFJFNIrKpuLjYi7c2xhjjLV+NcvkJcKaIfI5ztV0+zlV9R1DVFao6U1VnJid3+huDMcaY4+RNl0s+zsRDrTI9y9p4Jh+6DEBEBgKXdzGfhTHGmF7iTQv9E2CMiIwQkXDgSpypStuIyGARaT3Wz4GnfFumMcaY7nQb6J7Z624BVgKbgZdVNUdE7heR1luAzQO+FZGtQArO3BnGGGP6kGsXFs2cOVNtlIsxxvSMiHyqqjM7W2eX/htjTJCwybmMMaYX1TU2U1BZR35FLfkVteyvqGXB+BQmZ8b7/L0s0I0x5jipKuWHGtlfUUteuRPW+ytq2V9ZS355LfkVdZTU1B+13+CBERboxhjTlxqaWiiorCOv4hD7K+raArt9a7uuseWIfSLDBpCeEEVGQhTjU+PISIwiPSGK9IRIMhKiSI2PJCI0pFfqtUA3xvRLqkplbaMTzq2t69auEc/r4pp6Oo4bGTwwgoyESManxjJ/3BBPWDsBnpEYRWJ0GCLiys9kgW6MCUqNzS1tfdeHW9ZHvj7UcOQF7eGhA5xgTohi3rjkI8Pa07qODOud1rUvWKAbYwKOqlJV1+SEdLmnz7qilv0VdeSXO90jhdV1R7Wuk2LCyUiMYnTyQM4Yk9zWDdLaLZIUE+5a69oXLNCNMX6nqbmFwur6tq6PI1vZTnDX1DcdsU94yADSEyJJT4jitDGDPS3rSDISotuW+3Pr2hcs0I0xfa66rtFpTVccIr/9l42eAC+oqqOlQ+t6UEw46QmRZCXFMGfUYDLbvmx0vnAcHBPBgAEutK5bmqGpHprqjvyzub7z5U11kDkLksf6vBQLdGOMTzW3KEXVdZ5he7Wdjg6prjuydR0WIqTFO8F86qgkMhPah7WzPDq8k7hSheYGJyQPVXkCs2OIehGwne7n+bP1+F3t09LY85N0/u8s0I0x/qGlRdlTdoic/Ap27C+hqKyCkooqyiurqDpYQ2hLAxE0Ei5NRNDAoAhlXDScFgVDhiqDIpSECCUhrJm40BaiBzQi7QO3tAEKuwjR5g7LTpQMgNAoCI2A0EgIDff8GXH4z4hY58+QiKPXHfFnJ/uGRjqPkHbroo+6XYRPWKAbY5yWbkMN1JY7j0Nlbc8bD5ZRUVpIdXkx9dWl6KEywhsqiaOGhdRwgRzZ2iass+MDBz2Pjo4ZkhEQGdf5+pBjhGfrvl0Fb0i7dSHBE4PB85MYY5xgrq/qNJiprWj3vBxqy4583dLU6SHDgBiNoJEYGomlMSKehsTR1MQmIYnJJCYNITQi5hit1mO0XEPCIYBHlfgbC3Rj/FFLC9RXtgvm8g5h3FUwV4AedbOww8IHQlQiGpVAfWg8FZEjKArJJi80kp0Hw9lTG0GlxlChAwmNGURqahrDM9IZmzmE7PR4xiRGufPFo/GKBboxvamlGeoqu2gxdxPMHGNq64g4iEqAqETnEZ95+HnUoHbPE2mMiGfXwXBySgfwdWEduQcqyd1fRZXni8kBAiOTB5I9Oo6JaXFMTHf+TBoY0TfnyPiMBbox3mhugrqKngdzXeWxjxsZf0T4kpjVZTAffiRASGcd1c5wwC0F1eTuryJnRyW5B6rYWrCHhmZnvpHIsAGMT43jwqnpTEyPIzs9nnEpsUSFB/f47P7CAt30T6pOQFcXQvUBqCmE6gLnUVPYIZgrnH7pLsmRreXoJEga3X0wR8Yf9xdyqkpRdb0T3Pud4M7dX8Xu0kNt2yTFhDMxPY6lp2UxMc0J7xGDYwixLpOgZYFugouqE8LVBZ0EdcGRod3ZkLfwgTBwiBPKA1MgefwxgjnhcDAP6L0WbnOLsqvkILkHPOG9v4rNB6ooqWlo22Z4UjTZ6XF8b0Ym2enxTEyPY0hsREBfxm56zgLdBIaWFqfFfEQ4H3Ba2G1B7Xne3HD0/hFxTkDHpsLQWc6fA1OdP9uepzjjjV1U29DMt4XVbcGde6CKLQeqqW10vugMDxnA2NSBzB8/xGl1Z8QzPjWW2MjOu2BM/2KBbtzV0gKHSjoJ5/bB7Vne2bC6yHiITXPCevjsI8O5dXlsKoTH9P3P1o2ygw1HdZnsKK5pu+Q9LjKUielxXDVrmKe/O45RyQMJD7U7R5rOWaCb3tHSDAeLvQjqws6H2UUlHg7kwWM7aVF7gjosqu9/th5SVfaV1R4R3Dn7qyioOtzlk5EQxYS0OL4zOY1szyiTzMQo6zIxPWKBbnqmuQkOFh0O6q76qQ8Wg7YcvX900uGgHpLttKTbB3VrWIcG5pC5hqYWthVVk7PfCe7W/u5qz8yAIQOE0ckDmT0qyfNFZRwT0uJIjAl3uXITDCzQjaOpwQnmtnBuDeoOLeyDJRw9PlogJvlwOKdOadeKTjsc1DFDnCsGg0RVXWNbaDtfWFaxvaiaxmbn/ESHhzAhLY5LpmU4re70OMamxAb9FK7GPRbowU4VqvKhan/Xoz2qD8Ch0qP3lQFOCMemQFwGpE8/+kvE2DQnzLsYFx0MVJWCqjpy8qsOjzQ5UMW+stq2bZJjI5iYFse8ccltXSZZSTF2VaXpUxbowUoVtq2C934H+z46cp2EeFrPKZAwDDJP7hDUrS3q5F4djuePVJUdxTV80z6891dRfsiZIlUERiTFMCUzwfmy0nNl5ZDYSJcrN8YCPfi0NEPOa7DhASj8BuKHwsL/44ynbg3q6KR+F9TH0tTcwqY95azMKWBVTiH5FU7LOzx0AONTYzk3O7Wty2R8ahwxEfaxMf7J/mUGi6Z6+PLPsOFBKN/ljAy55HGYvCiou0OOV11jMxu2lbAyp4C1W4ooO9hAeOgAzhgzmFvnj2basERGJccQGmJDBE3gsEAPdPU18OnTsPFRpy88fRqc8zyMOx8GWBi1V1nbyPotRazMKeCfW4s51NBMbGQoC8YP4dzsVM4Ym2ytbxPQ7F9voDpUBh+vgI/+4FzqnnU6XPIYjDzL5pdup6CyjtW5BazKLWTjjlKaWpQhsRFcNj2Dc7NTOWVEkl2oY4KGBXqgqTrgtMY3/QkaD8K478Jpd8LQk92uzG/sKK5p6w//Yl8FACMHx3D96SM5NzuFqZkJNvrEBCUL9EBRthPefwi+eNG5BH7S9+C0ZZAy0e3KXKeqfJVXycqcAlbmFLCj2LnP2dTMeH567jjOzU5h9BB352gxpi9YoPu7gm+cESs5f4MBoTDtBzDnNhg0wu3KXNXY3MJHO8tYleu0xAuq6ggZIJw6chBL5mSxcGIKafH+Py2AMb5kge6v9n3sjCHf+rYzpevsW2D2j51hh/3UoYYm3t1azKqcQtZuKaKytpHIsAGcOTaZu7LHMX/8EBKig+dKVGN6ygLdn6jCjnXw3u9hzwZn/u2z/g1Ovh6iB7ldnSvKDzawZnMhK3MKeW9bMfVNLSREh3H2hBTOzU7h9DHJdrcdYzws0P1BSwts+YcT5Ae+gNh0OPf/wYwlfjnta2/Lr6hllac//JPd5TS3KOnxkVw1axjnZKcwK2uQjQ83phMW6G5qboSvXnb6yEu3waCRcNEjMGVxwM42eDxUla2FNU6I5xbwTb5zu7exKQP50bxRnDMxlUkZcTaVrDHd8CrQReQ84CEgBHhSVX/bYf0w4BkgwbPN3ar6po9rDR4Nh+Dz5+D9h6EqD1Imw/eegomX9JtL8ltalM/3lbMqp5CVOQVt98KcPiyBn39nPOdkpzJicP/77cSYE9FtoItICLAcWAjkAZ+IyOuqmttus18CL6vq4yIyEXgTyOqFegNbbQV88iR8+Lhzl55hs+HCB2H02f3iYqCGphY+2FHCypxCVucWUlJTT1iIMHvUYG44YyQLJ6QwJM4muTLmeHnTQp8FbFfVnQAi8hJwMdA+0BWI8zyPB/b7ssiAV1MEHz4Gn/yXc/f40Qvh9Dth+By3K+t1NfVNvPNtEStzCnlnSxHV9U3EhIcwb9wQzslO4azxQ4iz+2Ea4xPeBHoGsK/d6zzglA7b3AesEpFbgRjg7M4OJCI3AjcCDBs2rKe1Bp6KvU63yufPOZNnZV/iXAyUNtXtynpVcXU9azc7XSnvby+lobmFpJhwzp+SxjnZKcwZNdhu8mBML/DVl6JXAU+r6u9EZDbwnIhMUj3yHmSqugJYATBz5syOt70JHkVb4P0H4etXAIGpV8LcO2DwaLcr6zV7Sw+xKtcZmbJpTzmqMHRQFD+cPZxzslOZMTyRELvc3phe5U2g5wND273O9Cxr7zrgPABV3SgikcBgoMgXRQaM/E+doYdb3oCwaJh1o3NBUHyG25X5nKqSe6CKlTmFrMopYEtBNQAT0uK4fcEYzpmYyoS0WBuZYkwf8ibQPwHGiMgInCC/Evh+h232AguAp0VkAhAJFPuyUL+lCrvfc67q3PkORMbDGXfBKTdBTJLb1flUc4uyaXeZE+K5BeSV1yICJw8fxC/Pn8A5E1MZlhTtdpnG9FvdBrqqNonILcBKnCGJT6lqjojcD2xS1deBfwWeEJFlOF+QXquqwdulAs7FQFvfdoI8f5Nz782F98OMpRAZ1/3+AaKusZn3tzs3gliz+fCNIE4b7dwIYsGEFAYP7D9j5o3xZ171oXvGlL/ZYdk97Z7nAnN9W5qfam5yJsp67/dQvNm5J+f5v4eTroaw4BhyV1nb6BmZUsA733puBBERyvwJQzhnYipnjktmoN0Iwhi/Y59KbzXWwRcvOFPYVuyB5Alw2ROQfRmEBP5pLKyqY3WuMzLlw52lNDYrybERXDotg3OyU5k90m4EYYy/C/wk6m311bDpKdi4HGoKIWMmnPf/YOx3Av4WbzuLa9r6wz/f69wIYsTgGP7ltBGcm53KSXYjCGMCigV6Vw6WOrd3+/iPUFcJI+fB5U86t3oL0JEbqsrX+ZVtd/PZVlQDwOSMeH5yzljOzU5l9JCBNjLFmABlgd5RZb5zi7dPn4bGQzD+AueqzowZbld2XFSVD3eW8fY3B1iVW8iBSudGELOyBnH1KcNYmJ1KRoLdCMKYYGCB3qp0hzPr4ZcvgbbAlCuci4GGjHe7shPy/Ed7+dXfvyEidABnjE3mX88Zx4LxQ0iMsRtBGBNsLNAPfAUbfg+5/w0h4TDjWphzKyQOd7uyE1bX2MzDa7cxK2sQT//LyUSH21+3McGs/37C93zgDD3cvhrCY2Hu7XDqj2DgELcr85kXPtpLcXU9j1w1zcLcmH6gf33KVWH7GudioL0bIToJ5v/KucVbVILb1flUbUMzj7+zgzmjkjh1ZHBdsWqM6Vz/CPSWZqdLZcPvoeBriMuE7/w7TLsGwoPzUvXnP9xDSU09j/9gutulGGP6SHAHelMDfPUSbHgQynZA0hi4+DGYvAhCg/dLwYP1Tfzhnzs4fcxgTs7qnzeXNqY/Cs5AbzgInz4DHzwC1fud+ceveNYZgtgPbvH27MY9lB5s4I6zx7pdijGmDwVXoNeWw8dPOLd4qy2D4afBxY/CqPkBezFQT9XUN7Hi3R2cOTaZGcMT3S7HGNOHgiPQqwucS/M3PQUNNTD2PDjtThjW8cZKwe+ZD3ZTfqiRZQutdW5MfxPYgV62Cz54GD5/AVoanYmyTlsGqZPcrswV1XWNrHh3J/PHD+GkocE1ascY073ADPTCXOeqzm/+6vSJn/R9mHMbJI1yuzJX/en93VTWNrLM+s6N6ZcCL9DffwhW3wNhMXDqzc4t3uLS3K7KdZW1jTz53k7OnpDC5Mx4t8sxxrgg8AJ95Fkw7+fO/TqjbUheq6c27KKqrok7zh7jdinGGJcEXqCnTXEepk3loUae2rCLc7NTmJRhrXNj+qvAvkODAeDJDTuprm+ycefG9HMW6AGu/GADT23YxfmT05iQFjw3pzbG9JwFeoB74r2dHGps5nbrOzem37NAD2ClNfU8/cFuLpiSztiUWLfLMca4zAI9gK14bye1jc3cvmC026UYY/yABXqAKqmp59kP9nDx1HRGD7HWuTHGAj1g/fGfO6hvaua2BdZ3boxxWKAHoKLqOp77cA+XTMtgZPJAt8sxxvgJC/QA9Id3dtLYrNw231rnxpjDLNADTGFVHc9/tIfLpmWQNTjG7XKMMX7EAj3APLZ+Oy0tyq3WOjfGdGCBHkAOVNby54/38b0ZmQxLCs6bWxtjjp8FegBZvn47ivLjs2zcuTHmaBboASK/opa/fLKPRTOHMnSQtc6NMUezQA8Qj67bjiDWOjfGdMkCPQDsKzvEK5v2sfjkoWQkRLldjjHGT1mgB4BH121nwADhR2f173umGmOOzQLdz+0pPcirn+Xx/VnDSIu31rkxpmteBbqInCci34rIdhG5u5P1D4jIF57HVhGp8H2p/dMj67YTOkC4eZ61zo0xx9btPUVFJARYDiwE8oBPROR1Vc1t3UZVl7Xb/lZgWi/U2u/sKjnI3z7LY+ncEaTERbpdjjHGz3nTQp8FbFfVnaraALwEXHyM7a8C/uyL4vq7R9ZuIzx0ADedaa1zY0z3vAn0DGBfu9d5nmVHEZHhwAhgXRfrbxSRTSKyqbi4uKe19is7imv4+xf5/HB2FsmxEW6XY4wJAL7+UvRK4FVVbe5spaquUNWZqjozOTnZx28dXB5eu42I0BBuPGOk26UYYwKEN4GeDwxt9zrTs6wzV2LdLSdsW2E1r3+5nyVzshg80FrnxhjveBPonwBjRGSEiITjhPbrHTcSkfFAIrDRtyX2Pw+t3UZ0mLXOjTE9022gq2oTcAuwEtgMvKyqOSJyv4hc1G7TK4GXVFV7p9T+YUtBFf/z9QGunZvFoJhwt8sxxgSQboctAqjqm8CbHZbd0+H1fb4rq/96aM02YsJDueF0a50bY3rGrhT1I7n7q3jrmwL+ZW4WCdHWOjfG9IwFuh95cM1WYiNDue40a50bY3rOAt1PfJNfyarcQq47bQTx0WFul2OMCUAW6H7iwTVbiYsM5V9OG+F2KcaYAGWB7ge+yqtgzeYibjh9JHGR1jo3xhwfC3Q/8MDqrSREh3Ht3Cy3SzHGBDALdJd9vrec9d8Wc8PpI4m11rkx5gRYoLvsgTXbGBQTzpI5WW6XYowJcBboLvp0Txnvbi3mxjNGMjDCq2u8jDGmSxboLnpg9TaSYsL54ezhbpdijAkCFugu+XhXGRu2l3DzvFFEh1vr3Bhz4izQXfLA6q0MHhjB1adY69wY4xsW6C7YuKOUjTtL+dG8UUSFh7hdjjEmSFig9zFV5YE1WxkSG8H3TxnmdjnGmCBigd7HPthRyse7yvjxWaOJDLPWuTHGdyzQ+5Cq8sDqraTGRbL45KHd72CMMT1ggd6H3ttWwqY95fx4vrXOjTG+Z4HeR1SV36/eSnp8JFfMzHS7HGNMELJA7yPvbC3mi30V3DJ/DBGh1jo3xvieBXofaO07z0yM4nszrHVujOkdFuh9YN2WIr7Kq+TW+aMJD7VTbozpHZYuvax13PmwQdFcNt1a58aY3mOB3stW5xbyTX4Vt84fTViInW5jTO+xhOlFLS3KA2u2kZUUzaXTMtwuxxgT5CzQe9HKnAI2H6jitgVjCLXWuTGml1nK9JKWFuXBNdsYmRzDRVPT3S7HGNMPWKD3kje/OcC3hdXcbq1zY0wfsaTpBc0tykNrtjF6yEAumGKtc2NM37BA7wVvfLWfbUU13HH2GEIGiNvlGGP6CQt0H2tuUR5au41xKbF8d1Ka2+UYY/oRC3Qfe/3LfHYWH+SOs8cwwFrnxpg+ZIHuQ03NLTy8djvjU2M5NzvV7XKMMf2MBboP/f2L/ewqOciyhWOtdW6M6XMW6D7S2NzCw2u3kZ0exzkTU9wuxxjTD1mg+8hrn+Wzt+wQy84ei4i1zo0xfc8C3Qcamlp4eN02pmTGs2DCELfLMcb0U14FuoicJyLfish2Ebm7i22uEJFcEckRkRd9W6Z/++tneeSV11rr3BjjqtDuNhCREGA5sBDIAz4RkddVNbfdNmOAnwNzVbVcRPpNM7WhqYVH123npKEJzBuX7HY5xph+zJsW+ixgu6ruVNUG4CXg4g7b3AAsV9VyAFUt8m2Z/uvlTfvIr6hl2UJrnRtj3OVNoGcA+9q9zvMsa28sMFZE3heRD0XkvM4OJCI3isgmEdlUXFx8fBX7kfqmZpav386M4YmcMWaw2+UYY/o5X30pGgqMAeYBVwFPiEhCx41UdYWqzlTVmcnJgd898ZdP9nGgss76zo0xfsGbQM8HhrZ7nelZ1l4e8LqqNqrqLmArTsAHrbpGp3U+K2sQc0cnuV2OMcZ4FeifAGNEZISIhANXAq932ObvOK1zRGQwThfMTh/W6Xde/GgvhVX13LFwjLXOjTF+odtAV9Um4BZgJbAZeFlVc0TkfhG5yLPZSqBURHKB9cBPVbW0t4p2W21DM4//cwenjhzEnFHWd26M8Q/dDlsEUNU3gTc7LLun3XMF7vQ8gt4LH+2huLqeR6+a5nYpxhjTxq4U7aFDDU384Z87mDs6iVNGWt+5McZ/eNVCN4c9t3EPJTUN/OHssW6XYowxR7AWeg8crG/ij+/u5PQxg5mZNcjtcowx5ggW6D3wzMbdlB1sYNlCa50bY/yPBbqXqusaWfHuTuaNS2b6sES3yzHGmKNYoHvpmQ92U3GokWXWd26M8VMW6F6o8rTOF4wfwtShR81oYIwxfsEC3Qt/2rCbqrom6zs3xvg1C/RuVNY28uSGnSycmMKkjHi3yzHGmC5ZoHfjvzbsorquiTvODuq5xowxQcAC/RgqDjXw1IZdfGdSKtnp1jo3xvg3C/RjePK9XdTUN3G7tc6NMQHAAr0LZQcb+NP7uzh/ShrjU+PcLscYY7plgd6FFe/u5FBjM3cssNa5MSYwWKB3oqSmnmc37ubCKemMSYl1uxxjjPGKBXonVry7k7rGZm6z1rkxJoBYoHdQXO20zi8+KYPRQwa6XY4xxnjNAr2DP/xzBw1NLdw6f7TbpRhjTI9YoLdTVFXH8x/u4dJpmYxMtta5MSawWKC389g7O2hqUW5bYK1zY0zgsUD3KKis48WP93L59AyGJ8W4XY4xxvSYBbrHY+9sp6VFuXW+jWwxxgQmC3Rgf0UtL328j0UzMxk6KNrtcowx5rhYoAPL129HUX58lvWdG2MCV78P9LzyQ7y8aR9XzBxKZqK1zo0xgavfB/ry9dsRxFrnxpiA168DfV/ZIV7ZlMeVs4aSnhDldjnGGHNC+nWgP7JuGwMGCD+aZ61zY0zg67eBvrvkIH/9LJ+rTxlGanyk2+UYY8wJ67eB/si67YQOEG4+c5TbpRhjjE/0y0DfWVzDa5/ncc2pwxkSZ61zY0xw6JeB/vDabYSHDuB/WevcGBNE+l2gby+q5vUv97NkdhbJsRFul2OMMT7T7wL9obXbiQwL4cYzRrpdijHG+FS/CvSthdW88dV+lszJImmgtc6NMcEl1O0C+tJDa7YRHRbCjadb69wEnsbGRvLy8qirq3O7FNMHIiMjyczMJCwszOt9vAp0ETkPeAgIAZ5U1d92WH8t8B9AvqUYC84AABJ/SURBVGfRo6r6pNdV9IEtBVX8z9cHuOWs0STGhLtdjjE9lpeXR2xsLFlZWYiI2+WYXqSqlJaWkpeXx4gRI7zer9suFxEJAZYD3wEmAleJyMRONv2Lqp7kefhVmAM8uHobsRGhXH+69yfHGH9SV1dHUlKShXk/ICIkJSX1+Lcxb/rQZwHbVXWnqjYALwEXH0eNrsnZX8nbOQUsPW0ECdHWOjeBy8K8/ziev2tvAj0D2NfudZ5nWUeXi8hXIvKqiAztosAbRWSTiGwqLi7ucbHH68E124iNDOW606x1bowJXr4a5fIPIEtVpwCrgWc620hVV6jqTFWdmZyc7KO3Prav8ypZnVvI9aeNJD7K+y8XjDEm0HgT6PlA+xZ3Joe//ARAVUtVtd7z8klghm/KO3EPrtlKfFQYS0/LcrsUYwJaRUUFjz32WI/3++53v0tFRcUxt7nnnntYs2bN8ZZmPLwZ5fIJMEZERuAE+ZXA99tvICJpqnrA8/IiYLNPqzxOX+yrYO2WIn5yzljiIq11boLH//5HDrn7q3x6zInpcdx7YXaX61sD/Uc/+tERy5uamggN7TpK3nzzzW7f+/777/e+UD/T3c/fl7ptoatqE3ALsBInqF9W1RwRuV9ELvJsdpuI5IjIl8BtwLW9VXBPPLhmKwnRYVw71/rOjTlRd999Nzt27OCkk07i5JNP5vTTT+eiiy5i4kRn0Nsll1zCjBkzyM7OZsWKFW37ZWVlUVJSwu7du5kwYQI33HAD2dnZnHPOOdTW1gJw7bXX8uqrr7Ztf++99zJ9+nQmT57Mli1bACguLmbhwoVkZ2dz/fXXM3z4cEpKSrqst6t63n77baZPn87UqVNZsGABADU1NSxdupTJkyczZcoU/vrXvwIwcODAtv1effVVrr322rZ6b7rpJk455RTuuusuPv74Y2bPns20adOYM2cO3377LQDNzc385Cc/YdKkSUyZMoVHHnmEdevWcckll7Qdd/Xq1Vx66aXH95fSkaq68pgxY4b2pk27y3T4z97Q5eu39er7GNNXcnNzXX3/Xbt2aXZ2tqqqrl+/XqOjo3Xnzp1t60tLS1VV9dChQ5qdna0lJSWqqjp8+HAtLi7WXbt2aUhIiH7++eeqqrpo0SJ97rnnVFV1yZIl+sorr7Rt//DDD6uq6vLly/W6665TVdUf//jH+pvf/EZVVd966y0FtLi4uMt6O6unqKhIMzMz2+pu3eauu+7S22+/vW3fsrIyVVWNiYlpW/bKK6/okiVL2uo9//zztampSVVVKysrtbGxUVVVV69erZdddpmqqj722GN6+eWXt60rLS3VlpYWHTdunBYVFamq6lVXXaWvv/56pz9DZ3/nwCbtIlf94/eEXvDgmq0Miglnyewst0sxJijNmjXriIteHn74YV577TUA9u3bx7Zt20hKSjpinxEjRnDSSScBMGPGDHbv3t3psS+77LK2bf72t78BsGHDhrbjn3feeSQmJh6zvs7qKS4u5owzzmire9CgQQCsWbOGl156qW3f7o4NsGjRIkJCQgCorKxkyZIlbNu2DRGhsbGx7bg33XRTW5dM6/tdc801PP/88yxdupSNGzfy7LPPdvt+3gjKQP9kdxnvbSvhF98dT0xEUP6IxrguJiam7fk777zDmjVr2LhxI9HR0cybN6/Ti2IiIg7PoRQSEtLW5dLVdiEhITQ1NfW4Nm/r6U77seAd92//8//qV7/irLPO4rXXXmP37t3MmzfvmMddunQpF154IZGRkSxatMhnffBBOTnXA6u3MnhgOD84dbjbpRgTNGJjY6muru50XWVlJYmJiURHR7NlyxY+/PBDn7//3LlzefnllwFYtWoV5eXlXW7bVT2nnnoq7777Lrt27QKgrKwMgIULF7J8+fK2/VuPnZKSwubNm2lpaWlr7Xf1fhkZzuU5Tz/9dNvyhQsX8sc//rHtP6XW90tPTyc9PZ1f//rXLF26tEfn4ViCLtA/3FnKBztKuenMUUSHW+vcGF9JSkpi7ty5TJo0iZ/+9KdHrDvvvPNoampiwoQJ3H333Zx66qk+f/97772XVatWMWnSJF555RVSU1OJjY3tdNuu6klOTmbFihVcdtllTJ06lcWLFwPwy1/+kvLyciZNmsTUqVNZv349AL/97W+54IILmDNnDmlpaV3Wdtddd/Hzn/+cadOmHfEbxfXXX8+wYcOYMmUKU6dO5cUXX2xbd/XVVzN06FAmTJhwwuemlTh97H1v5syZumnTJp8fd/EfN7Kz5CDv3XUWkWEhPj++MW7ZvHmzTz/8gaa+vp6QkBBCQ0PZuHEjN998M1988YXbZR23W265hWnTpnHdddd1uU1nf+ci8qmqzuxs+6Bqwn6wo4SPdpVx74UTLcyNCTJ79+7liiuuoKWlhfDwcJ544gm3SzpuM2bMICYmht/97nc+PW7QBLqq8sDqraTERXDVrGFul2OM8bExY8bw+eefH7GstLS0bSx5e2vXrj1qhI0/+fTTT3vluEET6O9vL+WT3eXcf3G2tc6N6SeSkpICutvF14LiS1FV5fervyUtPpLFJ3c60aMxxgS9oAj0d7eV8NneCn581mgiQq11bozpnwI+0J3W+VYyEqK4Yqa1zo0x/VfAB/r6b4v4cl8Ft8wfTXhowP84xhhz3AI6AZ2RLdsYOiiK783IdLscY0w77WcqNH0joEe5rNlcxNf5lfz75VMICwno/5uM6Zm37oaCr317zNTJ8J3f+vaYfsCf5ivvbQGbgq3jzocnRXPp9M5ucWqM8aW77777iPlO7rvvPn7961+zYMGCtrnL//u//9urY9XU1HS537PPPtt2qfw111wDQGFhIZdeeilTp05l6tSpfPDBB+zevZtJkya17fef//mf3HfffQDMmzePO+64g5kzZ/LQQw/xj3/8g1NOOYVp06Zx9tlnU1hY2FZHx3nQn3rqKe6444624z7xxBMsW7bsuM9bn+pqXt3efpzofOhvfX1Ah//sDX1l074TOo4xgcLt+dA/++wzPeOMM9peT5gwQffu3auVlZWqqlpcXKyjRo3SlpYWVT1yLvGOGhsbO93vm2++0TFjxrTNc946X/kVV1yhDzzwgKqqNjU1aUVFxRHzs6uq/sd//Ifee++9qqp65pln6s0339y2rqysrK2uJ554Qu+8805V7Xwe9Orqah05cqQ2NDSoqurs2bP1q6++6unp8ol+MR96S4vy4JqtjBgcwyUnpbtdjjH9wrRp0ygqKmL//v0UFxeTmJhIamoqy5Yt491332XAgAHk5+dTWFhIamrqMY+lqvziF784ar9169axaNEiBg8eDByeP3zdunVtc4aHhIQQHx9/zNkWgbaJtwDy8vJYvHgxBw4coKGhoW0+9K7mQZ8/fz5vvPEGEyZMoLGxkcmTJ/fwbLkjIAP97ZwCthRU88DiqYRa37kxfWbRokW8+uqrFBQUsHjxYl544QWKi4v59NNPCQsLIysry6t5x493v/ZCQ0NpaWlpe32s+cpvvfVW7rzzTi666CLeeeedtq6Zrlx//fX85je/Yfz48T6d3ra3BVwatrbORybHcNFU6zs3pi8tXryYl156iVdffZVFixZRWVnJkCFDCAsLY/369ezZs8er43S13/z583nllVcoLS0FDs8fvmDBAh5//HHAuU9nZWUlKSkpFBUVUVpaSn19PW+88cYx3691vvJnnnmmbXlX86Cfcsop7Nu3jxdffJGrrrrK29PjuoAL9P/5+gBbC2u44+yxhAyQ7ncwxvhMdnY21dXVZGRkkJaWxtVXX82mTZuYPHkyzz77LOPHj/fqOF3tl52dzb/9279x5plnMnXqVO68804AHnroIdavX8/kyZOZMWMGubm5hIWFcc899zBr1iwWLlx4zPe+7777WLRoETNmzGjrzoGu50EHuOKKK5g7d65Xt6PzFwE3H/q6LYX8+eN9/OEHMyzQTb/S3+dD72sXXHABy5Yt63Q2x77S0/nQA66FPn98Ck/8cKaFuTGmV1RUVDB27FiioqJcDfPjEZBfihpjAsPXX3/dNpa8VUREBB999JFLFXUvISGBrVu3ul3GcbFANyaAqOoRd6L3d5MnT7b5yo/T8XSHB1yXizH9VWRkJKWlpcf1QTeBRVUpLS0lMjKyR/tZC92YAJGZmUleXh7FxcVul2L6QGRkJJmZPZt00ALdmAARFhbWdoWjMZ2xLhdjjAkSFujGGBMkLNCNMSZIuHalqIgUA95N/HC0wUCJD8vxFaurZ6yunvPX2qyunjmRuoaranJnK1wL9BMhIpu6uvTVTVZXz1hdPeevtVldPdNbdVmXizHGBAkLdGOMCRKBGugr3C6gC1ZXz1hdPeevtVldPdMrdQVkH7oxxpijBWoL3RhjTAcW6MYYEyT8OtBF5DwR+VZEtovI3Z2sjxCRv3jWfyQiWX5S17UiUiwiX3ge1/dRXU+JSJGIfNPFehGRhz11fyUi0/2krnkiUtnufN3TBzUNFZH1IpIrIjkicnsn2/T5+fKyLjfOV6SIfCwiX3rq+t+dbNPnn0cv63Ll8+h57xAR+VxEjrrhaa+cL1X1ywcQAuwARgLhwJfAxA7b/Aj4g+f5lcBf/KSua4FHXThnZwDTgW+6WP9d4C1AgFOBj/ykrnnAG318rtKA6Z7nscDWTv4e+/x8eVmXG+dLgIGe52HAR8CpHbZx4/PoTV2ufB49730n8GJnf1+9cb78uYU+C9iuqjtVtQF4Cbi4wzYXA6238H4VWCC9P/u/N3W5QlXfBcqOscnFwLPq+BBIEJE0P6irz6nqAVX9zPO8GtgMZHTYrM/Pl5d19TnPOajxvAzzPDqOqOjzz6OXdblCRDKB84Enu9jE5+fLnwM9A9jX7nUeR//DbttGVZuASiDJD+oCuNzza/qrIjK0l2vylre1u2G259fmt0Qkuy/f2POr7jSc1l17rp6vY9QFLpwvT/fBF0ARsFpVuzxfffh59KYucOfz+CBwF9DSxXqfny9/DvRA9g8gS1WnAKs5/L+w6dxnOPNTTAUeAf7eV28sIgOBvwJ3qGpVX71vd7qpy5XzparNqnoSkAnMEpFJffG+3fGirj7/PIrIBUCRqn7a2+/Vnj8Hej7Q/n/STM+yTrcRkVAgHih1uy5VLVXVes/LJ4EZvVyTt7w5p31OVataf21W1TeBMBEZ3NvvKyJhOKH5gqr+rZNNXDlf3dXl1vlq9/4VwHrgvA6r3Pg8dluXS5/HucBFIrIbp1t2vog832Ebn58vfw70T4AxIjJCRMJxvjR4vcM2rwNLPM+/B6xTzzcMbtbVoZ/1Ipx+UH/wOvBDz+iNU4FKVT3gdlEiktradygis3D+XfZqEHje77+Azar6+y426/Pz5U1dLp2vZBFJ8DyPAhYCWzps1uefR2/qcuPzqKo/V9VMVc3CyYh1qvqDDpv5/Hz57S3oVLVJRG4BVuKMLHlKVXNE5H5gk6q+jvMP/zkR2Y7zpduVflLXbSJyEdDkqeva3q4LQET+jDMCYrCI5AH34nxJhKr+AXgTZ+TGduAQsNRP6voecLOINAG1wJV98B/zXOAa4GtP/yvAL4Bh7epy43x5U5cb5ysNeEZEQnD+A3lZVd9w+/PoZV2ufB4709vnyy79N8aYIOHPXS7GGGN6wALdGGOChAW6McYECQt0Y4wJEhboxhgTJCzQTdARkeZ2M+t9IZ3MiHkCx86SLmaNNMZtfjsO3ZgTUOu5FNyYfsVa6KbfEJHdIvLvIvK1Zw7t0Z7lWSKyzjN501oRGeZZniIir3kmwfpSROZ4DhUiIk+IM//2Ks8ViojIbeLMY/6ViLzk0o9p+jELdBOMojp0uSxut65SVScDj+LMhgfOBFfPeCZvegF42LP8YeCfnkmwpgM5nuVjgOWqmg1UAJd7lt8NTPMc56be+uGM6YpdKWqCjojUqOrATpbvBuar6k7PBFgFqpokIiVAmqo2epYfUNXBIlIMZLab2Kl1StvVqjrG8/pnQJiq/lpE3gZqcGY//Hu7ebqN6RPWQjf9jXbxvCfq2z1v5vB3UecDy3Fa8594ZtAzps9YoJv+ZnG7Pzd6nn/A4YmRrgbe8zxfC9wMbTdRiO/qoCIyABiqquuBn+FMhXrUbwnG9CZrQZhgFNVupkKAt1W1dehiooh8hdPKvsqz7FbgTyLyU6CYw7Mq3g6sEJHrcFriNwNdTZ8bAjzvCX0BHvbMz21Mn7E+dNNvePrQZ6pqidu1GNMbrMvFGGOChLXQjTEmSFgL3RhjgoQFujHGBAkLdGOMCRIW6MYYEyQs0I0xJkj8f8wivyXAurZaAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pDi4i0H16tSa"
      },
      "source": [
        "From the look of the EfficientNetB0 model's loss curves, it looks like if we kept training our model for longer, it might improve even further. Perhaps that's something you might want to try?\n",
        "\n",
        "Let's check out the model summary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "clJzUbKpODXA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3538504b-1cc9-4569-db9e-b191ea37d20f"
      },
      "source": [
        "efficientnet_model.summary()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "feature_extraction_layer (Ke (None, 1280)              4049564   \n",
            "_________________________________________________________________\n",
            "output_layer (Dense)         (None, 10)                12810     \n",
            "=================================================================\n",
            "Total params: 4,062,374\n",
            "Trainable params: 12,810\n",
            "Non-trainable params: 4,049,564\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KHTMjJG07ElO"
      },
      "source": [
        "It seems despite having over four times less parameters (4,049,564 vs. 23,564,800) than the ResNet50V2 extraction layer, the  EfficientNetB0 feature extraction layer yields better performance. Now it's clear where the \"efficient\" name came from."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YV_ZWKC8SkE_"
      },
      "source": [
        "## Comparing models using TensorBoard\n",
        "\n",
        "Alright, even though we've already compared the performance of our two models by looking at the accuracy scores. But what if you had more than two models? \n",
        "\n",
        "That's where an experiment tracking tool like [TensorBoard](https://www.tensorflow.org/tensorboard) (preinstalled in Google Colab) comes in.\n",
        "\n",
        "The good thing is, since we set up a TensorBoard callback, all of our model's training logs have been saved automatically. To visualize them, we can upload the results to [TensorBoard.dev](https://tensorboard.dev/).\n",
        "\n",
        "Uploading your results to TensorBoard.dev enables you to track and share multiple different modelling experiments. So if you needed to show someone your results, you could send them a link to your TensorBoard.dev as well as the accompanying Colab notebook.\n",
        "\n",
        "> 🔑 **Note:** These experiments are public, do not upload sensitive data. You can delete experiments if needed.\n",
        "\n",
        "### Uploading experiments to TensorBoard\n",
        "\n",
        "To upload a series of TensorFlow logs to TensorBoard, we can use the following command:\n",
        "\n",
        "```\n",
        "Upload TensorBoard dev records\n",
        "\n",
        "!tensorboard dev upload --logdir ./tensorflow_hub/ \\\n",
        "  --name \"EfficientNetB0 vs. ResNet50V2\" \\ \n",
        "  --description \"Comparing two different TF Hub feature extraction models architectures using 10% of training images\" \\ \n",
        "  --one_shot\n",
        "```\n",
        "\n",
        "Where:\n",
        "* `--logdir` is the target upload directory\n",
        "* `--name` is the name of the experiment\n",
        "* `--description` is a brief description of the experiment\n",
        "* `--one_shot` exits the TensorBoard uploader once uploading is finished\n",
        "\n",
        "Running the `tensorboard dev upload` command will first ask you to authorize the upload to TensorBoard.dev. After you've authorized the upload, your log files will be uploaded."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tbKgWdIVNncW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 258
        },
        "outputId": "e344cf7c-7064-484d-f31b-4079735dc921"
      },
      "source": [
        "# Upload TensorBoard dev records\n",
        "!tensorboard dev upload --logdir ./tensorflow_hub/ \\\n",
        "  --name \"EfficientNetB0 vs. ResNet50V2\" \\\n",
        "  --description \"Comparing two different TF Hub feature extraction models architectures using 10% of training images\" \\\n",
        "  --one_shot"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-09-14 05:02:46.516878: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
            "Data for the \"graphs\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Data for the \"histograms\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Data for the \"hparams\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Upload started and will continue reading any new data as it's added\n",
            "to the logdir. To stop uploading, press Ctrl-C.\n",
            "\n",
            "View your TensorBoard live at: https://tensorboard.dev/experiment/73taSKxXQeGPQsNBcVvY3g/\n",
            "\n",
            "\u001b[1m[2020-09-14T05:02:48]\u001b[0m Uploader started.\n",
            "\u001b[1m[2020-09-14T05:02:50]\u001b[0m Total uploaded: 40 scalars, 0 tensors, 2 binary objects (3.2 MB)\n",
            "\u001b[2K\u001b[33mListening for new data in logdir...\u001b[0m\n",
            "Done. View your TensorBoard at https://tensorboard.dev/experiment/73taSKxXQeGPQsNBcVvY3g/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FlVfmBdBOPvf"
      },
      "source": [
        "Every time you upload something to TensorBoad.dev you'll get a new experiment ID. The experiment ID will look something like this: https://tensorboard.dev/experiment/73taSKxXQeGPQsNBcVvY3g/ (this is the actual experiment from this notebook).\n",
        "\n",
        "If you upload the same directory again, you'll get a new experiment ID to go along with it.\n",
        "\n",
        "This means to track your experiments, you may want to look into how you name your uploads. That way when you find them on TensorBoard.dev you can tell what happened during each experiment (e.g. \"efficientnet0_10_percent_data\").\n",
        "\n",
        "### Listing experiments you've saved to TensorBoard\n",
        "\n",
        "To see all of the experiments you've uploaded you can use the command:\n",
        "\n",
        "```tensorboard dev list```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sDamroaMOFJx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 496
        },
        "outputId": "d086e0da-e091-4504-87d6-a50aa4636577"
      },
      "source": [
        "# Check out experiments\n",
        "!tensorboard dev list"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-09-14 05:04:21.965097: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
            "Data for the \"graphs\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Data for the \"histograms\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Data for the \"hparams\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "https://tensorboard.dev/experiment/73taSKxXQeGPQsNBcVvY3g/\n",
            "\tName                 EfficientNetB0 vs. ResNet50V2\n",
            "\tDescription          Comparing two different TF Hub feature extraction models architectures using 10% of training images\n",
            "\tId                   73taSKxXQeGPQsNBcVvY3g\n",
            "\tCreated              2020-09-14 05:02:48 (1 minute ago)\n",
            "\tUpdated              2020-09-14 05:02:50 (1 minute ago)\n",
            "\tRuns                 4\n",
            "\tTags                 3\n",
            "\tScalars              40\n",
            "\tTensor bytes         0\n",
            "\tBinary object bytes  3402042\n",
            "https://tensorboard.dev/experiment/n6kd8XZ3Rdy1jSgSLH5WjA/\n",
            "\tName                 EfficientNetB0 vs. ResNet50V2\n",
            "\tDescription          Comparing two different TF Hub feature extraction models architectures using 10% of training images\n",
            "\tId                   n6kd8XZ3Rdy1jSgSLH5WjA\n",
            "\tCreated              2020-09-14 05:01:17 (3 minutes ago)\n",
            "\tUpdated              2020-09-14 05:01:23 (3 minutes ago)\n",
            "\tRuns                 10\n",
            "\tTags                 3\n",
            "\tScalars              100\n",
            "\tTensor bytes         0\n",
            "\tBinary object bytes  7619131\n",
            "Total: 2 experiment(s)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mdLUjm-xADQ4"
      },
      "source": [
        "### Deleting experiments from TensorBoard\n",
        "\n",
        "Remember, all uploads to TensorBoard.dev are public, so to delete an experiment you can use the command:\n",
        "\n",
        "`tensorboard dev delete --experiment_id [INSERT_EXPERIMENT_ID]`\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qj69wuAlT-xS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        },
        "outputId": "fcc4a104-85eb-4f66-aa43-8bc25b75a51a"
      },
      "source": [
        "# Delete an experiment\n",
        "!tensorboard dev delete --experiment_id n6kd8XZ3Rdy1jSgSLH5WjA"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-09-14 05:06:06.959717: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
            "Data for the \"graphs\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Data for the \"histograms\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Data for the \"hparams\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Deleted experiment n6kd8XZ3Rdy1jSgSLH5WjA.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oov6qKvbU1lL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 309
        },
        "outputId": "9d2997d3-7278-4290-e8b8-f4fb2c9aab76"
      },
      "source": [
        "# Check to see if experiments still exist\n",
        "!tensorboard dev list"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-09-14 05:06:11.214919: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
            "Data for the \"graphs\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Data for the \"histograms\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "Data for the \"hparams\" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the \"--plugins\" command line argument.\n",
            "https://tensorboard.dev/experiment/73taSKxXQeGPQsNBcVvY3g/\n",
            "\tName                 EfficientNetB0 vs. ResNet50V2\n",
            "\tDescription          Comparing two different TF Hub feature extraction models architectures using 10% of training images\n",
            "\tId                   73taSKxXQeGPQsNBcVvY3g\n",
            "\tCreated              2020-09-14 05:02:48 (3 minutes ago)\n",
            "\tUpdated              2020-09-14 05:02:50 (3 minutes ago)\n",
            "\tRuns                 4\n",
            "\tTags                 3\n",
            "\tScalars              40\n",
            "\tTensor bytes         0\n",
            "\tBinary object bytes  3402042\n",
            "Total: 1 experiment(s)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KLvUjrL2Y1Ed"
      },
      "source": [
        "## 🛠 Exercises\n",
        "\n",
        "1. Build and fit a model using the same data we have here but with the MobileNetV2 architecture feature extraction ([`mobilenet_v2_100_224/feature_vector`](https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/4)) from TensorFlow Hub, how does it perform compared to our other models?\n",
        "2. Name 3 different image classification models on TensorFlow Hub that we haven't used.\n",
        "3. Build a model to classify images of two different things you've taken photos of.\n",
        "  * You can use any feature extraction layer from TensorFlow Hub you like for this.\n",
        "  * You should aim to have at least 10 images of each class, for example to build a fridge versus oven classifier, you'll want 10 images of fridges and 10 images of ovens.\n",
        "4. What is the current best performing model on ImageNet?\n",
        "  * Hint: you might want to check [sotabench.com](https://www.sotabench.com) for this."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w_YxwuhfRzD5"
      },
      "source": [
        "## 📖 Extra-curriculum\n",
        "\n",
        "* Read through the [TensorFlow Transfer Learning Guide](https://www.tensorflow.org/tutorials/images/transfer_learning) and define the main two types of transfer learning in your own words.\n",
        "* Go through the [Transfer Learning with TensorFlow Hub tutorial](https://www.tensorflow.org/tutorials/images/transfer_learning_with_hub) on the TensorFlow website and rewrite all of the code yourself into a new Google Colab notebook making comments about what each step does along the way.\n",
        "* We haven't covered fine-tuning with TensorFlow Hub in this notebook, but if you'd like to know more, go through the [fine-tuning a TensorFlow Hub model tutorial](https://www.tensorflow.org/hub/tf2_saved_model#fine-tuning) on the TensorFlow homepage.How to fine-tune a tensorflow hub model:  \n",
        "* Look into [experiment tracking with Weights & Biases](https://www.wandb.com/experiment-tracking), how could you integrate it with our existing TensorBoard logs?"
      ]
    }
  ]
}