PublicTensor2TensorTraining4All.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "PublicTensor2TensorTraining4All.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Santosh-Gupta/ScientificSummarizationDataSets/blob/master/PublicTensor2TensorTraining4All.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5BjZu-wS7jmg",
        "colab_type": "text"
      },
      "source": [
        "In addition to importing the data, you need to add the following directories in the top level of your Google Drive. \n",
        "\n",
        "An empty folder called 'SummarizationCheckPoints' . This is where Tensor2Tensor will store your checkpoint files. \n",
        "\n",
        "An empty folder called 'tfRecordsSummaryFiles' . This is where the generated tfRecords files will be stores"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q5qa7bunwEp8",
        "colab_type": "code",
        "outputId": "15c91bd6-9648-471a-e6ba-2d59e5528b31",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#Mount your Drive\n",
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s64ux_9fYy7U",
        "colab_type": "code",
        "outputId": "7aa2738f-027a-49db-e0e5-0e7e0dae5f66",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        }
      },
      "source": [
        "# install some dependencies and my version of Tensor2Tensor\n",
        "\n",
        "# !pip install -q -U tensor2tensor\n",
        "!pip install -q -U https://github.com/Santosh-Gupta/tensor2tensor/archive/master.zip\n",
        "!pip install -q tensorflow matplotlib\n",
        "!pip install fastparquet\n",
        "\n",
        "# Set up stuff \n",
        "\n",
        "import tensorflow as tf\n",
        "import os\n",
        "Modes = tf.estimator.ModeKeys\n",
        "\n",
        "# Setup some directories\n",
        "data_dir = os.path.expanduser(\"~/t2t/data\")\n",
        "tmp_dir = os.path.expanduser(\"~/t2t/tmp\")\n",
        "train_dir = os.path.expanduser(\"~/t2t/train\")\n",
        "output_dir = os.path.expanduser(\"~/t2t/output\")\n",
        "checkpoint_dir = os.path.expanduser(\"~/t2t/checkpoints\")\n",
        "tf.gfile.MakeDirs(data_dir)\n",
        "tf.gfile.MakeDirs(tmp_dir)\n",
        "tf.gfile.MakeDirs(train_dir)\n",
        "tf.gfile.MakeDirs(checkpoint_dir)\n",
        "gs_data_dir = \"gs://tensor2tensor-data\"\n",
        "gs_ckpt_dir = \"gs://tensor2tensor-checkpoints/\""
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     | 11.3MB 3.9MB/s\n",
            "\u001b[K     |████████████████████████████████| 143kB 3.4MB/s \n",
            "\u001b[K     |████████████████████████████████| 655kB 63.8MB/s \n",
            "\u001b[K     |████████████████████████████████| 686kB 48.2MB/s \n",
            "\u001b[?25h  Building wheel for tensor2tensor (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Building wheel for pypng (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: fastparquet in /usr/local/lib/python3.6/dist-packages (0.3.1)\n",
            "Requirement already satisfied: numba>=0.28 in /usr/local/lib/python3.6/dist-packages (from fastparquet) (0.40.1)\n",
            "Requirement already satisfied: thrift>=0.11.0 in /usr/local/lib/python3.6/dist-packages (from fastparquet) (0.11.0)\n",
            "Requirement already satisfied: pandas>=0.19 in /usr/local/lib/python3.6/dist-packages (from fastparquet) (0.24.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from fastparquet) (1.12.0)\n",
            "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.6/dist-packages (from fastparquet) (1.16.4)\n",
            "Requirement already satisfied: llvmlite>=0.25.0dev0 in /usr/local/lib/python3.6/dist-packages (from numba>=0.28->fastparquet) (0.29.0)\n",
            "Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.19->fastparquet) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.19->fastparquet) (2.5.3)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xMpYTvCnY7oL",
        "colab_type": "code",
        "outputId": "24b64db1-bcd7-44b2-c2aa-c037a28f7b2e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "#Convert the raw data to a format usable by Tensor2Tensor\n",
        "\n",
        "!t2t-datagen \\\n",
        "  --data_dir='gdrive/My Drive/SummarizationCheckPoints' \\\n",
        "  --tmp_dir=~/t2t/tmp \\\n",
        "  --problem=summarize_scientific_sections_gdrive65k\n",
        "\n",
        "#After this is run and the files are saved to your google drive, you can skip this step and go directly to training. ."
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0716 16:07:48.747938 140196621469568 lazy_loader.py:50] \n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "W0716 16:07:50.354530 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:68: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "W0716 16:07:57.294870 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/sci_sum.py:73: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n",
            "W0716 16:07:59.296963 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/gym_utils.py:235: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n",
            "W0716 16:07:59.301079 140196621469568 deprecation_wrapper.py:119] From /usr/local/bin/t2t-datagen:27: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
            "\n",
            "W0716 16:07:59.301221 140196621469568 deprecation_wrapper.py:119] From /usr/local/bin/t2t-datagen:27: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
            "\n",
            "W0716 16:07:59.301321 140196621469568 deprecation_wrapper.py:119] From /usr/local/bin/t2t-datagen:28: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.\n",
            "\n",
            "W0716 16:07:59.304210 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/bin/t2t_datagen.py:204: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.\n",
            "\n",
            "I0716 16:07:59.304662 140196621469568 t2t_datagen.py:207] Generating problems:\n",
            "    summarize:\n",
            "      * summarize_scientific_sections_gdrive65k\n",
            "W0716 16:07:59.304804 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/bin/t2t_datagen.py:156: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
            "\n",
            "I0716 16:07:59.308906 140196621469568 t2t_datagen.py:280] Generating data for summarize_scientific_sections_gdrive65k.\n",
            "W0716 16:07:59.310134 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:345: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.\n",
            "\n",
            "I0716 16:07:59.310524 140196621469568 generator_utils.py:351] Generating vocab file: gdrive/My Drive/tfRecordsArxivTitleAbstractDataSet/vocab.summarize_scientific_sections_gdrive65k.65536.subwords\n",
            "gdrive/My Drive/CombinedArxivSummaryDataSets/ArxivAbstractionSectionalSummaryDataset/ArxivStructuredAbstractSectionalSummaries.parquet.gzip\n",
            "I0716 16:08:10.336785 140196621469568 text_encoder.py:722] Trying min_count 500\n",
            "I0716 16:08:10.363996 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:08:12.878664 140196621469568 text_encoder.py:866] vocab_size = 8148\n",
            "I0716 16:08:12.878886 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:08:14.224359 140196621469568 text_encoder.py:866] vocab_size = 3420\n",
            "I0716 16:08:14.224579 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:08:15.696834 140196621469568 text_encoder.py:866] vocab_size = 3582\n",
            "I0716 16:08:15.697109 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:08:17.135668 140196621469568 text_encoder.py:866] vocab_size = 3546\n",
            "I0716 16:08:17.147055 140196621469568 text_encoder.py:722] Trying min_count 250\n",
            "I0716 16:08:17.171352 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:08:19.765794 140196621469568 text_encoder.py:866] vocab_size = 12441\n",
            "I0716 16:08:19.766016 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:08:21.048086 140196621469568 text_encoder.py:866] vocab_size = 5053\n",
            "I0716 16:08:21.048305 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:08:22.477073 140196621469568 text_encoder.py:866] vocab_size = 5249\n",
            "I0716 16:08:22.477298 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:08:23.862138 140196621469568 text_encoder.py:866] vocab_size = 5208\n",
            "I0716 16:08:23.872906 140196621469568 text_encoder.py:722] Trying min_count 125\n",
            "I0716 16:08:23.897351 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:08:26.476781 140196621469568 text_encoder.py:866] vocab_size = 18468\n",
            "I0716 16:08:26.477020 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:08:27.727782 140196621469568 text_encoder.py:866] vocab_size = 7463\n",
            "I0716 16:08:27.728008 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:08:29.070323 140196621469568 text_encoder.py:866] vocab_size = 7706\n",
            "I0716 16:08:29.070548 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:08:30.369773 140196621469568 text_encoder.py:866] vocab_size = 7663\n",
            "I0716 16:08:30.380874 140196621469568 text_encoder.py:722] Trying min_count 62\n",
            "I0716 16:08:30.404569 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:08:33.057821 140196621469568 text_encoder.py:866] vocab_size = 26870\n",
            "I0716 16:08:33.058041 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:08:34.261327 140196621469568 text_encoder.py:866] vocab_size = 10902\n",
            "I0716 16:08:34.261573 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:08:35.538717 140196621469568 text_encoder.py:866] vocab_size = 11212\n",
            "I0716 16:08:35.538958 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:08:36.818098 140196621469568 text_encoder.py:866] vocab_size = 11154\n",
            "I0716 16:08:36.828863 140196621469568 text_encoder.py:722] Trying min_count 31\n",
            "I0716 16:08:36.852510 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:08:39.616913 140196621469568 text_encoder.py:866] vocab_size = 38612\n",
            "I0716 16:08:39.617160 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:08:40.782280 140196621469568 text_encoder.py:866] vocab_size = 15782\n",
            "I0716 16:08:40.782533 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:08:42.022303 140196621469568 text_encoder.py:866] vocab_size = 16179\n",
            "I0716 16:08:42.022541 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:08:43.224056 140196621469568 text_encoder.py:866] vocab_size = 16098\n",
            "I0716 16:08:43.235378 140196621469568 text_encoder.py:722] Trying min_count 15\n",
            "I0716 16:08:43.259122 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:08:46.032806 140196621469568 text_encoder.py:866] vocab_size = 56049\n",
            "I0716 16:08:46.033042 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:08:47.186045 140196621469568 text_encoder.py:866] vocab_size = 23090\n",
            "I0716 16:08:47.186276 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:08:48.392608 140196621469568 text_encoder.py:866] vocab_size = 23648\n",
            "I0716 16:08:48.392840 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:08:49.585987 140196621469568 text_encoder.py:866] vocab_size = 23551\n",
            "I0716 16:08:49.598384 140196621469568 text_encoder.py:722] Trying min_count 7\n",
            "I0716 16:08:49.623110 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:08:52.706712 140196621469568 text_encoder.py:866] vocab_size = 82132\n",
            "I0716 16:08:52.706981 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:08:53.897996 140196621469568 text_encoder.py:866] vocab_size = 33943\n",
            "I0716 16:08:53.898219 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:08:55.147520 140196621469568 text_encoder.py:866] vocab_size = 34714\n",
            "I0716 16:08:55.147758 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:08:56.340670 140196621469568 text_encoder.py:866] vocab_size = 34520\n",
            "I0716 16:08:56.353706 140196621469568 text_encoder.py:722] Trying min_count 3\n",
            "I0716 16:08:56.377950 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:09:00.007776 140196621469568 text_encoder.py:866] vocab_size = 124190\n",
            "I0716 16:09:00.008026 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:09:01.234271 140196621469568 text_encoder.py:866] vocab_size = 50047\n",
            "I0716 16:09:01.234508 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:09:02.453585 140196621469568 text_encoder.py:866] vocab_size = 51043\n",
            "I0716 16:09:02.453818 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:09:03.635010 140196621469568 text_encoder.py:866] vocab_size = 50724\n",
            "I0716 16:09:03.648672 140196621469568 text_encoder.py:722] Trying min_count 1\n",
            "I0716 16:09:03.672336 140196621469568 text_encoder.py:802] Iteration 0\n",
            "I0716 16:09:08.274942 140196621469568 text_encoder.py:866] vocab_size = 180451\n",
            "I0716 16:09:08.275178 140196621469568 text_encoder.py:802] Iteration 1\n",
            "I0716 16:09:09.515289 140196621469568 text_encoder.py:866] vocab_size = 66610\n",
            "I0716 16:09:09.515541 140196621469568 text_encoder.py:802] Iteration 2\n",
            "I0716 16:09:10.675758 140196621469568 text_encoder.py:866] vocab_size = 66610\n",
            "I0716 16:09:10.675986 140196621469568 text_encoder.py:802] Iteration 3\n",
            "I0716 16:09:11.822018 140196621469568 text_encoder.py:866] vocab_size = 66610\n",
            "W0716 16:09:11.878058 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/text_encoder.py:944: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n",
            "W0716 16:09:12.038422 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:166: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
            "\n",
            "I0716 16:09:12.193210 140196621469568 sci_sum_gdrive.py:92] Writing summaryData.train\n",
            "gdrive/My Drive/CombinedArxivSummaryDataSets/ArxivAbstractionSectionalSummaryDataset/ArxivStructuredAbstractSectionalSummaries.parquet.gzip\n",
            "gdrive/My Drive/CombinedArxivSummaryDataSets/ArxivAbstractionSectionalSummaryDataset/ArxivStructuredAbstractSectionalSummaries.parquet.gzip\n",
            "I0716 16:09:13.058475 140196621469568 generator_utils.py:172] Generating case 0.\n",
            "W0716 16:09:23.987187 140196621469568 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:185: The name tf.gfile.Rename is deprecated. Please use tf.io.gfile.rename instead.\n",
            "\n",
            "I0716 16:09:24.124346 140196621469568 generator_utils.py:195] Generated 6229 Examples\n",
            "I0716 16:09:24.125547 140196621469568 generator_utils.py:346] Found vocab file: gdrive/My Drive/tfRecordsArxivTitleAbstractDataSet/vocab.summarize_scientific_sections_gdrive65k.65536.subwords\n",
            "I0716 16:09:24.451530 140196621469568 sci_sum_gdrive.py:92] Writing summaryData.dev\n",
            "gdrive/My Drive/CombinedArxivSummaryDataSets/ArxivAbstractionSectionalSummaryDataset/ArxivStructuredAbstractSectionalSummaries.parquet.gzip\n",
            "gdrive/My Drive/CombinedArxivSummaryDataSets/ArxivAbstractionSectionalSummaryDataset/ArxivStructuredAbstractSectionalSummaries.parquet.gzip\n",
            "I0716 16:09:25.309220 140196621469568 generator_utils.py:172] Generating case 0.\n",
            "I0716 16:09:35.898481 140196621469568 generator_utils.py:195] Generated 6229 Examples\n",
            "I0716 16:09:35.899137 140196621469568 generator_utils.py:346] Found vocab file: gdrive/My Drive/tfRecordsArxivTitleAbstractDataSet/vocab.summarize_scientific_sections_gdrive65k.65536.subwords\n",
            "I0716 16:09:36.219978 140196621469568 sci_sum_gdrive.py:92] Writing summaryData.test\n",
            "gdrive/My Drive/CombinedArxivSummaryDataSets/ArxivAbstractionSectionalSummaryDataset/ArxivStructuredAbstractSectionalSummaries.parquet.gzip\n",
            "gdrive/My Drive/CombinedArxivSummaryDataSets/ArxivAbstractionSectionalSummaryDataset/ArxivStructuredAbstractSectionalSummaries.parquet.gzip\n",
            "I0716 16:09:37.100348 140196621469568 generator_utils.py:172] Generating case 0.\n",
            "I0716 16:09:47.803864 140196621469568 generator_utils.py:195] Generated 6229 Examples\n",
            "I0716 16:09:47.819801 140196621469568 generator_utils.py:529] Shuffling data...\n",
            "W0716 16:09:47.820035 140196621469568 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/generator_utils.py:471: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use eager execution and: \n",
            "`tf.data.TFRecordDataset(path)`\n",
            "I0716 16:09:49.201142 140196621469568 generator_utils.py:532] Data shuffled.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HnnZvnJB6aZC",
        "colab_type": "code",
        "outputId": "5cdc0291-f628-4a62-dd3c-60999ddb7fbd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#Check the files \n",
        "\n",
        "os.listdir('gdrive/My Drive/CombinedArxivSummaryDataSets/ArxivAbstractionSectionalSummaryDataset')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['ArxivStructuredAbstractSectionalSummaries.parquet.gzip']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TR-6NhG8Y7mL",
        "colab_type": "code",
        "outputId": "2bc78e60-fc5c-4fba-bbaf-1e4c9adf2819",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Start training! \n",
        "\n",
        "!t2t-trainer \\\n",
        "  --model=transformer \\\n",
        "  --hparams_set=transformer_prepend \\\n",
        "  --problem=summarize_scientific_sections_gdrive65k \\\n",
        "  --train_steps=4000 \\\n",
        "  --eval_steps=30 \\\n",
        "  --data_dir='gdrive/My Drive/tfRecordsSummaryFiles' \\\n",
        "  --tmp_dir=~/t2t/tmp \\\n",
        "  --output_dir='gdrive/My Drive/SummarizationCheckPoints' \\\n",
        "  --hparams='batch_size=2048' "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0716 16:11:16.562587 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:68: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "W0716 16:11:17.587020 140497369663360 lazy_loader.py:50] \n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "W0716 16:11:19.817484 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/adafactor.py:27: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "W0716 16:11:19.818099 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/multistep_optimizer.py:32: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n",
            "W0716 16:11:19.837353 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4237: The name tf.train.CheckpointSaverListener is deprecated. Please use tf.estimator.CheckpointSaverListener instead.\n",
            "\n",
            "W0716 16:11:19.837558 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4260: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
            "\n",
            "W0716 16:11:19.861151 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/neural_stack.py:38: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.\n",
            "\n",
            "W0716 16:11:19.887502 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/gym_utils.py:235: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n",
            "W0716 16:11:19.975629 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:111: The name tf.OptimizerOptions is deprecated. Please use tf.compat.v1.OptimizerOptions instead.\n",
            "\n",
            "W0716 16:11:24.707919 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/sci_sum.py:73: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n",
            "W0716 16:11:26.140439 140497369663360 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:32: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
            "\n",
            "W0716 16:11:26.140626 140497369663360 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:32: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
            "\n",
            "W0716 16:11:26.140757 140497369663360 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:33: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.\n",
            "\n",
            "W0716 16:11:26.141491 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/hparams_lib.py:49: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.\n",
            "\n",
            "I0716 16:11:26.142033 140497369663360 hparams_lib.py:55] Overriding hparams in transformer_prepend with batch_size=2048\n",
            "W0716 16:11:26.142342 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:839: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
            "\n",
            "W0716 16:11:26.143215 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:123: The name tf.GraphOptions is deprecated. Please use tf.compat.v1.GraphOptions instead.\n",
            "\n",
            "W0716 16:11:26.143388 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:129: The name tf.GPUOptions is deprecated. Please use tf.compat.v1.GPUOptions instead.\n",
            "\n",
            "W0716 16:11:26.143551 140497369663360 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:242: RunConfig.__init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.\n",
            "I0716 16:11:26.143737 140497369663360 trainer_lib.py:265] Configuring DataParallelism to replicate the model.\n",
            "I0716 16:11:26.143817 140497369663360 devices.py:76] schedule=continuous_train_and_eval\n",
            "I0716 16:11:26.143885 140497369663360 devices.py:77] worker_gpu=1\n",
            "I0716 16:11:26.143944 140497369663360 devices.py:78] sync=False\n",
            "W0716 16:11:26.144033 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/devices.py:139: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.\n",
            "\n",
            "W0716 16:11:26.144104 140497369663360 devices.py:141] Schedule=continuous_train_and_eval. Assuming that training is running on a single machine.\n",
            "I0716 16:11:26.144761 140497369663360 devices.py:170] datashard_devices: ['gpu:0']\n",
            "I0716 16:11:26.144831 140497369663360 devices.py:171] caching_devices: None\n",
            "I0716 16:11:26.145304 140497369663360 devices.py:172] ps_devices: ['gpu:0']\n",
            "W0716 16:11:26.145813 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/text_encoder.py:940: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n",
            "I0716 16:11:26.461180 140497369663360 estimator.py:209] Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fc7a10796d8>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_eval_distribute': None, '_experimental_max_worker_delay_secs': None, '_device_fn': None, '_tf_config': gpu_options {\n",
            "  per_process_gpu_memory_fraction: 1.0\n",
            "}\n",
            ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_protocol': None, '_session_config': gpu_options {\n",
            "  per_process_gpu_memory_fraction: 0.95\n",
            "}\n",
            "allow_soft_placement: true\n",
            "graph_options {\n",
            "  optimizer_options {\n",
            "    global_jit_level: OFF\n",
            "  }\n",
            "}\n",
            "isolate_session_state: true\n",
            ", '_save_checkpoints_steps': 1000, '_keep_checkpoint_max': 20, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'gdrive/My Drive/SummarizationCheckPoints', 'use_tpu': False, 't2t_device_info': {'num_async_replicas': 1}, 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7fc7a1079748>}\n",
            "W0716 16:11:26.461510 140497369663360 model_fn.py:630] Estimator's model_fn (<function T2TModel.make_estimator_model_fn.<locals>.wrapping_model_fn at 0x7fc7bae95e18>) includes params argument, but params are not passed to Estimator.\n",
            "W0716 16:11:26.462257 140497369663360 trainer_lib.py:783] ValidationMonitor only works with --schedule=train_and_evaluate\n",
            "I0716 16:11:26.474347 140497369663360 estimator_training.py:186] Not using Distribute Coordinator.\n",
            "I0716 16:11:26.474623 140497369663360 training.py:612] Running training and evaluation locally (non-distributed).\n",
            "I0716 16:11:26.474980 140497369663360 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.\n",
            "W0716 16:11:26.485336 140497369663360 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
            "I0716 16:11:26.495955 140497369663360 problem.py:644] Reading data files from gdrive/My Drive/tfRecordsArxivTitleAbstractDataSet/summarize_scientific_sections_gdrive65k-train*\n",
            "I0716 16:11:26.511197 140497369663360 problem.py:670] partition: 0 num_data_files: 100\n",
            "W0716 16:11:26.513390 140497369663360 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/problem.py:680: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.\n",
            "W0716 16:11:26.596098 140497369663360 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/data_reader.py:275: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use eager execution and: \n",
            "`tf.data.TFRecordDataset(path)`\n",
            "W0716 16:11:26.612076 140497369663360 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/data_reader.py:37: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "W0716 16:11:26.655771 140497369663360 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/data/experimental/ops/grouping.py:193: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "W0716 16:11:26.720481 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/data_reader.py:231: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
            "\n",
            "W0716 16:11:26.731303 140497369663360 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/data_reader.py:233: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "I0716 16:11:26.786887 140497369663360 estimator.py:1145] Calling model_fn.\n",
            "I0716 16:11:26.799174 140497369663360 t2t_model.py:2245] Setting T2TModel mode to 'train'\n",
            "W0716 16:11:26.874262 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:244: The name tf.summary.text is deprecated. Please use tf.compat.v1.summary.text instead.\n",
            "\n",
            "I0716 16:11:27.657939 140497369663360 api.py:255] Using variable initializer: uniform_unit_scaling\n",
            "I0716 16:11:28.135538 140497369663360 t2t_model.py:2245] Transforming feature 'inputs' with symbol_modality_66610_512.bottom\n",
            "I0716 16:11:28.454421 140497369663360 t2t_model.py:2245] Transforming feature 'targets' with symbol_modality_66610_512.targets_bottom\n",
            "I0716 16:11:28.466666 140497369663360 t2t_model.py:2245] Building model body\n",
            "W0716 16:11:28.542714 140497369663360 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/transformer.py:96: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
            "W0716 16:11:28.587941 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:3236: The name tf.layers.Dense is deprecated. Please use tf.compat.v1.layers.Dense instead.\n",
            "\n",
            "W0716 16:11:28.988707 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_attention.py:1249: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.\n",
            "\n",
            "I0716 16:11:34.934575 140497369663360 t2t_model.py:2245] Transforming body output with symbol_modality_66610_512.top\n",
            "W0716 16:11:35.311010 140497369663360 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/learning_rate.py:120: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
            "\n",
            "I0716 16:11:35.321029 140497369663360 learning_rate.py:114] Base learning rate: 0.200000\n",
            "I0716 16:11:35.330558 140497369663360 optimize.py:327] Trainable Variables Total size: 78224384\n",
            "I0716 16:11:35.330861 140497369663360 optimize.py:327] Non-trainable variables Total size: 5\n",
            "I0716 16:11:35.331209 140497369663360 optimize.py:182] Using optimizer adam\n",
            "I0716 16:11:45.709239 140497369663360 estimator.py:1147] Done calling model_fn.\n",
            "I0716 16:11:45.710800 140497369663360 basic_session_run_hooks.py:541] Create CheckpointSaverHook.\n",
            "I0716 16:11:49.219081 140497369663360 monitored_session.py:240] Graph was finalized.\n",
            "2019-07-16 16:11:49.231497: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz\n",
            "2019-07-16 16:11:49.233249: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5f98a00 executing computations on platform Host. Devices:\n",
            "2019-07-16 16:11:49.233282: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>\n",
            "2019-07-16 16:11:49.237998: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1\n",
            "2019-07-16 16:11:49.441171: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-16 16:11:49.441705: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5f98d80 executing computations on platform CUDA. Devices:\n",
            "2019-07-16 16:11:49.441733: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5\n",
            "2019-07-16 16:11:49.441957: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-16 16:11:49.442295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n",
            "name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59\n",
            "pciBusID: 0000:00:04.0\n",
            "2019-07-16 16:11:49.448913: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0\n",
            "2019-07-16 16:11:49.609363: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0\n",
            "2019-07-16 16:11:49.677435: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0\n",
            "2019-07-16 16:11:49.697065: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0\n",
            "2019-07-16 16:11:49.871729: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0\n",
            "2019-07-16 16:11:49.974830: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0\n",
            "2019-07-16 16:11:50.282951: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n",
            "2019-07-16 16:11:50.283213: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-16 16:11:50.283776: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-16 16:11:50.284127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n",
            "2019-07-16 16:11:50.286680: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0\n",
            "2019-07-16 16:11:50.288862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2019-07-16 16:11:50.288894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 \n",
            "2019-07-16 16:11:50.288905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N \n",
            "2019-07-16 16:11:50.290785: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-16 16:11:50.291270: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-16 16:11:50.291685: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "2019-07-16 16:11:50.291732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14325 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)\n",
            "2019-07-16 16:11:53.562530: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n",
            "I0716 16:11:56.313433 140497369663360 session_manager.py:500] Running local_init_op.\n",
            "I0716 16:11:56.527391 140497369663360 session_manager.py:502] Done running local_init_op.\n",
            "I0716 16:12:07.207109 140497369663360 basic_session_run_hooks.py:606] Saving checkpoints for 0 into gdrive/My Drive/SummarizationCheckPoints/model.ckpt.\n",
            "2019-07-16 16:12:25.455365: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0\n",
            "I0716 16:12:28.707125 140497369663360 basic_session_run_hooks.py:262] loss = 10.197866, step = 0\n",
            "I0716 16:13:22.443345 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 1.86066\n",
            "I0716 16:13:22.452297 140497369663360 basic_session_run_hooks.py:260] loss = 8.556887, step = 100 (53.745 sec)\n",
            "I0716 16:14:08.837083 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 2.15546\n",
            "I0716 16:14:08.846139 140497369663360 basic_session_run_hooks.py:260] loss = 7.75054, step = 200 (46.394 sec)\n",
            "I0716 16:14:53.963978 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 2.21597\n",
            "I0716 16:14:53.973404 140497369663360 basic_session_run_hooks.py:260] loss = 7.2219677, step = 300 (45.127 sec)\n",
            "I0716 16:15:38.866108 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 2.22707\n",
            "I0716 16:15:38.874092 140497369663360 basic_session_run_hooks.py:260] loss = 6.473357, step = 400 (44.901 sec)\n",
            "I0716 16:16:24.205215 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 2.2056\n",
            "I0716 16:16:24.217247 140497369663360 basic_session_run_hooks.py:260] loss = 6.5354266, step = 500 (45.343 sec)\n",
            "I0716 16:17:08.675126 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 2.24871\n",
            "I0716 16:17:08.685680 140497369663360 basic_session_run_hooks.py:260] loss = 6.1652474, step = 600 (44.468 sec)\n",
            "I0716 16:17:53.342605 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 2.23876\n",
            "I0716 16:17:53.352077 140497369663360 basic_session_run_hooks.py:260] loss = 5.9438124, step = 700 (44.666 sec)\n",
            "I0716 16:18:37.538886 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 2.26263\n",
            "I0716 16:18:37.548563 140497369663360 basic_session_run_hooks.py:260] loss = 5.8283453, step = 800 (44.196 sec)\n",
            "I0716 16:19:22.443572 140497369663360 basic_session_run_hooks.py:692] global_step/sec: 2.22694\n",
            "I0716 16:19:22.453545 140497369663360 basic_session_run_hooks.py:260] loss = 5.6550055, step = 900 (44.905 sec)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L_9XCbD8sgfP",
        "colab_type": "code",
        "outputId": "780f7cee-41cd-479a-88e6-3524d083ce4a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# If you want to check all the models \n",
        "\n",
        "!t2t-trainer --registry_help"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0716 00:15:41.755698 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:68: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "W0716 00:15:43.033833 140464506578816 lazy_loader.py:50] \n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "W0716 00:15:45.548528 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/adafactor.py:27: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "W0716 00:15:45.549373 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/multistep_optimizer.py:32: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n",
            "W0716 00:15:45.567197 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4237: The name tf.train.CheckpointSaverListener is deprecated. Please use tf.estimator.CheckpointSaverListener instead.\n",
            "\n",
            "W0716 00:15:45.567504 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4260: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
            "\n",
            "W0716 00:15:45.594980 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/neural_stack.py:38: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.\n",
            "\n",
            "W0716 00:15:45.629523 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/gym_utils.py:235: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n",
            "W0716 00:15:45.653687 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:111: The name tf.OptimizerOptions is deprecated. Please use tf.compat.v1.OptimizerOptions instead.\n",
            "\n",
            "W0716 00:15:49.393027 140464506578816 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/sci_sum.py:73: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n",
            "W0716 00:15:51.099856 140464506578816 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:32: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
            "\n",
            "W0716 00:15:51.100077 140464506578816 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:32: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
            "\n",
            "W0716 00:15:51.100199 140464506578816 deprecation_wrapper.py:119] From /usr/local/bin/t2t-trainer:33: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.\n",
            "\n",
            "I0716 00:15:51.102122 140464506578816 t2t_trainer.py:315] \n",
            "Registry contents:\n",
            "------------------\n",
            "\n",
            "  Models:\n",
            "    aligned:\n",
            "      * aligned\n",
            "    attention:\n",
            "      * attention_lm\n",
            "      * attention_lm_moe\n",
            "    autoencoder:\n",
            "      * autoencoder_autoregressive\n",
            "      * autoencoder_basic\n",
            "      * autoencoder_basic_discrete\n",
            "      * autoencoder_dual_discrete\n",
            "      * autoencoder_ordered_discrete\n",
            "      * autoencoder_residual\n",
            "      * autoencoder_residual_discrete\n",
            "      * autoencoder_residual_vae\n",
            "      * autoencoder_stacked\n",
            "    basic:\n",
            "      * basic_fc_relu\n",
            "    byte:\n",
            "      * byte_net\n",
            "    cycle:\n",
            "      * cycle_gan\n",
            "    dense:\n",
            "      * dense_bitwise_categorical_policy\n",
            "    diagonal:\n",
            "      * diagonal_neural_gpu\n",
            "    distillation:\n",
            "      * distillation\n",
            "    evolved:\n",
            "      * evolved_transformer\n",
            "    feed:\n",
            "      * feed_forward_categorical_policy\n",
            "      * feed_forward_cnn_small_categorical_policy\n",
            "      * feed_forward_cnn_small_categorical_policy_new\n",
            "    gene:\n",
            "      * gene_expression_conv\n",
            "    glow:\n",
            "      * glow\n",
            "    imagetransformer:\n",
            "      * imagetransformer\n",
            "    imagetransformer2d:\n",
            "      * imagetransformer2d\n",
            "    imagetransformer:\n",
            "      * imagetransformer_moe\n",
            "    img2img:\n",
            "      * img2img_transformer\n",
            "      * img2img_transformer_block_parallel\n",
            "    lstm:\n",
            "      * lstm_encoder\n",
            "      * lstm_seq2seq\n",
            "      * lstm_seq2seq_attention\n",
            "      * lstm_seq2seq_attention_bidirectional_encoder\n",
            "      * lstm_seq2seq_bidirectional_encoder\n",
            "    mtf:\n",
            "      * mtf_bitransformer\n",
            "      * mtf_image_transformer\n",
            "      * mtf_res_net\n",
            "      * mtf_transformer\n",
            "      * mtf_unitransformer\n",
            "    nas:\n",
            "      * nas_seq2_seq\n",
            "    neural:\n",
            "      * neural_gpu\n",
            "      * neural_queue_model\n",
            "      * neural_stack_model\n",
            "    next:\n",
            "      * next_frame_base\n",
            "      * next_frame_basic_deterministic\n",
            "      * next_frame_basic_recurrent\n",
            "      * next_frame_basic_stochastic\n",
            "      * next_frame_basic_stochastic_discrete\n",
            "      * next_frame_emily\n",
            "      * next_frame_epva\n",
            "      * next_frame_glow\n",
            "      * next_frame_savp\n",
            "      * next_frame_savp_rl\n",
            "      * next_frame_sv2p\n",
            "      * next_frame_sv2p_atari\n",
            "      * next_frame_sv2p_discrete\n",
            "      * next_frame_sv2p_legacy\n",
            "      * next_frame_sv2p_two_frames\n",
            "    random:\n",
            "      * random_policy\n",
            "    resnet:\n",
            "      * resnet\n",
            "    revnet:\n",
            "      * revnet\n",
            "    shake:\n",
            "      * shake_shake\n",
            "    similarity:\n",
            "      * similarity_transformer\n",
            "    slice:\n",
            "      * slice_net\n",
            "    sliced:\n",
            "      * sliced_gan\n",
            "    super:\n",
            "      * super_lm\n",
            "    text:\n",
            "      * text_cnn\n",
            "    transformer:\n",
            "      * transformer\n",
            "      * transformer_ae\n",
            "      * transformer_block_parallel\n",
            "      * transformer_encoder\n",
            "      * transformer_memory\n",
            "      * transformer_moe\n",
            "      * transformer_nat\n",
            "      * transformer_regressor\n",
            "      * transformer_revnet\n",
            "      * transformer_scorer\n",
            "      * transformer_sketch\n",
            "      * transformer_symshard\n",
            "    universal:\n",
            "      * universal_transformer\n",
            "      * universal_transformer_encoder\n",
            "    vqa:\n",
            "      * vqa_attention_baseline\n",
            "      * vqa_combined_self_attention\n",
            "      * vqa_iterative_combined_self_attention\n",
            "      * vqa_recurrent_self_attention\n",
            "      * vqa_self_attention\n",
            "      * vqa_simple_image_self_attention\n",
            "    xception:\n",
            "      * xception\n",
            "\n",
            "  HParams:\n",
            "    adaptive:\n",
            "      * adaptive_universal_transformer_base\n",
            "      * adaptive_universal_transformer_base_dropout03\n",
            "      * adaptive_universal_transformer_base_dropout05\n",
            "      * adaptive_universal_transformer_base_tpu\n",
            "      * adaptive_universal_transformer_concat_tiny\n",
            "      * adaptive_universal_transformer_global_base\n",
            "      * adaptive_universal_transformer_global_base_tpu\n",
            "      * adaptive_universal_transformer_mix_after_ut_base\n",
            "      * adaptive_universal_transformer_mix_before_ut_base\n",
            "      * adaptive_universal_transformer_multilayer_hard\n",
            "      * adaptive_universal_transformer_multilayer_tpu\n",
            "      * adaptive_universal_transformer_position_random_timing_tiny\n",
            "      * adaptive_universal_transformer_sepconv_tiny\n",
            "      * adaptive_universal_transformer_small\n",
            "      * adaptive_universal_transformer_tall\n",
            "      * adaptive_universal_transformer_tall_actlossw0\n",
            "      * adaptive_universal_transformer_tall_actlossw001\n",
            "      * adaptive_universal_transformer_tiny\n",
            "      * adaptive_universal_transformer_with_sru_base\n",
            "    afx:\n",
            "      * afx_adafactor\n",
            "      * afx_adam\n",
            "      * afx_base\n",
            "      * afx_clip\n",
            "      * afx_clip2\n",
            "      * afx_clip_factored\n",
            "      * afx_factored\n",
            "      * afx_fast\n",
            "      * afx_mimic_adam\n",
            "      * afx_pow05\n",
            "      * afx_pow08\n",
            "      * afx_pow08_clip\n",
            "      * afx_pow10\n",
            "      * afx_relative\n",
            "      * afx_small\n",
            "      * afx_small_bfloat16\n",
            "      * afx_small_p10\n",
            "      * afx_small_p11\n",
            "      * afx_small_p12\n",
            "      * afx_small_p16\n",
            "      * afx_small_p8\n",
            "      * afx_unscale\n",
            "      * afx_unscale_relative\n",
            "    aligned:\n",
            "      * aligned_8k\n",
            "      * aligned_8k_grouped\n",
            "      * aligned_base\n",
            "      * aligned_grouped\n",
            "      * aligned_local\n",
            "      * aligned_local_1k\n",
            "      * aligned_local_expert\n",
            "      * aligned_lsh\n",
            "      * aligned_memory_efficient\n",
            "      * aligned_moe\n",
            "      * aligned_no_att\n",
            "      * aligned_no_timing\n",
            "      * aligned_pos_emb\n",
            "      * aligned_pseudolocal\n",
            "      * aligned_pseudolocal_256\n",
            "    attention:\n",
            "      * attention_lm_11k\n",
            "      * attention_lm_12k\n",
            "      * attention_lm_16k\n",
            "      * attention_lm_ae_extended\n",
            "      * attention_lm_attention_moe_tiny\n",
            "      * attention_lm_base\n",
            "      * attention_lm_hybrid_v2\n",
            "      * attention_lm_moe_24b_diet\n",
            "      * attention_lm_moe_32b_diet\n",
            "      * attention_lm_moe_base\n",
            "      * attention_lm_moe_base_ae\n",
            "      * attention_lm_moe_base_hybrid\n",
            "      * attention_lm_moe_base_local\n",
            "      * attention_lm_moe_base_long_seq\n",
            "      * attention_lm_moe_base_memeff\n",
            "      * attention_lm_moe_large\n",
            "      * attention_lm_moe_large_diet\n",
            "      * attention_lm_moe_memory_efficient\n",
            "      * attention_lm_moe_small\n",
            "      * attention_lm_moe_tiny\n",
            "      * attention_lm_moe_translation\n",
            "      * attention_lm_moe_unscramble_base\n",
            "      * attention_lm_no_moe_small\n",
            "      * attention_lm_small\n",
            "      * attention_lm_translation\n",
            "      * attention_lm_translation_full_attention\n",
            "      * attention_lm_translation_l12\n",
            "    autoencoder:\n",
            "      * autoencoder_autoregressive\n",
            "      * autoencoder_basic\n",
            "      * autoencoder_basic_discrete\n",
            "      * autoencoder_discrete_cifar\n",
            "      * autoencoder_discrete_pong\n",
            "      * autoencoder_discrete_tiny\n",
            "      * autoencoder_ordered_discrete\n",
            "      * autoencoder_ordered_discrete_hs256\n",
            "      * autoencoder_ordered_discrete_image64\n",
            "      * autoencoder_ordered_discrete_patched\n",
            "      * autoencoder_ordered_discrete_single\n",
            "      * autoencoder_ordered_discrete_vq\n",
            "      * autoencoder_ordered_text\n",
            "      * autoencoder_ordered_text_small\n",
            "      * autoencoder_residual\n",
            "      * autoencoder_residual_discrete\n",
            "      * autoencoder_residual_discrete_big\n",
            "      * autoencoder_residual_text\n",
            "      * autoencoder_stacked\n",
            "    basic:\n",
            "      * basic_1\n",
            "      * basic_fc_small\n",
            "      * basic_policy_parameters\n",
            "    bytenet:\n",
            "      * bytenet_base\n",
            "    cycle:\n",
            "      * cycle_gan_small\n",
            "    denoise:\n",
            "      * denoise_dense_2_m30\n",
            "      * denoise_m15\n",
            "      * denoise_m30\n",
            "      * denoise_t15\n",
            "      * denoise_v1_m15\n",
            "      * denoise_v1_m30\n",
            "      * denoise_v1_m50\n",
            "      * denoise_v1_t15\n",
            "      * denoise_v1_z15\n",
            "      * denoise_z15\n",
            "    discrete:\n",
            "      * discrete_random_action_base\n",
            "    distill:\n",
            "      * distill_resnet_32_to_15_cifar20x5\n",
            "    dqn:\n",
            "      * dqn_10m_replay_buffer_params\n",
            "      * dqn_2m_replay_buffer_params\n",
            "      * dqn_atari_base\n",
            "      * dqn_guess1_params\n",
            "      * dqn_guess1_params_eval\n",
            "      * dqn_guess1_rainbow_params\n",
            "      * dqn_original_params\n",
            "    evolved:\n",
            "      * evolved_transformer_base\n",
            "      * evolved_transformer_base_tpu\n",
            "      * evolved_transformer_big\n",
            "      * evolved_transformer_big_tpu\n",
            "      * evolved_transformer_deep\n",
            "    frame:\n",
            "      * frame_glow_hparams\n",
            "    gene:\n",
            "      * gene_expression_conv_base\n",
            "    glow:\n",
            "      * glow_hparams\n",
            "    image:\n",
            "      * image_transformer2d_base\n",
            "      * image_transformer_base\n",
            "    imagetransformer1d:\n",
            "      * imagetransformer1d_base_12l_64by64\n",
            "      * imagetransformer1d_base_8l_64by64\n",
            "    imagetransformer2d:\n",
            "      * imagetransformer2d_base\n",
            "      * imagetransformer2d_base_12l_8_16_big\n",
            "      * imagetransformer2d_base_12l_8_64_64by64\n",
            "      * imagetransformer2d_base_14l_8_16_big\n",
            "      * imagetransformer2d_base_14l_8_16_big_uncond\n",
            "      * imagetransformer2d_base_8l_8_16\n",
            "      * imagetransformer2d_base_8l_8_16_big\n",
            "      * imagetransformer2d_base_8l_8_16_big_16k\n",
            "      * imagetransformer2d_base_8l_8_16_ls\n",
            "      * imagetransformer2d_base_8l_8_32_big\n",
            "      * imagetransformer2d_base_8l_8_64_64by64\n",
            "      * imagetransformer2d_tiny\n",
            "    imagetransformer:\n",
            "      * imagetransformer_ae_cifar\n",
            "      * imagetransformer_b10l_4h_big_uncond_dr01_tpu\n",
            "      * imagetransformer_b10l_4h_big_uncond_dr03_lr025_tpu\n",
            "      * imagetransformer_b10l_4h_big_uncond_dr03_tpu\n",
            "      * imagetransformer_b10l_dr03_moe_tpu\n",
            "      * imagetransformer_b12l_4h_b128_h512_uncond_dr01_im\n",
            "      * imagetransformer_b12l_4h_b128_h512_uncond_dr03_tpu\n",
            "      * imagetransformer_b12l_4h_b128_uncond_dr03_tpu\n",
            "      * imagetransformer_b12l_4h_b256_uncond_dr03_rel_tpu\n",
            "      * imagetransformer_b12l_4h_b256_uncond_dr03_tpu\n",
            "      * imagetransformer_b12l_4h_big_uncond_dr03_lr025_tpu\n",
            "      * imagetransformer_b12l_4h_big_uncond_dr03_tpu\n",
            "      * imagetransformer_b12l_4h_uncond_dr03_tpu\n",
            "      * imagetransformer_b12l_8h_b256_uncond_dr03_tpu\n",
            "      * imagetransformer_bas8l_8h_big_uncond_dr03_imgnet\n",
            "      * imagetransformer_base\n",
            "      * imagetransformer_base_10l_16h_big_dr01_imgnet\n",
            "      * imagetransformer_base_10l_16h_big_dr01_moe_imgnet\n",
            "      * imagetransformer_base_10l_16h_big_uncond_dr01_imgnet\n",
            "      * imagetransformer_base_10l_8h_big_cond_dr03_dan\n",
            "      * imagetransformer_base_10l_8h_big_uncond_dr03_dan\n",
            "      * imagetransformer_base_10l_8h_big_uncond_dr03_dan_64\n",
            "      * imagetransformer_base_10l_8h_big_uncond_dr03_dan_64_2d\n",
            "      * imagetransformer_base_12l_8h_big\n",
            "      * imagetransformer_base_12l_8h_big_uncond\n",
            "      * imagetransformer_base_14l_8h_big\n",
            "      * imagetransformer_base_14l_8h_big_dr01\n",
            "      * imagetransformer_base_14l_8h_big_uncond\n",
            "      * imagetransformer_base_8l_8h_big_cond_dr03_dan\n",
            "      * imagetransformer_base_8l_8h_big_cond_dr03_dan_128\n",
            "      * imagetransformer_base_8l_8h_big_cond_dr03_dan_dilated\n",
            "      * imagetransformer_base_8l_8h_big_cond_dr03_dan_dilated_b\n",
            "      * imagetransformer_base_8l_8h_big_cond_dr03_dan_dilated_c\n",
            "      * imagetransformer_base_8l_8h_big_cond_dr03_dan_dilated_d\n",
            "      * imagetransformer_base_imagenet_tpu\n",
            "      * imagetransformer_base_rel\n",
            "      * imagetransformer_base_tpu\n",
            "      * imagetransformer_cifar10_base\n",
            "      * imagetransformer_cifar10_base_dmol\n",
            "      * imagetransformer_imagenet32_base\n",
            "      * imagetransformer_moe_tiny\n",
            "      * imagetransformer_sep_channels\n",
            "      * imagetransformer_sep_channels_12l_16h_imagenet_large\n",
            "      * imagetransformer_sep_channels_16l_16h_imgnet_lrg_loc\n",
            "      * imagetransformer_sep_channels_16l_16h_imgnet_lrg_loc_128\n",
            "      * imagetransformer_sep_channels_8l\n",
            "      * imagetransformer_sep_channels_8l_8h\n",
            "      * imagetransformer_sep_channels_8l_8h_local_and_global_att\n",
            "      * imagetransformer_sep_channels_8l_multipos3\n",
            "      * imagetransformer_sep_channels_8l_tpu\n",
            "      * imagetransformer_sep_output_channels_8l_local_and_global_att\n",
            "      * imagetransformer_tiny\n",
            "      * imagetransformer_tiny_tpu\n",
            "    imagetransformerpp:\n",
            "      * imagetransformerpp_base_10l_8h_big_uncond_dr03_dan\n",
            "      * imagetransformerpp_base_10l_8h_big_uncond_dr03_dan_a\n",
            "      * imagetransformerpp_base_10l_8h_big_uncond_dr03_dan_b\n",
            "      * imagetransformerpp_base_10l_8h_big_uncond_dr03_dan_g\n",
            "      * imagetransformerpp_base_12l_8h_big_uncond_dr03_dan_k\n",
            "      * imagetransformerpp_base_12l_8h_big_uncond_dr03_dan_l\n",
            "      * imagetransformerpp_base_12l_8h_big_uncond_dr03_dan_m\n",
            "      * imagetransformerpp_base_12l_8h_big_uncond_dr03_dan_m_bs1\n",
            "      * imagetransformerpp_base_12l_8h_big_uncond_dr03_dan_m_rel\n",
            "      * imagetransformerpp_base_12l_8h_big_uncond_dr03_dan_m_relsh\n",
            "      * imagetransformerpp_base_14l_8h_big_uncond_dr03_dan_eval\n",
            "      * imagetransformerpp_base_14l_8h_big_uncond_dr03_dan_p\n",
            "      * imagetransformerpp_base_14l_8h_big_uncond_dr03_dan_p_bs1\n",
            "      * imagetransformerpp_base_5l_8h_big_uncond_dr00_dan_g_bs1\n",
            "      * imagetransformerpp_base_5l_8h_dr00_dan_g_bs1_adafactor\n",
            "      * imagetransformerpp_base_6l_8h_dr00_dan_g_bs1_adafactor\n",
            "      * imagetransformerpp_base_8l_8h_big_cond_dr03_dan\n",
            "      * imagetransformerpp_base_8l_8h_big_cond_dr03_dan_a\n",
            "      * imagetransformerpp_sep_channels_8l_8h\n",
            "      * imagetransformerpp_tiny\n",
            "    img2img:\n",
            "      * img2img_transformer2d_base\n",
            "      * img2img_transformer2d_n103\n",
            "      * img2img_transformer2d_n24\n",
            "      * img2img_transformer2d_n3\n",
            "      * img2img_transformer2d_n31\n",
            "      * img2img_transformer2d_n44\n",
            "      * img2img_transformer2d_q1\n",
            "      * img2img_transformer2d_q2\n",
            "      * img2img_transformer2d_q3\n",
            "      * img2img_transformer2d_tiny\n",
            "      * img2img_transformer_b1\n",
            "      * img2img_transformer_b2\n",
            "      * img2img_transformer_b3\n",
            "      * img2img_transformer_b3_bs1\n",
            "      * img2img_transformer_b3_bs10\n",
            "      * img2img_transformer_b3_bs2\n",
            "      * img2img_transformer_b3_bs3\n",
            "      * img2img_transformer_b3_bs4\n",
            "      * img2img_transformer_b3_bs5\n",
            "      * img2img_transformer_b3_bs6\n",
            "      * img2img_transformer_b3_bs7\n",
            "      * img2img_transformer_b3_bs8\n",
            "      * img2img_transformer_b3_bs9\n",
            "      * img2img_transformer_base\n",
            "      * img2img_transformer_base_tpu\n",
            "      * img2img_transformer_dilated\n",
            "      * img2img_transformer_tiny\n",
            "      * img2img_transformer_tiny_tpu\n",
            "    lmx:\n",
            "      * lmx_base\n",
            "      * lmx_h1k_f4k\n",
            "      * lmx_h1k_f64k\n",
            "      * lmx_h2k_f8k\n",
            "      * lmx_h3k_f12k\n",
            "      * lmx_h4k_f16k\n",
            "      * lmx_moe\n",
            "      * lmx_moe_h1k_f4k_x32\n",
            "      * lmx_moe_h1k_f8k_x16\n",
            "      * lmx_relative\n",
            "      * lmx_relative_nopos\n",
            "    lstm:\n",
            "      * lstm_area_attention_base\n",
            "      * lstm_area_attention_char\n",
            "      * lstm_area_attention_char_enfr\n",
            "      * lstm_area_attention_enfr\n",
            "      * lstm_asr_v1\n",
            "      * lstm_attention\n",
            "      * lstm_bahdanau_attention\n",
            "      * lstm_bahdanau_attention_multi\n",
            "      * lstm_luong_attention\n",
            "      * lstm_luong_attention_multi\n",
            "      * lstm_seq2seq\n",
            "      * lstm_transduction\n",
            "    mqp:\n",
            "      * mqp_ende_base\n",
            "      * mqp_ende_h1_ff6784\n",
            "      * mqp_ende_h1_kv1024\n",
            "      * mqp_ende_h2_ff6400\n",
            "      * mqp_ende_h2_kv512\n",
            "      * mqp_ende_h2_kv64_ff6784\n",
            "      * mqp_ende_h4_ff5632\n",
            "      * mqp_ende_h4_kv256\n",
            "      * mqp_ende_h4_kv32_ff6784\n",
            "      * mqp_ende_h8_kv16_ff6784\n",
            "      * mqp_ende_local\n",
            "      * mqp_ende_mq8\n",
            "      * mqp_ende_mq8_ff5440\n",
            "      * mqp_ende_mq8_ff5440_local\n",
            "      * mqp_lm1b_base\n",
            "      * mqp_lm1b_h1_ff9984\n",
            "      * mqp_lm1b_h2_kv64_ff9984\n",
            "      * mqp_lm1b_h4_kv32_ff9984\n",
            "      * mqp_lm1b_h8_kv16_ff9984\n",
            "      * mqp_lm1b_mq8\n",
            "      * mqp_lm1b_mq8_ff9088\n",
            "    mtf:\n",
            "      * mtf_bitransformer_all_layers_tiny\n",
            "      * mtf_bitransformer_base\n",
            "      * mtf_bitransformer_tiny\n",
            "      * mtf_image_transformer_base\n",
            "      * mtf_image_transformer_base_cifar\n",
            "      * mtf_image_transformer_base_imagenet\n",
            "      * mtf_image_transformer_base_imagenet_mp\n",
            "      * mtf_image_transformer_base_imagenet_mp128\n",
            "      * mtf_image_transformer_base_imagenet_mp64\n",
            "      * mtf_image_transformer_base_imagenet_mp_sp\n",
            "      * mtf_image_transformer_base_single\n",
            "      * mtf_image_transformer_cifar_4x\n",
            "      * mtf_image_transformer_cifar_mp_4x\n",
            "      * mtf_image_transformer_length_sharded\n",
            "      * mtf_image_transformer_single\n",
            "      * mtf_image_transformer_tiny\n",
            "      * mtf_image_transformer_tiny_8gpu\n",
            "      * mtf_image_transformer_tiny_spatial1d\n",
            "      * mtf_image_transformer_tiny_spatial2d\n",
            "      * mtf_resnet_base\n",
            "      * mtf_resnet_base_cifar\n",
            "      * mtf_resnet_base_single\n",
            "      * mtf_resnet_single\n",
            "      * mtf_resnet_tiny\n",
            "      * mtf_transformer_base\n",
            "      * mtf_transformer_base_lm\n",
            "      * mtf_transformer_enc_single\n",
            "      * mtf_transformer_lm_baseline\n",
            "      * mtf_transformer_paper_lm_0\n",
            "      * mtf_transformer_paper_lm_1\n",
            "      * mtf_transformer_paper_lm_2\n",
            "      * mtf_transformer_paper_lm_3\n",
            "      * mtf_transformer_paper_lm_4\n",
            "      * mtf_transformer_paper_lm_5\n",
            "      * mtf_transformer_paper_lm_m1\n",
            "      * mtf_transformer_paper_tr_0\n",
            "      * mtf_transformer_paper_tr_0_a32\n",
            "      * mtf_transformer_paper_tr_0_mesh_128\n",
            "      * mtf_transformer_paper_tr_0_mesh_512\n",
            "      * mtf_transformer_paper_tr_0_mesh_8\n",
            "      * mtf_transformer_paper_tr_0_mesh_8_v2\n",
            "      * mtf_transformer_paper_tr_0_nf\n",
            "      * mtf_transformer_paper_tr_1\n",
            "      * mtf_transformer_paper_tr_2\n",
            "      * mtf_transformer_paper_tr_3\n",
            "      * mtf_transformer_paper_tr_4\n",
            "      * mtf_transformer_paper_tr_4_mesh_16_8\n",
            "      * mtf_transformer_paper_tr_6_mesh_64_8\n",
            "      * mtf_transformer_paper_tr_m1\n",
            "      * mtf_transformer_single\n",
            "      * mtf_transformer_tiny\n",
            "      * mtf_transformer_tiny_8gpu\n",
            "      * mtf_transformer_tiny_denoising\n",
            "      * mtf_transformer_tiny_lm\n",
            "      * mtf_unitransformer_all_layers_tiny\n",
            "      * mtf_unitransformer_base\n",
            "      * mtf_unitransformer_tiny\n",
            "    mtr:\n",
            "      * mtr_lm_dense\n",
            "      * mtr_lm_dense_0\n",
            "      * mtr_lm_dense_0_h1_16\n",
            "      * mtr_lm_dense_1\n",
            "      * mtr_lm_dense_2\n",
            "      * mtr_lm_dense_3\n",
            "      * mtr_lm_v1\n",
            "      * mtr_lm_v1_h1_8\n",
            "      * mtr_tr_dense_0\n",
            "      * mtr_tr_dense_0_extra_logit\n",
            "      * mtr_tr_dense_0_h16\n",
            "      * mtr_tr_dense_0_h1_1\n",
            "      * mtr_tr_dense_0_h1_16\n",
            "      * mtr_tr_dense_0_h1_8\n",
            "      * mtr_tr_dense_0_h2_16\n",
            "      * mtr_tr_dense_0_h4\n",
            "      * mtr_tr_dense_0_shared_kv\n",
            "      * mtr_tr_dense_1\n",
            "      * mtr_tr_dense_2\n",
            "      * mtr_tr_dense_3\n",
            "      * mtr_tr_dense_3_88\n",
            "      * mtr_tr_dense_3_fast\n",
            "      * mtr_tr_dense_local_0\n",
            "      * mtr_tr_dense_local_0_h1_16\n",
            "      * mtr_tr_dense_local_0_h1_16_shared\n",
            "      * mtr_tr_dense_local_0_h1_16_shared_kv\n",
            "      * mtr_tr_dense_local_0_h1_8_kv256\n",
            "      * mtr_tr_dense_local_0_w8\n",
            "      * mtr_tr_ende_deep\n",
            "      * mtr_tr_ende_v0\n",
            "      * mtr_tr_enfr_v0\n",
            "    nas:\n",
            "      * nas_seq2seq_base\n",
            "    neural:\n",
            "      * neural_gpu\n",
            "      * neural_stack\n",
            "    next:\n",
            "      * next_frame_ae\n",
            "      * next_frame_ae_tiny\n",
            "      * next_frame_basic_deterministic\n",
            "      * next_frame_basic_recurrent\n",
            "      * next_frame_basic_stochastic\n",
            "      * next_frame_basic_stochastic_discrete\n",
            "      * next_frame_basic_stochastic_discrete_long\n",
            "      * next_frame_emily\n",
            "      * next_frame_epva\n",
            "      * next_frame_glow_bair_qual\n",
            "      * next_frame_glow_bair_quant\n",
            "      * next_frame_glow_hparams\n",
            "      * next_frame_glow_shapes\n",
            "      * next_frame_l1\n",
            "      * next_frame_l2\n",
            "      * next_frame_pixel_noise\n",
            "      * next_frame_pixel_noise_long\n",
            "      * next_frame_sampling\n",
            "      * next_frame_sampling_stochastic\n",
            "      * next_frame_savp\n",
            "      * next_frame_savp_gan\n",
            "      * next_frame_savp_l2\n",
            "      * next_frame_savp_vae\n",
            "      * next_frame_small\n",
            "      * next_frame_sv2p\n",
            "      * next_frame_sv2p_atari\n",
            "      * next_frame_sv2p_atari_deterministic\n",
            "      * next_frame_sv2p_atari_softmax\n",
            "      * next_frame_sv2p_atari_softmax_deterministic\n",
            "      * next_frame_sv2p_cutoff\n",
            "      * next_frame_sv2p_discrete\n",
            "      * next_frame_sv2p_tiny\n",
            "      * next_frame_sv2p_tiny_external\n",
            "      * next_frame_tiny\n",
            "      * next_frame_tpu\n",
            "    ppo:\n",
            "      * ppo_atari_base\n",
            "      * ppo_base_v1\n",
            "      * ppo_discrete_action_base\n",
            "      * ppo_dist_params\n",
            "      * ppo_original_params\n",
            "      * ppo_original_params_gamma90\n",
            "      * ppo_original_params_gamma95\n",
            "      * ppo_original_tiny\n",
            "      * ppo_original_world_model\n",
            "      * ppo_original_world_model_stochastic_discrete\n",
            "      * ppo_pong_ae_base\n",
            "      * ppo_tiny_world_model\n",
            "      * ppo_ttt_params\n",
            "    resnet:\n",
            "      * resnet_101\n",
            "      * resnet_152\n",
            "      * resnet_18\n",
            "      * resnet_200\n",
            "      * resnet_34\n",
            "      * resnet_50\n",
            "      * resnet_cifar_15\n",
            "      * resnet_cifar_32\n",
            "      * resnet_cifar_32_td_unit_05_05\n",
            "      * resnet_cifar_32_td_unit_no_drop\n",
            "      * resnet_cifar_32_td_weight_05_05\n",
            "      * resnet_imagenet_102\n",
            "      * resnet_imagenet_34\n",
            "      * resnet_imagenet_34_td_unit_05_05\n",
            "      * resnet_imagenet_34_td_unit_no_drop\n",
            "      * resnet_imagenet_34_td_weight_05_05\n",
            "    revnet:\n",
            "      * revnet_104\n",
            "      * revnet_110_cifar\n",
            "      * revnet_164_cifar\n",
            "      * revnet_38_cifar\n",
            "    rlmf:\n",
            "      * rlmf_base\n",
            "      * rlmf_dist\n",
            "      * rlmf_dist_threshold\n",
            "      * rlmf_dqn_tiny\n",
            "      * rlmf_eval\n",
            "      * rlmf_eval_dist\n",
            "      * rlmf_eval_dist_threshold\n",
            "      * rlmf_original\n",
            "      * rlmf_tictactoe\n",
            "      * rlmf_tiny\n",
            "    shake:\n",
            "      * shake_shake_quick\n",
            "    shakeshake:\n",
            "      * shakeshake_big\n",
            "      * shakeshake_small\n",
            "      * shakeshake_tpu\n",
            "    sliced:\n",
            "      * sliced_gan\n",
            "    slicenet:\n",
            "      * slicenet_1\n",
            "      * slicenet_1noam\n",
            "      * slicenet_1tiny\n",
            "    super:\n",
            "      * super_lm_b8k\n",
            "      * super_lm_base\n",
            "      * super_lm_big\n",
            "      * super_lm_big_tpu\n",
            "      * super_lm_conv\n",
            "      * super_lm_high_mix\n",
            "      * super_lm_low_mix\n",
            "      * super_lm_moe\n",
            "      * super_lm_moe_4b_diet\n",
            "      * super_lm_moe_h4\n",
            "      * super_lm_tpu\n",
            "      * super_lm_tpu_memtest\n",
            "    text:\n",
            "      * text_cnn_base\n",
            "    transformer:\n",
            "      * transformer_ada_lmpackedbase\n",
            "      * transformer_ada_lmpackedbase_dialog\n",
            "      * transformer_ada_lmpackedbase_relative\n",
            "      * transformer_ae_a3\n",
            "      * transformer_ae_a6\n",
            "      * transformer_ae_a8\n",
            "      * transformer_ae_base\n",
            "      * transformer_ae_base_ablation_1\n",
            "      * transformer_ae_base_ablation_2\n",
            "      * transformer_ae_base_ablation_3\n",
            "      * transformer_ae_base_ablation_4\n",
            "      * transformer_ae_base_ablation_5\n",
            "      * transformer_ae_base_iaf\n",
            "      * transformer_ae_base_noatt\n",
            "      * transformer_ae_base_tpu\n",
            "      * transformer_ae_small\n",
            "      * transformer_ae_small_noatt\n",
            "      * transformer_base\n",
            "      * transformer_base_bs1\n",
            "      * transformer_base_bs10\n",
            "      * transformer_base_bs2\n",
            "      * transformer_base_bs3\n",
            "      * transformer_base_bs4\n",
            "      * transformer_base_bs5\n",
            "      * transformer_base_bs6\n",
            "      * transformer_base_bs7\n",
            "      * transformer_base_bs8\n",
            "      * transformer_base_bs9\n",
            "      * transformer_base_multistep8\n",
            "      * transformer_base_single_gpu\n",
            "      * transformer_base_v1\n",
            "      * transformer_base_v2\n",
            "      * transformer_base_v3\n",
            "      * transformer_base_vq1_16_nb1_packed_dan_b01_scales\n",
            "      * transformer_base_vq1_16_nb1_packed_nda_b01_scales\n",
            "      * transformer_base_vq1_16_nb1_packed_nda_b01_scales_dialog\n",
            "      * transformer_base_vq_ada_32ex_packed\n",
            "      * transformer_big\n",
            "      * transformer_big_bs1\n",
            "      * transformer_big_dr1\n",
            "      * transformer_big_dr2\n",
            "      * transformer_big_enfr\n",
            "      * transformer_big_enfr_tpu\n",
            "      * transformer_big_single_gpu\n",
            "      * transformer_big_tpu\n",
            "      * transformer_cifar10_memory_v0\n",
            "      * transformer_clean\n",
            "      * transformer_clean_big\n",
            "      * transformer_clean_big_tpu\n",
            "      * transformer_common_voice\n",
            "      * transformer_common_voice_tpu\n",
            "      * transformer_dr0\n",
            "      * transformer_dr2\n",
            "      * transformer_fairseq_fp16_activation_big\n",
            "      * transformer_ff1024\n",
            "      * transformer_ff4096\n",
            "      * transformer_h1\n",
            "      * transformer_h16\n",
            "      * transformer_h32\n",
            "      * transformer_h4\n",
            "      * transformer_hs1024\n",
            "      * transformer_hs256\n",
            "      * transformer_imagenet64_memory_v0\n",
            "      * transformer_k128\n",
            "      * transformer_k256\n",
            "      * transformer_l10\n",
            "      * transformer_l2\n",
            "      * transformer_l4\n",
            "      * transformer_l8\n",
            "      * transformer_librispeech\n",
            "      * transformer_librispeech_tpu\n",
            "      * transformer_librispeech_tpu_v1\n",
            "      * transformer_librispeech_tpu_v2\n",
            "      * transformer_librispeech_v1\n",
            "      * transformer_librispeech_v2\n",
            "      * transformer_lm_tpu_0\n",
            "      * transformer_lm_tpu_1\n",
            "      * transformer_ls0\n",
            "      * transformer_ls2\n",
            "      * transformer_mlperf_tpu\n",
            "      * transformer_moe_12k\n",
            "      * transformer_moe_2k\n",
            "      * transformer_moe_8k\n",
            "      * transformer_moe_8k_lm\n",
            "      * transformer_moe_base\n",
            "      * transformer_moe_prepend_8k\n",
            "      * transformer_nat_base\n",
            "      * transformer_nat_big\n",
            "      * transformer_nat_small\n",
            "      * transformer_packed_tpu\n",
            "      * transformer_parameter_attention_a\n",
            "      * transformer_parameter_attention_b\n",
            "      * transformer_parsing_base\n",
            "      * transformer_parsing_big\n",
            "      * transformer_parsing_ice\n",
            "      * transformer_prepend\n",
            "      * transformer_prepend_v1\n",
            "      * transformer_prepend_v2\n",
            "      * transformer_relative\n",
            "      * transformer_relative_big\n",
            "      * transformer_relative_tiny\n",
            "      * transformer_revnet_base\n",
            "      * transformer_revnet_big\n",
            "      * transformer_sketch\n",
            "      * transformer_small\n",
            "      * transformer_small_tpu\n",
            "      * transformer_supervised_attention\n",
            "      * transformer_symshard_base\n",
            "      * transformer_symshard_h4\n",
            "      * transformer_symshard_lm_0\n",
            "      * transformer_symshard_sh4\n",
            "      * transformer_tall\n",
            "      * transformer_tall_big\n",
            "      * transformer_tall_finetune_textclass\n",
            "      * transformer_tall_finetune_tied\n",
            "      * transformer_tall_finetune_uniencdec\n",
            "      * transformer_tall_pretrain_lm\n",
            "      * transformer_tall_pretrain_lm_tpu\n",
            "      * transformer_tall_pretrain_lm_tpu_adafactor\n",
            "      * transformer_tall_pretrain_lm_tpu_adafactor_large\n",
            "      * transformer_tall_train_tied\n",
            "      * transformer_tall_train_uniencdec\n",
            "      * transformer_teeny\n",
            "      * transformer_test\n",
            "      * transformer_timeseries\n",
            "      * transformer_timeseries_tpu\n",
            "      * transformer_tiny\n",
            "      * transformer_tiny_bs1\n",
            "      * transformer_tiny_bs2\n",
            "      * transformer_tiny_bs3\n",
            "      * transformer_tiny_tpu\n",
            "      * transformer_topk_16_packed\n",
            "      * transformer_tpu\n",
            "      * transformer_tpu_1b\n",
            "      * transformer_tpu_bf16_activation\n",
            "      * transformer_tpu_with_conv\n",
            "      * transformer_wikitext103_l16k_memory_v0\n",
            "      * transformer_wikitext103_l4k_memory_v0\n",
            "      * transformer_wikitext103_l4k_v0\n",
            "    universal:\n",
            "      * universal_transformer_base\n",
            "      * universal_transformer_base_fp16\n",
            "      * universal_transformer_base_tpu\n",
            "      * universal_transformer_big\n",
            "      * universal_transformer_dwa_base\n",
            "      * universal_transformer_gru_base\n",
            "      * universal_transformer_highway_base\n",
            "      * universal_transformer_lstm_base\n",
            "      * universal_transformer_lstm_tall\n",
            "      * universal_transformer_mix_after_ut_base\n",
            "      * universal_transformer_mix_before_ut_base\n",
            "      * universal_transformer_position_random_timing_tiny\n",
            "      * universal_transformer_position_step_timing_tiny\n",
            "      * universal_transformer_sepconv_base\n",
            "      * universal_transformer_sepconv_big\n",
            "      * universal_transformer_sepconv_tiny\n",
            "      * universal_transformer_skip_base\n",
            "      * universal_transformer_small\n",
            "      * universal_transformer_small_dropconnect\n",
            "      * universal_transformer_step_sinusoid_timing_tiny\n",
            "      * universal_transformer_tall\n",
            "      * universal_transformer_teeny\n",
            "      * universal_transformer_tiny\n",
            "    vqa:\n",
            "      * vqa_attention_base\n",
            "      * vqa_attention_drop01_dna\n",
            "      * vqa_attention_feature_base\n",
            "      * vqa_attention_feature_batch1024\n",
            "      * vqa_attention_feature_batch1024_dnz\n",
            "      * vqa_attention_feature_batch1024_dnz_l2\n",
            "      * vqa_attention_feature_batch1024_dnz_noscaledp\n",
            "      * vqa_attention_feature_batch1024_drop01\n",
            "      * vqa_attention_feature_batch1024_drop01_dna\n",
            "      * vqa_attention_feature_batch1024_drop01_dna_concat\n",
            "      * vqa_attention_feature_batch1024_lstmlayernorm\n",
            "      * vqa_attention_feature_batch1024_numglimps1\n",
            "      * vqa_attention_feature_batch512\n",
            "      * vqa_attention_feature_dna\n",
            "      * vqa_attention_feature_dnz\n",
            "      * vqa_attention_feature_dnz_l2\n",
            "      * vqa_attention_feature_dnz_noscaledp\n",
            "      * vqa_attention_feature_hidden1024\n",
            "      * vqa_attention_feature_imagefeat1024\n",
            "      * vqa_attention_feature_imagefeat512\n",
            "      * vqa_attention_feature_initializer\n",
            "      * vqa_attention_feature_lstmlayernorm\n",
            "      * vqa_attention_feature_nonormalization\n",
            "      * vqa_attention_feature_numglimps1\n",
            "      * vqa_attention_numglimps1\n",
            "      * vqa_recurrent_self_attention_base\n",
            "      * vqa_recurrent_self_attention_big\n",
            "      * vqa_recurrent_self_attention_big_l4\n",
            "      * vqa_recurrent_self_attention_drop1\n",
            "      * vqa_recurrent_self_attention_drop3\n",
            "      * vqa_recurrent_self_attention_gru\n",
            "      * vqa_recurrent_self_attention_highway\n",
            "      * vqa_recurrent_self_attention_l4\n",
            "      * vqa_recurrent_self_attention_l8\n",
            "      * vqa_recurrent_self_attention_ls2\n",
            "      * vqa_recurrent_self_attention_mix_before_ut\n",
            "      * vqa_recurrent_self_attention_small\n",
            "      * vqa_self_attention_base\n",
            "      * vqa_self_attention_feature\n",
            "      * vqa_self_attention_feature_batch1024\n",
            "      * vqa_self_attention_feature_batch1024_big\n",
            "      * vqa_self_attention_feature_batch1024_drop03\n",
            "      * vqa_self_attention_feature_batch1024_exp\n",
            "      * vqa_self_attention_feature_batch1024_hidden6\n",
            "      * vqa_self_attention_feature_batch1024_hidden6_big\n",
            "      * vqa_self_attention_feature_lr5\n",
            "    wiki:\n",
            "      * wiki_2x2_base\n",
            "      * wiki_2x2_local\n",
            "      * wiki_2x2_v1\n",
            "    xception:\n",
            "      * xception_base\n",
            "      * xception_tiny\n",
            "      * xception_tiny_tpu\n",
            "    xmoe2:\n",
            "      * xmoe2_dense\n",
            "      * xmoe2_dense_0\n",
            "      * xmoe2_dense_1\n",
            "      * xmoe2_dense_2\n",
            "      * xmoe2_dense_3\n",
            "      * xmoe2_tiny\n",
            "      * xmoe2_v1\n",
            "      * xmoe2_v1_l4k\n",
            "      * xmoe2_v1_l4k_compressed_c4\n",
            "      * xmoe2_v1_l4k_compressed_c8\n",
            "      * xmoe2_v1_l4k_global_only\n",
            "      * xmoe2_v1_l4k_local_only\n",
            "      * xmoe2_v1_x128\n",
            "    xmoe:\n",
            "      * xmoe_2d\n",
            "      * xmoe_2d_c15\n",
            "      * xmoe_2d_debug\n",
            "      * xmoe_2d_x64\n",
            "      * xmoe_dense_4k\n",
            "      * xmoe_dense_64k\n",
            "      * xmoe_dense_8k\n",
            "      * xmoe_top_2\n",
            "      * xmoe_top_2_c15\n",
            "      * xmoe_tr_1d\n",
            "      * xmoe_tr_2d\n",
            "      * xmoe_tr_dense_2k\n",
            "      * xmoe_tr_dense_32k\n",
            "\n",
            "  RangedHParams:\n",
            "    adaptive:\n",
            "      * adaptive_universal_transformer_base_range\n",
            "    autoencoder:\n",
            "      * autoencoder_discrete_pong_range\n",
            "      * autoencoder_range\n",
            "    basic1:\n",
            "      * basic1\n",
            "    basic:\n",
            "      * basic_moe_range\n",
            "    imagetransformer:\n",
            "      * imagetransformer_cifar_tpu_range\n",
            "    next:\n",
            "      * next_frame_ae_range\n",
            "      * next_frame_base_range\n",
            "      * next_frame_clipgrad_range\n",
            "      * next_frame_doubling_range\n",
            "      * next_frame_stochastic_discrete_latent_range\n",
            "      * next_frame_stochastic_discrete_range\n",
            "      * next_frame_xent_cutoff_range\n",
            "    revnet:\n",
            "      * revnet_range\n",
            "    rlmf:\n",
            "      * rlmf_5runs\n",
            "      * rlmf_5runs_atari\n",
            "    slicenet1:\n",
            "      * slicenet1\n",
            "    transformer:\n",
            "      * transformer_base_range\n",
            "      * transformer_tiny_tpu_range\n",
            "      * transformer_tpu_range\n",
            "    universal:\n",
            "      * universal_transformer_base_range\n",
            "    vqa:\n",
            "      * vqa_attention_base_range\n",
            "\n",
            "  Problems:\n",
            "    algorithmic:\n",
            "      * algorithmic_addition_binary40\n",
            "      * algorithmic_addition_decimal40\n",
            "      * algorithmic_cipher_shift200\n",
            "      * algorithmic_cipher_shift5\n",
            "      * algorithmic_cipher_vigenere200\n",
            "      * algorithmic_cipher_vigenere5\n",
            "      * algorithmic_identity_binary40\n",
            "      * algorithmic_identity_decimal40\n",
            "      * algorithmic_identity_vocab95_train20_eval30\n",
            "      * algorithmic_math_deepmind_all\n",
            "      * algorithmic_math_two_variables\n",
            "      * algorithmic_multiplication_binary40\n",
            "      * algorithmic_multiplication_decimal40\n",
            "      * algorithmic_reverse_binary40\n",
            "      * algorithmic_reverse_binary40_test\n",
            "      * algorithmic_reverse_decimal40\n",
            "      * algorithmic_reverse_nlplike32k\n",
            "      * algorithmic_reverse_nlplike8k\n",
            "      * algorithmic_shift_decimal40\n",
            "      * algorithmic_sort_problem\n",
            "    audio:\n",
            "      * audio_timit_characters_tune\n",
            "      * audio_timit_tokens8k_test\n",
            "      * audio_timit_tokens8k_tune\n",
            "    babi:\n",
            "      * babi_qa_concat_all_tasks_10k\n",
            "      * babi_qa_concat_all_tasks_1k\n",
            "      * babi_qa_concat_task10_10k\n",
            "      * babi_qa_concat_task10_1k\n",
            "      * babi_qa_concat_task11_10k\n",
            "      * babi_qa_concat_task11_1k\n",
            "      * babi_qa_concat_task12_10k\n",
            "      * babi_qa_concat_task12_1k\n",
            "      * babi_qa_concat_task13_10k\n",
            "      * babi_qa_concat_task13_1k\n",
            "      * babi_qa_concat_task14_10k\n",
            "      * babi_qa_concat_task14_1k\n",
            "      * babi_qa_concat_task15_10k\n",
            "      * babi_qa_concat_task15_1k\n",
            "      * babi_qa_concat_task16_10k\n",
            "      * babi_qa_concat_task16_1k\n",
            "      * babi_qa_concat_task17_10k\n",
            "      * babi_qa_concat_task17_1k\n",
            "      * babi_qa_concat_task18_10k\n",
            "      * babi_qa_concat_task18_1k\n",
            "      * babi_qa_concat_task19_10k\n",
            "      * babi_qa_concat_task19_1k\n",
            "      * babi_qa_concat_task1_10k\n",
            "      * babi_qa_concat_task1_1k\n",
            "      * babi_qa_concat_task20_10k\n",
            "      * babi_qa_concat_task20_1k\n",
            "      * babi_qa_concat_task2_10k\n",
            "      * babi_qa_concat_task2_1k\n",
            "      * babi_qa_concat_task3_10k\n",
            "      * babi_qa_concat_task3_1k\n",
            "      * babi_qa_concat_task4_10k\n",
            "      * babi_qa_concat_task4_1k\n",
            "      * babi_qa_concat_task5_10k\n",
            "      * babi_qa_concat_task5_1k\n",
            "      * babi_qa_concat_task6_10k\n",
            "      * babi_qa_concat_task6_1k\n",
            "      * babi_qa_concat_task7_10k\n",
            "      * babi_qa_concat_task7_1k\n",
            "      * babi_qa_concat_task8_10k\n",
            "      * babi_qa_concat_task8_1k\n",
            "      * babi_qa_concat_task9_10k\n",
            "      * babi_qa_concat_task9_1k\n",
            "    cola:\n",
            "      * cola\n",
            "      * cola_characters\n",
            "    common:\n",
            "      * common_voice\n",
            "      * common_voice_clean\n",
            "      * common_voice_noisy\n",
            "      * common_voice_train_full_test_clean\n",
            "    copy:\n",
            "      * copy_sequence\n",
            "      * copy_sequence_small\n",
            "    flip:\n",
            "      * flip_bi_gram_sequence\n",
            "    genomics:\n",
            "      * genomics_expression_cage10\n",
            "      * genomics_expression_gm12878\n",
            "      * genomics_expression_l262k\n",
            "    github:\n",
            "      * github_function_docstring\n",
            "    gym:\n",
            "      * gym_air_raid-v0_random\n",
            "      * gym_air_raid-v4_random\n",
            "      * gym_air_raid_deterministic-v0_random\n",
            "      * gym_air_raid_deterministic-v4_random\n",
            "      * gym_air_raid_no_frameskip-v0_random\n",
            "      * gym_air_raid_no_frameskip-v4_random\n",
            "      * gym_alien-v0_random\n",
            "      * gym_alien-v4_random\n",
            "      * gym_alien_deterministic-v0_random\n",
            "      * gym_alien_deterministic-v4_random\n",
            "      * gym_alien_no_frameskip-v0_random\n",
            "      * gym_alien_no_frameskip-v4_random\n",
            "      * gym_amidar-v0_random\n",
            "      * gym_amidar-v4_random\n",
            "      * gym_amidar_deterministic-v0_random\n",
            "      * gym_amidar_deterministic-v4_random\n",
            "      * gym_amidar_no_frameskip-v0_random\n",
            "      * gym_amidar_no_frameskip-v4_random\n",
            "      * gym_assault-v0_random\n",
            "      * gym_assault-v4_random\n",
            "      * gym_assault_deterministic-v0_random\n",
            "      * gym_assault_deterministic-v4_random\n",
            "      * gym_assault_no_frameskip-v0_random\n",
            "      * gym_assault_no_frameskip-v4_random\n",
            "      * gym_asterix-v0_random\n",
            "      * gym_asterix-v4_random\n",
            "      * gym_asterix_deterministic-v0_random\n",
            "      * gym_asterix_deterministic-v4_random\n",
            "      * gym_asterix_no_frameskip-v0_random\n",
            "      * gym_asterix_no_frameskip-v4_random\n",
            "      * gym_asteroids-v0_random\n",
            "      * gym_asteroids-v4_random\n",
            "      * gym_asteroids_deterministic-v0_random\n",
            "      * gym_asteroids_deterministic-v4_random\n",
            "      * gym_asteroids_no_frameskip-v0_random\n",
            "      * gym_asteroids_no_frameskip-v4_random\n",
            "      * gym_atlantis-v0_random\n",
            "      * gym_atlantis-v4_random\n",
            "      * gym_atlantis_deterministic-v0_random\n",
            "      * gym_atlantis_deterministic-v4_random\n",
            "      * gym_atlantis_no_frameskip-v0_random\n",
            "      * gym_atlantis_no_frameskip-v4_random\n",
            "      * gym_bank_heist-v0_random\n",
            "      * gym_bank_heist-v4_random\n",
            "      * gym_bank_heist_deterministic-v0_random\n",
            "      * gym_bank_heist_deterministic-v4_random\n",
            "      * gym_bank_heist_no_frameskip-v0_random\n",
            "      * gym_bank_heist_no_frameskip-v4_random\n",
            "      * gym_battle_zone-v0_random\n",
            "      * gym_battle_zone-v4_random\n",
            "      * gym_battle_zone_deterministic-v0_random\n",
            "      * gym_battle_zone_deterministic-v4_random\n",
            "      * gym_battle_zone_no_frameskip-v0_random\n",
            "      * gym_battle_zone_no_frameskip-v4_random\n",
            "      * gym_beam_rider-v0_random\n",
            "      * gym_beam_rider-v4_random\n",
            "      * gym_beam_rider_deterministic-v0_random\n",
            "      * gym_beam_rider_deterministic-v4_random\n",
            "      * gym_beam_rider_no_frameskip-v0_random\n",
            "      * gym_beam_rider_no_frameskip-v4_random\n",
            "      * gym_berzerk-v0_random\n",
            "      * gym_berzerk-v4_random\n",
            "      * gym_berzerk_deterministic-v0_random\n",
            "      * gym_berzerk_deterministic-v4_random\n",
            "      * gym_berzerk_no_frameskip-v0_random\n",
            "      * gym_berzerk_no_frameskip-v4_random\n",
            "      * gym_bowling-v0_random\n",
            "      * gym_bowling-v4_random\n",
            "      * gym_bowling_deterministic-v0_random\n",
            "      * gym_bowling_deterministic-v4_random\n",
            "      * gym_bowling_no_frameskip-v0_random\n",
            "      * gym_bowling_no_frameskip-v4_random\n",
            "      * gym_boxing-v0_random\n",
            "      * gym_boxing-v4_random\n",
            "      * gym_boxing_deterministic-v0_random\n",
            "      * gym_boxing_deterministic-v4_random\n",
            "      * gym_boxing_no_frameskip-v0_random\n",
            "      * gym_boxing_no_frameskip-v4_random\n",
            "      * gym_breakout-v0_random\n",
            "      * gym_breakout-v4_random\n",
            "      * gym_breakout_deterministic-v0_random\n",
            "      * gym_breakout_deterministic-v4_random\n",
            "      * gym_breakout_no_frameskip-v0_random\n",
            "      * gym_breakout_no_frameskip-v4_random\n",
            "      * gym_carnival-v0_random\n",
            "      * gym_carnival-v4_random\n",
            "      * gym_carnival_deterministic-v0_random\n",
            "      * gym_carnival_deterministic-v4_random\n",
            "      * gym_carnival_no_frameskip-v0_random\n",
            "      * gym_carnival_no_frameskip-v4_random\n",
            "      * gym_centipede-v0_random\n",
            "      * gym_centipede-v4_random\n",
            "      * gym_centipede_deterministic-v0_random\n",
            "      * gym_centipede_deterministic-v4_random\n",
            "      * gym_centipede_no_frameskip-v0_random\n",
            "      * gym_centipede_no_frameskip-v4_random\n",
            "      * gym_chopper_command-v0_random\n",
            "      * gym_chopper_command-v4_random\n",
            "      * gym_chopper_command_deterministic-v0_random\n",
            "      * gym_chopper_command_deterministic-v4_random\n",
            "      * gym_chopper_command_no_frameskip-v0_random\n",
            "      * gym_chopper_command_no_frameskip-v4_random\n",
            "      * gym_crazy_climber-v0_random\n",
            "      * gym_crazy_climber-v4_random\n",
            "      * gym_crazy_climber_deterministic-v0_random\n",
            "      * gym_crazy_climber_deterministic-v4_random\n",
            "      * gym_crazy_climber_no_frameskip-v0_random\n",
            "      * gym_crazy_climber_no_frameskip-v4_random\n",
            "      * gym_demon_attack-v0_random\n",
            "      * gym_demon_attack-v4_random\n",
            "      * gym_demon_attack_deterministic-v0_random\n",
            "      * gym_demon_attack_deterministic-v4_random\n",
            "      * gym_demon_attack_no_frameskip-v0_random\n",
            "      * gym_demon_attack_no_frameskip-v4_random\n",
            "      * gym_double_dunk-v0_random\n",
            "      * gym_double_dunk-v4_random\n",
            "      * gym_double_dunk_deterministic-v0_random\n",
            "      * gym_double_dunk_deterministic-v4_random\n",
            "      * gym_double_dunk_no_frameskip-v0_random\n",
            "      * gym_double_dunk_no_frameskip-v4_random\n",
            "      * gym_elevator_action-v0_random\n",
            "      * gym_elevator_action-v4_random\n",
            "      * gym_elevator_action_deterministic-v0_random\n",
            "      * gym_elevator_action_deterministic-v4_random\n",
            "      * gym_elevator_action_no_frameskip-v0_random\n",
            "      * gym_elevator_action_no_frameskip-v4_random\n",
            "      * gym_enduro-v0_random\n",
            "      * gym_enduro-v4_random\n",
            "      * gym_enduro_deterministic-v0_random\n",
            "      * gym_enduro_deterministic-v4_random\n",
            "      * gym_enduro_no_frameskip-v0_random\n",
            "      * gym_enduro_no_frameskip-v4_random\n",
            "      * gym_fishing_derby-v0_random\n",
            "      * gym_fishing_derby-v4_random\n",
            "      * gym_fishing_derby_deterministic-v0_random\n",
            "      * gym_fishing_derby_deterministic-v4_random\n",
            "      * gym_fishing_derby_no_frameskip-v0_random\n",
            "      * gym_fishing_derby_no_frameskip-v4_random\n",
            "      * gym_freeway-v0_random\n",
            "      * gym_freeway-v4_random\n",
            "      * gym_freeway_deterministic-v0_random\n",
            "      * gym_freeway_deterministic-v4_random\n",
            "      * gym_freeway_no_frameskip-v0_random\n",
            "      * gym_freeway_no_frameskip-v4_random\n",
            "      * gym_frostbite-v0_random\n",
            "      * gym_frostbite-v4_random\n",
            "      * gym_frostbite_deterministic-v0_random\n",
            "      * gym_frostbite_deterministic-v4_random\n",
            "      * gym_frostbite_no_frameskip-v0_random\n",
            "      * gym_frostbite_no_frameskip-v4_random\n",
            "      * gym_gopher-v0_random\n",
            "      * gym_gopher-v4_random\n",
            "      * gym_gopher_deterministic-v0_random\n",
            "      * gym_gopher_deterministic-v4_random\n",
            "      * gym_gopher_no_frameskip-v0_random\n",
            "      * gym_gopher_no_frameskip-v4_random\n",
            "      * gym_gravitar-v0_random\n",
            "      * gym_gravitar-v4_random\n",
            "      * gym_gravitar_deterministic-v0_random\n",
            "      * gym_gravitar_deterministic-v4_random\n",
            "      * gym_gravitar_no_frameskip-v0_random\n",
            "      * gym_gravitar_no_frameskip-v4_random\n",
            "      * gym_hero-v0_random\n",
            "      * gym_hero-v4_random\n",
            "      * gym_hero_deterministic-v0_random\n",
            "      * gym_hero_deterministic-v4_random\n",
            "      * gym_hero_no_frameskip-v0_random\n",
            "      * gym_hero_no_frameskip-v4_random\n",
            "      * gym_ice_hockey-v0_random\n",
            "      * gym_ice_hockey-v4_random\n",
            "      * gym_ice_hockey_deterministic-v0_random\n",
            "      * gym_ice_hockey_deterministic-v4_random\n",
            "      * gym_ice_hockey_no_frameskip-v0_random\n",
            "      * gym_ice_hockey_no_frameskip-v4_random\n",
            "      * gym_jamesbond-v0_random\n",
            "      * gym_jamesbond-v4_random\n",
            "      * gym_jamesbond_deterministic-v0_random\n",
            "      * gym_jamesbond_deterministic-v4_random\n",
            "      * gym_jamesbond_no_frameskip-v0_random\n",
            "      * gym_jamesbond_no_frameskip-v4_random\n",
            "      * gym_journey_escape-v0_random\n",
            "      * gym_journey_escape-v4_random\n",
            "      * gym_journey_escape_deterministic-v0_random\n",
            "      * gym_journey_escape_deterministic-v4_random\n",
            "      * gym_journey_escape_no_frameskip-v0_random\n",
            "      * gym_journey_escape_no_frameskip-v4_random\n",
            "      * gym_kangaroo-v0_random\n",
            "      * gym_kangaroo-v4_random\n",
            "      * gym_kangaroo_deterministic-v0_random\n",
            "      * gym_kangaroo_deterministic-v4_random\n",
            "      * gym_kangaroo_no_frameskip-v0_random\n",
            "      * gym_kangaroo_no_frameskip-v4_random\n",
            "      * gym_krull-v0_random\n",
            "      * gym_krull-v4_random\n",
            "      * gym_krull_deterministic-v0_random\n",
            "      * gym_krull_deterministic-v4_random\n",
            "      * gym_krull_no_frameskip-v0_random\n",
            "      * gym_krull_no_frameskip-v4_random\n",
            "      * gym_kung_fu_master-v0_random\n",
            "      * gym_kung_fu_master-v4_random\n",
            "      * gym_kung_fu_master_deterministic-v0_random\n",
            "      * gym_kung_fu_master_deterministic-v4_random\n",
            "      * gym_kung_fu_master_no_frameskip-v0_random\n",
            "      * gym_kung_fu_master_no_frameskip-v4_random\n",
            "      * gym_montezuma_revenge-v0_random\n",
            "      * gym_montezuma_revenge-v4_random\n",
            "      * gym_montezuma_revenge_deterministic-v0_random\n",
            "      * gym_montezuma_revenge_deterministic-v4_random\n",
            "      * gym_montezuma_revenge_no_frameskip-v0_random\n",
            "      * gym_montezuma_revenge_no_frameskip-v4_random\n",
            "      * gym_ms_pacman-v0_random\n",
            "      * gym_ms_pacman-v4_random\n",
            "      * gym_ms_pacman_deterministic-v0_random\n",
            "      * gym_ms_pacman_deterministic-v4_random\n",
            "      * gym_ms_pacman_no_frameskip-v0_random\n",
            "      * gym_ms_pacman_no_frameskip-v4_random\n",
            "      * gym_name_this_game-v0_random\n",
            "      * gym_name_this_game-v4_random\n",
            "      * gym_name_this_game_deterministic-v0_random\n",
            "      * gym_name_this_game_deterministic-v4_random\n",
            "      * gym_name_this_game_no_frameskip-v0_random\n",
            "      * gym_name_this_game_no_frameskip-v4_random\n",
            "      * gym_phoenix-v0_random\n",
            "      * gym_phoenix-v4_random\n",
            "      * gym_phoenix_deterministic-v0_random\n",
            "      * gym_phoenix_deterministic-v4_random\n",
            "      * gym_phoenix_no_frameskip-v0_random\n",
            "      * gym_phoenix_no_frameskip-v4_random\n",
            "      * gym_pitfall-v0_random\n",
            "      * gym_pitfall-v4_random\n",
            "      * gym_pitfall_deterministic-v0_random\n",
            "      * gym_pitfall_deterministic-v4_random\n",
            "      * gym_pitfall_no_frameskip-v0_random\n",
            "      * gym_pitfall_no_frameskip-v4_random\n",
            "      * gym_pong-v0_random\n",
            "      * gym_pong-v4_random\n",
            "      * gym_pong_deterministic-v0_random\n",
            "      * gym_pong_deterministic-v4_random\n",
            "      * gym_pong_no_frameskip-v0_random\n",
            "      * gym_pong_no_frameskip-v4_random\n",
            "      * gym_pooyan-v0_random\n",
            "      * gym_pooyan-v4_random\n",
            "      * gym_pooyan_deterministic-v0_random\n",
            "      * gym_pooyan_deterministic-v4_random\n",
            "      * gym_pooyan_no_frameskip-v0_random\n",
            "      * gym_pooyan_no_frameskip-v4_random\n",
            "      * gym_private_eye-v0_random\n",
            "      * gym_private_eye-v4_random\n",
            "      * gym_private_eye_deterministic-v0_random\n",
            "      * gym_private_eye_deterministic-v4_random\n",
            "      * gym_private_eye_no_frameskip-v0_random\n",
            "      * gym_private_eye_no_frameskip-v4_random\n",
            "      * gym_qbert-v0_random\n",
            "      * gym_qbert-v4_random\n",
            "      * gym_qbert_deterministic-v0_random\n",
            "      * gym_qbert_deterministic-v4_random\n",
            "      * gym_qbert_no_frameskip-v0_random\n",
            "      * gym_qbert_no_frameskip-v4_random\n",
            "      * gym_riverraid-v0_random\n",
            "      * gym_riverraid-v4_random\n",
            "      * gym_riverraid_deterministic-v0_random\n",
            "      * gym_riverraid_deterministic-v4_random\n",
            "      * gym_riverraid_no_frameskip-v0_random\n",
            "      * gym_riverraid_no_frameskip-v4_random\n",
            "      * gym_road_runner-v0_random\n",
            "      * gym_road_runner-v4_random\n",
            "      * gym_road_runner_deterministic-v0_random\n",
            "      * gym_road_runner_deterministic-v4_random\n",
            "      * gym_road_runner_no_frameskip-v0_random\n",
            "      * gym_road_runner_no_frameskip-v4_random\n",
            "      * gym_robotank-v0_random\n",
            "      * gym_robotank-v4_random\n",
            "      * gym_robotank_deterministic-v0_random\n",
            "      * gym_robotank_deterministic-v4_random\n",
            "      * gym_robotank_no_frameskip-v0_random\n",
            "      * gym_robotank_no_frameskip-v4_random\n",
            "      * gym_seaquest-v0_random\n",
            "      * gym_seaquest-v4_random\n",
            "      * gym_seaquest_deterministic-v0_random\n",
            "      * gym_seaquest_deterministic-v4_random\n",
            "      * gym_seaquest_no_frameskip-v0_random\n",
            "      * gym_seaquest_no_frameskip-v4_random\n",
            "      * gym_skiing-v0_random\n",
            "      * gym_skiing-v4_random\n",
            "      * gym_skiing_deterministic-v0_random\n",
            "      * gym_skiing_deterministic-v4_random\n",
            "      * gym_skiing_no_frameskip-v0_random\n",
            "      * gym_skiing_no_frameskip-v4_random\n",
            "      * gym_solaris-v0_random\n",
            "      * gym_solaris-v4_random\n",
            "      * gym_solaris_deterministic-v0_random\n",
            "      * gym_solaris_deterministic-v4_random\n",
            "      * gym_solaris_no_frameskip-v0_random\n",
            "      * gym_solaris_no_frameskip-v4_random\n",
            "      * gym_space_invaders-v0_random\n",
            "      * gym_space_invaders-v4_random\n",
            "      * gym_space_invaders_deterministic-v0_random\n",
            "      * gym_space_invaders_deterministic-v4_random\n",
            "      * gym_space_invaders_no_frameskip-v0_random\n",
            "      * gym_space_invaders_no_frameskip-v4_random\n",
            "      * gym_star_gunner-v0_random\n",
            "      * gym_star_gunner-v4_random\n",
            "      * gym_star_gunner_deterministic-v0_random\n",
            "      * gym_star_gunner_deterministic-v4_random\n",
            "      * gym_star_gunner_no_frameskip-v0_random\n",
            "      * gym_star_gunner_no_frameskip-v4_random\n",
            "      * gym_tennis-v0_random\n",
            "      * gym_tennis-v4_random\n",
            "      * gym_tennis_deterministic-v0_random\n",
            "      * gym_tennis_deterministic-v4_random\n",
            "      * gym_tennis_no_frameskip-v0_random\n",
            "      * gym_tennis_no_frameskip-v4_random\n",
            "      * gym_time_pilot-v0_random\n",
            "      * gym_time_pilot-v4_random\n",
            "      * gym_time_pilot_deterministic-v0_random\n",
            "      * gym_time_pilot_deterministic-v4_random\n",
            "      * gym_time_pilot_no_frameskip-v0_random\n",
            "      * gym_time_pilot_no_frameskip-v4_random\n",
            "      * gym_tutankham-v0_random\n",
            "      * gym_tutankham-v4_random\n",
            "      * gym_tutankham_deterministic-v0_random\n",
            "      * gym_tutankham_deterministic-v4_random\n",
            "      * gym_tutankham_no_frameskip-v0_random\n",
            "      * gym_tutankham_no_frameskip-v4_random\n",
            "      * gym_up_n_down-v0_random\n",
            "      * gym_up_n_down-v4_random\n",
            "      * gym_up_n_down_deterministic-v0_random\n",
            "      * gym_up_n_down_deterministic-v4_random\n",
            "      * gym_up_n_down_no_frameskip-v0_random\n",
            "      * gym_up_n_down_no_frameskip-v4_random\n",
            "      * gym_venture-v0_random\n",
            "      * gym_venture-v4_random\n",
            "      * gym_venture_deterministic-v0_random\n",
            "      * gym_venture_deterministic-v4_random\n",
            "      * gym_venture_no_frameskip-v0_random\n",
            "      * gym_venture_no_frameskip-v4_random\n",
            "      * gym_video_pinball-v0_random\n",
            "      * gym_video_pinball-v4_random\n",
            "      * gym_video_pinball_deterministic-v0_random\n",
            "      * gym_video_pinball_deterministic-v4_random\n",
            "      * gym_video_pinball_no_frameskip-v0_random\n",
            "      * gym_video_pinball_no_frameskip-v4_random\n",
            "      * gym_wizard_of_wor-v0_random\n",
            "      * gym_wizard_of_wor-v4_random\n",
            "      * gym_wizard_of_wor_deterministic-v0_random\n",
            "      * gym_wizard_of_wor_deterministic-v4_random\n",
            "      * gym_wizard_of_wor_no_frameskip-v0_random\n",
            "      * gym_wizard_of_wor_no_frameskip-v4_random\n",
            "      * gym_yars_revenge-v0_random\n",
            "      * gym_yars_revenge-v4_random\n",
            "      * gym_yars_revenge_deterministic-v0_random\n",
            "      * gym_yars_revenge_deterministic-v4_random\n",
            "      * gym_yars_revenge_no_frameskip-v0_random\n",
            "      * gym_yars_revenge_no_frameskip-v4_random\n",
            "      * gym_zaxxon-v0_random\n",
            "      * gym_zaxxon-v4_random\n",
            "      * gym_zaxxon_deterministic-v0_random\n",
            "      * gym_zaxxon_deterministic-v4_random\n",
            "      * gym_zaxxon_no_frameskip-v0_random\n",
            "      * gym_zaxxon_no_frameskip-v4_random\n",
            "    image:\n",
            "      * image_celeba\n",
            "      * image_celeba32\n",
            "      * image_celeba64\n",
            "      * image_celeba_multi_resolution\n",
            "      * image_celebahq128\n",
            "      * image_celebahq128_dmol\n",
            "      * image_celebahq256\n",
            "      * image_celebahq256_dmol\n",
            "      * image_cifar10\n",
            "      * image_cifar100\n",
            "      * image_cifar100_plain\n",
            "      * image_cifar100_plain8\n",
            "      * image_cifar100_plain_gen\n",
            "      * image_cifar100_tune\n",
            "      * image_cifar10_plain\n",
            "      * image_cifar10_plain8\n",
            "      * image_cifar10_plain_gen\n",
            "      * image_cifar10_plain_gen_dmol\n",
            "      * image_cifar10_plain_gen_flat\n",
            "      * image_cifar10_plain_random_shift\n",
            "      * image_cifar10_tune\n",
            "      * image_cifar20\n",
            "      * image_cifar20_plain\n",
            "      * image_cifar20_plain8\n",
            "      * image_cifar20_plain_gen\n",
            "      * image_cifar20_tune\n",
            "      * image_fashion_mnist\n",
            "      * image_fsns\n",
            "      * image_imagenet\n",
            "      * image_imagenet224\n",
            "      * image_imagenet224_no_normalization\n",
            "      * image_imagenet256\n",
            "      * image_imagenet32\n",
            "      * image_imagenet32_gen\n",
            "      * image_imagenet32_small\n",
            "      * image_imagenet64\n",
            "      * image_imagenet64_gen\n",
            "      * image_imagenet64_gen_flat\n",
            "      * image_imagenet_multi_resolution_gen\n",
            "      * image_lsun_bedrooms\n",
            "      * image_mnist\n",
            "      * image_mnist_tune\n",
            "      * image_ms_coco_characters\n",
            "      * image_ms_coco_tokens32k\n",
            "      * image_text_ms_coco\n",
            "      * image_text_ms_coco_multi_resolution\n",
            "      * image_vqav2_rcnn_feature_tokens10k_labels3k\n",
            "      * image_vqav2_tokens10k_labels3k\n",
            "    img2img:\n",
            "      * img2img_allen_brain\n",
            "      * img2img_allen_brain_dim16to16_paint1\n",
            "      * img2img_allen_brain_dim48to64\n",
            "      * img2img_allen_brain_dim8to32\n",
            "      * img2img_celeba\n",
            "      * img2img_celeba64\n",
            "      * img2img_cifar10\n",
            "      * img2img_cifar100\n",
            "      * img2img_imagenet\n",
            "    lambada:\n",
            "      * lambada_lm\n",
            "      * lambada_lm_control\n",
            "      * lambada_rc\n",
            "      * lambada_rc_control\n",
            "    languagemodel:\n",
            "      * languagemodel_de_en_fr_ro_wiki64k\n",
            "      * languagemodel_de_en_fr_ro_wiki64k_fitb_packed1k\n",
            "      * languagemodel_de_wiki32k\n",
            "      * languagemodel_de_wiki64k\n",
            "      * languagemodel_en_wiki32k\n",
            "      * languagemodel_en_wiki64k\n",
            "      * languagemodel_en_wiki64k_shorter\n",
            "      * languagemodel_en_wiki_lm_multi_nli_subwords\n",
            "      * languagemodel_en_wiki_lm_multi_nli_subwords64k\n",
            "      * languagemodel_en_wiki_lm_multi_nli_subwords_v2\n",
            "      * languagemodel_en_wiki_lm_short_multi_nli_subwords64k\n",
            "      * languagemodel_en_wiki_lm_squad_concat_subwords\n",
            "      * languagemodel_en_wiki_lm_summarize_cnndm_subwords\n",
            "      * languagemodel_en_wiki_lm_summarize_cnndm_subwords64k\n",
            "      * languagemodel_en_wiki_lm_summarize_frac10_cnndm_subwords64k\n",
            "      * languagemodel_en_wiki_lm_summarize_frac1_cnndm_subwords64k\n",
            "      * languagemodel_en_wiki_lm_summarize_frac20_cnndm_subwords64k\n",
            "      * languagemodel_en_wiki_lm_summarize_frac2_cnndm_subwords64k\n",
            "      * languagemodel_en_wiki_lm_summarize_frac50_cnndm_subwords64k\n",
            "      * languagemodel_en_wiki_lm_summarize_frac5_cnndm_subwords64k\n",
            "      * languagemodel_fr_wiki32k\n",
            "      * languagemodel_fr_wiki64k\n",
            "      * languagemodel_lm1b32k\n",
            "      * languagemodel_lm1b32k_packed\n",
            "      * languagemodel_lm1b8k\n",
            "      * languagemodel_lm1b8k_packed\n",
            "      * languagemodel_lm1b_characters\n",
            "      * languagemodel_lm1b_characters_packed\n",
            "      * languagemodel_lm1b_multi_nli\n",
            "      * languagemodel_lm1b_multi_nli_subwords\n",
            "      * languagemodel_lm1b_sentiment_imdb\n",
            "      * languagemodel_multi_wiki_translate\n",
            "      * languagemodel_multi_wiki_translate_fr\n",
            "      * languagemodel_multi_wiki_translate_packed1k\n",
            "      * languagemodel_multi_wiki_translate_packed1k_v2\n",
            "      * languagemodel_ptb10k\n",
            "      * languagemodel_ptb_characters\n",
            "      * languagemodel_ro_wiki32k\n",
            "      * languagemodel_ro_wiki64k\n",
            "      * languagemodel_wiki_noref_v128k_l1k\n",
            "      * languagemodel_wiki_noref_v32k_l16k\n",
            "      * languagemodel_wiki_noref_v32k_l1k\n",
            "      * languagemodel_wiki_noref_v8k_l16k\n",
            "      * languagemodel_wiki_noref_v8k_l1k\n",
            "      * languagemodel_wiki_scramble_l128\n",
            "      * languagemodel_wiki_scramble_l1k\n",
            "      * languagemodel_wiki_xml_v8k_l1k\n",
            "      * languagemodel_wiki_xml_v8k_l4k\n",
            "      * languagemodel_wikitext103\n",
            "      * languagemodel_wikitext103_characters\n",
            "      * languagemodel_wikitext103_l16k\n",
            "      * languagemodel_wikitext103_l4k\n",
            "    librispeech:\n",
            "      * librispeech\n",
            "      * librispeech_clean\n",
            "      * librispeech_clean_small\n",
            "      * librispeech_noisy\n",
            "      * librispeech_train_full_test_clean\n",
            "    msr:\n",
            "      * msr_paraphrase_corpus\n",
            "      * msr_paraphrase_corpus_characters\n",
            "    multi:\n",
            "      * multi_nli\n",
            "      * multi_nli_characters\n",
            "      * multi_nli_shared_vocab\n",
            "      * multi_nli_text2text\n",
            "      * multi_nli_text2text_multi64k_packed1k\n",
            "      * multi_nli_wiki_lm_multi_vocab64k\n",
            "      * multi_nli_wiki_lm_shared_vocab\n",
            "      * multi_nli_wiki_lm_shared_vocab64k\n",
            "    ocr:\n",
            "      * ocr_test\n",
            "    paraphrase:\n",
            "      * paraphrase_generation_ms_coco_problem1d\n",
            "      * paraphrase_generation_ms_coco_problem1d_characters\n",
            "      * paraphrase_generation_ms_coco_problem2d\n",
            "      * paraphrase_generation_ms_coco_problem2d_characters\n",
            "    parsing:\n",
            "      * parsing_english_ptb16k\n",
            "      * parsing_english_ptb8k\n",
            "      * parsing_icelandic16k\n",
            "    program:\n",
            "      * program_search_algolisp\n",
            "    programming:\n",
            "      * programming_desc2code_cpp\n",
            "      * programming_desc2code_py\n",
            "    question:\n",
            "      * question_nli\n",
            "      * question_nli_characters\n",
            "    quora:\n",
            "      * quora_question_pairs\n",
            "      * quora_question_pairs_characters\n",
            "    reverse:\n",
            "      * reverse_sequence\n",
            "      * reverse_sequence_small\n",
            "    rte:\n",
            "      * rte\n",
            "      * rte_characters\n",
            "    sci:\n",
            "      * sci_tail\n",
            "      * sci_tail_characters\n",
            "      * sci_tail_shared_vocab\n",
            "    sentiment:\n",
            "      * sentiment_imdb\n",
            "      * sentiment_imdb_characters\n",
            "      * sentiment_sst_binary\n",
            "      * sentiment_sst_binary_characters\n",
            "      * sentiment_yelp_full\n",
            "      * sentiment_yelp_full_characters\n",
            "      * sentiment_yelp_polarity\n",
            "      * sentiment_yelp_polarity_characters\n",
            "    squad:\n",
            "      * squad\n",
            "      * squad_concat\n",
            "      * squad_concat_multi64k\n",
            "      * squad_concat_positioned\n",
            "      * squad_concat_shared_vocab\n",
            "      * squad_text2text\n",
            "      * squad_text2text_multi64k_packed1k\n",
            "    stanford:\n",
            "      * stanford_nli\n",
            "      * stanford_nli_characters\n",
            "      * stanford_nli_shared_vocab\n",
            "      * stanford_nli_wiki_lm_shared_vocab\n",
            "      * stanford_nli_wiki_lm_shared_vocab64k\n",
            "    style:\n",
            "      * style_transfer_modern_to_shakespeare\n",
            "      * style_transfer_modern_to_shakespeare_characters\n",
            "      * style_transfer_shakespeare_to_modern\n",
            "      * style_transfer_shakespeare_to_modern_characters\n",
            "    summarize:\n",
            "      * summarize_cnn_dailymail32k\n",
            "      * summarize_cnn_dailymail_multi64k_packed1k\n",
            "      * summarize_cnn_dailymail_wiki_lm_multi_vocab64k\n",
            "      * summarize_cnn_dailymail_wiki_lm_shared_vocab\n",
            "      * summarize_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_frac0p1_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_frac10_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_frac1_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_frac20_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_frac2_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_frac50_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_frac5_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_frac_cnn_dailymail_wiki_lm_shared_vocab64k\n",
            "      * summarize_scientific_sections65k\n",
            "      * summarize_scientific_sections_gdrive65k\n",
            "    sva:\n",
            "      * sva_language_modeling\n",
            "      * sva_number_prediction\n",
            "    text2text:\n",
            "      * text2text_copyable_tokens\n",
            "      * text2text_tmpdir\n",
            "      * text2text_tmpdir_tokens\n",
            "    timeseries:\n",
            "      * timeseries_synthetic_data_series10_samples100k\n",
            "      * timeseries_toy_problem\n",
            "      * timeseries_toy_problem_no_inputs\n",
            "    tiny:\n",
            "      * tiny_algo\n",
            "    translate:\n",
            "      * translate_encs_wmt32k\n",
            "      * translate_encs_wmt_characters\n",
            "      * translate_ende2018_wmt32k\n",
            "      * translate_ende_pc32k\n",
            "      * translate_ende_pc_clean32k\n",
            "      * translate_ende_wmt32k\n",
            "      * translate_ende_wmt32k_packed\n",
            "      * translate_ende_wmt8k\n",
            "      * translate_ende_wmt8k_packed\n",
            "      * translate_ende_wmt_characters\n",
            "      * translate_ende_wmt_clean32k\n",
            "      * translate_ende_wmt_clean_pc32k\n",
            "      * translate_ende_wmt_clean_pc_clean32k\n",
            "      * translate_ende_wmt_multi64k\n",
            "      * translate_ende_wmt_multi64k_packed1k\n",
            "      * translate_ende_wmt_pc32k\n",
            "      * translate_ende_wmt_pc_clean32k\n",
            "      * translate_enet_wmt32k\n",
            "      * translate_enet_wmt_characters\n",
            "      * translate_enfr_wmt32k\n",
            "      * translate_enfr_wmt32k_packed\n",
            "      * translate_enfr_wmt32k_with_backtranslate_en\n",
            "      * translate_enfr_wmt32k_with_backtranslate_fr\n",
            "      * translate_enfr_wmt8k\n",
            "      * translate_enfr_wmt_characters\n",
            "      * translate_enfr_wmt_multi64k\n",
            "      * translate_enfr_wmt_multi64k_packed1k\n",
            "      * translate_enfr_wmt_small32k\n",
            "      * translate_enfr_wmt_small8k\n",
            "      * translate_enfr_wmt_small_characters\n",
            "      * translate_enid_iwslt32k\n",
            "      * translate_enmk_setimes32k\n",
            "      * translate_enmk_setimes_characters\n",
            "      * translate_enro_wmt32k\n",
            "      * translate_enro_wmt8k\n",
            "      * translate_enro_wmt_characters\n",
            "      * translate_enro_wmt_multi64k\n",
            "      * translate_enro_wmt_multi_small64k\n",
            "      * translate_enro_wmt_multi_tiny64k\n",
            "      * translate_enro_wmt_multi_tiny64k_packed1k\n",
            "      * translate_envi_iwslt32k\n",
            "      * translate_enzh_wmt32k\n",
            "      * translate_enzh_wmt8k\n",
            "    video:\n",
            "      * video_bair_robot_pushing\n",
            "      * video_bair_robot_pushing_with_actions\n",
            "      * video_google_robot_pushing\n",
            "      * video_moving_mnist\n",
            "      * video_stochastic_shapes10k\n",
            "    wiki:\n",
            "      * wiki_revision\n",
            "      * wiki_revision_packed1k\n",
            "      * wiki_revision_packed256\n",
            "    wikisum:\n",
            "      * wikisum_commoncrawl\n",
            "      * wikisum_commoncrawl_lead_section\n",
            "      * wikisum_web\n",
            "      * wikisum_web_lead_section\n",
            "    winograd:\n",
            "      * winograd_nli\n",
            "      * winograd_nli_characters\n",
            "    wsj:\n",
            "      * wsj_parsing\n",
            "\n",
            "  Optimizers:\n",
            "    adafactor:\n",
            "      * adafactor\n",
            "    adagrad:\n",
            "      * adagrad\n",
            "    adam:\n",
            "      * adam\n",
            "      * adam_w\n",
            "    ftrl:\n",
            "      * ftrl\n",
            "    momentum:\n",
            "      * momentum\n",
            "    multistep:\n",
            "      * multistep_adam\n",
            "    rms:\n",
            "      * rms_prop\n",
            "    sgd:\n",
            "      * sgd\n",
            "    true:\n",
            "      * true_adam\n",
            "    yellow:\n",
            "      * yellow_fin\n",
            "\n",
            "  Attacks:\n",
            "\n",
            "\n",
            "  Attack HParams:\n",
            "    resnet:\n",
            "      * resnet_fgsm\n",
            "      * resnet_madry\n",
            "      * resnet_random\n",
            "    shake:\n",
            "      * shake_shake_fgsm\n",
            "\n",
            "  Pruning HParams:\n",
            "    resnet:\n",
            "      * resnet_unit\n",
            "      * resnet_weight\n",
            "\n",
            "  Pruning Strategies:\n",
            "\n",
            "\n",
            "  Env Problems:\n",
            "    reacher:\n",
            "      * reacher_env_problem\n",
            "    tic:\n",
            "      * tic_tac_toe_env_problem\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rXQ3hSx-shjK",
        "colab_type": "code",
        "outputId": "5d9da21e-3272-4152-9066-b342d635b518",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "#If you want to check what are all the hyper parameters \n",
        "\n",
        "from tensor2tensor.models import transformer\n",
        "hps = transformer.transformer_base_v3()\n",
        "[print('%48s: %s' % (i, str(hps.__dict__[i]))) for i in hps.__dict__ if i!= '_hparam_types']"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0716 00:43:44.347541 139976359049088 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:68: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "W0716 00:43:45.647855 139976359049088 lazy_loader.py:50] \n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "W0716 00:43:48.159514 139976359049088 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/adafactor.py:27: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "W0716 00:43:48.161128 139976359049088 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/multistep_optimizer.py:32: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n",
            "W0716 00:43:48.181633 139976359049088 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4237: The name tf.train.CheckpointSaverListener is deprecated. Please use tf.estimator.CheckpointSaverListener instead.\n",
            "\n",
            "W0716 00:43:48.182691 139976359049088 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4260: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
            "\n",
            "W0716 00:43:48.208230 139976359049088 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/neural_stack.py:38: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.\n",
            "\n",
            "W0716 00:43:48.242801 139976359049088 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/gym_utils.py:235: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n",
            "W0716 00:43:48.267923 139976359049088 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:111: The name tf.OptimizerOptions is deprecated. Please use tf.compat.v1.OptimizerOptions instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "                                _model_structure: None\n",
            "                                      batch_size: 4096\n",
            "                              batch_shuffle_size: 512\n",
            "                            use_fixed_batch_size: False\n",
            "                               num_hidden_layers: 6\n",
            "                                   kernel_height: 3\n",
            "                                    kernel_width: 1\n",
            "                                     hidden_size: 512\n",
            "                                  compress_steps: 0\n",
            "                                         dropout: 0.2\n",
            "                                  clip_grad_norm: 0.0\n",
            "                                grad_noise_scale: 0.0\n",
            "                                 summarize_grads: False\n",
            "                                     mlperf_mode: False\n",
            "                                  summarize_vars: False\n",
            "                                     initializer: uniform_unit_scaling\n",
            "                                initializer_gain: 1.0\n",
            "                                 label_smoothing: 0.1\n",
            "                                       optimizer: adam\n",
            "                          optimizer_adam_epsilon: 1e-09\n",
            "                            optimizer_adam_beta1: 0.9\n",
            "                            optimizer_adam_beta2: 0.997\n",
            "                     optimizer_momentum_momentum: 0.9\n",
            "                     optimizer_momentum_nesterov: False\n",
            "                       optimizer_adafactor_beta1: 0.0\n",
            "                       optimizer_adafactor_beta2: 0.999\n",
            "                    optimizer_adafactor_factored: True\n",
            "                  optimizer_adafactor_decay_type: pow\n",
            "             optimizer_adafactor_memory_exponent: 0.8\n",
            "          optimizer_adafactor_clipping_threshold: 1.0\n",
            " optimizer_adafactor_multiply_by_parameter_scale: True\n",
            "            optimizer_multistep_accumulate_steps: 0\n",
            "           mixed_precision_optimizer_loss_scaler: exponential\n",
            "       mixed_precision_optimizer_init_loss_scale: 32768\n",
            "                            optimizer_zero_grads: False\n",
            "                                    weight_decay: 0.0\n",
            "                                    weight_noise: 0.0\n",
            "                          learning_rate_schedule: constant*linear_warmup*rsqrt_decay*rsqrt_hidden_size\n",
            "                          learning_rate_constant: 2.0\n",
            "                      learning_rate_decay_scheme: noam\n",
            "                       learning_rate_decay_steps: 5000\n",
            "                   learning_rate_decay_staircase: False\n",
            "                           learning_rate_minimum: None\n",
            "                        learning_rate_decay_rate: 1.0\n",
            "                      learning_rate_warmup_steps: 8000\n",
            "                learning_rate_cosine_cycle_steps: 250000\n",
            "                                   learning_rate: 0.2\n",
            "                                 sampling_method: argmax\n",
            "                                   sampling_temp: 1.0\n",
            "                             sampling_keep_top_k: -1\n",
            "                                 factored_logits: False\n",
            "                         multiply_embedding_mode: sqrt_depth\n",
            "                                moe_hidden_sizes: 2048\n",
            "                                 moe_num_experts: 16\n",
            "                                           moe_k: 2\n",
            "                                   moe_loss_coef: 0.001\n",
            "                       layer_preprocess_sequence: n\n",
            "                      layer_postprocess_sequence: da\n",
            "                    layer_prepostprocess_dropout: 0.1\n",
            "     layer_prepostprocess_dropout_broadcast_dims: \n",
            "                                  symbol_dropout: 0.0\n",
            "                                       norm_type: layer\n",
            "                                    norm_epsilon: 1e-06\n",
            "                                   vocab_divisor: 1\n",
            "                                      min_length: 0\n",
            "                                      max_length: 256\n",
            "                                    pack_dataset: False\n",
            "                                  use_custom_ops: True\n",
            "                      split_targets_chunk_length: 0\n",
            "                        split_targets_max_chunks: 100\n",
            "                  split_targets_strided_training: False\n",
            "                               min_length_bucket: 8\n",
            "                              length_bucket_step: 1.1\n",
            "                        eval_drop_long_sequences: False\n",
            "                         eval_run_autoregressive: False\n",
            "            shared_embedding_and_softmax_weights: True\n",
            "                                shared_embedding: False\n",
            "                      symbol_modality_num_shards: 16\n",
            "                                          bottom: {}\n",
            "                                            loss: {}\n",
            "                                            name: {}\n",
            "                                             top: {}\n",
            "                                      weights_fn: {}\n",
            "                            max_input_seq_length: 0\n",
            "                           max_target_seq_length: 0\n",
            "                                 split_to_length: 0\n",
            "                          video_num_input_frames: 1\n",
            "                         video_num_target_frames: 1\n",
            "                                    prepend_mode: none\n",
            "                         scheduled_sampling_prob: 0.0\n",
            "                       scheduled_sampling_method: parallel\n",
            "                 scheduled_sampling_warmup_steps: 50000\n",
            "              scheduled_sampling_gold_mixin_prob: 0.5\n",
            "                   scheduled_sampling_num_passes: 1\n",
            "              scheduled_sampling_warmup_schedule: exp\n",
            "                           daisy_chain_variables: True\n",
            "                              force_full_predict: False\n",
            "                             no_data_parallelism: False\n",
            "                                activation_dtype: float32\n",
            "                                    weight_dtype: float32\n",
            "                            pretrained_model_dir: \n",
            "                 multiproblem_schedule_threshold: 0.5\n",
            "                 multiproblem_per_task_threshold: \n",
            "              multiproblem_schedule_max_examples: 10000000.0\n",
            "                    multiproblem_mixing_schedule: constant\n",
            "                multiproblem_reweight_label_loss: False\n",
            "                       multiproblem_label_weight: 0.5\n",
            "                           max_relative_position: 0\n",
            "                  heads_share_relative_embedding: False\n",
            "                          add_relative_to_values: False\n",
            "                            tpu_enable_host_call: False\n",
            "                                       pad_batch: False\n",
            "                   multiproblem_target_eval_only: False\n",
            "                         multiproblem_vocab_size: -1\n",
            "                   multiproblem_max_input_length: -1\n",
            "                  multiproblem_max_target_length: -1\n",
            "                 multiproblem_fixed_train_length: -1\n",
            "                          warm_start_from_second: \n",
            "                                 area_value_mode: none\n",
            "                                   area_key_mode: none\n",
            "                                 num_area_layers: 0\n",
            "                                  max_area_width: 1\n",
            "                                 max_area_height: 1\n",
            "                                   memory_height: 1\n",
            "                             num_sampled_classes: 0\n",
            "                                     filter_size: 2048\n",
            "                              num_encoder_layers: 0\n",
            "                              num_decoder_layers: 0\n",
            "                                       num_heads: 8\n",
            "                          attention_key_channels: 0\n",
            "                        attention_value_channels: 0\n",
            "                                       ffn_layer: dense_relu_dense\n",
            "                parameter_attention_key_channels: 0\n",
            "              parameter_attention_value_channels: 0\n",
            "                               attention_dropout: 0.1\n",
            "                attention_dropout_broadcast_dims: \n",
            "                                    relu_dropout: 0.1\n",
            "                     relu_dropout_broadcast_dims: \n",
            "                                             pos: timing\n",
            "                            nbr_decoder_problems: 1\n",
            "                                  proximity_bias: False\n",
            "                   causal_decoder_self_attention: True\n",
            "                                 use_pad_remover: True\n",
            "                             self_attention_type: dot_product\n",
            "                               conv_first_kernel: 3\n",
            "                          attention_variables_3d: False\n",
            "                      use_target_space_embedding: True\n",
            "                              moe_overhead_train: 1.0\n",
            "                               moe_overhead_eval: 2.0\n",
            "                       overload_eval_metric_name: \n",
            "                          unidirectional_encoder: False\n",
            "                                hard_attention_k: 0\n",
            "                             gumbel_noise_weight: 0.0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None,\n",
              " None]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zd2PamvDciiu",
        "colab_type": "text"
      },
      "source": [
        "### Test the trained summarizar "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "acGyPgb-dPRd",
        "colab_type": "code",
        "outputId": "d51017f8-afde-4f2a-b9df-eec9bf261971",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 490
        }
      },
      "source": [
        "#First, I'll show how to try the summarizar directly in the Colab cells. \n",
        "\n",
        "# Imports we need.\n",
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import os\n",
        "import collections\n",
        "\n",
        "from tensor2tensor import models\n",
        "from tensor2tensor import problems\n",
        "from tensor2tensor.layers import common_layers\n",
        "from tensor2tensor.utils import trainer_lib\n",
        "from tensor2tensor.utils import t2t_model\n",
        "from tensor2tensor.utils import registry\n",
        "from tensor2tensor.utils import metrics\n",
        "\n",
        "# Enable TF Eager execution\n",
        "tfe = tf.contrib.eager\n",
        "tfe.enable_eager_execution()\n",
        "\n",
        "# Other setup\n",
        "Modes = tf.estimator.ModeKeys"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0717 19:48:17.115717 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:68: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "W0717 19:48:20.349527 140550544508800 lazy_loader.py:50] \n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "W0717 19:48:22.807933 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/adafactor.py:27: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "W0717 19:48:22.810198 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/multistep_optimizer.py:32: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n",
            "W0717 19:48:22.844276 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4237: The name tf.train.CheckpointSaverListener is deprecated. Please use tf.estimator.CheckpointSaverListener instead.\n",
            "\n",
            "W0717 19:48:22.845558 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4260: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
            "\n",
            "W0717 19:48:22.881502 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/neural_stack.py:38: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.\n",
            "\n",
            "W0717 19:48:22.939158 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/gym_utils.py:235: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n",
            "W0717 19:48:23.051343 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:111: The name tf.OptimizerOptions is deprecated. Please use tf.compat.v1.OptimizerOptions instead.\n",
            "\n",
            "W0717 19:48:29.116462 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/sci_sum.py:73: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QBKVLKjCccuG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Specify the 'problem' we're using\n",
        "ende_problem = problems.problem(\"summarize_scientific_sections_gdrive65k\")\n",
        "\n",
        "#Path to volcab file\n",
        "vocab_file  = \"vocab.summarize_scientific_sections_gdrive65k.65536.subwords\"\n",
        "\n",
        "#Path to encoder\n",
        "encoders = ende_problem.feature_encoders('gdrive/My Drive/tfRecordsSummaryFiles')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o1PvSXLQccdm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Setup helper functions for encoding and decoding\n",
        "def encode(input_str, output_str=None):\n",
        "  \"\"\"Input str to features dict, ready for inference\"\"\"\n",
        "  inputs = encoders[\"inputs\"].encode(input_str) + [1]  # add EOS id\n",
        "  batch_inputs = tf.reshape(inputs, [1, -1, 1])  # Make it 3D.\n",
        "  return {\"inputs\": batch_inputs}\n",
        "\n",
        "def decode(integers):\n",
        "  \"\"\"List of ints to str\"\"\"\n",
        "  integers = list(np.squeeze(integers))\n",
        "  if 1 in integers:\n",
        "    integers = integers[:integers.index(1)]\n",
        "  return encoders[\"inputs\"].decode(np.squeeze(integers))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9lmeTEKwccUf",
        "colab_type": "code",
        "outputId": "29995f6f-228f-4242-84a4-876c58b5caf4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "source": [
        "data_dir = 'gdrive/My Drive/tfRecordsSummaryFiles'\n",
        "model_name = \"transformer\"\n",
        "hparams_set = \"transformer_prepend\"\n",
        "\n",
        "hparams = trainer_lib.create_hparams(hparams_set, data_dir=data_dir, problem_name=\"summarize_scientific_sections_gdrive65k\")\n",
        "\n",
        "# NOTE: Only create the model once when restoring from a checkpoint; it's a\n",
        "# Layer and so subsequent instantiations will have different variable scopes\n",
        "# that will not match the checkpoint.\n",
        "translate_model = registry.model(model_name)(hparams, Modes.EVAL)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0717 19:54:03.459699 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:94: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n",
            "W0717 19:54:04.699697 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:244: The name tf.summary.text is deprecated. Please use tf.compat.v1.summary.text instead.\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CcDFE4A_iJE_",
        "colab_type": "code",
        "outputId": "a6bd2263-39c9-4d59-9c51-422acc4ea5b1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        }
      },
      "source": [
        "# Check the checkpoint files\n",
        "\n",
        "os.listdir('gdrive/My Drive/SummarizationCheckPoints')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['flags.txt',\n",
              " 'flags_t2t.txt',\n",
              " 'hparams.json',\n",
              " 'events.out.tfevents.1563293506.9f8fc6d3dde4',\n",
              " 'graph.pbtxt',\n",
              " 'model.ckpt-0.index',\n",
              " 'model.ckpt-0.meta',\n",
              " 'model.ckpt-0.data-00000-of-00002',\n",
              " 'model.ckpt-0.data-00001-of-00002',\n",
              " 'model.ckpt-1000.index',\n",
              " 'checkpoint',\n",
              " 'model.ckpt-1000.meta',\n",
              " 'eval',\n",
              " 'eval_eval',\n",
              " 'model.ckpt-1000.data-00000-of-00002',\n",
              " 'model.ckpt-1000.data-00001-of-00002']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SnU0dqpefTsg",
        "colab_type": "code",
        "outputId": "f107fe81-3e11-4d37-e81e-058e733271fb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Copy the pretrained checkpoint locall\n",
        "# !gsutil -q cp -R {gs_ckpt} {checkpoint_dir}\n",
        "ckpt_path = tf.train.latest_checkpoint('gdrive/My Drive/SummarizationCheckPoints')\n",
        "ckpt_path"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'gdrive/My Drive/SummarizationCheckPoints/model.ckpt-1000'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TpRqjj6MiQUe",
        "colab_type": "code",
        "outputId": "752eb291-66e6-4eab-9cf4-2f4934000716",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 373
        }
      },
      "source": [
        "\n",
        "# Restore and translate!\n",
        "def translate(inputs):\n",
        "  encoded_inputs = encode(inputs)\n",
        "  with tfe.restore_variables_on_create(ckpt_path):\n",
        "    model_output = translate_model.infer(encoded_inputs)[\"outputs\"]\n",
        "  return decode(model_output)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0717 20:10:34.771781 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:2296: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.\n",
            "\n",
            "W0717 20:10:34.782998 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/modalities.py:475: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
            "\n",
            "W0717 20:10:36.432304 140550544508800 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_attention.py:857: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "W0717 20:10:36.487500 140550544508800 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:621: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "W0717 20:10:36.492521 140550544508800 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:621: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "W0717 20:10:36.523204 140550544508800 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:3236: The name tf.layers.Dense is deprecated. Please use tf.compat.v1.layers.Dense instead.\n",
            "\n",
            "W0717 20:10:39.839085 140550544508800 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/transformer.py:1221: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Inputs: The animal didn't cross the street because it was too tired\n",
            "Outputs: . Results. The results of the same same same same same same same same same same and the same same same same of the same of the same of the same of the same and the same and the same and the same and the same and the same of the same of the same of the same and the same and the same and\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iEXgfNWikLb9",
        "colab_type": "code",
        "outputId": "133238d4-ed64-4862-8fe6-f29d6ca3001a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        }
      },
      "source": [
        "# You can type in your own text you want to summarize, and then check the output. \n",
        "\n",
        "inputs = \"In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles. We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video. Making generation less sequential is another research goals of ours. The code we used to train and evaluate our models is available at https://github.com/ tensorflow/tensor2tensor.\"\n",
        "outputs = translate(inputs)\n",
        "\n",
        "print(\"Inputs: %s\" % inputs)\n",
        "print(\"Outputs: %s\" % outputs)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Inputs: In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers. On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art. In the former task our best model outperforms even all previously reported ensembles. We are excited about the future of attention-based models and plan to apply them to other tasks. We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video. Making generation less sequential is another research goals of ours. The code we used to train and evaluate our models is available at https://github.com/ tensorflow/tensor2tensor.\n",
            "Outputs: . Conclusions. We have have a large of the same of the same of the same of the same of the same of the same of the same of the same of the same of the same of the same of the same same same same of the same of the same of the same of the same of the same and the same same same of the same of the same same same and the same of the same than the same of the same of the same of the same of the same of the same-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ewGpzdBB9-yS",
        "colab_type": "text"
      },
      "source": [
        "For batch processing of summaries, you can write them to a file, and then use the Tensor2Tensor decoder to process the summaries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LcyuCE0L9057",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# This is where the tensor2tensor decoder will output the summaries.\n",
        "DECODE_FILE=Testtext.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y1V5_UwhMqai",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%writefile Testtext3.txt\n",
        "Recent empirical improvements due to transfer learning with language models have demonstrated that rich, unsupervised pre-training is an integral part of many language understanding systems. In particular, these results enable even low-resource tasks to benefit from deep unidirectional architectures. Our major contribution is further generalizing these findings to deep bidirectional architectures, allowing the same pre-trained model to successfully tackle a broad set of NLP tasks.\n",
        "We demonstrate the importance of the deep bidirectionality of BERT by evaluating two pretraining objectives using exactly the same pretraining data, fine-tuning scheme, and hyperparameters as BERTBASE: No NSP: A bidirectional model which is trained using the “masked LM” (MLM) but without the “next sentence prediction” (NSP) task. LTR & No NSP: A left-context-only model which is trained using a standard Left-to-Right (LTR) LM, rather than an MLM. The left-only constraint was also applied at fine-tuning, because removing it introduced a pre-train/fine-tune mismatch that degraded downstream performance. Additionally, this model was pre-trained without the NSP task. This is directly comparable to OpenAI GPT, but using our larger training dataset, our input representation, and our fine-tuning scheme. We first examine the impact brought by the NSP task. In Table 5, we show that removing NSP hurts performance significantly on QNLI, MNLI, and SQuAD 1.1. Next, we evaluate the impact of training bidirectional representations by comparing “No NSP” to “LTR & No NSP”. The LTR model performs worse than the MLM model on all tasks, with large drops on MRPC and SQuAD. For SQuAD it is intuitively clear that a LTR model will perform poorly at token predictions, since the token-level hidden states have no rightside context. In order to make a good faith attempt at strengthening the LTR system, we added a randomly initialized BiLSTM on top. This does significantly improve results on SQuAD, but theHe spread his light among us like a candle.\n",
        "In this section, we present BERT fine-tuning results on 11 NLP tasks. 4.1 GLUE The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018a) is a collection of diverse natural language understanding tasks. Detailed descriptions of GLUE datasets are included in Appendix B.1. To fine-tune on GLUE, we represent the input sequence (for single sentence or sentence pairs) as described in Section 3, and use the final hidden vector C ∈ R H corresponding to the first input token ([CLS]) as the aggregate representation. The only new parameters introduced during fine-tuning are classification layer weights W ∈ R K×H, where K is the number of labels. We compute a standard classification loss with C and W, i.e., log(softmax(CWT )).\n",
        "             \n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "keX37D_xR7DE",
        "colab_type": "code",
        "outputId": "cb48691e-d8ef-4304-e42d-0ab416deae8c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# Now use the tensor2tesnor decoder to process Testtext3.txt\n",
        "\n",
        "!t2t-decoder \\\n",
        "  --data_dir='gdrive/My Drive/tfRecordsSummaryFiles' \\\n",
        "  --problem=summarize_scientific_sections_gdrive65k \\\n",
        "  --model=transformer \\\n",
        "  --hparams_set=transformer_prepend \\\n",
        "  --output_dir='gdrive/My Drive/SummarizationCheckPoints' \\\n",
        "  --decode_hparams=\"beam_size=3,alpha=0.6\" \\\n",
        "  --decode_from_file=Testtext3.txt\n",
        "  \n",
        "#     --t2t_usr_dir=./poetry/trainer \\"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0717 23:43:25.305890 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:68: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "W0717 23:43:26.521717 140002720810880 lazy_loader.py:50] \n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "W0717 23:43:29.093972 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/adafactor.py:27: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "W0717 23:43:29.094941 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/multistep_optimizer.py:32: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
            "\n",
            "W0717 23:43:29.112589 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4237: The name tf.train.CheckpointSaverListener is deprecated. Please use tf.estimator.CheckpointSaverListener instead.\n",
            "\n",
            "W0717 23:43:29.112908 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/ops.py:4260: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
            "\n",
            "W0717 23:43:29.137330 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/research/neural_stack.py:38: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead.\n",
            "\n",
            "W0717 23:43:29.171424 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/rl/gym_utils.py:235: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n",
            "W0717 23:43:29.194288 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:111: The name tf.OptimizerOptions is deprecated. Please use tf.compat.v1.OptimizerOptions instead.\n",
            "\n",
            "W0717 23:43:32.973552 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/sci_sum.py:73: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.\n",
            "\n",
            "W0717 23:43:34.598912 140002720810880 deprecation_wrapper.py:119] From /usr/local/bin/t2t-decoder:16: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
            "\n",
            "W0717 23:43:34.599116 140002720810880 deprecation_wrapper.py:119] From /usr/local/bin/t2t-decoder:16: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
            "\n",
            "W0717 23:43:34.599260 140002720810880 deprecation_wrapper.py:119] From /usr/local/bin/t2t-decoder:17: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.\n",
            "\n",
            "W0717 23:43:34.599762 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:839: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
            "\n",
            "W0717 23:43:34.600700 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/text_encoder.py:938: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.\n",
            "\n",
            "W0717 23:43:34.601318 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/data_generators/text_encoder.py:940: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n",
            "W0717 23:43:34.961399 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:123: The name tf.GraphOptions is deprecated. Please use tf.compat.v1.GraphOptions instead.\n",
            "\n",
            "W0717 23:43:34.961669 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:129: The name tf.GPUOptions is deprecated. Please use tf.compat.v1.GPUOptions instead.\n",
            "\n",
            "W0717 23:43:34.961901 140002720810880 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/trainer_lib.py:242: RunConfig.__init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.\n",
            "I0717 23:43:34.962164 140002720810880 trainer_lib.py:265] Configuring DataParallelism to replicate the model.\n",
            "I0717 23:43:34.962288 140002720810880 devices.py:76] schedule=continuous_train_and_eval\n",
            "I0717 23:43:34.962430 140002720810880 devices.py:77] worker_gpu=1\n",
            "I0717 23:43:34.962538 140002720810880 devices.py:78] sync=False\n",
            "W0717 23:43:34.962714 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/devices.py:139: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.\n",
            "\n",
            "W0717 23:43:34.962814 140002720810880 devices.py:141] Schedule=continuous_train_and_eval. Assuming that training is running on a single machine.\n",
            "I0717 23:43:34.963728 140002720810880 devices.py:170] datashard_devices: ['gpu:0']\n",
            "I0717 23:43:34.963837 140002720810880 devices.py:171] caching_devices: None\n",
            "I0717 23:43:34.964405 140002720810880 devices.py:172] ps_devices: ['gpu:0']\n",
            "I0717 23:43:34.965086 140002720810880 estimator.py:209] Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f548eccbcf8>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_train_distribute': None, '_eval_distribute': None, '_experimental_max_worker_delay_secs': None, '_device_fn': None, '_tf_config': gpu_options {\n",
            "  per_process_gpu_memory_fraction: 1.0\n",
            "}\n",
            ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_protocol': None, '_session_config': gpu_options {\n",
            "  per_process_gpu_memory_fraction: 0.95\n",
            "}\n",
            "allow_soft_placement: true\n",
            "graph_options {\n",
            "  optimizer_options {\n",
            "    global_jit_level: OFF\n",
            "  }\n",
            "}\n",
            "isolate_session_state: true\n",
            ", '_save_checkpoints_steps': 1000, '_keep_checkpoint_max': 20, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'gdrive/My Drive/SummarizationCheckPoints', 'use_tpu': False, 't2t_device_info': {'num_async_replicas': 1}, 'data_parallelism': <tensor2tensor.utils.expert_utils.Parallelism object at 0x7f548eccbd68>}\n",
            "W0717 23:43:34.965349 140002720810880 model_fn.py:630] Estimator's model_fn (<function T2TModel.make_estimator_model_fn.<locals>.wrapping_model_fn at 0x7f548f0a4f28>) includes params argument, but params are not passed to Estimator.\n",
            "I0717 23:43:34.972451 140002720810880 decoding.py:404] decode_hp.batch_size not specified; default=32\n",
            "I0717 23:43:34.972567 140002720810880 decoding.py:415] Performing decoding from file (Testtext3.txt).\n",
            "I0717 23:43:34.972663 140002720810880 decoding.py:860] Getting sorted inputs\n",
            "I0717 23:43:34.983731 140002720810880 decoding.py:673]  batch 1\n",
            "I0717 23:43:34.983889 140002720810880 decoding.py:675] Decoding batch 0\n",
            "W0717 23:43:34.987493 140002720810880 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/decoding.py:617: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "tf.py_func is deprecated in TF V2. Instead, there are two\n",
            "    options available in V2.\n",
            "    - tf.py_function takes a python function which manipulates tf eager\n",
            "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
            "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
            "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
            "    being differentiable using a gradient tape.\n",
            "    - tf.numpy_function maintains the semantics of the deprecated tf.py_func\n",
            "    (it is not differentiable, and manipulates numpy arrays). It drops the\n",
            "    stateful argument making all functions stateful.\n",
            "    \n",
            "W0717 23:43:34.990216 140002720810880 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/decoding.py:950: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "W0717 23:43:34.998590 140002720810880 estimator.py:1000] Input graph does not use tf.data.Dataset or contain a QueueRunner. That means predict yields forever. This is probably a mistake.\n",
            "I0717 23:43:34.998994 140002720810880 estimator.py:1145] Calling model_fn.\n",
            "I0717 23:43:35.000635 140002720810880 t2t_model.py:2245] Setting T2TModel mode to 'infer'\n",
            "I0717 23:43:35.001108 140002720810880 t2t_model.py:2245] Setting hparams.dropout to 0.0\n",
            "I0717 23:43:35.001275 140002720810880 t2t_model.py:2245] Setting hparams.label_smoothing to 0.0\n",
            "I0717 23:43:35.001404 140002720810880 t2t_model.py:2245] Setting hparams.layer_prepostprocess_dropout to 0.0\n",
            "I0717 23:43:35.001516 140002720810880 t2t_model.py:2245] Setting hparams.symbol_dropout to 0.0\n",
            "I0717 23:43:35.001674 140002720810880 t2t_model.py:2245] Setting hparams.attention_dropout to 0.0\n",
            "I0717 23:43:35.001795 140002720810880 t2t_model.py:2245] Setting hparams.relu_dropout to 0.0\n",
            "W0717 23:43:35.102075 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:244: The name tf.summary.text is deprecated. Please use tf.compat.v1.summary.text instead.\n",
            "\n",
            "I0717 23:43:35.119224 140002720810880 t2t_model.py:2245] Beam Decoding with beam size 3\n",
            "W0717 23:43:35.336982 140002720810880 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_attention.py:857: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "W0717 23:43:35.342806 140002720810880 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
            "W0717 23:43:35.403017 140002720810880 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/models/transformer.py:96: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
            "W0717 23:43:35.410570 140002720810880 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/expert_utils.py:621: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "W0717 23:43:35.445972 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_layers.py:3236: The name tf.layers.Dense is deprecated. Please use tf.compat.v1.layers.Dense instead.\n",
            "\n",
            "W0717 23:43:35.923579 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/layers/common_attention.py:1249: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.\n",
            "\n",
            "W0717 23:43:44.262510 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/t2t_model.py:1741: The name tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY is deprecated. Please use tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY instead.\n",
            "\n",
            "I0717 23:43:44.262940 140002720810880 estimator.py:1147] Done calling model_fn.\n",
            "I0717 23:43:44.892267 140002720810880 monitored_session.py:240] Graph was finalized.\n",
            "2019-07-17 23:43:44.898299: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz\n",
            "2019-07-17 23:43:44.898580: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x76bca80 executing computations on platform Host. Devices:\n",
            "2019-07-17 23:43:44.898618: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>\n",
            "2019-07-17 23:43:44.900802: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1\n",
            "2019-07-17 23:43:44.967936: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-17 23:43:44.968495: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x76bd180 executing computations on platform CUDA. Devices:\n",
            "2019-07-17 23:43:44.968538: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7\n",
            "2019-07-17 23:43:44.968800: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-17 23:43:44.969208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: \n",
            "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n",
            "pciBusID: 0000:00:04.0\n",
            "2019-07-17 23:43:44.969549: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0\n",
            "2019-07-17 23:43:44.970920: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0\n",
            "2019-07-17 23:43:44.972516: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0\n",
            "2019-07-17 23:43:44.972986: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0\n",
            "2019-07-17 23:43:44.974916: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0\n",
            "2019-07-17 23:43:44.976405: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0\n",
            "2019-07-17 23:43:44.980915: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7\n",
            "2019-07-17 23:43:44.981119: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-17 23:43:44.981809: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-17 23:43:44.982327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0\n",
            "2019-07-17 23:43:44.982405: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0\n",
            "2019-07-17 23:43:44.983867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
            "2019-07-17 23:43:44.983902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 \n",
            "2019-07-17 23:43:44.983919: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N \n",
            "2019-07-17 23:43:44.984305: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-17 23:43:44.984768: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2019-07-17 23:43:44.985174: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
            "2019-07-17 23:43:44.985246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10869 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)\n",
            "W0717 23:43:44.986013 140002720810880 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n",
            "I0717 23:43:44.988441 140002720810880 saver.py:1280] Restoring parameters from gdrive/My Drive/SummarizationCheckPoints/model.ckpt-1000\n",
            "2019-07-17 23:43:46.729637: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n",
            "I0717 23:43:46.755821 140002720810880 session_manager.py:500] Running local_init_op.\n",
            "I0717 23:43:46.840225 140002720810880 session_manager.py:502] Done running local_init_op.\n",
            "2019-07-17 23:43:49.118622: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0\n",
            "I0717 23:44:32.279644 140002720810880 decoding.py:152] Inference results INPUT: We demonstrate the importance of the deep bidirectionality of BERT by evaluating two pretraining objectives using exactly the same pretraining data, fine-tuning scheme, and hyperparameters as BERTBASE: No NSP: A bidirectional model which is trained using the “masked LM” (MLM) but without the “next sentence prediction” (NSP) task. LTR & No NSP: A left-context-only model which is trained using a standard Left-to-Right (LTR) LM, rather than an MLM. The left-only constraint was also applied at fine-tuning, because removing it introduced a pre-train/fine-tune mismatch that degraded downstream performance. Additionally, this model was pre-trained without the NSP task. This is directly comparable to OpenAI GPT, but using our larger training dataset, our input representation, and our fine-tuning scheme. We first examine the impact brought by the NSP task. In Table 5, we show that removing NSP hurts performance significantly on QNLI, MNLI, and SQuAD 1.1. Next, we evaluate the impact of training bidirectional representations by comparing “No NSP” to “LTR & No NSP”. The LTR model performs worse than the MLM model on all tasks, with large drops on MRPC and SQuAD. For SQuAD it is intuitively clear that a LTR model will perform poorly at token predictions, since the token-level hidden states have no rightside context. In order to make a good faith attempt at strengthening the LTR system, we added a randomly initialized BiLSTM on top. This does significantly improve results on SQuAD, but theHe spread his light among us like a candle.\n",
            "I0717 23:44:32.285180 140002720810880 decoding.py:167] Inference results OUTPUT: AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been used in this work, we have been been been been been been been used in the two-ray, we have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been used to the same than than, and the two two two-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray, and the data, and the two-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-based on the two-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-based on the two-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray. \n",
            "I0717 23:44:32.286712 140002720810880 decoding.py:152] Inference results INPUT: In this section, we present BERT fine-tuning results on 11 NLP tasks. 4.1 GLUE The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018a) is a collection of diverse natural language understanding tasks. Detailed descriptions of GLUE datasets are included in Appendix B.1. To fine-tune on GLUE, we represent the input sequence (for single sentence or sentence pairs) as described in Section 3, and use the final hidden vector C ∈ R H corresponding to the first input token ([CLS]) as the aggregate representation. The only new parameters introduced during fine-tuning are classification layer weights W ∈ R K×H, where K is the number of labels. We compute a standard classification loss with C and W, i.e., log(softmax(CWT )).\n",
            "I0717 23:44:32.291226 140002720810880 decoding.py:167] Inference results OUTPUT: AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been a large-to the data. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been a a different and, we use of the data. This work, and the data. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been to the data. We have been been been been been been been to the data. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been to the data. This work, we have been been been been been been been been been been been been been been been to the data. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been a large-to a large-to a large-to the two-to the two two-to the \n",
            "I0717 23:44:32.292053 140002720810880 decoding.py:152] Inference results INPUT: Recent empirical improvements due to transfer learning with language models have demonstrated that rich, unsupervised pre-training is an integral part of many language understanding systems. In particular, these results enable even low-resource tasks to benefit from deep unidirectional architectures. Our major contribution is further generalizing these findings to deep bidirectional architectures, allowing the same pre-trained model to successfully tackle a broad set of NLP tasks.\n",
            "I0717 23:44:32.297667 140002720810880 decoding.py:167] Inference results OUTPUT: . Conclusions Conclusions. We have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have a a a a a a a a a a a given by a large-to a given by a large-to a given by a given by a large of the most of the most of the most of the data. The results of the data. We have have have have have have have have been been been been a large of the other-to a given by a given by a given by a given by a single-ray of this work of the data. This is a large-ray of the most of the two-ray of the two-ray of the most of the most of this work of the two-ray of the data. We have a large-ray of the two-ray of the two-ray of the two-ray. This is not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not\n",
            "I0717 23:44:32.297939 140002720810880 decoding.py:152] Inference results INPUT: \n",
            "I0717 23:44:32.303428 140002720810880 decoding.py:167] Inference results OUTPUT: . Results. The ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////\n",
            "I0717 23:44:32.377710 140002720810880 decoding.py:520] Elapsed Time: 57.40431\n",
            "I0717 23:44:32.377928 140002720810880 decoding.py:524] Averaged Single Token Generation Time: 0.0181340 (time 57.3033507 count 3160)\n",
            "I0717 23:44:32.378035 140002720810880 decoding.py:532] Inference time 57.4043 seconds (Throughput = 0.0697 sentences/second)\n",
            "I0717 23:44:32.378167 140002720810880 decoding.py:542] Writing decodes into Testtext3.txt.transformer.transformer_prepend.summarize_scientific_sections_gdrive65k.beam3.alpha0.6.decodes\n",
            "W0717 23:44:32.378633 140002720810880 deprecation_wrapper.py:119] From /usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/decoding.py:550: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "geaAB_yw-PvB",
        "colab_type": "text"
      },
      "source": [
        "Read the output summaries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qM9eW5nWTOOJ",
        "colab_type": "code",
        "outputId": "61beefa6-24b3-4026-a786-3e853dd01272",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 104
        }
      },
      "source": [
        "%%bash  \n",
        "DECODE_FILE=Testtext.txt\n",
        "cat ${DECODE_FILE}.*.decodes"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            ". Conclusions Conclusions. We have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have a a a a a a a a a a a given by a large-to a given by a large-to a given by a given by a large of the most of the most of the most of the data. The results of the data. We have have have have have have have have been been been been a large of the other-to a given by a given by a given by a given by a single-ray of this work of the data. This is a large-ray of the most of the two-ray of the two-ray of the most of the most of this work of the two-ray of the data. We have a large-ray of the two-ray of the two-ray of the two-ray. This is not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not not\n",
            "AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been used in this work, we have been been been been been been been used in the two-ray, we have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been used to the same than than, and the two two two-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray, and the data, and the two-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-based on the two-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-based on the two-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray-ray. \n",
            "AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND AND. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been a large-to the data. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been a a different and, we use of the data. This work, and the data. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been to the data. We have been been been been been been been to the data. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been to the data. This work, we have been been been been been been been been been been been been been been been to the data. We have been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been been a large-to a large-to a large-to the two-to the two two-to the \n",
            ". Results. The ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}