7_djl/4.load_your_own_mxnet_bert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load your own MXNet BERT model\n",
    "\n",
    "In the previous [example](https://github.com/awslabs/djl/blob/master/jupyter/BERTQA.ipynb), you run BERT inference with the model from Model Zoo. You can also load the model on your own pre-trained BERT and use custom classes as the input and output.\n",
    "\n",
    "In general, the MXNet BERT model requires these three inputs:\n",
    "\n",
    "- word indices: The index of each word in a sentence\n",
    "- word types: The type index of the word.\n",
    "- valid length: The actual length of the question and resource document tokens\n",
    "\n",
    "We will dive deep into these details later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "This tutorial requires the installation of Java Kernel. To install the Java Kernel, see the [README](https://github.com/awslabs/djl/blob/master/jupyter/README.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are dependencies we will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:18+0000",
     "start_time": "2020-07-29T08:18:09.074Z"
    }
   },
   "outputs": [],
   "source": [
    "// %mavenRepo snapshots https://oss.sonatype.org/content/repositories/snapshots/\n",
    "\n",
    "%maven ai.djl:api:0.6.0\n",
    "%maven ai.djl.mxnet:mxnet-engine:0.6.0\n",
    "%maven ai.djl.mxnet:mxnet-model-zoo:0.6.0\n",
    "%maven org.slf4j:slf4j-api:1.7.26\n",
    "%maven org.slf4j:slf4j-simple:1.7.26\n",
    "%maven net.java.dev.jna:jna:5.3.0\n",
    "        \n",
    "// See https://github.com/awslabs/djl/blob/master/mxnet/mxnet-engine/README.md\n",
    "// for more MXNet library selection options\n",
    "%maven ai.djl.mxnet:mxnet-native-auto:1.7.0-b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import java packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:18+0000",
     "start_time": "2020-07-29T08:18:11.109Z"
    }
   },
   "outputs": [],
   "source": [
    "import java.io.*;\n",
    "import java.nio.file.*;\n",
    "import java.util.*;\n",
    "import java.util.stream.*;\n",
    "\n",
    "import ai.djl.*;\n",
    "import ai.djl.ndarray.*;\n",
    "import ai.djl.ndarray.types.*;\n",
    "import ai.djl.inference.*;\n",
    "import ai.djl.translate.*;\n",
    "import ai.djl.training.util.*;\n",
    "import ai.djl.repository.zoo.*;\n",
    "import ai.djl.modality.nlp.*;\n",
    "import ai.djl.modality.nlp.qa.*;\n",
    "import ai.djl.mxnet.zoo.nlp.qa.*;\n",
    "import ai.djl.modality.nlp.bert.*;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reuse the previous input**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:18+0000",
     "start_time": "2020-07-29T08:18:12.023Z"
    }
   },
   "outputs": [],
   "source": [
    "var question = \"When did BBC Japan start broadcasting?\";\n",
    "var resourceDocument = \"BBC Japan was a general entertainment Channel.\\n\" +\n",
    "    \"Which operated between December 2004 and April 2006.\\n\" +\n",
    "    \"It ceased operations after its Japanese distributor folded.\";\n",
    "\n",
    "QAInput input = new QAInput(question, resourceDocument);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dive deep into Translator\n",
    "\n",
    "Inference in deep learning is the process of predicting the output for a given input based on a pre-defined model.\n",
    "DJL abstracts away the whole process for ease of use. It can load the model, perform inference on the input, and provide\n",
    "output. DJL also allows you to provide user-defined inputs. The workflow looks like the following:\n",
    "\n",
    "![https://github.com/awslabs/djl/blob/master/examples/docs/img/workFlow.png?raw=true](https://github.com/awslabs/djl/blob/master/examples/docs/img/workFlow.png?raw=true)\n",
    "\n",
    "The red block (\"Images\") in the workflow is the input that DJL expects from you. The green block (\"Images\n",
    "bounding box\") is the output that you expect. Because DJL does not know which input to expect and which output format that you prefer, DJL provides the `Translator` interface so you can define your own\n",
    "input and output.\n",
    "\n",
    "The `Translator` interface encompasses the two white blocks: Pre-processing and Post-processing. The pre-processing\n",
    "component converts the user-defined input objects into an NDList, so that the `Predictor` in DJL can understand the\n",
    "input and make its prediction. Similarly, the post-processing block receives an NDList as the output from the\n",
    "`Predictor`. The post-processing block allows you to convert the output from the `Predictor` to the desired output\n",
    "format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-processing\n",
    "\n",
    "Now, you need to convert the sentences into tokens. We provide a powerful tool `BertTokenizer` that you can use to convert questions and answers into tokens, and batchify your sequence together. Once you have properly formatted tokens, you can use `Vocabulary` to map your token to BERT index.\n",
    "\n",
    "The following code block demonstrates tokenizing the question and answer defined earlier into BERT-formatted tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:18+0000",
     "start_time": "2020-07-29T08:18:16.777Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question Token: [when, did, bbc, japan, start, broadcasting, ?]\n",
      "Answer Token: [bbc, japan, was, a, general, entertainment, channel, ., which, operated, between, december, 2004, and, april, 2006, ., it, ceased, operations, after, its, japanese, distributor, folded, .]\n"
     ]
    }
   ],
   "source": [
    "var tokenizer = new BertTokenizer();\n",
    "List<String> tokenQ = tokenizer.tokenize(question.toLowerCase());\n",
    "List<String> tokenA = tokenizer.tokenize(resourceDocument.toLowerCase());\n",
    "\n",
    "System.out.println(\"Question Token: \" + tokenQ);\n",
    "System.out.println(\"Answer Token: \" + tokenA);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BertTokenizer` can also help you batchify questions and resource documents together by calling `encode()`.\n",
    "The output contains information that BERT ingests.\n",
    "\n",
    "- getTokens: It returns a list of strings, including the question, resource document and special word to let the model tell which part is the question and which part is the resource document. Because MXNet BERT was trained with a fixed sequence length, you see the `[PAD]` in the tokens as well.\n",
    "- getTokenTypes: It returns a list of type indices of the word to indicate the location of the resource document. All Questions will be labelled with 0 and all resource documents will be labelled with 1.\n",
    "\n",
    "    [Question tokens...DocResourceTokens...padding tokens] => [000000...11111....0000]\n",
    "    \n",
    "\n",
    "- getValidLength: It returns the actual length of the question and tokens, which are required by MXNet BERT.\n",
    "- getAttentionMask: It returns the mask for the model to indicate which part should be paid attention to and which part is the padding. It is required by PyTorch BERT.\n",
    "\n",
    "    [Question tokens...DocResourceTokens...padding tokens] => [111111...11111....0000]\n",
    "    \n",
    "MXNet BERT was trained with fixed sequence length 384, so we need to pass that in when we encode the question and resource doc.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:18+0000",
     "start_time": "2020-07-29T08:18:20.045Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded tokens: [[CLS], when, did, bbc, japan, start, broadcasting, ?, [SEP], bbc, japan, was, a, general, entertainment, channel, ., which, operated, between, december, 2004, and, april, 2006, ., it, ceased, operations, after, its, japanese, distributor, folded, ., [SEP], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD], [PAD]]\n",
      "Encoded token type: [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
      "Valid length: 33\n"
     ]
    }
   ],
   "source": [
    "BertToken token = tokenizer.encode(question.toLowerCase(), resourceDocument.toLowerCase(), 384);\n",
    "System.out.println(\"Encoded tokens: \" + token.getTokens());\n",
    "System.out.println(\"Encoded token type: \" + token.getTokenTypes());\n",
    "System.out.println(\"Valid length: \" + token.getValidLength());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, words and sentences are represented as indices instead of tokens for training. \n",
    "They typically work like a vector in a n-dimensional space. In this case, you need to map them into indices.\n",
    "DJL provides `Vocabulary` to take care of you vocabulary mapping.\n",
    "\n",
    "Assume your vocab.json is of the following format\n",
    "```\n",
    "{'token_to_idx':{'\"slots\": 19832,...}, 'idx_to_token':[\"[UNK]\", \"[PAD]\", ...]}\n",
    "```\n",
    "We provide the `vocab.json` from our pre-trained BERT for demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:19+0000",
     "start_time": "2020-07-29T08:18:26.896Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading: 100% |████████████████████████████████████████| vocab.json\n"
     ]
    }
   ],
   "source": [
    "DownloadUtils.download(\"https://djl-ai.s3.amazonaws.com/mlrepo/model/nlp/question_answer/ai/djl/mxnet/bertqa/vocab.json\", \"build/mxnet/bertqa/vocab.json\", new ProgressBar());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:19+0000",
     "start_time": "2020-07-29T08:19:17.908Z"
    }
   },
   "outputs": [],
   "source": [
    "var file = Paths.get(\"build/mxnet/bertqa/vocab.json\");\n",
    "var inputStream = Files.newInputStream(file);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:19+0000",
     "start_time": "2020-07-29T08:19:18.738Z"
    }
   },
   "outputs": [],
   "source": [
    "var vocabulary = MxBertVocabulary.parse(inputStream);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily convert the token to the index using `vocabulary.getIndex(token)` and the other way around using `vocabulary.getToken(index)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:19+0000",
     "start_time": "2020-07-29T08:19:22.078Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The index of the car is 2482\n",
      "The token of the index 2482 is car\n"
     ]
    }
   ],
   "source": [
    "long index = vocabulary.getIndex(\"car\");\n",
    "String token = vocabulary.getToken(2482);\n",
    "System.out.println(\"The index of the car is \" + index);\n",
    "System.out.println(\"The token of the index 2482 is \" + token);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To properly convert them into `float[]` for `NDArray` creation, use the following helper function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:19+0000",
     "start_time": "2020-07-29T08:19:24.946Z"
    }
   },
   "outputs": [],
   "source": [
    "/**\n",
    " * Convert a List of Number to float array.\n",
    " *\n",
    " * @param list the list to be converted\n",
    " * @return float array\n",
    " */\n",
    "public static float[] toFloatArray(List<? extends Number> list) {\n",
    "    float[] ret = new float[list.size()];\n",
    "    int idx = 0;\n",
    "    for (Number n : list) {\n",
    "        ret[idx++] = n.floatValue();\n",
    "    }\n",
    "    return ret;\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have everything you need, you can create an NDList and populate all of the inputs you formatted earlier. You're done with pre-processing! \n",
    "\n",
    "#### Construct `Translator`\n",
    "\n",
    "You need to do this processing within an implementation of the `Translator` interface. `Translator` is designed to do pre-processing and post-processing. You must define the input and output objects. It contains the following two override classes:\n",
    "- `public NDList processInput(TranslatorContext ctx, I)`\n",
    "- `public String processOutput(TranslatorContext ctx, O)`\n",
    "\n",
    "Every translator takes in input and returns output in the form of generic objects. In this case, the translator takes input in the form of `QAInput` (I) and returns output as a `String` (O). `QAInput` is just an object that holds questions and answer; We have prepared the Input class for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Armed with the needed knowledge, you can write an implementation of the `Translator` interface. `BertTranslator` uses the code snippets explained previously to implement the `processInput`method. For more information, see [`NDManager`](https://javadoc.io/static/ai.djl/api/0.6.0/index.html?ai/djl/ndarray/NDManager.html).\n",
    "\n",
    "```\n",
    "manager.create(Number[] data, Shape)\n",
    "manager.create(Number[] data)\n",
    "```\n",
    "\n",
    "The `Shape` for `data0` and `data1` is sequence_length. For `data2` the `Shape` is just 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:19+0000",
     "start_time": "2020-07-29T08:19:28.846Z"
    }
   },
   "outputs": [],
   "source": [
    "public class BertTranslator implements Translator<QAInput, String> {\n",
    "    private List<String> tokens;\n",
    "    \n",
    "    @Override\n",
    "    public Batchifier getBatchifier() {\n",
    "        return null;\n",
    "    }\n",
    "        \n",
    "    @Override\n",
    "    public NDList processInput(TranslatorContext ctx, QAInput input) throws IOException {\n",
    "        var inputStream = Files.newInputStream(Paths.get(\"build/mxnet/bertqa/vocab.json\"));\n",
    "        Vocabulary vocabulary = MxBertVocabulary.parse(inputStream);\n",
    "        BertTokenizer tokenizer = new BertTokenizer();\n",
    "        BertToken token =\n",
    "            tokenizer.encode(\n",
    "                input.getQuestion().toLowerCase(),\n",
    "                input.getParagraph().toLowerCase(),\n",
    "                384);\n",
    "        // get the encoded tokens that would be used in precessOutput\n",
    "        tokens = token.getTokens();\n",
    "        // map the tokens(String) to indices(long)\n",
    "        List<Long> indices =\n",
    "            token.getTokens().stream().map(vocabulary::getIndex).collect(Collectors.toList());\n",
    "        float[] indexesFloat = toFloatArray(indices);\n",
    "        float[] types = toFloatArray(token.getTokenTypes());\n",
    "        int validLength = token.getValidLength();\n",
    "\n",
    "        NDManager manager = ctx.getNDManager();\n",
    "        NDArray data0 = manager.create(indexesFloat);\n",
    "        data0.setName(\"data0\");\n",
    "        NDArray data1 = manager.create(types);\n",
    "        data1.setName(\"data1\");\n",
    "        NDArray data2 = manager.create(new float[] {validLength});\n",
    "        data2.setName(\"data2\");\n",
    "        return new NDList(data0, data1, data2);\n",
    "    }\n",
    "\n",
    "    @Override\n",
    "    public String processOutput(TranslatorContext ctx, NDList list) {\n",
    "        NDArray array = list.singletonOrThrow();\n",
    "        NDList output = array.split(2, 2);\n",
    "        // Get the formatted logits result\n",
    "        NDArray startLogits = output.get(0).reshape(new Shape(1, -1));\n",
    "        NDArray endLogits = output.get(1).reshape(new Shape(1, -1));\n",
    "        int startIdx = (int) startLogits.argMax(1).getLong();\n",
    "        int endIdx = (int) endLogits.argMax(1).getLong();\n",
    "        return tokens.subList(startIdx, endIdx + 1).toString();\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congrats! You have created your first Translator! We have pre-filled the `processOutput()` function to process the `NDList` and return it in a desired format. `processInput()` and `processOutput()` offer the flexibility to get the predictions from the model in any format you desire. \n",
    "\n",
    "With the Translator implemented, you need to bring up the predictor that uses your `Translator` to start making predictions. You can find the usage for `Predictor` in the [Predictor Javadoc](https://javadoc.io/static/ai.djl/api/0.6.0/index.html?ai/djl/inference/Predictor.html). Create a translator and use the `question` and `resourceDocument` provided previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-29T08:33+0000",
     "start_time": "2020-07-29T08:19:38.640Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading: 100% |████████████████████████████████████████| bertqa-symbol.json\n",
      "Downloading:   2% |█                                       | bertqa-0000.params"
     ]
    },
    {
     "ename": "EvalException",
     "evalue": "Connection reset",
     "output_type": "error",
     "traceback": [
      "\u001b[1m\u001b[31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1m\u001b[31mjavax.net.ssl.SSLException: Connection reset\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:327)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:270)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:265)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:144)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1199)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1166)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:832)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.net.www.MeteredStream.read(MeteredStream.java:134)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.io.FilterInputStream.read(FilterInputStream.java:133)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3510)\u001b[0m",
      "\u001b[1m\u001b[31m\tat ai.djl.training.util.DownloadUtils$ProgressInputStream.read(DownloadUtils.java:114)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:243)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.io.InputStream.transferTo(InputStream.java:704)\u001b[0m",
      "\u001b[1m\u001b[31m\tat java.base/java.nio.file.Files.copy(Files.java:3077)\u001b[0m",
      "\u001b[1m\u001b[31m\tat ai.djl.training.util.DownloadUtils.download(DownloadUtils.java:80)\u001b[0m",
      "\u001b[1m\u001b[31m\tat ai.djl.training.util.DownloadUtils.download(DownloadUtils.java:50)\u001b[0m",
      "\u001b[1m\u001b[31m\tat .(#75:1)\u001b[0m"
     ]
    }
   ],
   "source": [
    "DownloadUtils.download(\"https://djl-ai.s3.amazonaws.com/mlrepo/model/nlp/question_answer/ai/djl/mxnet/bertqa/0.0.1/static_bert_qa-symbol.json\", \"build/mxnet/bertqa/bertqa-symbol.json\", new ProgressBar());\n",
    "DownloadUtils.download(\"https://djl-ai.s3.amazonaws.com/mlrepo/model/nlp/question_answer/ai/djl/mxnet/bertqa/0.0.1/static_bert_qa-0002.params.gz\", \"build/mxnet/bertqa/bertqa-0000.params\", new ProgressBar());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BertTranslator translator = new BertTranslator();\n",
    "\n",
    "Criteria<QAInput, String> criteria = Criteria.builder()\n",
    "        .setTypes(QAInput.class, String.class)\n",
    "        .optModelUrls(\"build/mxnet/bertqa/\") // Search for models in the build/mxnet/bert folder\n",
    "        .optTranslator(translator)\n",
    "        .optProgress(new ProgressBar()).build();\n",
    "\n",
    "ZooModel model = ModelZoo.loadModel(criteria);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "String predictResult = null;\n",
    "QAInput input = new QAInput(question, resourceDocument);\n",
    "\n",
    "// Create a Predictor and use it to predict the output\n",
    "try (Predictor<QAInput, String> predictor = model.newPredictor(translator)) {\n",
    "    predictResult = predictor.predict(input);\n",
    "}\n",
    "\n",
    "System.out.println(question);\n",
    "System.out.println(predictResult);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the input, the following result will be shown:\n",
    "```\n",
    "[december, 2004]\n",
    "```\n",
    "That's it! \n",
    "\n",
    "You can try with more questions and answers. Here are the samples:\n",
    "\n",
    "**Answer Material**\n",
    "\n",
    "The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants would gradually merge with the Carolingian-based cultures of West Francia. The distinct cultural and ethnic identity of the Normans emerged initially in the first half of the 10th century, and it continued to evolve over the succeeding centuries.\n",
    "\n",
    "\n",
    "**Question**\n",
    "\n",
    "Q: When were the Normans in Normandy?\n",
    "A: 10th and 11th centuries\n",
    "\n",
    "Q: In what country is Normandy located?\n",
    "A: france\n",
    "\n",
    "For the full source code,see the [DJL repo](https://github.com/awslabs/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/BertQaInference.java) and translator implementation [MXNet](https://github.com/awslabs/djl/blob/master/mxnet/mxnet-model-zoo/src/main/java/ai/djl/mxnet/zoo/nlp/qa/MxBertQATranslator.java) [PyTorch](https://github.com/awslabs/djl/blob/master/pytorch/pytorch-model-zoo/src/main/java/ai/djl/pytorch/zoo/nlp/qa/PtBertQATranslator.java)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "11.0.8+10-LTS"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}