demo07-transformers.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VvYQfJ72Bln-"
      },
      "source": [
        "# Terrific Translation with Transformers\n",
        "\n",
        "Transformers were developed in the natural language processing (NLP) domain and now form a key component of the cutting edge deep learning architectures, including the recent architecture [Stable Diffusion](https://en.wikipedia.org/wiki/Stable_Diffusion).\n",
        "\n",
        "In this demo, we will focus on the NLP domain and build a transformer from scratch to translate German to English. Our code is based on the excellent pytorch tutorial [here](https://pytorch.org/tutorials/beginner/translation_transformer.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ah5BCnVYDk3-"
      },
      "source": [
        "## Libraries! Bibliotheken!\n",
        "\n",
        "We start by loading the (numerous) libraries we will use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "U6stAzq8BlId"
      },
      "outputs": [],
      "source": [
        "# All libraries\n",
        "from torchtext.data.utils import get_tokenizer\n",
        "from torchtext.vocab import build_vocab_from_iterator\n",
        "from torchtext.datasets import multi30k, Multi30k\n",
        "from typing import Iterable, List\n",
        "from torch import Tensor\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.nn import Transformer\n",
        "import math\n",
        "from torch.nn.utils.rnn import pad_sequence\n",
        "from torch.utils.data import DataLoader\n",
        "from timeit import default_timer as timer\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r06ydV4BCKen"
      },
      "source": [
        "## Language Set Up\n",
        "\n",
        "In order to work with language, we need to convert words as we read them in letters to \"tokens\" which are a numeric representation. This process is called tokenization and we will install both the dataset and tokenizers for English and German on our machine. (The following code should only be run once and should take about a minute to run.)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "P0NUQRXRCQiz"
      },
      "outputs": [],
      "source": [
        "first_run = False\n",
        "# Run once to install dataset and tokenizers for English and German\n",
        "if first_run:\n",
        "  !pip install -U torchdata\n",
        "  !pip install -U spacy\n",
        "  !python -m spacy download en_core_web_sm\n",
        "  !python -m spacy download de_core_news_sm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UBjtWZLeCstO"
      },
      "source": [
        "Now we will load the datasets and store the tokenizers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "zvBUhoJCBsrg"
      },
      "outputs": [],
      "source": [
        "multi30k.URL[\"train\"] = \"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz\"\n",
        "multi30k.URL[\"valid\"] = \"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz\"\n",
        "\n",
        "SRC_LANGUAGE = 'de'\n",
        "TGT_LANGUAGE = 'en'\n",
        "\n",
        "token_transform = {}\n",
        "vocab_transform = {}\n",
        "\n",
        "token_transform[SRC_LANGUAGE] = get_tokenizer('spacy', language='de_core_news_sm')\n",
        "token_transform[TGT_LANGUAGE] = get_tokenizer('spacy', language='en_core_web_sm')\n",
        "\n",
        "def yield_tokens(data_iter: Iterable, language: str) -> List[str]:\n",
        "  language_index = {SRC_LANGUAGE: 0, TGT_LANGUAGE: 1}\n",
        "  for data_sample in data_iter:\n",
        "    yield token_transform[language](data_sample[language_index[language]])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X0n9JCGdJXFJ"
      },
      "source": [
        "There are special characters which will use to structure the sequences of tokens (representing sentences of words). The `<box>` token starts a sequence while the `<eos>` token ends it. `<pad>` is used for sequences of length less than the maximum and `<unk>` is used for unknown words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "gJMmjqnSJWlH"
      },
      "outputs": [],
      "source": [
        "UNK_IDX, PAD_IDX, BOS_IDX, EOS_IDX = 0,1,2,3\n",
        "special_symbols = ['<unk>', '<pad>', '<box>', '<eos>']\n",
        "\n",
        "for language in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
        "  train_iter = Multi30k(split='train', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
        "  vocab_transform[language] = build_vocab_from_iterator(yield_tokens(train_iter, language),\n",
        "                                                        specials=special_symbols)\n",
        "                                                        \n",
        "for language in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
        "  vocab_transform[language].set_default_index(UNK_IDX)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rtldDo01Dl_P"
      },
      "source": [
        "## Seq2Seq Network\n",
        "\n",
        "We will use the Seq2Seq model introduced in \"[Attention Is All You Need](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\".\n",
        "\n",
        "The first step of the model is to embed the tokenized words into a smaller latent space that captures the meaning of the words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "JKLGbIZcEo8n"
      },
      "outputs": [],
      "source": [
        "class TokenEmbedding(nn.Module):\n",
        "    def __init__(self, vocab_size: int, emb_size: int):\n",
        "        super(TokenEmbedding, self).__init__()\n",
        "        self.embedding = nn.Embedding(vocab_size, emb_size)\n",
        "        self.emb_size = emb_size\n",
        "\n",
        "    def forward(self, tokens: Tensor):\n",
        "        return self.embedding(tokens.long()) * math.sqrt(self.emb_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fLKQDYyhEpOJ"
      },
      "source": [
        "Unlike recurrent networks, self-attention does not automatically capture the order that words appear. We need to fix this with a special positional encoding. Naive positional encodings like a one-hot approach or normalizing the index between $[0,1]$ fail due to variable sequence lenghts. The very clever solution in the original paper is to use a vector of sinusoidal functions at different frequencies. The video linked [here](https://www.youtube.com/watch?v=dichIcUZfOw) provides a high-level introduction while the article [here](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/) explains more of the math behind why they work. In particular, the positional encoding is able to capture the *relative* distance between indices."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "7cza0HMZD3hH"
      },
      "outputs": [],
      "source": [
        "class PositionalEncoding(nn.Module):\n",
        "    def __init__(self,\n",
        "                 emb_size: int,\n",
        "                 dropout: float,\n",
        "                 maxlen: int = 5000):\n",
        "        super(PositionalEncoding, self).__init__()\n",
        "        den = torch.exp(- torch.arange(0, emb_size, 2)* math.log(10000) / emb_size)\n",
        "        pos = torch.arange(0, maxlen).reshape(maxlen, 1)\n",
        "        pos_embedding = torch.zeros((maxlen, emb_size))\n",
        "        pos_embedding[:, 0::2] = torch.sin(pos * den)\n",
        "        pos_embedding[:, 1::2] = torch.cos(pos * den)\n",
        "        pos_embedding = pos_embedding.unsqueeze(-2)\n",
        "\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.register_buffer('pos_embedding', pos_embedding)\n",
        "\n",
        "    def forward(self, token_embedding: Tensor):\n",
        "        return self.dropout(token_embedding + self.pos_embedding[:token_embedding.size(0), :])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzk7-HLIKvNR"
      },
      "source": [
        "Now that we have the tools to embed the tokens, we will define the transformer architecture. Notice that all the exciting self-attention gets hidden under the hood in `Transformer`. We will peel this back a little at the end when we investigate the weights of the self-attention mechanism."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "B61YzMRyFIdA"
      },
      "outputs": [],
      "source": [
        "class Seq2SeqTransformer(nn.Module):\n",
        "    def __init__(self, num_encoder_layers: int, num_decoder_layers: int, emb_size: int,\n",
        "                 nhead: int, src_vocab_size: int, tgt_vocab_size: int, dim_feedforward: int = 512,\n",
        "                 dropout: float = 0.1):\n",
        "        super(Seq2SeqTransformer, self).__init__()\n",
        "        self.transformer = Transformer(d_model=emb_size, nhead=nhead,\n",
        "                                       num_encoder_layers=num_encoder_layers,\n",
        "                                       num_decoder_layers=num_decoder_layers,\n",
        "                                       dim_feedforward=dim_feedforward,\n",
        "                                       dropout=dropout)\n",
        "        self.generator = nn.Linear(emb_size, tgt_vocab_size)\n",
        "        self.src_tok_emb = TokenEmbedding(src_vocab_size, emb_size)\n",
        "        self.tgt_tok_emb = TokenEmbedding(tgt_vocab_size, emb_size)\n",
        "        self.positional_encoding = PositionalEncoding(\n",
        "            emb_size, dropout=dropout)\n",
        "\n",
        "    def forward(self, src: Tensor, trg: Tensor, src_mask: Tensor, tgt_mask: Tensor,\n",
        "                src_padding_mask: Tensor, tgt_padding_mask: Tensor,\n",
        "                memory_key_padding_mask: Tensor):\n",
        "        src_emb = self.positional_encoding(self.src_tok_emb(src))\n",
        "        tgt_emb = self.positional_encoding(self.tgt_tok_emb(trg))\n",
        "        outs = self.transformer(src_emb, tgt_emb, src_mask, tgt_mask, None,\n",
        "                                src_padding_mask, tgt_padding_mask, memory_key_padding_mask)\n",
        "        return self.generator(outs)\n",
        "\n",
        "    def encode(self, src: Tensor, src_mask: Tensor):\n",
        "        return self.transformer.encoder(self.positional_encoding(\n",
        "                            self.src_tok_emb(src)), src_mask)\n",
        "\n",
        "    def decode(self, tgt: Tensor, memory: Tensor, tgt_mask: Tensor):\n",
        "        return self.transformer.decoder(self.positional_encoding(\n",
        "                          self.tgt_tok_emb(tgt)), memory,\n",
        "                          tgt_mask)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M9Olx53NK-46"
      },
      "source": [
        "A key part of training is hiding the tokens we haven't seen yet (otherwise, the problem would be very easy!)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "OauEompFGQJh"
      },
      "outputs": [],
      "source": [
        "def generate_square_subsequent_mask(sz):\n",
        "    mask = (torch.triu(torch.ones((sz, sz), device=DEVICE)) == 1).transpose(0, 1)\n",
        "    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n",
        "    return mask\n",
        "\n",
        "\n",
        "def create_mask(src, tgt):\n",
        "    src_seq_len = src.shape[0]\n",
        "    tgt_seq_len = tgt.shape[0]\n",
        "\n",
        "    tgt_mask = generate_square_subsequent_mask(tgt_seq_len)\n",
        "    src_mask = torch.zeros((src_seq_len, src_seq_len),device=DEVICE).type(torch.bool)\n",
        "\n",
        "    src_padding_mask = (src == PAD_IDX).transpose(0, 1)\n",
        "    tgt_padding_mask = (tgt == PAD_IDX).transpose(0, 1)\n",
        "    return src_mask, tgt_mask, src_padding_mask, tgt_padding_mask"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ru8zZsoDQsD"
      },
      "source": [
        "## Initialization\n",
        "\n",
        "We initialize the model with the standard hyperparameters chosen in the original paper. The resulting number of parameters is huge!!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "gPmTO9uIHm-O"
      },
      "outputs": [],
      "source": [
        "torch.manual_seed(0)\n",
        "\n",
        "SRC_VOCAB_SIZE = len(vocab_transform[SRC_LANGUAGE])\n",
        "TGT_VOCAB_SIZE = len(vocab_transform[TGT_LANGUAGE])\n",
        "EMB_SIZE = 512\n",
        "NHEAD = 8\n",
        "FFN_HID_DIM = 512\n",
        "BATCH_SIZE = 128\n",
        "NUM_ENCODER_LAYERS = 3\n",
        "NUM_DECODER_LAYERS = 3\n",
        "\n",
        "DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "transformer = Seq2SeqTransformer(NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, EMB_SIZE,\n",
        "                                 NHEAD, SRC_VOCAB_SIZE, TGT_VOCAB_SIZE, FFN_HID_DIM)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HIMEtKIgLZm0"
      },
      "source": [
        "We will use the Xavier initialization (Gaussian with a special variance) in order to keep the norm of the weights close to 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mDCljThSLYul",
        "outputId": "34851cbb-61fa-4966-a649-e0d4b4107dd2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are 33570389 parameters in the model!\n"
          ]
        }
      ],
      "source": [
        "count = 0\n",
        "for p in transformer.parameters():\n",
        "    count += p.numel()\n",
        "    if p.dim() > 1:\n",
        "        nn.init.xavier_uniform_(p)\n",
        "\n",
        "# Architecture\n",
        "transformer = transformer.to(DEVICE)\n",
        "# Loss function\n",
        "loss_fn = torch.nn.CrossEntropyLoss(ignore_index=PAD_IDX)\n",
        "# Optimization method\n",
        "optimizer = torch.optim.Adam(transformer.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9)\n",
        "\n",
        "print(f'There are {count} parameters in the model!')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JR1rAgl-LmbI"
      },
      "source": [
        "We will also need some helper functions for processing the inputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "evkcDw5hKCGO"
      },
      "outputs": [],
      "source": [
        "def sequential_transforms(*transforms):\n",
        "    def func(txt_input):\n",
        "        for transform in transforms:\n",
        "            txt_input = transform(txt_input)\n",
        "        return txt_input\n",
        "    return func\n",
        "\n",
        "# function to add BOS/EOS and create tensor for input sequence indices\n",
        "def tensor_transform(token_ids: List[int]):\n",
        "    return torch.cat((torch.tensor([BOS_IDX]),\n",
        "                      torch.tensor(token_ids),\n",
        "                      torch.tensor([EOS_IDX])))\n",
        "\n",
        "# src and tgt language text transforms to convert raw strings into tensors indices\n",
        "text_transform = {}\n",
        "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
        "    text_transform[ln] = sequential_transforms(token_transform[ln], #Tokenization\n",
        "                                               vocab_transform[ln], #Numericalization\n",
        "                                               tensor_transform) # Add BOS/EOS and create tensor\n",
        "\n",
        "\n",
        "# function to collate data samples into batch tesors\n",
        "def collate_fn(batch):\n",
        "    src_batch, tgt_batch = [], []\n",
        "    for src_sample, tgt_sample in batch:\n",
        "        src_batch.append(text_transform[SRC_LANGUAGE](src_sample.rstrip(\"\\n\")))\n",
        "        tgt_batch.append(text_transform[TGT_LANGUAGE](tgt_sample.rstrip(\"\\n\")))\n",
        "\n",
        "    src_batch = pad_sequence(src_batch, padding_value=PAD_IDX)\n",
        "    tgt_batch = pad_sequence(tgt_batch, padding_value=PAD_IDX)\n",
        "    return src_batch, tgt_batch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JZSoS8JiDWoR"
      },
      "source": [
        "## Training\n",
        "\n",
        "Finally, we will get to train the model!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "NRDrvU87Kj2o"
      },
      "outputs": [],
      "source": [
        "def run(model, runtype: str, optimizer=None):\n",
        "  assert runtype in ['train', 'valid']\n",
        "  if runtype == 'train': model.train()\n",
        "  else: model.eval()\n",
        "  \n",
        "  losses = 0\n",
        "  data_iter = Multi30k(split=runtype, language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
        "  dataloader = DataLoader(data_iter, batch_size=BATCH_SIZE, collate_fn=collate_fn)\n",
        "  num_points = 0\n",
        "\n",
        "  for src, tgt in dataloader:\n",
        "    num_points += src.shape[1]\n",
        "    src = src.to(DEVICE)\n",
        "    tgt = tgt.to(DEVICE)\n",
        "\n",
        "    tgt_input = tgt[:-1, :]\n",
        "\n",
        "    src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input)\n",
        "\n",
        "    logits = model(src, tgt_input, src_mask, tgt_mask, src_padding_mask,\n",
        "                   tgt_padding_mask, src_padding_mask)\n",
        "    \n",
        "    if runtype == 'train': optimizer.zero_grad()\n",
        "\n",
        "    tgt_out = tgt[1:, :]\n",
        "    loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))\n",
        "\n",
        "    if runtype == 'train':\n",
        "      loss.backward()\n",
        "      optimizer.step()\n",
        "    \n",
        "    losses += loss.item()\n",
        "  \n",
        "  return losses / num_points\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4iFRGdXwL48C"
      },
      "source": [
        "Even though the model has more than 30 million parameters, we'll be able to train it for 10 epochs (in about 6 minutes on colab GPU). While not perfect, the results are pretty good!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LEEUl_NFOsMJ",
        "outputId": "e8fb2114-d3a7-4095-83be-864493270ff4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch: 0, Train loss: 0.042, Val loss: 0.032, Epoch time = 40.146s\n",
            "Epoch: 1, Train loss: 0.029, Val loss: 0.026, Epoch time = 39.424s\n",
            "Epoch: 2, Train loss: 0.025, Val loss: 0.023, Epoch time = 38.848s\n",
            "Epoch: 3, Train loss: 0.022, Val loss: 0.021, Epoch time = 38.938s\n",
            "Epoch: 4, Train loss: 0.019, Val loss: 0.019, Epoch time = 39.020s\n",
            "Epoch: 5, Train loss: 0.018, Val loss: 0.018, Epoch time = 39.290s\n",
            "Epoch: 6, Train loss: 0.016, Val loss: 0.017, Epoch time = 39.032s\n",
            "Epoch: 7, Train loss: 0.015, Val loss: 0.017, Epoch time = 39.010s\n",
            "Epoch: 8, Train loss: 0.014, Val loss: 0.016, Epoch time = 39.075s\n",
            "Epoch: 9, Train loss: 0.013, Val loss: 0.016, Epoch time = 38.981s\n"
          ]
        }
      ],
      "source": [
        "NUM_EPOCHS = 10\n",
        "\n",
        "for epoch in range(NUM_EPOCHS):\n",
        "  start_time = timer()\n",
        "  train_loss = run(transformer, runtype='train', optimizer=optimizer)\n",
        "  end_time = timer()\n",
        "  val_loss = run(transformer, runtype='valid')\n",
        "  print((f\"Epoch: {epoch}, Train loss: {train_loss:.3f}, Val loss: {val_loss:.3f}, \"f\"Epoch time = {(end_time - start_time):.3f}s\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4OLSrlp4DZwA"
      },
      "source": [
        "## Actual Translation\n",
        "\n",
        "We've trained the weights to give a good estimate of what German sentences correspond to in English. In order to do the translation, we'll need to pick the English words with the highest probability given by the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "_YVpu8aMPsj8"
      },
      "outputs": [],
      "source": [
        "def greedy_decode(model, src, src_mask, max_len, start_symbol):\n",
        "  src = src.to(DEVICE)\n",
        "  src_mask = src_mask.to(DEVICE)\n",
        "\n",
        "  memory = model.encode(src, src_mask)\n",
        "  ys = torch.ones(1,1).fill_(start_symbol).type(torch.long).to(DEVICE)\n",
        "  for i in range(max_len-1):\n",
        "    memory = memory.to(DEVICE)\n",
        "    tgt_mask = (generate_square_subsequent_mask(ys.size(0)).type(torch.bool)).to(DEVICE)\n",
        "    out = model.decode(ys, memory, tgt_mask)\n",
        "    out = out.transpose(0,1)\n",
        "    prob = model.generator(out[:,-1])\n",
        "    _, next_word = torch.max(prob, dim=1)\n",
        "    next_word = next_word.item()\n",
        "    ys = torch.cat([ys, torch.ones(1,1).type_as(src.data).fill_(next_word)], dim=0)\n",
        "    if next_word == EOS_IDX:\n",
        "      break\n",
        "  return ys\n",
        "\n",
        "def translate(model: torch.nn.Module, src_sentence: str):\n",
        "    model.eval()\n",
        "    src = text_transform[SRC_LANGUAGE](src_sentence).view(-1, 1)\n",
        "    num_tokens = src.shape[0]\n",
        "    src_mask = (torch.zeros(num_tokens, num_tokens)).type(torch.bool)\n",
        "    tgt_tokens = greedy_decode(\n",
        "        model,  src, src_mask, max_len=num_tokens + 5, start_symbol=BOS_IDX).flatten()\n",
        "    return \" \".join(vocab_transform[TGT_LANGUAGE].lookup_tokens(list(tgt_tokens.cpu().numpy()))).replace(\"<box>\", \"\").replace(\"<eos>\", \"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-k1TqwXMX5T"
      },
      "source": [
        "Now let's translate some German into English. For fun, we can also run this code *before* and *after* we train. Amazingly, the model works pretty well! "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xNG950FIUm-W",
        "outputId": "bb484c2f-307a-4639-fe99-327c9278ea93"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " A group of people standing in front of an orange cone . \n"
          ]
        }
      ],
      "source": [
        "sentences = [\n",
        "    'Eine Gruppe von Menschen steht vor einem Iglu .',\n",
        "]\n",
        "for sentence in sentences:\n",
        "  print(translate(transformer, sentence))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_hOiwpsmDcrQ"
      },
      "source": [
        "## Investigating the Self-Attention Weights\n",
        "\n",
        "Remember that the key component of this model is self-attention. Let's take a specia look at it. First, we'll need to get some validation data to investigate with."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "7haKKx5JOagJ"
      },
      "outputs": [],
      "source": [
        "\n",
        "data_iter = Multi30k(split='valid', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
        "dataloader = DataLoader(data_iter, batch_size=BATCH_SIZE, collate_fn=collate_fn)\n",
        "source, target = next(iter(dataloader))\n",
        "source = source.to(DEVICE)\n",
        "target = target.to(DEVICE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KqiUAxdwMzSK"
      },
      "source": [
        "In order to run the model on the data, we'll need to get masks and embedding. Then we'll add a positional encoding and feed these through the first self-attention head in the transformer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "HSXuW5nWPpOG"
      },
      "outputs": [],
      "source": [
        "src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(source, target)\n",
        "src_emb = transformer.positional_encoding(transformer.src_tok_emb(source))\n",
        "encoderlayer = transformer.transformer.encoder.layers[0]\n",
        "x = encoderlayer.norm1(src_emb)\n",
        "output, weights = encoderlayer.self_attn(x, x, x, attn_mask = src_mask, key_padding_mask=src_padding_mask)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zGFEWRQgNPxm"
      },
      "source": [
        "Now let's look at an example sentence and visualize the similarity that the model sees between embedded words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jBVHFyL8OnLO",
        "outputId": "777166d8-5d8a-4711-d526-79e5f014132d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " Eine Gruppe von Männern lädt Baumwolle auf einen Lastwagen                         \n",
            " A group of men are loading cotton onto a truck                   \n"
          ]
        }
      ],
      "source": [
        "index = 0\n",
        "src_sentence = vocab_transform[SRC_LANGUAGE].lookup_tokens(list(source[:,index]))\n",
        "tgt_sentence = vocab_transform[TGT_LANGUAGE].lookup_tokens(list(target[:,index]))\n",
        "end_index = src_sentence.index('<eos>')\n",
        "print(' '.join(src_sentence).replace('<box>', '').replace('<pad>', '').replace('<eos>', ''))\n",
        "print(' '.join(tgt_sentence).replace('<box>', '').replace('<pad>', '').replace('<eos>', ''))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 266
        },
        "id": "FmA6kyT4-JTj",
        "outputId": "3202652d-409e-4376-e603-b81936bd2020"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAD5CAYAAACEcub7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAY3UlEQVR4nO3df7RdZX3n8fcnN7kESExsUQaTCNFGMRUXYBo6VSlTwAZ1JWumvwLtFFzq7awh/qydxrYrVpzOVMdiWWsyrQFBOi2kgLbe2jsGbf0xbTXcqEFJQuw1MOQG+SGCkZ83957v/LE3eLjee/Y5yd7P3WfzebH2Yp999nm+z0nge5/73c/ejyICMzNLY95cd8DM7LnESdfMLCEnXTOzhJx0zcwSctI1M0vISdfMLKH5VQc4b/kFSeaknTBvMEUYAF6nn0gW6/on9ieL9dBTh5PFWjq4KEmcJ6aeShIH4JEnH0sWa+H8BcliLV5wfLJYdz10u461jSPfO9B1zllw0ks6xpO0DrgKGACuiYg/nvb+i4HrgaX5OZsjYqRTmx7pmpnNQNIAsBW4CFgNXCxp9bTT/gC4KSLOAjYC/6uo3cpHumZmSbWmymppLTAWEQcAJG0HNgB7284J4Hn5/hLg3qJGnXTNrFmmJstqaRlwsO31OHDOtHP+ELhV0tuBE4ELihp1ecHMGiWi1fUmaUjSrrZtqMdwFwOfiIjlwBuA/y2pY171SNfMmqXV6vrUiNgGbJvl7UPAirbXy/Nj7d4CrMvb+oqkhcBJwAOzxfRI18yaJVrdb52NAqskrZQ0SHahbHjaOfcA5wNIegWwEHiwU6Me6ZpZs5R0IS0iJiVtAnaQTQe7NiL2SLoC2BURw8BvA1dLejfZRbXLouDRjU66ZtYsxSPY7pvK5tyOTDu2pW1/L/CaXtosTLqSTiebJrEsP3QIGI6Ifb0EMjNLIcqbvVCJjjVdSb8LbAcE3JZvAm6UtLn67pmZ9ajV6n6bA0Uj3bcAPx0RR9oPSroS2AP88UwfyqddDAGsWno6Lzpx2UynmZmVr8TyQhWKZi+0gBfNcPyU/L0ZRcS2iFgTEWuccM0sqdZU99scKBrpvgv4B0n/yo/uzHgx8FPApio7ZmZ2VGo+0u2YdCPis5JeRnYPcvuFtNGImJsfE2ZmndT8Qlrh7IWIaAFfTdAXM7NjN0cXyLrlebpm1ih1/yXcSdfMmqWfa7pmZn3H5QUzs4Q80jUzS2jqSPE5c8hJ18ya5bleXtjzw4PFJ5Xg0uefnSQOwGuffDJZrE/PPzFZrEcmHk0W6/HJNH+GP5x4IkkcgImEI6wjrXRzUY/M0Z1bR83lBTOzhJ7rI10zs6ScdM3M0glfSDMzS6jmNV0vTGlmzVLiQ8wlrZO0X9LYTAs3SPqopN359m1JjxS16ZGumTVLSSNdSQPAVuBCYBwYlTScr4uWhYp4d9v5bwfOKmrXI10za5byRrprgbGIOBARE2RLl23ocP7FwI1FjXqka2bNUl5Ndxk/WrwBstHuOTOdKOlUYCXwj0WNOumaWbNMdn/jSPt6jrltEbHtKKJuBG7pZnGHo066kt4cEdcd7efNzCrRw0g3T7CzJdlDwIq218vzYzPZCFzeTcxjqel+YLY3JA1J2iVp15MThRfzzMzKU15NdxRYJWmlpEGyxDo8/SRJpwPPB77STfc6jnQlfXO2t4CTZ/tc+0+PFyx5eXTTETOzUpRU042ISUmbgB3AAHBtROyRdAWwKyKeTsAbge0R0VWuKyovnAz8IvDwtOMC/qXr3puZpVLibcARMQKMTDu2ZdrrP+ylzaKk+xlgUUTsnv6GpC/2EsjMLIma35FWtAT7Wzq8d0n53TEzO0Y9zF6YC54yZmbN0l1pdc446ZpZs/jRjmZmCTnpmpkl1M8X0szM+s5Uvdd0qzzpTkyluZJ49fdu49RFL0wS6/KTFyWJA7DykaXJYj21MN0T9x+aOJwkzlTCUU+Xc+P7zlOT9V6J4ce4vJBGqoRrZjXnpGtmlpBrumZm6USr3mUeJ10zaxaXF8zMEnquz14wM0vKI10zs4ScdM3MEqr5fGknXTNrlpqPdAvXSJN0uqTzJS2adnxddd0yMztKreh+mwMdk66kdwCfBt4O3CFpQ9vb/63KjpmZHZWpqe63ApLWSdovaUzS5lnO+VVJeyXtkXRDUZtF5YW3Aa+OiEclnQbcIum0iLiKbJ202Tr6zFryCwdPYnDB84r6YWZWiiipvCBpANgKXAiMA6OShiNib9s5q4D3Aa+JiIclFT6PoCjpzouIRwEi4m5J55El3lPpkHTbVwNesuil9a5qm1mzlFc2WAuMRcQBAEnbgQ3A3rZz3gZsjYiHASLigaJGi2q690s68+kXeQJ+E3AScEZP3TczSyFa3W+dLQMOtr0ez4+1exnwMkn/LOmr3VzrKhrp/ibwrGczRsQk8JuSPlbUuJlZcj2MdNtLoblt+W/q3ZoPrALOA5YDX5Z0RkQ80ukDs4qI8Q7v/XMPHTMzS2Oy+9uA20uhMzgErGh7vTw/1m4c2BkRR4C7JH2bLAmPzhazcMqYmVlfKa+8MAqskrRS0iCwERieds7fko1ykXQSWbnhQKdGfXOEmTVLSRfSImJS0iZgBzAAXBsReyRdAeyKiOH8vddL2gtMAb8TEQ91atdJ18wapawpYwARMQKMTDu2pW0/gPfkW1ecdM2sWfwQczOzhJ7rSfexiSerDgHAnQ8fLD6pJBe1TkkWa+Px6RbcnH9cuuuql887OUmcNxz5VpI4kHbV3JQrD0fU+6HgP8YPMTczS8drpJmZpeSka2aWUM2fp+uka2bN4pGumVlCTrpmZunElMsLZmbpeKRrZpaOp4yZmaXU70lX0lqy5zqMSloNrAPuzB8EYWZWL/Uu6XZOupLeD1wEzJf0OeAc4AvAZklnRcQfzfK5Z57GroElzJt3Yrm9NjObRUzWO+sWjXR/GTgTOA64D1geEYclfQTYCcyYdNufxj5/cFm9x/pm1iz1zrmFSXcysqddPC7pOxFxGCAinpBU869mZs9F/X4hbULSCRHxOPDqpw9KWkLtf56Y2XNSzTNT0bP8zs0TLhHPWlBoAXBpZb0yMztK0YqutyKS1knaL2lM0uYZ3r9M0oOSdufbW4vaLFoN+KlZjn8P+F5hj83MUitppCtpANgKXEi26u+opOGI2Dvt1L+OiE3dtut5umbWKDFZWlNrgbGIOAAgaTuwAZiedHviJdjNrFHKW4GdZUD7kjTj+bHpfknSNyXdImlFUaNOumbWLK3uN0lDkna1bUM9Rvs74LSIeBXwOeD6og+4vGBmjdLFCPZH57bdUzCDQ0D7yHV5fqz98w+1vbwG+HBRTI90zaxRSiwvjAKrJK2UNAhsBIbbT5DUvkrtemBfUaONGem2Eq6Oetfh+5LFuvrI48li/dqSM5LFetV7FyeJs+i/LkwSB+DwU+n+rlKq960GPy6mVE47EZOSNgE7gAHg2ojYI+kKYFdEDAPvkLQemAS+D1xW1G5jkq6ZGfRWXihsK3uw18i0Y1va9t8HvK+XNp10zaxRolXOSLcqTrpm1ihljnSr4KRrZo0S4ZGumVkyHumamSXUKmn2QlWcdM2sUXwhzcwsobon3Z7vSJP0F1V0xMysDBHdb3OhaGHK4emHgH8naSlARKyvqmNmZkej7iPdovLCcrJnR15DdjeggDXAn3T6kFcDNrO5UvcpY0XlhTXA14DfB34QEV8EnoiIL0XEl2b7UERsi4g1EbHGCdfMUpqaUtfbXCharqcFfFTSzfm/7y/6jJnZXKr7SLerBBoR48CvSHojcLjaLpmZHb1+r+k+S0T8PfD3FfXFzOyYzdWshG65VGBmjdKoka6ZWd1Nteq9II6Trpk1issLZmYJtWo+e6He43Azsx5FqOutiKR1kvZLGpO0ucN5vyQpJK0patNJ18wapaxnL0gaALYCFwGrgYslrZ7hvMXAO4Gd3fTP5YWjMNVK95TkBx57JFmsT0x+LVms//6m65LEWXvV/UniAPzto99PFiulev+y/uNKLC+sBcYi4gCApO3ABrJHI7T7IPAh4He6adQjXTNrlKnWvK63AsuAg22vx/Njz5B0NrAiv4ehK066ZtYo0cMmaUjSrrZtqNs4kuYBVwK/3Uv/XF4ws0bppbwQEduAbbO8fQhY0fZ6eX7saYuBVwJflATwb4BhSesjYtdsMZ10zaxRSnzgzSiwStJKsmS7EbjkR3HiB8BJT7+W9EXgvZ0SLri8YGYN0+ph6yQiJoFNwA5gH3BTROyRdIWko17AwSNdM2uUKHG+RUSMACPTjm2Z5dzzumnTSdfMGmWy5nekOemaWaOUOdKtQk9JV9JrySYM3xERt1bTJTOzo5fu1qWj0/FCmqTb2vbfBvxPsmkS7+90H7KZ2VwJ1PU2F4pmLyxo2x8CLoyIDwCvB359tg+1TzhutR4roZtmZt0pa/ZCVYrKC/MkPZ8sOSsiHgSIiMckTc72ofYJx/MHl9X86ZZm1iRTfV7TXUK2BLuAkHRKRHxX0iL67zkYZvYcUPPVegqXYD9tlrdawL8vvTdmZseoVfPx4FFNGYuIx4G7Su6Lmdkxq3s90/N0zaxR6j5lzEnXzBqlpQaWF8zM6mpqrjtQwEnXzBqlr2cvmJn1m0bOXrB0Ul6JfXTiiWSxbjv3T5PE+cSbFxSfVJKfuDLd/+ytoqVsS6Sa10in8+wFM7OEXF4wM0vIU8bMzBKaqvlI12ukmVmjlPmUMUnrJO2XNDbT42wl/SdJ35K0W9I/SVpd1KaTrpk1SllJV9IAsBW4CFgNXDxDUr0hIs6IiDOBDwNXFvXPSdfMGiXU/VZgLTAWEQciYgLYDmx4VqyIw20vT6SLyROu6ZpZo5R4IW0ZcLDt9ThwzvSTJF0OvAcYBH6hqFGPdM2sUaZ62NpXucm3oV7jRcTWiHgp8LvAHxSd75GumTVKL/N021e5mcEhYEXb6+X5sdlsB/6sKGbRwpTnSHpevn+8pA9I+jtJH5K0pKhxM7PUSpy9MAqskrRS0iCwERhuP0HSqraXbwT+tajRovLCtcDj+f5VZMv3fCg/dl1xn83M0ior6UbEJLAJ2AHsA26KiD2SrpC0Pj9tk6Q9knaT1XUvLepf4cKUeWCANRFxdr7/T3mQGeV1kSEADSxh3rwTi/phZlaKMp+9EBEjwMi0Y1va9t/Za5tFI907JL05379d0hoASS8DjnTo6LaIWBMRa5xwzSyllrrf5kJR0n0r8POSvkM2Ofgrkg4AV+fvmZnVSi+zF+ZC0WrAPwAuyy+mrczPH4+I+1N0zsysV62aP9yxqylj+V0Xt1fcFzOzY+anjJmZJVTvca6Trpk1jEe6ZmYJTareY10nXTNrlHqnXCddM2sYlxesb6RcYfbGhWkecLfm9JckiQNw/IJZb9Is3ZOTE8livXjxC5PFKkMjpoyZmfWLeqdcJ10zaxiXF8zMEpqq+VjXSdfMGsUjXTOzhMIjXTOzdDzSNTNLqO5TxrwasJk1SvSwFZG0TtJ+SWOSNs/w/nsk7ZX0TUn/IOnUojaddM2sUSaJrrdOJA0AW4GLyBZxuFjS6mmnfYNsKbNXAbcAHy7qX9FqwO+QtKLTOWZmdRI9/FNgLTAWEQciYoJsifUNz4oV8YWIeHrx3q+SLdPeUdFI94PATkn/V9J/lvSCogYhW5hS0i5Ju1qtx7r5iJlZKUpcgn0ZcLDt9Xh+bDZvAf5PUaNFSfcAWeb+IPBqYK+kz0q6VNLi2T7khSnNbK70MtJtHyDm29DRxJT0G8Aa4H8UnVs0eyEiogXcCtwqaQFZfeNi4CNAVyNfM7NUepkyFhHbgG2zvH0IaC+vLs+PPYukC4DfB34+Ip4qilmUdJ+1SHFEHAGGgWFJJxQ1bmaW2lR5T8sbBVZJWkmWbDcCl7SfIOks4GPAuoh4oJtGi5Lur832Rlvx2MysNsqapxsRk5I2ATuAAeDaiNgj6QpgV0QMk5UTFgE3SwK4JyLWd2q3aAn2b5fSezOzRMq8DTgiRoCRace2tO1f0GubviPNzBrFtwGbmSVU99uAnXTNrFH8lDEzs4RKnL1QCSddM2sUlxfMZnDT99OsnHvflieTxAFYtGBhslhrlr40WaxP/2q671UGX0gzM0vINV0zs4RcXjAzSyh8Ic3MLB0vwW5mlpDLC2ZmCbm8YGaWkEe6ZmYJ9fWUMUmDZA/uvTciPi/pEuDngH3Atvyh5mZmtdHvtwFfl59zgqRLyR7W+yngfLKVMi+ttntmZr3p9/LCGRHxKknzyZareFFETEn6S+D22T6UL+42BKCBJXhxSjNLpe5Jt2g14Hl5iWExcAKwJD9+HLBgtg95NWAzmysR0fU2F4qS7seBO4HdZKtd3izparIF27ZX3Dczs561iK63IpLWSdovaUzS5hneP1fS1yVNSvrlbvpXtEbaRyX9db5/r6S/AC4Aro6I27oJYGaWUlmzFyQNAFuBC4FxYFTScETsbTvtHuAy4L3dtls4ZSwi7m3bfwS4pdvGzcxSm4rSHu64FhiLiAMAkrYDG4Bnkm5E3J2/13VQz9M1s0YpsVa7DDjY9nocOOdYGy2q6ZqZ9ZVearqShiTtatuGqu6fR7pm1ii91HQjYhuwbZa3DwEr2l4vz48dEyddM2uUVnnlhVFglaSVZMl2I3DJsTbq8oKZNUr08E/HdiImgU3ADrJHH9wUEXskXSFpPYCkn5E0DvwK8DFJe4r655GumTVKibMXiIgRYGTasS1t+6NkZYeuOenanHhiciJJnC8/fGeSOAA/nHgiWax75qdbofe437s5WawylFheqISTrpk1Sl8/2tHMrN94pGtmlpBHumZmCU3F1Fx3oSMnXTNrFC9MaWaWUN0fYu6ka2aN4pGumVlCfT97QdJLgP9A9uCHKeDbwA0RcbjivpmZ9azusxc6PntB0juAPwcWAj9DtjbaCuCrks6rvHdmZj2ailbX21woGum+DTgzXwH4SmAkIs6T9DHg08BZM33IqwGb2VxpQk13PllZ4ThgEUBE3COp42rA5M+onD+4rN5/AmbWKP1e072GbDG2ncDrgA8BSHoB8P2K+2Zm1rO+HulGxFWSPg+8AviTiLgzP/4gcG6C/pmZ9aTv5+lGxB6g8MG8ZmZ10NcjXTOzfjNXsxK65aRrZo3S7xfSzMz6issLZmYJ1f2ONCddM2sUj3TNzBKqe02XiKjlBgw1KY5j9VesJn6nJsfqp63jA2/m2FDD4jhWf8Vq4ndqcqy+Ueeka2bWOE66ZmYJ1TnpbmtYHMfqr1hN/E5NjtU3lBe8zcwsgTqPdM3MGqd2SVfSOkn7JY1J2lxhnGslPSDpjqpitMVaIekLkvZK2iPpnRXGWijpNkm357E+UFWsPN6ApG9I+kzFce6W9C1JuyXtqjjWUkm3SLpT0j5J/7aiOC/Pv8/T22FJ76oo1rvz/x7ukHSjpIVVxMljvTOPs6eq79PX5nrO2rR5fQPAd4CXAIPA7cDqimKdC5wN3JHge50CnJ3vLyZb3LOq7yVgUb6/ANgJ/GyF3+09wA3AZyr+M7wbOKnqv6s81vXAW/P9QWBpgpgDwH3AqRW0vQy4Czg+f30TcFlF3+OVwB3ACWQ3X30e+KkUf2/9stVtpLsWGIuIAxExAWwHNlQRKCK+TKLVLyLiuxHx9Xz/h8A+sv8RqogVEfFo/nJBvlVSuJe0HHgj2QojjSBpCdkP5I8DRMRERDySIPT5wHci4v9V1P584HhJ88kS4r0VxXkFsDMiHo+ISeBLZKuJW65uSXcZcLDt9TgVJae5Iuk0sgU9d1YYY0DSbuAB4HMRUVWsPwX+C5DiAaYB3Crpa/nCp1VZCTwIXJeXTa6RlGJl1Y3AjVU0HBGHgI8A9wDfBX4QEbdWEYtslPs6ST8p6QTgDWQriFuubkm30SQtAj4JvCsiDlcVJyKmIuJMYDmwVtIry44h6U3AAxHxtbLbnsVrI+Js4CLgcklVLRc1n6zs9GcRcRbwGFDZtQUASYPAeuDmitp/PtlvjCuBFwEnSvqNKmJFxD6ytRRvBT4L7CZb2NZydUu6h3j2T8Xl+bG+l6+e/EngryLiUyli5r8WfwFYV0HzrwHWS7qbrAz0C5L+soI4wDOjNSLiAeBvyEpRVRgHxtt+O7iFLAlX6SLg6xFxf0XtXwDcFREPRsQR4FPAz1UUi4j4eES8OiLOBR4mu4Zhubol3VFglaSV+U//jcDwHPfpmEkSWY1wX0RcWXGsF0hamu8fD1wI3Fl2nIh4X0Qsj4jTyP6e/jEiKhk9STpR0uKn94HXk/0aW7qIuA84KOnl+aHzgb1VxGpzMRWVFnL3AD8r6YT8v8Xzya4rVELSC/N/v5isnntDVbH6Ua0e7RgRk5I2ATvIruZeG9nCmKWTdCNwHnCSpHHg/RHx8SpikY0K/yPwrbzWCvB7ETFSQaxTgOslDZD9UL0pIiqdzpXAycDfZPmC+cANEfHZCuO9Hfir/Af/AeDNVQXKf4hcCPxWVTEiYqekW4CvA5PAN6j2brFPSvpJ4AhweaILkX3Dd6SZmSVUt/KCmVmjOemamSXkpGtmlpCTrplZQk66ZmYJOemamSXkpGtmlpCTrplZQv8fp8Vt+kuoi4MAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "sns.heatmap(weights[index].detach().cpu().numpy()[:end_index, :end_index])\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "authorship_tag": "ABX9TyMy/xBXa7+SgjIAjBxe9O6u",
      "collapsed_sections": [],
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.10.0 64-bit ('3.10.0')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.10.0"
    },
    "vscode": {
      "interpreter": {
        "hash": "6aea8d84a86a1280a34780710c5e478b9c6b3c016841b907182e7484363820b1"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}