workshops/10_transformer_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Трансформер. Варианты архитектур"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "План на сегодня:\n",
    "- Decoder-only архитектура: GPT\n",
    "- Стратегии семплирования при генерации\n",
    "- Encoder-only архитектура: BERT\n",
    "- Перекрёстное внимание\n",
    "- Encoder-decoder: T5\n",
    "- Мультимодальные трансформеры"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torch import Tensor, nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Decoder-only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generative pretrained transformer (GPT): $p(\\mathbf{x_n} | \\mathbf{x_1}, ..., \\mathbf{x_{n-1}})$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обозначения:\n",
    "- `<start>` — специальный токен начала генерации\n",
    "- $x_i$ — токены\n",
    "- embedding — слой эмбеддинга токенов\n",
    "- positional encoding — позиционное кодирование\n",
    "- masked transformer layer — слой трансформера с маской на будущие токены\n",
    "- LSM — linear softmax\n",
    "- $y_i$ — вероятности токенов\n",
    "\n",
    "<img src=\"../attachments/gpt.png\" style=\"background:white\" height=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Эти модели обучаются в *self-supervised* режиме, обрабатывая токены параллельно, но без подглядывания вперёд — для этого нужно наложить маску на матрицу внимания (*masked*, или *causal attention*):\n",
    "\n",
    "<img src=\"../attachments/masked_attention.png\" style=\"background:white\" height=\"300\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_causal_mask(token_ids: Tensor, pad_token_id: int = 0) -> Tensor:\n",
    "    B, T = token_ids.shape\n",
    "    # маска для <pad> токенов\n",
    "    pad_mask = (token_ids != pad_token_id).unsqueeze(1)  # B x 1 x T\n",
    "    # маска нижнетреугольной матрицы\n",
    "    causal_mask = torch.tril(\n",
    "        torch.ones((T, T), device=token_ids.device)\n",
    "    ).bool()  # T x T\n",
    "    return pad_mask & causal_mask  # B x T x T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6MAAAE9CAYAAAD+lbQGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABH/UlEQVR4nO3deXiM9/7/8dckIYssdrFEkKg9lNKipEpriaKK01Jb7aetrUWqp7XU1h8tSqmqUmt7LFWHLy2ntI62WlVLlRa1L8cSCUEiy+f3R0+mpkmIyWQmmTwf1zXXlbnvz9zz/gx5Zd4z92IxxhgBAAAAAOBEHq4uAAAAAACQ/9CMAgAAAACcjmYUAAAAAOB0NKMAAAAAAKejGQUAAAAAOB3NKAAAAADA6WhGAQAAAABORzMKAAAAAHA6mlEAAAAAgNPRjAIAAABADli/fr327Nljvb927VodOHDAdQXlMjSjAAC4wKJFi2SxWGSxWPSf//wn3XpjjEJCQmSxWNS2bVsXVAjAlW7PiLRbyZIl1axZM23cuNHV5SGL9u/fryFDhujw4cP67rvvNHDgQF27ds3VZeUaNKPIsrRQ3LVrV7p1vXr1ksViUc2aNV1QGQDkXT4+Plq+fHm65V999ZVOnz4tb29vF1QFILcYP368lixZosWLF2vkyJG6ePGi2rRpo/Xr17u6NGRB3759dfbsWd13331q2LChGjdurIceesjVZeUaXq4uAHnfkSNHtHTpUleXAQB5Ups2bbRy5Uq988478vL688/y8uXLVa9ePV26dMmF1QFwtdatW+uBBx6w3u/Tp49KlSqlFStWsNdEHlCiRAn9/PPP+vnnn+Xn56dq1aq5uqRchW9GkW0TJ05UgQIFVKVKFVeXAgB5zjPPPKPLly9r8+bN1mW3bt3SqlWr1LVr13Tjp02bpkaNGqlYsWLy9fVVvXr1tGrVqgy3vXTpUjVo0EB+fn4qUqSImjZtqi+++MK6vkKFCurVq5fNY/r37y8fHx9t27bNuuyzzz5TVFSUypQpI29vb4WFhemNN95QSkpK9iYP4J4VLlxYvr6+Nh9eSVJqaqpmzJihGjVqyMfHR6VKldKAAQN05cqVdNvYuHGjIiMjFRAQoMDAQNWvX99mD43t27erc+fOKl++vLy9vRUSEqJhw4bp5s2bWaoxNjZWw4YNU4UKFeTt7a1y5cqpR48ed/1wLW135BkzZqRbV7VqVVksFr3wwgvWZTExMXr55ZdVq1Yt+fv7KzAwUK1bt9bevXvTPT4hIUFjx47VfffdJx8fH5UuXVodO3bU0aNHJUnHjx+XxWLRokWLrI+5du2a6tWrp4oVK+rcuXPW5devX9dLL72kkJAQeXt7q0qVKpo2bZqMMenmM3bsWHl7e6tevXqqVq2apk6dKovFokceeSQLr6T7oxlFthw9elRLly7VgAEDFBwcnOGYO70Z+vLLL+Xh4aHXX3/d5jHLly+XxWLR3LlzrcuSk5P1xhtvKCwsTN7e3qpQoYJGjx6txMTEdM+ZFigZ3Y4fP24dl9mux5cuXbIGSJqxY8fKYrHcMUgzemMXGxuroUOHWgMrPDxcb775plJTUzPdzu3bS6vbw8NDwcHB+tvf/qaTJ0/ajMtNf4AA3JsKFSqoYcOGWrFihXXZxo0bFRcXp6effjrd+JkzZ+r+++/X+PHjNWnSJHl5ealz587asGGDzbhx48ape/fuKlCggMaPH69x48YpJCREX375Zaa1jBkzRgsWLNDSpUtt3igtWrRI/v7+Gj58uGbOnKl69erp9ddfV3R0dPZfAAB3FBcXp0uXLunixYs6cOCABg0apPj4eD377LM24wYMGKARI0aocePGmjlzpnr37q1ly5apZcuWSkpKso5btGiRoqKiFBMTo1deeUVTpkxRnTp1tGnTJuuYlStX6saNGxo0aJBmzZqlli1batasWerRo8dd642Pj1eTJk00a9YsPf7445o5c6YGDhyoQ4cO6fTp03d9vI+PjxYuXGiz7JtvvtGJEyfSjf3999+1du1atW3bVm+//bZGjBih/fv3KzIyUmfPnrWOS0lJUdu2bTVu3DjVq1dPb731loYMGaK4uDj9/PPPGdaRlJSkp556SidPntTnn3+u0qVLS/rjeP527dpp+vTpatWqld5++21VqVJFI0aM0PDhw+84t9jYWE2ePPmur0G+YoAsWrhwoZFkfvjhB+uyXr16GR8fH3P27FkTGRlpatSoYfOYsWPHGkmmUaNGZurUqWbmzJmma9euZtSoUdYxzz//vPHy8jI//vijMcaYs2fPmqJFi5oWLVqY1NRU67iePXsaSaZTp07m3XffNT169DCSTIcOHdLVeuzYMSPJPPPMM2bJkiVmyZIl5plnnjGSzLFjx+44J2OMuXjxopFkxowZY102ZswYI8lcvHgx09coNDTU9OzZ03r/+vXrJiIiwhQrVsyMHj3avPfee6ZHjx7GYrGYIUOGZLqd27fXpEkTs2TJEvPRRx+ZESNGmAIFCpiHH37YZlzfvn2Nl5eX6devn3nvvffMqFGjTKFChUz9+vXNrVu3bOZrsVhMzZo1zcSJE827775r+vbta7p3724d8+KLL5o2bdqYSZMmmXnz5pk+ffoYT09P06lTp7vWCyDrbs+f2bNnm4CAAHPjxg1jjDGdO3c2zZo1M8b8kQNRUVHWx6WNSXPr1i1Ts2ZN8+ijj1qXHT582Hh4eJgnn3zSpKSk2Iy/PVdvz6x58+YZSWbWrFnpav3rcxpjzIABA4yfn59JSEi4x5kDyIq0jPjrzdvb2yxatMhm7Pbt240ks2zZMpvlmzZtslkeGxtrAgICzIMPPmhu3rxpM/b2bMjod37y5MnGYrGYEydO3LHu119/3Ugya9asSbfu9ufISNr7PC8vL7Nr1y7r8j59+piuXbsaSeb555+3Lk9ISEiXcceOHTPe3t5m/Pjx1mUffvihkWTefvvtTGtKe++4cOFCk5qaarp162b8/PzMzp07bcavXbvWSDITJkywWd6pUydjsVjMkSNHbOZz+3vJkSNHmpIlS5p69eqZyMjIO74W+QXNKLLsr43b0aNHjZeXlxk8eLAxxqRrRrP6Zuj69esmPDzc1KhRwyQkJJioqCgTGBhoE3Z79uwxkkzfvn1ttvPyyy8bSebLL7+0Wf7bb78ZSWbatGnWZVOnTnV6M/rGG2+YQoUKmd9++81mXHR0tPH09DQnT57MdFsZbc8YY7p27Wr8/Pys93PTHyAAWXd7/ly4cMF4eXmZf/7zn+bq1avG19fXzJ8/3xiTvhm9XUxMjLl48aIZNGiQKVy4sHV5Wt799NNPd6whLWPWrl1rPD09zYgRI+5a99WrV83FixfN0qVLjSSzZ8+erE8aQJalZcS7775rNm/ebDZv3myWLl1qWrVqZby8vMzq1autYwcPHmyCgoLMhQsXzMWLF21u/v7+1vdPK1euNJLMp59+muU64uPjzcWLF81XX31lJJm1a9fecXyNGjVM7dq17Zmytdls3769tem8fv26CQgIMJs3b07XjN4uOTnZXLp0yVy8eNFERETYfFkRFRVlihcvbpKSkjJ97tub0ZdeeslIMuvXr083rn///sbT09NcvXrVZvm3336b7gO9299Lnj592vj6+ppZs2aZyMhImtH/YTdd2G3ChAny8vLKdDettWvXKjU1Va+//ro8PGz/q1ksFuvPfn5+WrRokQ4ePKimTZtqw4YNmj59usqXL28d83//93+SlG73h5deekmS0u2elpCQIOmPXT2yIm0XmLRbTExMpmNjYmJ06dIlXb9+/a7bXblypZo0aaIiRYrYbL9FixZKSUnR119/fddtJCYm6tKlS7pw4YI2b96sL7/8Us2bN7d5jqCgID322GM2z1GvXj35+/tr69atkqTNmzfr2rVrio6OTve63P7v4evra/35+vXrunTpkho1aiRjjH766ae71gvg3pUoUUItWrTQ8uXLtWbNGqWkpKhTp04Zjl2/fr0eeugh+fj4qGjRoipRooTmzp2ruLg465ijR4/Kw8ND1atXv+tz79mzR88884xSUlIyzb4DBw7oySefVFBQkAIDA1WiRAnrLoK3Py8Ax2vQoIFatGihFi1aqFu3btqwYYOqV6+uF154Qbdu3ZIkHT58WHFxcSpZsqRKlChhc4uPj9eFCxckyXp85N2ufnDy5En16tVLRYsWlb+/v0qUKKHIyEhJd/+dP3r0aLavrtC7d28tX75ciYmJWrlypYoUKaJHH3003bjU1FRNnz5dlStXlre3t4oXL64SJUpo37596TKxSpUq6Y6zzci8efP01ltvSVKGhzudOHFCZcqUUUBAgM3ytBMTZbQ7sfTHYRBlypTRgAED7lpDfsLZdGGX33//XUuWLNHf//536z70f3Uvb4YaN26sQYMG6d1331XLli313HPP2aw/ceKEPDw8FB4ebrM8ODhYhQsXTveLn3ZcZ1BQUJbm06JFiyyNk2RzoqaSJUuqX79+GjdunDw9PdONPXz4sPbt26cSJUpkuK20Pw538vHHH+vjjz+23q9fv74++OADm+dI+wN0p+e4lz9Ar7/+utatW5cuhHnTCeScrl27ql+/fjp//rxat26twoULpxuzfft2tWvXTk2bNtWcOXNUunRpFShQQAsXLszw8jBZsXfvXrVu3VrNmzfXiBEj9Oyzz9ocLxobG6vIyEgFBgZq/PjxCgsLk4+Pj3bv3q1Ro0Zl6fh3AI7j4eGhZs2aaebMmTp8+LBq1Kih1NRUlSxZUsuWLcvwMZm9D8lISkqKHnvsMcXExGjUqFGqWrWqChUqpDNnzqhXr15O+Z2PiopSwYIFtXbtWi1cuFA9e/ZM98WGJE2aNEmvvfaannvuOb3xxhsqWrSoPDw8NHToULvr/O677zRx4kT98MMPGjZsmFq1aqXixYtnaz4HDx7UokWLtHTpUhUoUCBb23I3NKOwy8SJE+Xl5aVRo0Y5ZHuJiYnWMzcePXpUN27ckJ+fX7pxt3+DdydpJymqUKFClsa/++67uu+++6z3r169qqeeeirDsatXr1ZgYKBu3LihTz/9VBMnTlRgYKBGjhyZbmxqaqoee+yxDNdJsnnOzDz++OMaMWKEJOn06dN688031axZM+3atUu+vr5u9wcIyK+efPJJDRgwQN99950++eSTDMesXr1aPj4++vzzz22uP/rXk32EhYUpNTVVv/zyi+rUqXPH561Vq5ZWrlwpX19frVy5Uv3799e+ffuse1Bs27ZNly9f1po1a9S0aVPr444dO2bnTAFkV3JysqQ/ThYk/fE7v2XLFjVu3NhmD6e/CgsLkyT9/PPP6T7gT7N//3799ttv+uijj2xOWHT7Gb/vJCwsLNOTAmWVl5eXunfvrokTJ+rAgQP68MMPMxy3atUqNWvWTAsWLLBZHhsba9NAhoWFaefOnUpKSrprM/jcc89p9OjROnv2rKpXr65hw4ZpyZIl1vWhoaHasmWLrl27ZvPt6KFDh6zr/+qVV15RnTp19Le//e3uk89n2E0X9+zYsWNavHix+vXrpzJlymQ67vY3Q3czZswYHTx4UNOmTdOxY8fS7fobGhqq1NRUHT582Gb5f//7X8XGxqb7xd+1a5e8vLzu+iYsze27wLRo0cLmDddfNW3aVC1atFC7du20cOFCVa5c2eYMdLcLCwtTfHy8zbZvv92+K3JmSpcubR3fq1cvffjhh/rll1+0du1a63NcvnxZjRs3zvA5ateubR0n6Y5/INL+AL311lsaNWqU2rdvrxYtWtzx3xmAY/j7+2vu3LkaO3asnnjiiQzHeHp6ymKx2FxS5fjx49Y8SNOhQwd5eHho/Pjx6T5EMn+59EDdunVVqFAheXh46IMPPtDx48c1fvx4m+f86+Nu3bqlOXPm2DVPANmTlJSkL774QgULFrTuGtqlSxelpKTojTfeSDc+OTlZsbGxkv74gDsgIECTJ0+2HtKUJu13PKPfeWOMZs6cmaX6nnrqKe3du1effvppunV/zZ87ee6557R//341bdpUlSpVynCMp6dnum2uXLlSZ86cSVfTpUuXNHv27LvW1KRJE0lSmTJl9Oabb2rp0qU2l8Rq06aNUlJS0m1r+vTpslgsat26tc3yb7/9Vp999pmmTJmS5S9V8hO+GcU9mzRpkjw9Pe96Sv8OHTpo1KhRGj9+vFatWmWze4UxxvoLuXPnTk2bNk1Dhw7VSy+9pEuXLunNN9/UU089ZT0+oU2bNho9erRmzJihefPmWbfz9ttvS/pjd440t27d0rp16/Too4/K39/fYfPOiPnjJGAZ7qIr/fHHYezYsfr888/VsmVLm3WxsbHy9/fP0vELt0u7xEraJW26dOmiOXPm6I033tCkSZNsxiYnJys+Pl6FCxe2+QPUqlUrm+NG0/49svsHCED29OzZ847ro6Ki9Pbbb6tVq1bq2rWrLly4oHfffVfh4eHat2+fdVx4eLheffVVvfHGG2rSpIk6duwob29v/fDDDypTpkymlxaoWbOmRo0apSlTpujpp59WRESEGjVqpCJFiqhnz54aPHiwLBaLlixZck9vKgHYb+PGjdZv3S5cuKDly5fr8OHDio6OVmBgoCQpMjJSAwYM0OTJk7Vnzx49/vjjKlCggA4fPqyVK1dq5syZ6tSpkwIDAzV9+nT17dtX9evXV9euXVWkSBHt3btXN27c0EcffaSqVasqLCxML7/8ss6cOaPAwECtXr06w+MnMzJixAitWrVKnTt31nPPPad69eopJiZG69at03vvvWf9kPxuqlWrpkuXLt3xm962bdtq/Pjx6t27txo1aqT9+/dr2bJl6ZrXHj16aPHixRo+fLi+//57NWnSRNevX9eWLVv097//Xe3bt89w+/3799fy5cs1cOBA/fzzz/Lz89MTTzyhZs2a6dVXX9Xx48dVu3ZtffHFF/rss880dOhQ64f/ab744gs99thj93RIWL7i/HMmIa+6/RTjL7zwQrr1GV3a5bXXXrNe2mXatGlm1qxZpkePHiY6OtoYY8zNmzdNlSpVTNWqVa1neE1MTDQ1atQwFStWNPHx8dZtpV3apUuXLubdd9+13r/9bGl79+41HTt2NJLMgAEDrJd1uf3SLtOnTzfnz5+3mdO9nE33k08+MRs3bjSrV682zz77rJFkZs+ebYzJ+NIudevWNV5eXqZv375m7ty5Ztq0aaZnz56mUKFCdzwzb9r20i7tsmTJEjNlyhRTrlw5ExgYaHMm3gEDBhhJpnXr1mb69Olm9uzZZsiQIaZMmTJm5cqV1nEffPCBkWRq1qxpJk2aZObOnWsGDhxoevToYYz54xIRYWFhpnjx4mbixIlm1qxZ5pFHHjG1a9e2nmEOgGNklj9/9dez6S5YsMBUrlzZeHt7m6pVq5qFCxda8+mvPvzwQ3P//fcbb29vU6RIERMZGWk2b95ss+2/nrE7ISHBVK1a1dSvX98kJycbY4zZsWOHeeihh4yvr68pU6aMGTlypPn888+NJLN161b7XwQAmcro0i4+Pj6mTp06Zu7cuRleJuX999839erVM76+viYgIMDUqlXLjBw50pw9e9Zm3Lp160yjRo2Mr6+vCQwMNA0aNDArVqywrv/ll19MixYtjL+/vylevLjp16+f2bt3b5bfC1y+fNm88MILpmzZsqZgwYKmXLlypmfPnubSpUt3fJzucLbcjNYnJCSYl156yZQuXdr4+vqaxo0bm2+//TbDs9XeuHHDvPrqq6ZixYqmQIECJjg42HTq1MkcPXrUGGN7Nt3b/frrr8bHx8cMGzbMuuzatWtm2LBhpkyZMqZAgQKmcuXKZurUqen+TSQZi8VivXxhGs6m+yeaUWRZWih6e3ub06dPp1ufUTNqzJ3fDA0bNsx4enqmu4bTrl27jJeXlxk0aJB1WVJSkhk3bpw1REJCQswrr7xic427tDdkd7ulvXmypxlNu/n6+prq1aub6dOnW8dk9Mbu2rVr5pVXXjHh4eGmYMGCpnjx4tbm/PZrgGYkNDTU5jmLFy9uHn/8cfPtt9+mG5tb/gABAAAAWWExhv1s4D7Gjh2rbdu2WU+GlJEKFSpo0aJFNmeLBAAAAOBcnMAIAAAAAOB0nMAIbiUiIuKup+x+8sknVapUKSdVBAAAACAj7KYLAAAAAHA6dtMFAAAAADgdzSgAAAAAwOk4ZjSHpaam6uzZswoICJDFYnF1OQD+xxija9euqUyZMvLw4HO5NGQWkDuRWemRV0DudC95RTOaw86ePauQkBBXlwEgE6dOnVK5cuVcXUauQWYBuRuZ9SfyCsjdspJXNKM5LCAgQJJUpEgRt/8kMzU11dUl5LipU6e6ugSn2LJli6tLyHFJSUlavXq19XcUf0h7PU6dOqXAwEAXV5OzgoKCXF0CcM/IrD+RV0DulpW8ohnNYWm7jXh4eLh9M5of+Pn5uboEpyhYsKCrS3Aadu2ylfZ6BAYGuv2bOyAvIrP+RF4BuVtW8oruCAAAAADgdDSjAAAAAACnoxkFAAAAADgdzSgAAAAAwOloRgEAAAAATkczCgAAAABwOppRAAAAAIDT0YwCAAAAAJyOZhQAAAAA4HQ0owAAAAAAp6MZBQAAAAA4Hc0oAAAAAMDpaEYBAAAAAE5HMwoAAAAAcDqaUQAAAACA09GMAgAAAACcjmYUAAAAAOB0NKMAAAAAAKejGQUAAAAAOB3NKAAAAADA6WhGAQAAAABORzMKAAAAAHA6mlEAAAAAgNPRjEpasWKFfH19de7cOeuy3r17KyIiQnFxcS6sDADSI7MA5CVkFoDM0IxKevrpp3Xfffdp0qRJkqQxY8Zoy5Yt2rhxo4KCglxcHQDYIrMA5CVkFoDMeLm6gNzAYrFo4sSJ6tSpk4KDgzVr1ixt375dZcuWlSQ9+eST2rZtm5o3b65Vq1bdcVuJiYlKTEy03r969WqO1g4g/yGzAOQljsos8gpwP3wz+j9t27ZV9erVNX78eH366aeqUaOGdd2QIUO0ePHiLG1n8uTJCgoKst5CQkJyqmQA+RiZBSAvcURmkVeA+6EZ/Z9Nmzbp0KFDSklJUalSpWzWPfLIIwoICMjSdl555RXFxcVZb6dOncqJcgHkc2QWgLzEEZlFXgHuh2ZU0u7du9WlSxctWLBAzZs312uvvWb3try9vRUYGGhzAwBHIrMA5CWOyizyCnA/+f6Y0ePHjysqKkqjR4/WM888o0qVKqlhw4bavXu36tat6+ryAMAGmQUgLyGzANxJvv5mNCYmRq1atVL79u0VHR0tSXrwwQfVunVrjR492sXVAYAtMgtAXkJmAbibfP3NaNGiRXXo0KF0yzds2OCCagDgzsgsAHkJmQXgbvJ1M5pVLVq00N69e3X9+nWVK1dOK1euVMOGDV1dFgBkiMwCkJeQWUD+RTOaBVu2bHF1CQCQZWQWgLyEzALyr3x9zCgAAAAAwDVoRgEAAAAATkczCgAAAABwOppRAAAAAIDT0YwCAAAAAJyOZhQAAAAA4HQ0owAAAAAAp6MZBQAAAAA4Hc0oAAAAAMDpaEYBAAAAAE5HMwoAAAAAcDqaUQAAAACA09GMAgAAAACcjmYUAAAAAOB0XvY8aPHixXdc36NHD7uKAQAAAADkD3Y1o0OGDMl0ncVioRkFAAAAANyRXc3olStXHF0HAAAAACAfyfYxo2fPnlW7du1Uvnx5RUVF6dSpU46oCwAAAADgxrLdjA4fPlxnzpxRdHS0bt68qRdeeMERdQEAAAAA3Jhdu+ne7ptvvtHHH3+sRo0aKSoqSnXr1nVEXQAAAAAAN5btZjQ2NlbBwcGSpODgYMXGxmZ3k26pVatWKliwoKvLyFHly5d3dQk5bujQoa4uwSkqVKjg6hJyXEpKiqtLgIsZY1xdQo6zWCyuLgHIcdHR0fL29nZ1GTlq+vTpri4hxw0bNszVJcAF7GpG9+3bZ/05NTVVhw4dUnx8vBITEx1WGAAAAADAfdnVjNapU0cWi8X6qXLbtm2t9/kUFgAAAABwN3Y1o8eOHXN0HQAAAACAfMSuZjQ0NNTRdQAAAAAA8hG7mtF169bdcX27du3sKgYAAAAAkD/Y1Yx26NDB+vPtx46m3ecslQAAAACAO/Gw50GpqanWm5+fn44cOWK9TyMKAAAAALgbu5pRAAAAAACyg2YUAAAAAOB0dh0zevXqVevPFotF8fHxNssCAwOzXxkAAAAAwG3Z1YwWLlxYFotFkmSM0f3332/9mRMYAQAAAADuxq5mdOvWrY6uAwAAAACQj9jVjEZGRjq6DgAAAABAPmL3CYy2b9+uZ599Vo0aNdKZM2ckSUuWLNF//vMfhxUHAAAAAHBPdjWjq1evVsuWLeXr66vdu3crMTFRkhQXF6dJkyY5tEAAAAAAgPuxqxmdMGGC3nvvPc2fP18FChSwLm/cuLF2797tsOIAAAAAAO7Jrmb0119/VdOmTdMtDwoKUmxsbHZrAgAAAAC4Obua0eDgYB05ciTd8v/85z+qVKlStosCAAAAALg3u5rRfv36aciQIdq5c6csFovOnj2rZcuW6eWXX9agQYMcXSMAAAAAwM3YdWmX6Ohopaamqnnz5rpx44aaNm0qb29vvfzyy3rxxRcdXSMAAAAAwM3Y1YxaLBa9+uqrGjFihI4cOaL4+HhVr15d/v7+jq4PAAAAAOCG7GpG0xQsWFDVq1e3WfbLL7+kWwYAAAAAwO3sOma0V69eSk1NtVmWmpqqiRMnqn79+g4pDAAAAADgvuxqRn/66Sd17txZSUlJkqQDBw7owQcf1KJFi7Rx40aHFggAAAAAcD92NaPbtm3TuXPn1KZNG02YMEEPPPCAGjZsqL1792Z4/VEAAAAAAG5nVzNapEgRbd68WcYYjRkzRitWrNA777wjPz8/R9fnFCtWrJCvr6/OnTtnXda7d29FREQoLi7OhZUBQHpkFoC8hMwCkBm7mtGrV68qJSVFy5cv16OPPqoxY8boxIkTunr1qq5everoGnPc008/rfvuu0+TJk2SJI0ZM0ZbtmzRxo0bFRQU5OLqAMAWmQUgLyGzAGTGrrPpFi5cWBaLRZJkjJEkVapUScYYWSwWpaSkOK5CJ7BYLJo4caI6deqk4OBgzZo1S9u3b1fZsmV16tQpde/eXRcuXJCXl5dee+01de7c2dUlA8jHyCwAeQmZBSAzdjWjW7dudXQdLte2bVtVr15d48eP1xdffKEaNWpIkry8vDRjxgzVqVNH58+fV7169dSmTRsVKlQow+0kJiYqMTHRej8vflMMIPcjswDkJY7ILPIKcD92NaORkZGOrsPlNm3apEOHDiklJUWlSpWyLi9durRKly4tSQoODlbx4sUVExOT6Ru7yZMna9y4cU6pGUD+RWYByEsckVnkFeB+7DpmVJJiY2P11ltvqW/fvurbt6+mT5+eZw9C3717t7p06aIFCxaoefPmeu211zIc9+OPPyolJUUhISGZbuuVV15RXFyc9Xbq1KmcKhtAPkVmAchLHJVZ5BXgfuz6ZnTXrl1q2bKlfH191aBBA0nS22+/rYkTJ+qLL75Q3bp1HVpkTjp+/LiioqI0evRoPfPMM6pUqZIaNmyo3bt328wjJiZGPXr00Pz58++4PW9vb3l7e+d02QDyKTILQF7iyMwirwD3Y9c3o8OGDVO7du10/PhxrVmzRmvWrNGxY8fUtm1bDR061MEl5pyYmBi1atVK7du3V3R0tCTpwQcfVOvWrTV69GjruMTERHXo0EHR0dFq1KiRq8oFkM+RWQDyEjILwN3Y/c3o/Pnz5eX158O9vLw0cuRIPfDAAw4rLqcVLVpUhw4dSrd8w4YN1p+NMerVq5ceffRRde/e3ZnlAYANMgtAXkJmAbgbu74ZDQwM1MmTJ9MtP3XqlAICArJdVG6yY8cOffLJJ1q7dq3q1KmjOnXqaP/+/a4uCwAyRGYByEvILCB/s+ub0b/97W/q06ePpk2bZt2dYseOHRoxYoSeeeYZhxboag8//LBSU1NdXQYAZAmZBSAvIbOA/M2uZnTatGmyWCzq0aOHkpOTJUkFChTQoEGDNGXKFIcWCAAAAABwP3Y1owULFtTMmTM1efJkHT16VJIUFhYmPz8/hxYHAAAAAHBPdh0z+txzz+natWvy8/NTrVq1VKtWLRpRAAAAAECW2dWMfvTRR7p586ajawEAAAAA5BN2NaPGGFksFkfXAgAAAADIJ+w6ZlSSBg8eLF9f3wzXffjhh3YXBAAAAABwf3Y3o8YYGWMcWQsAAAAAIJ+wqxm1WCx65513VLJkSUfXAwAAAADIB+w+ZhQAAAAAAHvZ1Yz27Nkz0+NFAQAAAAC4G7t20124cKGj6wAAAAAA5CN2fTMKAAAAAEB20IwCAAAAAJyOZhQAAAAA4HQ0owAAAAAAp7O7GV2yZIkaN26sMmXK6MSJE5KkGTNm6LPPPnNYcQAAAAAA92RXMzp37lwNHz5cbdq0UWxsrFJSUiRJhQsX1owZMxxZHwAAAADADdnVjM6aNUvz58/Xq6++Kk9PT+vyBx54QPv373dYcQAAAAAA92RXM3rs2DHdf//96ZZ7e3vr+vXr2S4KAAAAAODe7GpGK1asqD179qRbvmnTJlWrVi27NQEAAAAA3JyXPQ8aPny4nn/+eSUkJMgYo++//14rVqzQ5MmT9cEHHzi6RgAAAACAm7EYY4w9D1y2bJnGjh2ro0ePSpLKlCmjcePGqU+fPg4tMK+7evWqgoKCFBYWZnN8rTv69ddfXV1CjrNYLK4uwSl8fHxcXUKOM8YoMTFRcXFxCgwMdHU5uUZaZvG6IC/JL9ksid/N26TlVaVKleTh4d5XK1y3bp2rS8hxtWvXdnUJTpGUlOTqEpwmK3ll1zejktStWzd169ZNN27cUHx8vEqWLGnvpgAAAAAA+YzdzWgaPz8/+fn5OaIWAAAAAEA+YVczWrFixTvuEvP777/bXRAAAAAAwP3Z1YwOHTrUwWUAAAAAAPITu5rRIUOG2Nz/8ccftXfvXtWsWVMNGjRwSGEAAAAAAPeV7VOPLVy4UA0aNFB0dLQaNWqk9957zxF1AQAAAADcWLab0ZkzZ2rq1Km6cOGCli1bpunTpzuiLgAAAACAG8t2M3ry5ElFRUVJkqKionT8+PHsbhIAAAAA4Oay3YwmJSWpYMGCkqQCBQooOTk520UBAAAAANybXScw6tixo/XnhIQEDRw4UIUKFVJqaqrDCgMAAAAAuC+7mtGgoCDrz88++6zNuh49emSvIgAAAACA27OrGV24cKGj6wAAAAAA5CPZPmYUAAAAAIB7Zdc3o/fff78sFkum63fv3m13QQAAAAAA92dXM9qhQwdJkjFGkydP1sCBA1W0aFFH1gUAAAAAcGN2NaNjxoyx/vzWW29pyJAhqlSpksOKAgAAAAC4N44ZBQAAAAA4Hc0oAAAAAMDp7NpNd/jw4dafb926pYkTJ9pce/Ttt9/OfmUAAAAAALdlVzP6008/WX9u1KiRfv/9d+v9O51lFwAAAAAAyc5mdOvWrY6uAwAAAACQj9jVjGYmOTlZ33zzjSSpYMGCeuihhxy5eQAAAACAm7CrGV23bl2Gy2NjY9W7d2+1b99eRYoUoRkFAAAAAGTIrma0Q4cOma6zWCxas2aNvfUAAAAAAPIBuy/tcu7cOaWmptrczp4968jaAAAAAABuyu5m1MMj/UPz6pl0V6xYIV9fX507d866rHfv3oqIiFBcXJwLKwOA9MgsAHkJmQUgM3Y3o/Pnz9fcuXO1bNkyffPNN7p165Yj63Kqp59+Wvfdd58mTZokSRozZoy2bNmijRs32lw/FQByAzILQF5CZgHIjF3HjJYvX17z589XYmKiYmJidOvWLfn5+al9+/aOrs8pLBaLJk6cqE6dOik4OFizZs3S9u3bVbZsWcXGxqpFixZKTk5WcnKyhgwZon79+rm6ZAD5GJkFIC8hswBkxmKMMdndyJkzZ7Rt2za9+eabOnDggI4fPy4PDw+VLVvWETU6Td26dXXgwAF98cUXioyMlCSlpKQoMTFRfn5+un79umrWrKldu3apWLFiGW4jMTFRiYmJ1vtXr15VSEiIwsLC5Onp6ZR5uMqvv/7q6hJyXF7dFf1e+fj4uLqEHGeMUWJiouLi4hQYGOjqcuySk5mVl18X5D/5JZsl5enfzexmVmZ5ValSpQwPH3MnmV3Jwp3Url3b1SU4RVJSkqtLcJqs5JVDfnPLli2rbt266dNPP1VISIiaNm2qjh07OmLTTrNp0yYdOnRIKSkpKlWqlHW5p6en/Pz8JP0RgsYY3al/nzx5soKCgqy3kJCQHK8dQP5DZgHISxyRWeQV4H4c+jFSWFiYjh8/rmPHjmnnzp2O3HSO2r17t7p06aIFCxaoefPmeu2112zWx8bGqnbt2ipXrpxGjBih4sWLZ7qtV155RXFxcdbbqVOncrp8APkMmQUgL3FUZpFXgPux65hR6Y/dKtauXauDBw9KkqpXr6727dvnuV1Rjx8/rqioKI0ePVrPPPOMKlWqpIYNG2r37t2qW7euJKlw4cLau3ev/vvf/6pjx47q1KmTzad6t/P29pa3t7czpwAgHyGzAOQljsws8gpwP3Z9M3rkyBFVr15dPXr00Jo1a7RmzRp1795dNWrU0NGjRx1dY46JiYlRq1at1L59e0VHR0uSHnzwQbVu3VqjR49ON75UqVKqXbu2tm/f7uxSAYDMApCnkFkA7saub0YHDx6sSpUq6dtvv1XRokUlSZcvX9azzz6rwYMHa8OGDQ4tMqcULVpUhw4dSrf89vr/+9//ys/PTwEBAYqLi9PXX3+tQYMGObNMAJBEZgHIW8gsAHdjVzP61Vdf6bvvvrM2opJUrFgxTZkyRY0bN3ZYcbnBiRMn1L9/f+sB9S+++KJq1arl6rIAIENkFoC8hMwC8je7mlFvb29du3Yt3fL4+HgVLFgw20XlJg0aNNCePXtcXQYAZAmZBSAvIbOA/M2uY0bbtm2r/v37a+fOndZPsr777jsNHDhQ7dq1c3SNAAAAAAA3Y1cz+s477ygsLEwNGzaUj4+PfHx81LhxY4WHh2vmzJmOrhEAAAAA4Gbs2k23cOHC+uyzz3TkyBHrpV2qVaum8PBwhxYHAAAAAHBPdjWjCQkJ8vHxUXh4eLoG9Ny5cypdurRDigMAAAAAuCe7dtOtW7duhgebr169WhEREdmtCQAAAADg5uxqRh955BE99NBDevPNNyVJ169fV69evdS9e/cML2IMAAAAAMDt7NpNd86cOYqKilLfvn21fv16nTt3Tv7+/vr+++9Vs2ZNR9cIAAAAAHAzdjWjktS6dWt17NhRc+fOlZeXl/71r3/RiAIAAAAAssSu3XSPHj2qhg0bav369fr88881cuRItWvXTiNHjlRSUpKjawQAAAAAuBm7mtE6deqoYsWK2rt3rx577DFNmDBBW7du1Zo1a9SgQQNH1wgAAAAAcDN2NaNz5szRxx9/rMKFC1uXNWrUSD/99JPq1q3rqNoAAAAAAG7Krma0e/fuGS4PCAjQggULslUQAAAAAMD92XUCo8WLF2e6zmKxZNqsAgAAAAAg2dmMDhkyJNN1NKMAAAAAgLuxqxm9cuWKo+sAAAAAAOQjdh0zeruzZ8+qXbt2Kl++vKKionTq1ClH1AUAAAAAcGPZbkaHDx+uM2fOKDo6Wjdv3tQLL7zgiLoAAAAAAG7Mrt10b/fNN9/o448/VqNGjRQVFcWlXQAAAAAAd5Xtb0ZjY2MVHBwsSQoODlZsbGx2NwkAAAAAcHN2fTO6b98+68+pqak6dOiQ4uPjlZiY6LDCAAAAAADuy65mtE6dOrJYLDLGSJLatm1rvW+xWBxaIAAAAADA/djVjB47dszRdbi9woULy9PT09Vl5Kjw8HBXl5DjJk2a5OoSnOLbb791dQk5LikpSZs2bXJ1GQAcIO3DcXd29epVBQUFubqMXMnPz8/t32O1adPG1SXkuEGDBrm6BKfYv3+/q0vIccnJydq+fXuWxtrVjIaGhtrzMAAAAAAAJNnZjF6+fFnFihWTJJ06dUrz58/XzZs31a5dOzVp0sShBQIAAAAA3M89nU13//79qlChgkqWLKmqVatqz549ql+/vqZPn673339fzZo109q1a3OoVAAAAACAu7inZnTkyJGqVauWvv76az3yyCNq27atoqKiFBcXpytXrmjAgAGaMmVKTtUKAAAAAHAT97Sb7g8//KAvv/xSERERql27tt5//339/e9/l4fHHz3tiy++qIceeihHCgUAAAAAuI97+mY0JiZGwcHBkiR/f38VKlRIRYoUsa4vUqSIrl275tgKAQAAAABu556aUUnpriPKdUUBAAAAAPfqns+m26tXL3l7e0uSEhISNHDgQBUqVEiSlJiY6NjqAAAAAABu6Z6a0Z49e9rcf/bZZ9ON6dGjR/YqAgAAAAC4vXtqRhcuXJhTdQAAAAAA8pF7PmYUAAAAAIDsohkFAAAAADgdzSgAAAAAwOloRgEAAAAATkczCgAAAABwOppRAAAAAIDT0YwCAAAAAJyOZhQAAAAA4HQ0owAAAAAAp6MZBQAAAAA4Hc0oAAAAAMDpaEYBAAAAAE5HMwoAAAAAcLp834xWqFBBM2bMcHUZAJAlZBaAvITMAnAn+b4ZBQAAAAA4X65uRq9cuaL4+HinPufJkyed+nwA3AeZBSAvIbMAuFqua0aTk5O1YcMGde7cWaVLl9bRo0clSadOnVKXLl1UuHBhFS1aVO3bt9fx48etj+vVq5c6dOigadOmqXTp0ipWrJief/55JSUlWcdcuHBBTzzxhHx9fVWxYkUtW7Ys3fP37NlTNWvW1NSpU3Xu3Lkcny+AvI3MApCXkFkAcpNc04zu379fL730ksqVK6cePXqoRIkS2rp1q2rXrq2kpCS1bNlSAQEB2r59u3bs2CF/f3+1atVKt27dsm5j69atOnr0qLZu3aqPPvpIixYt0qJFi6zre/XqpVOnTmnr1q1atWqV5syZowsXLtjU8c9//lP9+/fXJ598opCQELVp00affPKJEhISsjSPxMREXb161eYGwP2QWQDyEnfILPIKcD8ubUYvX76smTNnqm7dunrggQf0+++/a86cOTp37pzmzJmjhg0bSpI++eQTpaam6oMPPlCtWrVUrVo1LVy4UCdPntS2bdus2ytSpIhmz56tqlWrqm3btoqKitK///1vSdJvv/2mjRs3av78+XrooYdUr149LViwQDdv3rSpqUSJEho8eLB27dql/fv3KyIiQi+//LJKly6tgQMH6rvvvrvjnCZPnqygoCDrLSQkxLEvGgCXIbMA5CXullnkFeB+XNqMzpo1S0OHDpW/v7+OHDmiTz/9VB07dlTBggVtxu3du1dHjhxRQECA/P395e/vr6JFiyohIcG6e4kk1ahRQ56entb7pUuXtn4id/DgQXl5ealevXrW9VWrVlXhwoUzra9atWqaMmWKTpw4oejoaH344Ydq1arVHef0yiuvKC4uzno7derUvbwkAHIxMgtAXuJumUVeAe7Hy5VP3r9/f3l5eWnx4sWqUaOGnnrqKXXv3l2PPPKIPDz+7JPj4+NVr169DI89KFGihPXnAgUK2KyzWCxKTU21u75Tp05p2bJlWrJkiY4dO6bOnTurd+/ed3yMt7e3vL297X5OALkXmQUgL3G3zCKvAPfj0m9Gy5Qpo3/84x/67bfftGnTJhUsWFAdO3ZUaGiooqOjdeDAAUlS3bp1dfjwYZUsWVLh4eE2t6CgoCw9V9WqVZWcnKwff/zRuuzXX39VbGyszbhr165p0aJFevTRR1WhQgVt2LBBw4cP1/nz57Vs2TK1aNHCYfMHkLeQWQDyEjILQG6Xa05g1KhRI82bN0/nz5/X1KlTtWfPHtWuXVv79+9Xt27dVLx4cbVv317bt2/XsWPHtG3bNg0ePFinT5/O0varVKmiVq1aacCAAdq5c6d+/PFH9e3bV76+vjbjOnTooHHjxunhhx/Wb7/9pu3bt6tPnz4KDAzMiWkDyKPILAB5CZkFIDfKNc1oGh8fHz399NPatGmTTp48qdDQUPn5+enrr79W+fLl1bFjR1WrVk19+vRRQkLCPYXXwoULVaZMGUVGRqpjx47q37+/SpYsaTNmzpw5+v333zV+/HiFhYU5enoA3AyZBSAvIbMA5CYWY4xxdRHu7OrVqwoKClK9evVsDvp3R5cvX3Z1CTmuT58+ri7BKb799ltXl5DjkpKStGnTJsXFxfGJ/G3SMovXBchd+N1ML+01qVmzptu/x4qLi3N1CTmuXbt2ri7BKfbv3+/qEnJccnKytm/fnqW8ynXfjAIAAAAA3B/NKAAAAADA6WhGAQAAAABORzMKAAAAAHA6mlEAAAAAgNPRjAIAAAAAnI5mFAAAAADgdDSjAAAAAACnoxkFAAAAADgdzSgAAAAAwOloRgEAAAAATkczCgAAAABwOppRAAAAAIDT0YwCAAAAAJyOZhQAAAAA4HQ0owAAAAAAp6MZBQAAAAA4Hc0oAAAAAMDpaEYBAAAAAE5HMwoAAAAAcDqaUQAAAACA03m5ugB3Z4yRJKWkpLi4kpyXmprq6hJyXEJCgqtLcIqkpCRXl5DjkpOTJf35O4o/pL0eV69edXElAG6X9jtJZv2J91juJTEx0dUlOEXa+w93di/vsSyGVMtRp0+fVkhIiKvLAJCJU6dOqVy5cq4uI9cgs4Dcjcz6E3kF5G5ZySua0RyWmpqqs2fPKiAgQBaLxSnPefXqVYWEhOjUqVMKDAx0ynM6W36Yo8Q8c5IxRteuXVOZMmXk4cERC2mcnVn8H3cvzDPnkFnp8R4r5zBP9+Lsed5LXrGbbg7z8PBw2SeYgYGBbv2LJeWPOUrMM6cEBQU57bnyCldlFv/H3QvzzBlkli3eY+U85ulenDnPrOYVH60BAAAAAJyOZhQAAAAA4HQ0o27I29tbY8aMkbe3t6tLyTH5YY4S84T7yy//9szTveSXeSK9/PJvzzzdS26eJycwAgAAAAA4Hd+MAgAAAACcjmYUAAAAAOB0NKMAAAAAAKejGQUAAAAAOB3NaB5ksVi0du1ah2zr+PHjslgs2rNnj0O2lxNWrFghX19fnTt3zrqsd+/eioiIUFxcnAsrcyzm6V7zxJ/ILPf8P8483Wue+BOZ5Z7/x5lnLp2nQa4RExNjrl27dtdxksynn36a7edLTEw0x44dM5LMTz/9ZLPuwoUL5ubNm9l+DkdITU01ERER5oUXXjDGGPP666+bcuXKmdOnT7u4Msdinu41z/yAzMpYfvk/zjzda575AZmVsfzyf5x55s550oy6WFJSklm/fr3p1KmT8fb2Nnv27DGJiYnm+eefN8HBwcbb29uUL1/eTJo0yRhjTGhoqJFkvYWGhhpjjDly5Ihp166dKVmypClUqJB54IEHzObNm22eKzQ01IwfP950797dBAQEmJ49e9psS5KJjIw0xhizaNEiU7hwYTNgwADzzTffOPMlydC//vUv4+3tbSZMmGCKFClifv75Z+u6Dh06mMKFC5unnnrKhRU6RmbzPHnypImMjDTVqlUztWrVMv/85z9dXGn2ZDbPK1eumHr16pnatWubGjVqmPfff9/FleKvyKysIbPILOQOZFbWkFlklqvQjLrIvn37zPDhw02pUqVM0aJFzaBBg6xhNHXqVBMSEmK+/vprc/z4cbN9+3azfPlyY8wfn6RJMgsXLjTnzp0zFy5cMMYYs2fPHvPee++Z/fv3m99++8384x//MD4+PubEiRPW5wwNDTWBgYFm2rRp5siRI+bIkSPm+++/N5LMli1bzLlz58zly5eNMX+Gd5cuXYyPj4+57777zMSJE83Jkyed/Er96f777zcFCxY027Zts1m+detWs27dOrcISWMynufZs2etn6qeO3fOlClTxsTHx7uoQsfIaJ7Jycnm+vXrxhhj4uPjTYUKFcylS5dcVSJuQ2bdOzLrJ2MMmQXXILPuHZn1kzGGzHI2mlEnunTpkpkxY4b1P0eHDh3M6tWrTWJios24F1980Tz66KMmNTU1w+1kdfeRGjVqmFmzZlnvh4aGmg4dOtiMyWz3kdvFxsaa999/3zRp0sR4enqa5s2bm8WLF5sbN27ctQZH2bhxo/H19TWenp7m4MGD6dZv3brVLULybvNMExER4dI/WNmVlXlevnzZhIaGmosXLzq5OqQhs+xHZtkis+AMZJb9yCxbZJbz0Iw60ZgxY4wk06RJkzv+B//xxx9N0aJFTeXKlc2LL75oPv/8c5v1GYXktWvXzEsvvWSqVq1qgoKCTKFChYyHh4cZMWKEdUxoaKiZMGGCzeOyEpK327lzp6lUqZLDjqfIih9//NEEBASY5cuXm8cff9x06tQp3Rh3CMmszNMYY3bt2mVq1Kjh5Ooc527zvHLliomIiDC+vr5m9uzZLqoSxpBZ9iKzbJFZcBYyyz5kli0yy7loRp3ozJkz5o033jCVK1c2AQEBplevXubf//63SUlJSTc2Li7OfPzxx6Zv374mKCjIJgAyCqgBAwaYSpUqmTVr1ph9+/aZw4cPm9q1a5shQ4ZYx4SGhprp06fbPC4rIXnz5k3zz3/+07Rt29YUKFDANGjQwMyZM8dcvXrVnpfhnhw7dswEBwebyZMnG2OM+e6774zFYjE//vijzbi8HpJZnefly5dN9erVzY4dO1xRZrZldZ7GGHP+/HnTqFEjc/78eWeXif8hs+4dmUVmkVmuQ2bdOzKLzHJ1ZtGMusiOHTtM//79TVBQkClXrpwZNWqUzcHit9u0aZORZD3OoECBAmbVqlU2Y2rWrGnGjx9vvX/t2jUTFBR015A8c+aMkWR27dplszw1NdV8/fXX1pAuV66ciY6OvuMuDY52+fJlU6VKFTNgwACb5W3atDEtW7a0WZaXQzKr80xISDBNmjQxixcvdnaJDnEv/55pBg0aZFauXOmM8nAXZNbdkVlkFpmVe5BZd0dmkVm5IbNoRl3s5s2bZsWKFaZly5bG09PT7Nu3z7z11ltm+fLl5uDBg+bXX381ffr0McHBwdZP9ipXrmwGDRpkzp07Z2JiYowxxjz55JOmTp065qeffjJ79uwxTzzxhAkICLhrSCYlJRlfX18zYcIEc/78eRMbG2uMMWbx4sXG19fXdO3a1Xz++ecZfqqYm+TlkMyK1NRU8/TTT5sxY8a4upQcdf78eesnwbGxsaZGjRpm3759Lq4KtyOzHIPMcg9kVu5HZjkGmeUecmNm0YzmImfOnDFxcXHm/fffN3Xq1DGFChUygYGBpnnz5mb37t3WcevWrTPh4eHGy8vLesrxY8eOmWbNmhlfX18TEhJiZs+ebSIjI+8aksYYM3/+fBMSEmI8PDyspxxPqyUvaN68uSlevLjx9fU1ZcuWzRWnSHe07du3G4vFYmrXrm29uTo8csLOnTtN7dq1TUREhKlVq5Z57733XF0S7oDMsg+Z5T7IrLyFzLIPmeU+cmNmWYwxRgAAAAAAOJGHqwsAAAAAAOQ/NKMAAAAAAKejGQUAAAAAOB3NKAAAAADA6WhGAQAAAABORzMKAAAAAHA6mlEAAAAAgNPRjAIAAAAAnI5mFAAAAADgdDSjAAAAAACnoxlFtvTq1UsWi8V6K1asmFq1aqV9+/a5ujQASIfMApCXkFlwdzSjyLZWrVrp3LlzOnfunP7973/Ly8tLbdu2dXVZAJAhMgtAXkJmwZ3RjCLbvL29FRwcrODgYNWpU0fR0dE6deqULl68KElavHix/P39dfjwYetj/v73v6tq1aq6ceOGJOnKlSvq0aOHihQpIj8/P7Vu3dpmfJoKFSrYfEJosVi0du1a6/pHHnlEQ4cOtXnM2LFjVadOHev9Xr16qUOHDhnOZcaMGapQoYLNsg8++EDVqlWTj4+Pqlatqjlz5mT6Wvz1E8zbb7169ZIkJSYmavDgwSpZsqR8fHz08MMP64cffrBuY9u2bbJYLIqNjbW+NhEREerRo4eMMZKk1NRUTZ48WRUrVpSvr69q166tVatWZbqNNLe/XsePH5fFYtGePXsynEvhwoW1aNGiTOcK5FVk1p/ILCD3I7P+RGa5H5pROFR8fLyWLl2q8PBwFStWTJLUo0cPtWnTRt26dVNycrI2bNigDz74QMuWLZOfn5+kP8Jl165dWrdunb799lsZY9SmTRslJSWle47x48dbPyHMacuWLdPrr7+uiRMn6uDBg5o0aZJee+01ffTRRxmOnzlzprW2Ll26qEuXLtb7M2fOlCSNHDlSq1ev1kcffaTdu3crPDxcLVu2VExMTLrtxcfHq02bNqpUqZI+/PBDWSwWSdLkyZO1ePFivffeezpw4ICGDRumZ599Vl999VXOvRiAGyKzyCwgLyGzyCx34+XqApD3rV+/Xv7+/pKk69evq3Tp0lq/fr08PP78rGPevHmKiIjQ4MGDtWbNGo0dO1b16tWTJB0+fFjr1q3Tjh071KhRI0l/hFNISIjWrl2rzp07W7eTmJiookWLKjg42ClzGzNmjN566y117NhRklSxYkX98ssvmjdvnnr27JlufFBQkIKCgiRJvr6+kmRT6/Xr1zV37lwtWrRIrVu3liTNnz9fmzdv1oIFCzRixAjr2MTERHXq1El+fn765JNP5OXlZV0+adIkbdmyRQ0bNpQkVapUSf/5z380b948RUZG5sArAbgPMutPZBaQ+5FZfyKz3A/fjCLbmjVrpj179mjPnj36/vvv1bJlS7Vu3VonTpywjilSpIgWLFiguXPnKiwsTNHR0dZ1Bw8elJeXlx588EHrsmLFiqlKlSo6ePCgzXPFxMQoMDDwjvXMmTNH/v7+1tukSZPSjUkL9iJFiqh27dr68MMP0425fv26jh49qj59+thsb8KECTp69GiWX5/bHT16VElJSWrcuLF1WYECBdSgQYN0c+3WrZv+/e9/KzIyUt7e3tblR44c0Y0bN/TYY4/Z1LV48eJ0dZUrV85mTEYaNWqkgIAAhYSE6G9/+5tOnz5t19yAvILMyjoyC3A9MivryKy8h29GkW2FChVSeHi49f4HH3ygoKAgzZ8/XxMmTLAu//rrr+Xp6alz587p+vXrCggIuKfnOX36tG7duqWKFSvecVy3bt306quvWu+/8847+vrrr23GNGvWTHPnzlVSUpL+7//+T3379lWtWrVsxsTHx0v64xO12wNckjw9Pe+pdnucP39eq1evVteuXfXkk09a60ura8OGDSpbtqzNY24PU0navn27zetcuXLldM/zySefqFq1ajp//rwGDx6sgQMHav369Y6eDpBrkFk5g8wCcgaZlTPIrNyBb0bhcBaLRR4eHrp586Z12TfffKM333xT//rXv+Tv768XXnjBuq5atWpKTk7Wzp07rcsuX76sX3/9VdWrV7cu++qrr+Tr66sHHnjgjs8fFBSk8PBw661o0aLpxqQFe7Vq1fTSSy+pWLFi2rt3r82YUqVKqUyZMvr9999tthceHn7XoM5MWFiYChYsqB07dliXJSUl6YcffrCZqyStW7dOHTt2VL9+/dS7d28lJydLkqpXry5vb2+dPHkyXV0hISE226hYsaLN+oyEhIQoPDxcDz/8sPr06ZPpgfaAuyKzMkdmAbkPmZU5Mivv4ZtRZFtiYqLOnz8v6Y8zks2ePVvx8fF64oknJEnXrl1T9+7dNXjwYLVu3VrlypVT/fr19cQTT6hTp06qXLmy2rdvr379+mnevHkKCAhQdHS0ypYtq/bt20v6Y7eLKVOmqH379unOXBYbG6tbt26pYMGCWa45NTVVCQkJ1k/sLl++rJo1a+q7776zGTdu3DgNHjxYQUFBatWqlRITE7Vr1y5duXJFw4cPv+fXqlChQho0aJBGjBihokWLqnz58vp//+//6caNG+rTp4/N2LRwnzJliiIiIjRlyhT94x//UEBAgF5++WUNGzZMqampevjhhxUXF6cdO3YoMDAww2Ms7uTWrVtKSEjQf//7X61atUo1a9a853kBeQmZlXVkFuB6ZFbWkVl5kAGyoWfPnkaS9RYQEGDq169vVq1aZR3Tu3dvU6tWLZOQkGBd9tZbb5miRYua06dPG2OMiYmJMd27dzdBQUHG19fXtGzZ0vz222/W8aGhoTbP89fb1q1bjTHGREZGmiFDhtjUOGbMGFO7du0Ma/by8jLh4eFm9uzZxhhjpk+fbkJDQ20ev2zZMlOnTh1TsGBBU6RIEdO0aVOzZs2aLL02PXv2TLf85s2b5sUXXzTFixc33t7epnHjxub777+3rt+6dauRZK5cuWJdtm3bNuPj42P2799vjDEmNTXVzJgxw1SpUsUUKFDAlChRwrRs2dJ89dVXmW7DGGMkmU8//dQYY8yxY8dsXsPChQubtm3bmuPHjxtjjAkKCjILFy686zyBvITMuvNrQ2YBuQuZdefXhszK+yzG/O+COkAuVqFCBW3bti3dtakkqUOHDho6dKgeeeQRp9cFABkhswDkJWQWXIVjRpEnlChRItOD2YsUKXJPu44AQE4jswDkJWQWXIVvRgEAAAAATsc3owAAAAAAp6MZBQAAAAA4Hc0oAAAAAMDpaEYBAAAAAE5HMwoAAAAAcDqaUQAAAACA09GMAgAAAACcjmYUAAAAAOB0NKMAAAAAAKf7/4n4hQyIvce0AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1100x300 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# токены предложения\n",
    "tokens = [\"<start>\", r\"$x_1$\", r\"$x_2$\", r\"$x_3$\", \"<end>\"]\n",
    "token_ids = torch.tensor([1, 3, 4, 5, 2])\n",
    "input_ids = token_ids[:-1].view(1, -1)\n",
    "target_ids = token_ids[1:].view(1, -1)\n",
    "\n",
    "# создаём маску\n",
    "causal_mask = create_causal_mask(target_ids)\n",
    "\n",
    "# получаем эмбеддинги токенов (без позиционного кодирования для простоты)\n",
    "embedding_layer = nn.Embedding(num_embeddings=7, embedding_dim=5)\n",
    "h = embedding_layer.forward(token_ids.view(1, -1))\n",
    "\n",
    "# внимание между входными и выходными токенами\n",
    "attention = (\n",
    "    torch.bmm(embedding_layer(input_ids), embedding_layer(target_ids).permute(0, 2, 1))\n",
    "    / 5**0.5\n",
    ")\n",
    "# накладываем маску\n",
    "masked_attention = attention.masked_fill(causal_mask == 0, float(\"-inf\")).softmax(-1)\n",
    "\n",
    "\n",
    "colormap = \"gist_gray\"\n",
    "fig, axes = plt.subplots(1, 3, figsize=(11, 3))\n",
    "axes[0].imshow(attention.squeeze().detach().softmax(-1), cmap=colormap)\n",
    "axes[1].imshow(causal_mask.squeeze(), cmap=colormap)\n",
    "axes[2].imshow(masked_attention.squeeze().detach(), cmap=colormap)\n",
    "\n",
    "axes[0].set_ylabel(\"Выходные токены\")\n",
    "for ax, title in zip(axes, [\"Исходные веса\", \"Маска\", \"Веса с маской\"]):\n",
    "    ax.set_title(title)\n",
    "    ax.set_xlabel(\"Входные токены\")\n",
    "    ax.set_xticks(torch.arange(len(tokens) - 1), tokens[:-1])\n",
    "    ax.set_yticks(torch.arange(len(tokens) - 1), tokens[1:])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 4, 5])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class ScaledDotProductAttention(nn.Module):\n",
    "    def __init__(self, input_dim: int, hidden_dim: int) -> None:\n",
    "        super().__init__()\n",
    "        self.W_q = nn.Linear(input_dim, hidden_dim)\n",
    "        self.W_k = nn.Linear(input_dim, hidden_dim)\n",
    "        self.W_v = nn.Linear(input_dim, input_dim)\n",
    "\n",
    "    def forward(self, x: Tensor, mask: Tensor) -> Tensor:\n",
    "        q = self.W_q(x)\n",
    "        k = self.W_k(x)\n",
    "        v = self.W_v(x)\n",
    "        _, _, D_k = k.shape\n",
    "\n",
    "        attention: Tensor = torch.bmm(q, k.permute(0, 2, 1)) / D_k**0.5\n",
    "        # apply mask\n",
    "        # 4. получаем веса внимания для разрешённых токенов\n",
    "        if mask is not None:\n",
    "            attention = attention.masked_fill(mask == 0, float(\"-inf\"))\n",
    "        attention_weights = attention.softmax(-1)\n",
    "        return torch.bmm(attention_weights, v)\n",
    "\n",
    "\n",
    "ScaledDotProductAttention(5, 3).forward(\n",
    "    embedding_layer(input_ids), mask=create_causal_mask(input_ids)\n",
    ").shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Encoder-only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bidirectional encoder representations from transformers (BERT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обозначения:\n",
    "- `<class>` — специальный токен класса\n",
    "- $x_i$ — токены\n",
    "- embedding — слой эмбеддинга токенов\n",
    "- positional encoding — позиционное кодирование\n",
    "- transformer layer — слой трансформера (multi-head attention + layer norm + skip connection + MLP)\n",
    "- **с** - метка класса\n",
    "- LSM — linear softmax\n",
    "- $y_i$ — вероятности скрытых токенов\n",
    "\n",
    "<img src=\"../attachments/bert.png\" style=\"background:white\" height=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучение идёт в два этапа:\n",
    "1. pre-training: выбирается 15% токенов предложения, задача модели — угадать их; из этих токенов\n",
    "   - 80% заменяются на специальный токен `<mask>`\n",
    "   - 10% заменяются на случайный токен из словаря\n",
    "   - 10% остаются неизменными\n",
    "2. fine-tuning: донастройка под нужную задачу\n",
    "   - классификация предложений: обучение MLP поверх выхода для `<class>` токена\n",
    "   - классификация токенов: обучение MLP, общего для всех токенов"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Encoder-decoder и перекрёстное внимание"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Энкодер обрабатывает входной контекст, предоставляя надбор ключей и значений для декодера.\n",
    "\n",
    "Декодер, помимо *masked self-attention*, теперь имеет второе внимание, направленное на контекст из энкодера — *cross-attention*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../attachments/seq2seq.png\" style=\"background:white\" height=\"400\"/>\n",
    "<img src=\"../attachments/cross-attention.png\" style=\"background:white\" height=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Посмотрим на реализацию этого типа архитектуры на примере модели `T5` из библиотеки `transformers`.\n",
    "\n",
    "`T5Block` — основной строительный блок как энкодера, так и декодера.\n",
    "\n",
    "В энкодере он принимает эмбеддинги токенов `hidden_states` с предыдущего блока и маску для внимания `attention_mask` — она скрывает `<pad>` токены.\n",
    "\n",
    "В декодере помимо эмбеддингов токенов `hidden_states` и маски `attention_mask` (она теперь также скрывает будущие токены) блоку нужно два дополнительных входа: эмбеддинги из энкодера `encoder_hidden_states` и соответствующую маску внимания `encoder_attention_mask`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Упражнение**. В T5 есть несколько модификаций по сравнению с архитектурой оригинального трансформера. В частности, эмбеддинг позиции токена теперь не абсолютный, а относительный. Разберите код класса `transformers.models.t5.modeling_t5.T5Attention`, чтобы понять, как это работает."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers.models.t5.modeling_t5 import T5Block"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Мультимодальные трансформеры"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В токены можно превращать что угодно: фрагменты изображения, участки аудиофайлов, атомы молекул\n",
    "\n",
    "А значит, можно даже передавать в трансформер токены, полученные и из разных модальностей\n",
    "\n",
    "Примеры задач:\n",
    "- ответы на вопросы по картинке\n",
    "- суммаризация видео\n",
    "- speech-to-speech между языками"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../attachments/vit.png\" style=\"background:white\" height=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dl-mcs",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}