04-train-word2vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d232a003-836f-4940-86b7-3be5da304a8b",
   "metadata": {},
   "source": [
    "# Train `word2vec` models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77d74700-e047-4d0b-a0bd-77597eb77e2b",
   "metadata": {},
   "source": [
    "In this notebook, we will train `word2vec` models that can represent math-specific tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a970c4d0-4f9e-4070-9bcf-29ceff2d1656",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mir\n"
     ]
    }
   ],
   "source": [
    "! hostname"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1eb2a74a-cc34-4c5d-9040-93cd13c5f0a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! pip install .[word2vec]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9b123fd-1783-48a2-8816-ed9768fa1d1e",
   "metadata": {},
   "source": [
    "## Train non-positional `word2vec` models\n",
    "\n",
    "In this section, we will train `word2vec` models without [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca548959-2b09-4408-a409-de40d4671c15",
   "metadata": {},
   "source": [
    "### The text + LaTeX format\n",
    "\n",
    "As our primary representation, we use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "92b52b98-714f-4680-abf2-d9d6cf69a4cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word2vec-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "846be871-a87a-4c12-b26e-53927c9c42cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "491M\tword2vec-text+latex/model/custom-en-word2vec_cbow-epochs=1\n",
      "491M\tword2vec-text+latex/model\n",
      "0\tword2vec-text+latex/cache/custom-en-word2vec_cbow-epochs=1\n",
      "0\tword2vec-text+latex/cache\n",
      "491M\tword2vec-text+latex\n",
      "491M\ttotal\n"
     ]
    }
   ],
   "source": [
    "! du -hc word2vec-text+latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6ff45500-c2f5-4743-8410-671eb3809ab1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71540 300\n",
      "Ġ -0.08767373 0.45843008 -0.008227732 -0.12825662 0.40604976 -0.08795726 0.25151646 0.6583432 -0.2843155 -0.36248145 -0.90115786 -0.040905494 0.5283086 -0.059527937 0.0679248 -0.9048535 -1.3493316 -0.029202132 0.537324 -1.0813957 -0.0958676 0.20301759 -0.42168352 0.2544472 0.40478817 -0.16245314 -0.87397313 0.1526849 -0.84534985 0.3666558 0.43923587 0.1550599 0.116662145 -0.545336 0.21840543 -0.5227038 -0.822009 0.64769906 -0.080697194 0.14055185 -0.39671564 -0.19012949 0.18417068 -0.1396354 -0.14135444 -0.009711535 0.5304928 0.11867654 0.29499966 -0.085327715 -0.34632125 -0.8764067 0.9665256 -0.3684227 0.4612774 -0.4608598 1.1773355 -0.25845176 0.7018656 -0.33604014 -0.6696823 -0.58044964 -0.6593275 0.4494126 -0.2941379 -0.0895813 0.17626098 0.326288 0.7238655 0.52115345 -0.36118248 -0.04045876 -0.740491 -0.8278218 -0.030319996 -0.21983442 -0.39075902 -0.23682156 -0.2234199 -0.8090305 -0.5750199 -0.38446173 -0.5580111 -0.35380092 -0.12416926 -0.40380785 0.08775886 -0.6330915 -0.40948898 0.14137323 -0.3217639 -0.056174345 -0.15840434 -0.62162894 0.1320803 -0.44423342 0.11852682 0.32392016 -0.40903822 0.3645926 -0.5848186 0.008668691 1.0376679 0.5706758 -0.33991644 -0.08167994 0.11818813 -0.6282519 0.11206024 -0.07510575 -0.5762489 -0.68117976 0.15575096 0.20001493 0.07470402 1.0600466 -0.670748 0.6146246 -0.27924186 0.84726334 0.7455679 -0.5079409 0.52235085 -0.09790807 0.27425402 0.61482745 0.2627966 0.23189926 -0.34064427 0.32675257 0.07505082 0.6977508 1.062599 -0.035859644 -0.6118634 -0.35036147 0.16329038 -0.4093189 0.5496668 0.87633353 -0.90940243 0.26335418 -0.15710287 -0.98359966 0.34571174 0.44936162 -0.0018540785 0.1545101 0.7140992 0.3241018 -0.17774296 -0.78474516 -0.16747466 0.13635491 0.057428926 -0.1431892 0.6635018 0.68636596 0.06688242 -0.4404522 0.07986421 0.025586799 -0.06530251 0.53884494 0.3485678 -0.019057099 -0.07262489 0.3904915 0.29909274 0.027896285 0.61303604 -0.48149627 -0.7915106 0.68934804 0.016549265 -0.1499346 1.0956509 0.34629893 0.5575733 -0.41357878 -0.41392168 -0.6716312 0.6500191 0.40095156 -0.55831087 -0.56142116 1.3261213 -0.39677054 0.1952396 -0.12442935 0.33384433 0.071405664 -0.1040462 0.3448097 -0.5673152 -0.8832012 0.6578483 0.2832926 0.023321822 -0.29410833 0.21995458 0.0069410093 0.6322188 -0.6466467 0.18078256 0.21426396 0.40271017 -0.9816196 0.24556477 0.43710402 -0.29173812 0.2050397 0.14551854 -0.17924523 -0.13611424 -0.15060024 0.16942094 0.6039859 -0.48329443 -0.114280686 -0.028682569 -0.42622736 -0.11365329 0.69146115 -0.5361356 -0.9855321 0.6187468 0.21207567 0.99783534 -0.5191399 0.087104216 -1.2373824 -0.6002799 -0.2105187 0.9440029 -0.71715915 -0.6508703 0.9776211 -0.05629038 -0.29268286 -0.5374582 1.0119452 -0.3331671 -0.28554472 0.74148434 -0.23046555 -0.7011179 -0.986338 0.8502456 0.8837246 -0.34374416 0.2253697 0.041501835 0.44571415 0.61615986 0.3843366 -0.60890824 -0.3337613 0.31139696 -0.4750477 0.4160278 -0.07978772 0.20922202 -0.17292652 -0.14533928 1.1868623 -0.73019534 -0.18034989 0.6425214 0.20317228 -0.2564635 -0.08594687 0.7591718 0.49438468 0.821218 0.21384524 0.24891214 -0.11722552 0.0766423 0.049412657 -0.87792885 -0.25032264 -0.50686145 -0.2993835 -0.32196322 -0.68571204 0.07937667 -0.774816 0.16331445 0.3771449 0.5321696 0.22740927 0.29289734 0.4675806 -0.13330528 -0.5665563 0.23658617 0.07748564 -0.20460369 -0.5185828\n"
     ]
    }
   ],
   "source": [
    "! head -n 2 word2vec-text+latex/model/*/model.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8462f26b-360c-4847-a392-8e5561c30969",
   "metadata": {},
   "source": [
    "### The text format\n",
    "\n",
    "To train a `word2vec` model just for text, we also have a separate dataset with just text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7bcfe66c-aa67-463c-a5bd-4b72566bbb7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word2vec-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "730f3401-7c4f-488a-bf07-d3a7a14a5c97",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "339M\tword2vec-text/model/custom-en-word2vec_cbow-epochs=2\n",
      "339M\tword2vec-text/model\n",
      "0\tword2vec-text/cache/custom-en-word2vec_cbow-epochs=2\n",
      "0\tword2vec-text/cache\n",
      "339M\tword2vec-text\n",
      "339M\ttotal\n"
     ]
    }
   ],
   "source": [
    "! du -hc word2vec-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d12f6e2f-d66c-4393-af8d-a50b6b590235",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "49310 300\n",
      "Ġthe 0.7032212 0.03634197 -0.5135605 0.3357469 0.7312258 0.351296 -0.896176 -0.3988427 -0.10211854 -0.11711154 0.32692328 0.29528573 0.12167 0.34439772 0.08338026 0.20829177 -0.36266616 -0.5905029 0.47458592 -0.35525286 0.013833059 0.6882711 -1.1641 -0.038543947 -1.1867766 -0.6374976 0.0148884 0.6491241 0.05241671 0.35373393 -0.103352964 -0.17781082 -0.05479559 -1.2284712 -0.590193 0.19130298 -0.31961197 -0.61971647 0.70202094 0.169576 0.69972306 0.2638736 0.32178205 0.283192 -0.9492635 -0.34824353 -0.017195312 -0.07383108 0.5235321 -0.98763424 -0.7479147 0.36549878 -0.16234569 -0.1586904 0.23776159 -1.0839185 0.24051921 0.11132975 -0.023195984 0.39197168 1.5016292 0.1194545 -0.15831332 0.6309355 0.6230399 -0.45612815 -0.33054242 -0.22435433 -0.5830698 -0.8330652 0.26828966 -1.0591255 0.62596726 0.117289886 0.35900852 -0.106096074 -0.4572636 0.53354627 0.15884255 0.055366952 0.101207554 -0.20706625 0.29750574 0.22175674 -0.12139011 0.38055658 0.5777266 -0.69356936 0.08338858 -0.12468156 0.4142652 0.7215577 -0.10814693 0.079819225 0.1652419 0.9701961 -0.17433813 0.023993889 0.22170807 0.08784258 0.14387059 0.14518297 0.91435957 0.34979132 0.023644615 -0.10192971 -0.7106374 0.22543693 -0.011796071 -0.7530082 0.17292611 -0.13373935 -0.11294071 -0.40253234 0.41702643 1.1383959 0.17800318 0.60980093 0.15823461 -0.71622825 0.10954955 0.75798887 -0.2011988 0.4635097 -0.116971605 -0.24899434 0.18632708 -0.5643058 0.23101267 -0.20835613 0.2553958 0.42387357 0.054583747 0.95209044 0.34605297 0.16842169 0.113921456 0.11262249 0.7717428 0.45101517 0.89610845 0.11031873 0.5878394 0.23339741 0.6991393 0.6743794 0.5628635 -0.37489337 0.0111964075 0.5820504 0.20210288 -0.18791339 -0.3407555 0.00353751 1.0972883 -0.36512902 -0.19475263 -0.46427214 -1.2256234 -1.2600694 0.24791919 0.23910682 -0.14240767 -0.2565167 -0.4947033 -0.22007321 -0.33788484 0.17184174 0.12118953 0.73373 0.39797947 -0.22141118 -0.5172849 0.17548381 -0.51482844 0.20291987 0.6496129 0.64327633 0.035723906 -0.32425112 0.03389794 0.20201845 -0.4562789 -0.32670966 0.69296086 0.41825798 -0.004201926 -0.13113388 0.30526623 -1.272194 0.018950913 0.40599248 0.5830895 0.367151 -0.32910654 -0.6715604 0.573865 0.5028504 -0.38303998 -0.11055435 -0.9473077 0.5689157 -0.8206967 -0.6346933 -1.0836716 0.21541505 -0.4139576 0.16068774 0.19791006 -0.24645741 -0.6675114 -0.84458166 -0.6464452 0.23288472 -0.0387988 -0.79467785 0.4417636 0.21014889 -0.55782366 0.15799816 0.105386734 0.038828157 -0.26303357 -0.0022730394 -0.29828674 -0.3189903 1.5600485 -0.42283416 0.20070507 0.13892803 -0.219938 -0.09631657 -0.70460105 -0.23347785 -0.6816922 0.09791499 -0.36452883 -0.116085015 0.20115876 -0.35879305 1.0202456 -0.58141947 0.0008911004 -0.36229464 -1.5053937 -1.1938794 0.18494572 -1.3094447 0.42081127 0.11061382 0.6581194 -1.1847813 0.45141038 -0.2593635 0.45969322 0.055821866 0.99714375 0.037813693 -0.10453319 0.45534766 0.06938248 -0.12983768 0.70297617 -0.2303147 0.030285187 -0.42075086 -0.25409964 -0.21302252 -1.4439908 -0.48591053 0.18736441 -0.38908225 -1.0066328 -1.1883599 0.5143463 -0.21521124 -0.3614359 -0.3159893 -0.25229445 -0.8268502 -1.4373244 -0.21027316 -0.048495814 -0.18162547 -0.3077641 -0.24173014 0.18933122 1.0984145 0.5472132 0.24571675 -0.13806412 0.00723782 0.9951707 0.6712722 0.0794823 -0.38129246 -0.50898063 -0.2219945 -1.15217 1.1673851\n"
     ]
    }
   ],
   "source": [
    "! head -n 2 word2vec-text/model/*/model.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d3e8508-8646-48d1-b35b-924fd0768ee2",
   "metadata": {},
   "source": [
    "### The LaTeX format\n",
    "\n",
    "To train a `word2vec` model just for math, we also have a separate dataset with just LaTeX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a7fb3a4c-1796-4c48-a896-6328fc452b41",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word2vec-latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c92ea198-929e-4601-8a55-fbab4d39fa4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "203M\tword2vec-latex/model/custom-en-word2vec_cbow-epochs=5\n",
      "203M\tword2vec-latex/model\n",
      "0\tword2vec-latex/cache/custom-en-word2vec_cbow-epochs=5\n",
      "0\tword2vec-latex/cache\n",
      "203M\tword2vec-latex\n",
      "203M\ttotal\n"
     ]
    }
   ],
   "source": [
    "! du -hc word2vec-latex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7b022310-db99-4a3b-b9ba-5bfbe7df8ef8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29675 300\n",
      "% -0.8123627 -0.4865128 -1.6376101 -1.0977508 -0.0026990217 -0.81645113 1.092308 0.4254648 -1.7375989 1.0688992 -0.9607899 -0.84274924 0.030448405 -0.16411988 -0.11348923 1.3030134 1.5414021 0.4524594 1.6606439 -0.1709696 0.468996 -0.006377744 0.03779586 0.19229876 1.8753405 -0.7870954 1.3225005 1.5892274 0.7606449 0.46662828 -1.3111341 -0.16214229 -0.43260443 -1.8870902 -0.39288288 -2.175066 0.6409486 -1.2540636 -0.69904655 -1.4974338 3.1012635 0.9241071 0.57255584 -0.4895081 0.404196 1.4884887 -0.98764694 0.28389123 -0.3655657 0.72464097 2.0262249 -0.0961223 -0.14714527 -0.13083251 -2.4775496 -1.8082969 -2.3877156 0.37305248 0.5121084 -3.3746302 0.06849546 -0.42158577 1.6311511 0.006520479 -0.5237138 2.7410126 0.8662932 1.7007933 1.372537 -0.01475599 0.26689458 1.2083758 0.8985544 0.35812837 -0.84248924 0.27347162 0.51197463 0.94877106 -0.8685739 1.3737218 0.26224765 -1.1566223 0.16475773 1.4718179 0.29538846 1.2587968 -0.5608657 0.85403377 0.61386347 1.0316347 -2.148078 0.3311078 2.7504566 0.9395646 1.6867167 -1.8202823 0.8181647 -1.2330543 1.5306771 -1.5110942 1.7264669 0.604235 0.9768641 -0.28888047 1.4056671 -1.1768188 -0.72768426 0.9345465 -1.3855999 0.4695429 -2.4020395 -0.18859299 -0.53428113 -0.20885769 0.68570966 0.28413305 0.9378714 -1.4513879 0.8486003 -0.5431686 0.916233 -0.4495686 -0.5717226 0.7825732 0.20561585 2.50433 1.7897633 -0.6473989 1.3329875 -1.292524 -1.8232738 0.17948511 -1.6432968 0.82835495 -1.6910813 0.5037663 -1.6832622 -1.9155686 -0.50602216 1.6560411 0.54117084 1.0524638 1.0510081 0.8804764 1.9866478 0.6705864 1.0114582 -1.3019656 0.10403357 0.32826656 1.3532912 0.23784822 0.1423988 -0.14446466 -0.57553315 0.2100854 -0.9779043 0.11986798 -0.42332637 0.054344825 -0.6736863 1.1670914 1.4187896 -0.9289039 0.75444007 1.4867086 -0.30810603 -0.4400249 -0.38666475 1.01341 -1.3834178 -0.53591603 0.5991456 1.3366655 -0.36113805 1.7849749 -0.580174 -1.262397 -0.6253308 -2.2188513 -0.9738565 2.3449106 -0.46274284 0.17434067 -1.0650723 0.59822726 -1.6130629 -1.0229404 -1.4639151 -0.08686826 0.18055655 0.1683726 0.40454042 1.4884297 -3.4949346 2.1560369 0.14648801 -1.0505072 0.018464614 1.2250947 -0.68011284 -0.67342454 0.7730886 0.3817088 -1.5059363 -0.45171103 1.384424 0.09044544 -2.4193814 1.0147784 -0.31977445 -1.3752204 1.7940927 -0.19548282 1.1604615 -3.4423327 -1.1255811 -0.57944685 1.9795127 1.051063 0.41773847 1.8612565 0.40292734 1.9315907 -0.27114946 0.7236178 -0.20810893 -2.9534962 -0.28063318 0.6352363 -1.724534 -1.4742657 0.8905435 2.4196699 -2.5617366 1.5607321 0.14000624 0.42309767 1.2795153 0.30613855 2.0044465 0.42001152 -0.210959 0.4365202 1.3050739 -0.49899662 -1.0764526 -0.9166407 1.5090113 0.4954564 0.36746556 -1.0061764 -0.16859661 -1.0634037 1.9037688 1.0653332 -0.3183008 0.9962681 -0.9255219 0.99048144 -1.3002526 1.2006431 0.9814633 -0.9595871 2.2052667 0.2743566 0.8299105 -0.22000125 -0.6890798 0.83639354 0.9158898 -1.1526989 -0.14381412 0.48132378 0.3294277 -2.7947447 2.1230867 1.9446428 2.0093582 0.68539387 -0.32059008 0.37481666 0.98511386 1.2046865 0.74780315 3.2603183 -0.70608217 -0.010069882 -0.63476324 -1.3298305 -1.1735759 2.3084 1.922498 0.14844276 2.6606627 -2.3418038 0.72223926 0.5944676 0.83826184 0.34922925\n"
     ]
    }
   ],
   "source": [
    "! head -n 2 word2vec-latex/model/*/model.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "756a1d58-877a-43d5-a34f-0f807b90b2a1",
   "metadata": {},
   "source": [
    "### The Tangent-L format\n",
    "\n",
    "To train a word2vec model just for math, we also have a separate dataset with just the format used by [the Tangent-L search engine from UWaterloo][1].\n",
    "\n",
    " [1]: http://ceur-ws.org/Vol-2936/paper-05.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1604ba54-cae9-4a9f-9247-1526c2eb284f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word2vec-tangentl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "25ac4a96-59fc-465c-ba0c-942ebb90f2f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14G\tword2vec-tangentl/model/custom-en-word2vec_cbow-epochs=2\n",
      "14G\tword2vec-tangentl/model\n",
      "0\tword2vec-tangentl/cache/custom-en-word2vec_cbow-epochs=2\n",
      "0\tword2vec-tangentl/cache\n",
      "14G\tword2vec-tangentl\n",
      "14G\ttotal\n"
     ]
    }
   ],
   "source": [
    "! du -hc word2vec-tangentl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b527865c-c707-4ba5-bf5a-433c670ea721",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1986439 300\n",
      "(start) -0.7883517 -0.4147513 0.109570004 0.30688292 3.6664822 -1.8741546 0.5164354 -0.3924657 0.7925373 -1.4477867 -0.23767693 -2.0926077 0.4927457 -0.75468713 0.54832906 1.0683583 -0.6374132 0.061881304 0.09244942 -2.2327147 1.1101563 0.5815448 0.2750769 1.1154897 -0.96233326 2.1798615 0.053582694 -0.71024233 0.01958334 0.11164918 -0.2490331 -1.1246321 1.8936965 2.0551243 -0.67028046 1.387179 0.37774748 2.308518 0.79442024 -0.7665613 -1.3263549 1.0679648 -1.3366199 0.31892794 1.0904627 0.6458117 -1.3810245 -0.059227582 0.41283834 -0.6207842 -1.3623626 1.7307568 0.70670766 1.8203899 -0.8782172 -0.6903082 2.0658896 -0.8475402 0.7414832 0.0026021323 -1.5228511 -1.5677015 0.38836935 -0.15504538 -1.664133 1.1359817 0.32770953 -0.7394564 -1.185888 -0.38944528 -1.1922969 1.1946704 2.3373098 -2.6557286 -0.37687904 -1.317471 -0.15844513 -0.19751823 0.62681144 -0.6336746 0.55956715 0.19937158 0.012083598 0.8902954 -0.10233464 0.8064445 0.29683325 0.23845148 -1.2934929 0.43259752 -1.0025017 0.053134415 -1.499047 0.4091567 -0.6436059 -3.061214 -2.5322685 1.2827375 -0.73186165 -0.65268266 -0.31773424 -2.0035229 0.058959518 0.88510114 0.9327762 -1.4798402 -2.345173 -1.8565704 -1.6285127 0.34957212 -1.1384103 -1.607012 -2.7363296 0.31607732 -1.0931888 0.942612 -0.68005055 -1.2658322 -2.1222417 0.12271517 0.650918 1.2890283 -0.24029745 0.11299459 1.465649 2.9507346 -0.5600176 -1.9008464 1.9499542 0.990931 -2.3694577 0.38949326 -2.1961265 -0.017982122 0.21786521 1.2681831 0.70216596 -1.1897635 0.015133598 -0.015395201 -0.19256479 0.3598189 -0.6428125 -0.9443753 1.355702 -1.6330523 0.8508862 -0.8199999 0.031606104 -1.627325 0.3143541 0.95261335 0.41943955 0.06492989 -0.14079733 0.53294766 0.51801 0.63648 0.44012275 -0.67208916 1.3528887 -0.42335886 -1.7800093 -1.4920665 3.3009877 0.21279933 3.2637026 2.6692657 0.97050136 -1.0215422 0.121348634 0.25435153 1.6263947 0.35032204 -1.6576772 -0.5270458 0.5449815 -0.28203213 2.0090976 -0.45495668 2.310208 0.27590835 1.6513096 -0.24747272 3.176068 1.8148326 2.0655355 -1.5356815 -1.4987398 0.5901769 -0.5433914 -0.25452474 -1.0399299 -0.9015507 1.7075961 1.9607157 1.6582023 -2.1851008 -0.23494855 -2.1118379 0.25436205 0.79399854 -0.26535133 0.9051056 -0.7972669 -0.23155178 2.8250303 -0.10529831 -2.4389393 -0.7683769 0.28255892 0.9706536 -1.7579983 -0.07468515 -2.4999425 1.9022057 -1.0962068 0.28886536 0.78561825 -1.4071286 -0.53934497 0.6720705 -1.7429113 2.2206776 1.4703506 -0.36994165 -1.5290967 -2.034656 -0.3777649 -0.76946425 -1.4999954 0.8658083 0.95371735 -0.9172984 -2.1872551 -2.0729647 1.2722569 -0.8010794 0.6116946 -0.7764798 -1.6006391 1.109636 3.3999288 1.0058513 -0.8646668 -0.4448434 0.5546474 0.9523334 -1.164235 0.01866254 -0.56853366 0.3744149 -0.15821528 -0.7253024 -1.2748411 -0.082098044 1.4542475 -2.738288 -2.680634 1.1139567 2.56887 1.313381 -0.7272672 3.23548 -1.0018919 0.8709408 0.6742339 -0.26821366 0.4020069 2.2450118 -2.2735097 0.8286834 -1.0707921 2.3511992 -1.8656586 1.5959777 1.121281 0.04910606 0.34743184 -1.7238557 -1.7034782 -0.057723504 -0.29123676 -0.049308274 -0.091668636 0.48792765 -0.43610862 0.8744989 -0.28056353 0.1418997 1.2280695 -0.68234324 1.8146248 0.73928463 -0.14755319 0.9735819 0.8933675 0.27920365 0.49371052 -2.06782\n"
     ]
    }
   ],
   "source": [
    "! head -n 2 word2vec-tangentl/model/*/model.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b48ae0e0-390f-4e7e-a4ee-1f3ed54cbe0a",
   "metadata": {},
   "source": [
    "## Train positional `word2vec` models\n",
    "\n",
    "In this section, we will train `word2vec` models with [positional weighting][1].\n",
    "\n",
    " [1]: https://github.com/MIR-MU/pine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afa26a57-9351-4921-ba20-c52f28a2c503",
   "metadata": {},
   "source": [
    "### The text + LaTeX format\n",
    "\n",
    "As our primary representation, we use text and LaTeX separated by special `[MATH]` and `[/MATH]` tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "37e3d10a-46f3-45ad-9def-d961162cb3e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word2vec-text+latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "85c90bd9-f14e-474f-97b5-85f68cdebe2c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "490M\tword2vec-text+latex-positional/model/custom-en-constrained_positional_word2vec_cbow-epochs=1\n",
      "490M\tword2vec-text+latex-positional/model\n",
      "0\tword2vec-text+latex-positional/cache/custom-en-constrained_positional_word2vec_cbow-epochs=1\n",
      "0\tword2vec-text+latex-positional/cache\n",
      "490M\tword2vec-text+latex-positional\n",
      "490M\ttotal\n"
     ]
    }
   ],
   "source": [
    "! du -hc word2vec-text+latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e61aea80-ed69-4520-8954-fe07229cb3d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71540 300\n",
      "Ġ -0.1875479 0.8918865 0.30134764 0.61537254 0.16202745 0.16520086 -0.031330816 -1.4616253 -0.66537344 0.12608404 1.8684675 -0.35645586 0.68418777 -0.75702 -0.10578573 -0.42978078 -0.48130286 0.7738346 0.5421547 0.2888756 0.65891993 0.45541206 0.014337258 -0.94012845 0.84462523 0.23259977 -0.37134096 0.40669978 -0.86422944 0.17734031 -0.62940836 -1.7463795 0.8235458 0.30762246 0.7477674 -0.24838255 1.5179554 -0.32424372 0.6378801 0.57103693 0.7150298 1.2847652 0.21121918 0.037991 0.5547662 -0.11271347 1.0556422 -0.47664535 0.23925139 -0.94804865 0.26941147 0.5826693 0.39958215 -0.0810523 -0.86759174 -0.46414074 -1.1166532 -0.8150435 1.0491073 -0.059790947 0.1920825 0.31189322 -0.25114658 0.8361882 -0.32525218 -0.48084393 -0.13130282 0.05037433 -0.19800007 0.583167 -0.19305679 0.25336298 -0.5069497 0.25286612 0.06388154 0.067826465 0.16921827 0.2411752 -0.06670237 -0.1617248 -0.96788156 0.5192936 0.13087822 -0.9724806 -0.050773334 0.60405374 -0.30630124 -0.6006769 -0.6523337 -0.090416856 -0.50178194 -0.27522486 -0.38215968 -0.093843184 0.4268409 0.22846344 0.72420204 -0.09429116 -0.18929848 0.2305726 0.4209774 0.6395536 -0.27030194 0.035327397 0.19187456 -0.2761 -0.5273337 0.31379548 -0.17281747 -0.24399602 0.07203875 -0.17436518 0.30976182 0.13409619 -0.14228305 0.012071814 0.26170382 0.15617642 0.050079264 -0.10996329 -0.001988597 -0.5657276 -0.12776057 -0.16067082 -0.14667086 0.28157058 -0.33820525 0.10578688 -0.34368423 -0.46554857 0.556898 -0.13849354 0.33068973 -0.09641061 -0.04475212 0.03538974 -0.28005892 0.24018337 0.18446656 -0.14506218 -0.49402896 0.11297398 0.3063739 0.25485903 -0.34681323 0.19977616 -0.12708259 -0.7008026 -0.025313001 -0.02010656 -0.40949965 0.21166638 -0.15068254 -0.152773 0.46235085 -0.09822817 -0.17311957 -0.2028609 0.1793367 -0.25487763 -0.35907304 0.2039339 0.2779899 -0.8498061 -0.18656339 0.4634143 0.31203294 0.46710843 0.24346653 -0.18626036 0.542164 -0.2015579 0.26167983 0.40929443 0.0073186713 0.16731553 0.22913942 -0.1802317 0.3809723 0.2888774 0.5070744 -0.10175196 0.006417045 0.014234353 -0.37702167 0.20488486 -0.3691689 0.32172915 -0.10072076 0.35102648 0.4338663 0.22532436 -0.047314372 0.05274775 0.13652714 -0.059277255 0.07935854 -0.44979775 0.68290555 -0.30278355 0.6534513 -0.58885497 -0.17456514 -0.77399373 -0.40127024 -0.8098032 0.1009861 0.5558518 -0.1449176 -0.4021928 -0.6859702 -0.8821405 0.00833318 0.6729845 -0.5632754 0.34283677 -0.42847207 0.29046285 0.0414267 -0.1275674 0.43086267 -0.2453801 -0.07947726 0.10361258 0.75858223 0.04351373 0.23232368 0.28248045 0.53699046 -0.10331799 -0.18573761 -0.43352476 0.37950426 -0.7569455 -0.021394879 0.25250003 0.07918312 -0.11323617 0.18236375 0.06788802 0.1257946 0.13635054 0.11283595 -0.0456362 -0.37482888 -0.27515304 0.0908392 0.16158347 -0.2959242 0.021860406 0.3607077 -0.076345384 0.436023 -0.06330103 0.12704606 0.22630952 0.4401505 0.62984693 -0.49300256 -0.2060631 -0.11108081 -0.03397448 -0.49660736 0.04799576 -0.22392556 -0.004915501 0.12948956 0.06575476 -0.9014949 0.9360621 -0.092884816 0.2691935 0.15423691 -0.49052492 0.11404328 -0.4500608 -0.5050492 -0.19625574 0.5288513 0.68178385 -0.13454193 -0.20586088 0.124659 -0.17768149 0.5915897 -0.07933269 0.22368115 0.165613 0.106340826 0.24258897 -0.49974456 -0.26225412 0.49279755 0.2686802 0.468745 0.124740444 -0.2776027 -0.13510704 0.4667588 -0.37842703\n"
     ]
    }
   ],
   "source": [
    "! head -n 2 word2vec-text+latex-positional/model/*/model.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abd6665c-cf5f-41e2-8c10-e1e6feb25c82",
   "metadata": {},
   "source": [
    "### The text format\n",
    "\n",
    "To train a `word2vec` model just for text, we also have a separate dataset with just text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "74e91186-9db5-44b3-8332-f5387586b929",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word2vec-text-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "69cd2e29-6d78-43e6-9c4b-f0a07e814249",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "338M\tword2vec-text-positional/model/custom-en-constrained_positional_word2vec_cbow-epochs=2\n",
      "338M\tword2vec-text-positional/model\n",
      "0\tword2vec-text-positional/cache/custom-en-constrained_positional_word2vec_cbow-epochs=2\n",
      "0\tword2vec-text-positional/cache\n",
      "338M\tword2vec-text-positional\n",
      "338M\ttotal\n"
     ]
    }
   ],
   "source": [
    "! du -hc word2vec-text-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d510add7-38d5-496e-ad14-0f33c63f1bf3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "49310 300\n",
      "Ġthe -0.0948922 0.46487233 0.53211135 -0.041445784 1.6362921 -0.10327671 1.3562237 -0.6604223 -0.5201821 -0.074263625 1.2274079 -0.5351085 0.05654604 -1.4765608 0.22930744 0.76847905 -0.22626773 -1.4716115 -0.24517034 1.646543 -0.31159335 0.026490277 0.38063404 0.6644411 -0.25424173 0.8107544 -1.7661531 2.2621965 0.02205728 -1.1229014 -0.2920755 1.4923047 -0.32814467 -2.2108057 0.32496676 -0.7461394 0.7928426 -0.1117733 0.54062027 -1.5501577 -0.2083572 -0.91348296 0.7764302 -0.64569443 0.32488403 -0.42126602 -1.657367 0.9719783 -0.17734364 0.046469208 0.8273238 -1.5106404 -0.37492773 1.3581058 0.08664954 -0.10760002 -0.6162666 -0.53794456 0.36314002 1.2475406 0.26962999 -0.17876142 0.015458746 0.9794636 0.08373761 -0.38949597 -0.22871017 0.1528613 -0.41417062 -0.3410043 -0.48524252 0.31217077 -0.1679793 -0.42378938 -0.20197037 -0.0029911604 0.59018666 -0.0024281521 -0.046227697 -0.74926484 0.24579398 0.5046935 -0.030378481 -0.21657053 0.22813272 0.13031432 -0.1576565 -0.20766057 0.37962636 -0.100297935 0.11839421 0.26767337 0.1944624 -0.28912115 -0.5467687 0.24880293 0.08436728 -0.016022887 -0.22509155 0.61973995 -0.05571367 0.15441006 -0.13125493 0.016458254 -0.3369991 -0.536278 0.015724922 -0.325089 -0.04308213 0.21561025 0.6466987 -0.17191955 -0.40566707 0.094122715 0.4909713 0.46185076 -0.29614833 -0.018521195 0.11475844 -0.81280553 0.36361912 0.5218304 -0.86172837 0.16615964 -0.5919934 -0.6705995 -0.30303714 0.33643195 -0.36013323 -0.43369684 -0.4232348 0.3289151 0.13196968 -0.047870662 0.16946815 0.6733917 0.14940976 -0.6193638 0.0183977 -0.06626867 0.23720813 -0.3099043 0.24511985 0.002050346 -0.25607812 0.22615135 -0.22930609 0.077634625 -0.4571406 0.2701932 0.1349929 -0.08575605 -0.07779897 -0.28398603 0.17822634 -0.714591 0.79434264 0.3711259 0.13105528 -0.037244517 0.39268345 -0.26900694 -0.012533018 0.16464616 0.5066293 0.25655112 -0.46981072 0.24205445 -0.5497719 0.5421995 -0.0925894 0.012961947 0.021217873 -0.7964931 -0.18980692 0.19714683 0.025466248 0.032327313 -0.07526694 -0.6901916 0.102057904 0.40988278 0.14828728 -0.016430113 1.079269 -0.060629565 0.20774364 -0.068320766 0.519794 0.36136076 -0.5974902 0.090371154 -0.06567787 -0.29659495 -0.06166303 0.07732721 0.6455489 0.5310287 -0.22705573 -0.5061386 0.29006043 -0.10687838 -0.4418074 0.11027829 -0.18012296 -0.57253116 -0.32226965 0.19295888 -0.49936596 0.3793013 -0.080602914 0.3118454 0.002082606 0.20797801 0.5756213 0.13771403 0.16862835 -0.112200536 -0.013501014 0.49715126 0.087013654 -0.51798886 0.7088602 -0.23840822 -0.19949144 0.07258242 -0.2915224 0.07219324 -0.42665526 -0.6635496 -0.24730572 0.08788957 -1.0102566 0.05880175 -0.38076538 0.71097887 -0.31087056 0.582946 -0.7477895 -0.15816173 0.008553023 0.02333851 -0.55589694 -0.68172455 0.40095487 -0.3276839 0.02884732 -0.6576986 -0.099606186 -0.19121127 0.020446194 0.4715886 -0.15726297 0.29254046 -0.62215894 0.47124317 0.1411517 -0.630044 -1.0892184 -0.32871428 0.05280602 -0.36826906 -0.5467138 0.075117186 -0.030592533 0.6691533 -0.31924367 0.07990982 -1.2051688 -0.02718387 0.18652381 -0.16628557 0.4107398 0.37093127 -0.30746254 0.1403109 -0.36297897 -0.35581088 -0.7253777 -0.20135966 -0.73208195 0.42599943 0.075905345 -0.09373599 -0.1377807 -0.31895185 -0.76665807 -0.5864012 -0.20228334 -0.79129803 0.49112904 0.52249646 0.23819853 0.06525556 0.077298716 -0.33033678 -0.0032464918 -0.17499177 0.13935627 -0.16631594\n"
     ]
    }
   ],
   "source": [
    "! head -n 2 word2vec-text-positional/model/*/model.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb84fe45-bd54-4df8-a723-421d33b99354",
   "metadata": {},
   "source": [
    "### The LaTeX format\n",
    "\n",
    "To train a `word2vec` model just for math, we also have a separate dataset with just LaTeX."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "08c5a02c-4356-492d-8b89-d3241fb8d863",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word2vec-latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4e6bfa8b-25fe-4228-bed7-d79ae4dc5580",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "202M\tword2vec-latex-positional/model/custom-en-constrained_positional_word2vec_cbow-epochs=5\n",
      "202M\tword2vec-latex-positional/model\n",
      "0\tword2vec-latex-positional/cache/custom-en-constrained_positional_word2vec_cbow-epochs=5\n",
      "0\tword2vec-latex-positional/cache\n",
      "202M\tword2vec-latex-positional\n",
      "202M\ttotal\n"
     ]
    }
   ],
   "source": [
    "! du -hc word2vec-latex-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "927bbe41-e9eb-47fb-bfe2-953c010d8ca2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29675 300\n",
      "% 0.8901723 0.47753775 -0.25580484 -0.49702826 -1.2510386 0.28111792 0.23857468 -1.0780758 -1.4763756 0.2956801 0.055865422 0.026966417 0.326713 0.6106123 -0.87188375 -1.1999911 0.34720817 -0.4468305 -0.3698747 -0.4408113 0.071058765 -1.0206571 0.4129372 -0.30090615 -0.94386333 -0.8120412 1.0945362 -0.10427095 0.99721193 0.29392707 1.0004305 -0.06176557 0.40482345 -0.30843467 0.42092946 0.3593929 -0.59519 1.3344753 0.85418326 0.67675304 0.31679824 -0.5394206 0.72658104 -0.07458251 -0.36788827 -0.80487925 -0.7956097 0.19825435 -0.043689813 -1.1933388 -0.5309262 0.18160331 -0.60350937 -0.5598332 0.34673214 0.10679278 -1.5410138 0.50901806 0.76687497 0.45944133 -0.52140075 -0.056415234 -1.2947371 0.6880816 0.41167077 -0.36029693 2.0548363 -0.26385704 0.22266854 -0.0026835317 -0.91878295 2.0201352 -0.44925925 -0.46337846 1.0135711 0.17934509 0.86974764 -1.5536085 0.051703487 -0.45952705 0.556976 1.0920143 0.5733894 -0.26399985 1.5638045 -1.2356725 0.28950712 -0.69601995 -0.87338513 0.09456474 -0.04665726 0.34483024 0.8121219 1.266905 1.024573 -0.75927234 0.1246664 0.7170658 0.8604843 0.81956404 -2.7452712 1.7152462 0.36710927 -1.6603361 -0.23355314 -0.07293337 -0.4754351 0.532039 -0.058265813 2.085058 -0.41918254 0.0068587665 0.9206174 0.119078636 -0.8617888 -0.3631117 -0.29497972 0.6344842 -0.69397193 -0.63280165 -0.023537312 -0.527134 -0.03761067 -1.4826596 -0.6378652 0.97365886 0.15129045 -1.2025206 -0.103320904 0.466007 1.550284 -0.7802185 0.74726504 0.45698768 -1.2798743 1.2624152 0.3126311 0.90415835 0.4583063 -0.41962564 0.61581254 -0.42433405 0.47986802 -0.21326183 0.007226813 -0.658114 0.7064646 0.6328973 0.9827275 -0.021880582 0.054142367 -0.017042134 -1.080414 -1.318031 -1.282127 0.835481 0.39098412 -1.4061803 0.88017714 0.045085467 -1.1973183 0.92895997 0.4203812 -0.32178006 -0.3285343 -0.1101595 0.26883 1.3841552 1.4901435 -0.2689317 0.79190004 0.42905626 -0.64837646 1.4658525 1.3803753 0.62023205 0.43113914 1.3267792 -1.0026082 -1.8564943 -1.0581224 0.5177784 0.18460274 2.2470136 -0.10454118 1.540145 -0.434532 1.7083253 -1.6431874 1.7458907 -0.23440042 0.06710033 0.5789458 -0.48487216 -1.9169787 -0.47857267 0.43410537 0.9603848 -0.88840705 -0.7288412 -0.17111997 -2.268441 0.6489332 1.5164206 -1.0386437 -1.2629896 0.560759 -0.026916709 -1.0886866 -1.020368 0.5736862 2.6086836 0.2054618 0.63768584 -0.30849546 0.13820083 0.27342582 0.27460766 -0.1033606 0.22879681 0.18055011 0.65606594 0.20199656 0.6625208 0.26646528 1.3530835 1.8140106 0.948962 0.7135855 1.4167676 0.11120978 -0.908606 0.72981423 -0.23395656 1.044645 -0.9715456 2.5041273 -0.5924533 0.36576605 1.3623512 -0.66235447 0.34357518 -1.337199 -1.0939572 -1.1184542 0.5928003 1.6584271 -0.17449854 0.77935404 -0.66212106 0.034197837 0.12068617 0.27800038 0.7567682 0.9051967 1.6103308 1.4210178 1.5619899 0.70119524 0.75821334 -0.6452251 1.3769978 0.6063633 0.022526417 -0.49578682 0.17537329 0.6493036 0.87859136 0.62150186 0.35534596 0.38754684 -0.0812787 0.43650827 -0.23584785 -0.88143146 0.09766546 -0.5928137 0.11067212 0.03230634 0.024209734 0.087339714 1.1200598 1.8487386 0.7007932 -0.62594754 1.5404825 -0.13932301 -0.97893745 -1.11789 1.2474853 -1.2892255 0.3495276 2.1913345 -1.1140983 -0.5886454 -1.301535 1.5289693 -1.5422949 1.1315149 -1.0406291\n"
     ]
    }
   ],
   "source": [
    "! head -n 2 word2vec-latex-positional/model/*/model.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63f452dc-bfd5-4b20-9a2b-8ec61548133e",
   "metadata": {},
   "source": [
    "### The Tangent-L format\n",
    "\n",
    "To train a word2vec model just for math, we also have a separate dataset with just the format used by [the Tangent-L search engine from UWaterloo][1].\n",
    "\n",
    " [1]: http://ceur-ws.org/Vol-2936/paper-05.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "432b0dda-2f1b-48e8-8fb4-de9798052aa8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "! make word2vec-tangentl-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0b8fa3cc-27d9-4004-a85c-1990674e8dd3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14G\tword2vec-tangentl-positional/model/custom-en-constrained_positional_word2vec_cbow-epochs=2\n",
      "14G\tword2vec-tangentl-positional/model\n",
      "0\tword2vec-tangentl-positional/cache/custom-en-constrained_positional_word2vec_cbow-epochs=2\n",
      "0\tword2vec-tangentl-positional/cache\n",
      "14G\tword2vec-tangentl-positional\n",
      "14G\ttotal\n"
     ]
    }
   ],
   "source": [
    "! du -hc word2vec-tangentl-positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "aae3c89c-8f32-4529-8ac0-03a3778227c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1986439 300\n",
      "(start) 0.3166078 0.32142597 0.34454095 -0.07255285 0.38778037 -0.14426847 -0.01732232 -0.26281923 1.252759 0.049788285 -0.37009966 -0.15158775 0.07610741 2.0655713 -2.5267181 0.22337817 -0.004000889 -0.5431778 -0.076354615 -0.03853108 1.0655208 -0.116475835 -0.10065177 -0.078265786 0.07453254 0.988562 0.2175998 1.3232359 0.87032974 -0.00044681778 0.176348 0.023796262 -0.053987995 -0.6664432 -0.6631214 0.090399295 0.13955821 1.2250364 -0.017805219 -0.989486 -0.27473718 0.14729865 -0.14013909 -0.056522164 -0.1962746 -0.10431214 -0.17020053 -0.0711098 0.17637534 0.23139422 1.329581 -0.07272253 -0.037196796 0.41047052 0.14671275 -1.2506131 -1.5762898 -0.17462981 -0.031680465 -1.0242909 0.47987965 -0.8490513 -0.23418312 0.6616664 0.80433255 0.5773198 -1.518877 -0.49992532 -0.7281841 -1.3296658 1.5090414 -0.8300803 -1.3984554 1.6109108 0.36564013 0.10850824 0.7570445 1.623565 1.4367251 1.0795465 -1.3869472 0.48184153 1.693365 1.3953366 0.32961982 0.09521597 0.69020426 2.0102963 -1.3748267 -2.270638 2.0010011 2.2665334 1.1567765 0.47882062 1.5496837 -1.1182609 0.07538759 0.06752641 1.6165599 -0.9913531 -0.2198654 -0.74294955 0.46905822 0.32877582 0.271151 -1.926091 0.04950454 -0.7577612 0.46084556 -0.40931177 -1.8226207 -1.6728257 0.06303381 0.17894289 0.04280059 -0.22275417 -0.88083285 0.11459722 0.715966 0.4971358 -0.52473456 -0.28044292 -0.0038706476 2.614319 -1.3524951 -2.420288 -0.6320869 0.4237997 -0.34196305 -0.018926607 0.58856755 2.1727557 0.09813921 -0.7259589 0.3141199 0.70916 1.5391192 0.4718461 1.2158141 -1.3383248 1.303291 0.19899811 -1.9656285 0.055231348 0.45815825 -1.4391018 -0.044102747 -1.3703172 0.97722274 -2.0147133 -0.8324939 -0.04988036 1.4773793 0.6035807 0.5328804 -2.2466986 0.60381913 1.8163457 1.7385136 1.7176719 1.189831 -1.5681031 0.45548037 -0.6455459 2.4971745 -0.9118949 0.3052036 2.2278793 -0.25144398 0.34503925 -0.4985481 1.236979 -1.6640456 1.1403431 0.04411476 0.9270714 2.2901704 -0.028806753 -0.56632537 -2.6387925 0.22498766 -0.055968143 0.9523764 -0.38954237 -1.3945193 -0.68024963 0.31487107 -0.11041278 0.3561033 -0.9880139 -0.2181298 -0.8979228 1.0366924 -2.2207341 -1.7011045 -2.8310072 1.8293055 -0.21377742 1.777952 -0.65660053 -1.9364955 -1.3104427 0.70092463 -0.007618532 2.1026182 -2.3430412 2.013348 -2.4409797 -0.6864526 -1.4332837 -0.28340966 -0.3555211 -3.005226 -1.2361859 -1.5151939 -0.45761302 -0.7316255 0.6008862 -0.6903792 -0.033405676 -1.4431115 0.3240994 -1.5181187 0.371064 -0.6723623 -0.64953166 1.5562918 -0.4010787 0.65162754 -1.322041 -0.95778203 0.9369039 2.1460972 -1.5791786 2.0900352 -1.0400112 -0.62326294 1.8197535 0.39638862 0.04837084 -1.3412919 0.77661586 0.9857937 -0.029907035 0.015005964 -0.73751247 -0.0567993 -0.85503155 0.09153577 1.9515264 -1.436403 0.23914066 -1.5409727 2.042277 0.0076455236 -0.3321087 1.0508469 -0.010856943 0.54931575 3.092635 0.44413733 0.39460063 -0.765865 -2.384519 -0.5533257 1.8071245 -1.0535407 0.011211348 -0.112978466 0.1975267 -1.944817 -0.6339633 1.0674963 -0.9810466 0.28114516 -0.33029366 1.2749081 1.7087222 1.3493327 -1.9841698 0.55812293 1.1181922 -1.4427557 0.86258554 0.8616149 1.050233 0.44504502 0.83059454 0.1947895 -0.95461255 -0.18290612 1.1364385 2.6725602 0.049029995 2.9522605 1.4928633 0.53799987 -0.97061336 0.2792981 0.90342844\n"
     ]
    }
   ],
   "source": [
    "! head -n 2 word2vec-tangentl-positional/model/*/model.vec"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}