Bash Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shell scripting tutorial\n",
    "\n",
    "Learning shell scripting can provide you quick and easy ways to perform a lot of work related with a machine learning / text processing project.\n",
    "\n",
    "Some of the things you can achieve with shell scripting:\n",
    "\n",
    "- Installing project dependencies\n",
    "- Build project dependencies from source\n",
    "- Download organize and clean data\n",
    "- Extract data statistics (e.g. word / character counts)\n",
    "- Perform text processing\n",
    "- Train and evaluate models using existing frameworks that provide a command line interface (Kaldi, openfst, fasttext, fairseq etc.)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all list current working directory and it's contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/geopar/projects/bash_tutorial\n"
     ]
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 80K\n",
      "drwxr-xr-x  6 geopar geopar 4.0K Oct 18 06:48  \u001b[0m\u001b[01;34m.\u001b[0m\n",
      "drwxrwxrwx 13 geopar geopar 4.0K Oct 17 19:45  \u001b[34;42m..\u001b[0m\n",
      "-rw-r--r--  1 geopar geopar  55K Oct 18 06:48 'Bash Tutorial.ipynb'\n",
      "drwxr-xr-x  2 geopar geopar 4.0K Oct 17 20:52  \u001b[01;34mbin\u001b[0m\n",
      "drwxr-xr-x  2 geopar geopar 4.0K Oct 18 06:42  \u001b[01;34mdata\u001b[0m\n",
      "drwxr-xr-x  2 geopar geopar 4.0K Oct 17 19:46  \u001b[01;34m.ipynb_checkpoints\u001b[0m\n",
      "drwxr-xr-x  9 geopar geopar 4.0K Oct 17 20:41  \u001b[01;34mkenlm\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -lah"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing project dependencies and building a project from source\n",
    "\n",
    "Let's say we need to build an N-Gram language model for some corpus. One commonly used tool for this is KenLM. Let's download and build it from source"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download KenLM from git repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fatal: destination path 'kenlm' already exists and is not an empty directory.\n"
     ]
    },
    {
     "ename": "",
     "evalue": "128",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "git clone https://github.com/kpu/kenlm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install necessary dependencies for building KenLM (follow docs: https://kheafield.com/code/kenlm/dependencies/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sudo apt-get install build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List files in current directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 80K\n",
      "drwxr-xr-x  6 geopar geopar 4.0K Oct 18 06:48  \u001b[0m\u001b[01;34m.\u001b[0m\n",
      "drwxrwxrwx 13 geopar geopar 4.0K Oct 17 19:45  \u001b[34;42m..\u001b[0m\n",
      "-rw-r--r--  1 geopar geopar  55K Oct 18 06:48 'Bash Tutorial.ipynb'\n",
      "drwxr-xr-x  2 geopar geopar 4.0K Oct 17 20:52  \u001b[01;34mbin\u001b[0m\n",
      "drwxr-xr-x  2 geopar geopar 4.0K Oct 18 06:42  \u001b[01;34mdata\u001b[0m\n",
      "drwxr-xr-x  2 geopar geopar 4.0K Oct 17 19:46  \u001b[01;34m.ipynb_checkpoints\u001b[0m\n",
      "drwxr-xr-x  9 geopar geopar 4.0K Oct 17 20:41  \u001b[01;34mkenlm\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -lah"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Navigate inside mosesdecoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd kenlm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display current working directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/geopar/projects/bash_tutorial/kenlm\n"
     ]
    }
   ],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 212K\n",
      "drwxr-xr-x 9 geopar geopar 4.0K Oct 17 20:41 \u001b[0m\u001b[01;34m.\u001b[0m\n",
      "drwxr-xr-x 6 geopar geopar 4.0K Oct 18 06:48 \u001b[01;34m..\u001b[0m\n",
      "drwxr-xr-x 7 geopar geopar 4.0K Oct 17 20:41 \u001b[01;34mbuild\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar  696 Oct 17 20:40 BUILDING\n",
      "-rwxr-xr-x 1 geopar geopar   81 Oct 17 20:40 \u001b[01;32mclean_query_only.sh\u001b[0m\n",
      "drwxr-xr-x 3 geopar geopar 4.0K Oct 17 20:40 \u001b[01;34mcmake\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar 3.7K Oct 17 20:40 CMakeLists.txt\n",
      "-rwxr-xr-x 1 geopar geopar 1.2K Oct 17 20:40 \u001b[01;32mcompile_query_only.sh\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar  26K Oct 17 20:40 COPYING\n",
      "-rw-r--r-- 1 geopar geopar  35K Oct 17 20:40 COPYING.3\n",
      "-rw-r--r-- 1 geopar geopar 7.5K Oct 17 20:40 COPYING.LESSER.3\n",
      "-rw-r--r-- 1 geopar geopar  63K Oct 17 20:40 Doxyfile\n",
      "drwxr-xr-x 6 geopar geopar 4.0K Oct 17 20:40 \u001b[01;34m.git\u001b[0m\n",
      "drwxr-xr-x 3 geopar geopar 4.0K Oct 17 20:40 \u001b[01;34m.github\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar  249 Oct 17 20:40 .gitignore\n",
      "-rw-r--r-- 1 geopar geopar 1.2K Oct 17 20:40 LICENSE\n",
      "drwxr-xr-x 7 geopar geopar 4.0K Oct 17 20:40 \u001b[01;34mlm\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar  220 Oct 17 20:40 MANIFEST.in\n",
      "drwxr-xr-x 2 geopar geopar 4.0K Oct 17 20:40 \u001b[01;34mpython\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar 5.3K Oct 17 20:40 README.md\n",
      "-rw-r--r-- 1 geopar geopar 1.9K Oct 17 20:40 setup.py\n",
      "drwxr-xr-x 4 geopar geopar 4.0K Oct 17 20:40 \u001b[01;34mutil\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -lah"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use cmake to compile project (follow instructions: https://github.com/kpu/kenlm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- The C compiler identification is GNU 9.3.0\n",
      "-- The CXX compiler identification is GNU 9.3.0\n",
      "-- Check for working C compiler: /usr/bin/cc\n",
      "-- Check for working C compiler: /usr/bin/cc -- works\n",
      "-- Detecting C compiler ABI info\n",
      "-- Detecting C compiler ABI info - done\n",
      "-- Detecting C compile features\n",
      "-- Detecting C compile features - done\n",
      "-- Check for working CXX compiler: /usr/bin/c++\n",
      "-- Check for working CXX compiler: /usr/bin/c++ -- works\n",
      "-- Detecting CXX compiler ABI info\n",
      "-- Detecting CXX compiler ABI info - done\n",
      "-- Detecting CXX compile features\n",
      "-- Detecting CXX compile features - done\n",
      "-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version \"1.71.0\", minimum required is \"1.41.0\") found components: program_options system thread unit_test_framework \n",
      "-- Check if compiler accepts -pthread\n",
      "-- Check if compiler accepts -pthread - yes\n",
      "-- Found Threads: TRUE  \n",
      "-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version \"1.2.11\") \n",
      "-- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version \"1.0.8\") \n",
      "-- Looking for BZ2_bzCompressInit\n",
      "-- Looking for BZ2_bzCompressInit - found\n",
      "-- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so\n",
      "-- Looking for lzma_auto_decoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n",
      "-- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so\n",
      "-- Looking for lzma_easy_encoder in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n",
      "-- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so\n",
      "-- Looking for lzma_lzma_preset in /usr/lib/x86_64-linux-gnu/liblzma.so - found\n",
      "-- Found LibLZMA: /usr/lib/x86_64-linux-gnu/liblzma.so (found version \"5.2.4\") \n",
      "-- Found OpenMP_C: -fopenmp (found version \"4.5\") \n",
      "-- Found OpenMP_CXX: -fopenmp (found version \"4.5\") \n",
      "-- Found OpenMP: TRUE (found version \"4.5\")  \n",
      "-- Configuring done\n",
      "-- Generating done\n",
      "-- Build files have been written to: /home/geopar/projects/bash_tutorial/kenlm/build\n",
      "\u001b[35m\u001b[1mScanning dependencies of target kenlm_util\u001b[0m\n",
      "[  1%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum.cc.o\u001b[0m\n",
      "[  2%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/diy-fp.cc.o\u001b[0m\n",
      "[  3%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/bignum-dtoa.cc.o\u001b[0m\n",
      "[  4%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/cached-powers.cc.o\u001b[0m\n",
      "[  5%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/double-conversion.cc.o\u001b[0m\n",
      "[  6%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fast-dtoa.cc.o\u001b[0m\n",
      "[  7%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/fixed-dtoa.cc.o\u001b[0m\n",
      "[  8%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/double-conversion/strtod.cc.o\u001b[0m\n",
      "[  9%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/chain.cc.o\u001b[0m\n",
      "[ 10%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/count_records.cc.o\u001b[0m\n",
      "[ 11%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/io.cc.o\u001b[0m\n",
      "[ 12%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/line_input.cc.o\u001b[0m\n",
      "[ 13%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/multi_progress.cc.o\u001b[0m\n",
      "[ 14%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/stream/rewindable_stream.cc.o\u001b[0m\n",
      "[ 15%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/bit_packing.cc.o\u001b[0m\n",
      "[ 16%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/ersatz_progress.cc.o\u001b[0m\n",
      "[ 17%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/exception.cc.o\u001b[0m\n",
      "[ 18%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/file.cc.o\u001b[0m\n",
      "[ 19%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/file_piece.cc.o\u001b[0m\n",
      "[ 20%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/float_to_string.cc.o\u001b[0m\n",
      "[ 21%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/integer_to_string.cc.o\u001b[0m\n",
      "[ 22%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/mmap.cc.o\u001b[0m\n",
      "[ 23%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/murmur_hash.cc.o\u001b[0m\n",
      "[ 25%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/parallel_read.cc.o\u001b[0m\n",
      "[ 26%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/pool.cc.o\u001b[0m\n",
      "[ 27%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/read_compressed.cc.o\u001b[0m\n",
      "[ 28%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/scoped.cc.o\u001b[0m\n",
      "[ 29%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/spaces.cc.o\u001b[0m\n",
      "[ 30%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/string_piece.cc.o\u001b[0m\n",
      "[ 31%] \u001b[32mBuilding CXX object util/CMakeFiles/kenlm_util.dir/usage.cc.o\u001b[0m\n",
      "[ 32%] \u001b[32m\u001b[1mLinking CXX static library ../lib/libkenlm_util.a\u001b[0m\n",
      "[ 32%] Built target kenlm_util\n",
      "\u001b[35m\u001b[1mScanning dependencies of target kenlm\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target probing_hash_table_benchmark\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target kenlm_filter\u001b[0m\n",
      "[ 33%] \u001b[32mBuilding CXX object util/CMakeFiles/probing_hash_table_benchmark.dir/probing_hash_table_benchmark_main.cc.o\u001b[0m\n",
      "[ 34%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/arpa_io.cc.o\u001b[0m\n",
      "[ 35%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/phrase.cc.o\u001b[0m\n",
      "[ 36%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/bhiksha.cc.o\u001b[0m\n",
      "[ 37%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/binary_format.cc.o\u001b[0m\n",
      "[ 38%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/kenlm_filter.dir/vocab.cc.o\u001b[0m\n",
      "[ 39%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/config.cc.o\u001b[0m\n",
      "[ 40%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/lm_exception.cc.o\u001b[0m\n",
      "[ 41%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/model.cc.o\u001b[0m\n",
      "[ 42%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/quantize.cc.o\u001b[0m\n",
      "[ 43%] \u001b[32m\u001b[1mLinking CXX static library ../../lib/libkenlm_filter.a\u001b[0m\n",
      "[ 43%] Built target kenlm_filter\n",
      "[ 44%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/read_arpa.cc.o\u001b[0m\n",
      "[ 45%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/search_hashed.cc.o\u001b[0m\n",
      "[ 46%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/search_trie.cc.o\u001b[0m\n",
      "[ 47%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/sizes.cc.o\u001b[0m\n",
      "[ 48%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/trie.cc.o\u001b[0m\n",
      "[ 50%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/trie_sort.cc.o\u001b[0m\n",
      "[ 51%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/value_build.cc.o\u001b[0m\n",
      "[ 52%] \u001b[32m\u001b[1mLinking CXX executable ../bin/probing_hash_table_benchmark\u001b[0m\n",
      "[ 52%] Built target probing_hash_table_benchmark\n",
      "[ 53%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/virtual_interface.cc.o\u001b[0m\n",
      "[ 54%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/vocab.cc.o\u001b[0m\n",
      "[ 55%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/model_buffer.cc.o\u001b[0m\n",
      "[ 56%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/print.cc.o\u001b[0m\n",
      "[ 57%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/renumber.cc.o\u001b[0m\n",
      "[ 58%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm.dir/common/size_option.cc.o\u001b[0m\n",
      "[ 59%] \u001b[32m\u001b[1mLinking CXX static library ../lib/libkenlm.a\u001b[0m\n",
      "[ 59%] Built target kenlm\n",
      "\u001b[35m\u001b[1mScanning dependencies of target fragment\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target kenlm_benchmark\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target query\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target build_binary\u001b[0m\n",
      "[ 60%] \u001b[32mBuilding CXX object lm/CMakeFiles/query.dir/query_main.cc.o\u001b[0m\n",
      "[ 61%] \u001b[32mBuilding CXX object lm/CMakeFiles/fragment.dir/fragment_main.cc.o\u001b[0m\n",
      "[ 62%] \u001b[32mBuilding CXX object lm/CMakeFiles/build_binary.dir/build_binary_main.cc.o\u001b[0m\n",
      "[ 63%] \u001b[32mBuilding CXX object lm/CMakeFiles/kenlm_benchmark.dir/kenlm_benchmark_main.cc.o\u001b[0m\n",
      "[ 64%] \u001b[32m\u001b[1mLinking CXX executable ../bin/fragment\u001b[0m\n",
      "[ 64%] Built target fragment\n",
      "\u001b[35m\u001b[1mScanning dependencies of target kenlm_builder\u001b[0m\n",
      "[ 65%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/adjust_counts.cc.o\u001b[0m\n",
      "[ 66%] \u001b[32m\u001b[1mLinking CXX executable ../bin/build_binary\u001b[0m\n",
      "[ 66%] Built target build_binary\n",
      "\u001b[35m\u001b[1mScanning dependencies of target phrase_table_vocab\u001b[0m\n",
      "[ 67%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/phrase_table_vocab.dir/phrase_table_vocab_main.cc.o\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 68%] \u001b[32m\u001b[1mLinking CXX executable ../bin/query\u001b[0m\n",
      "[ 68%] Built target query\n",
      "\u001b[35m\u001b[1mScanning dependencies of target filter\u001b[0m\n",
      "[ 69%] \u001b[32mBuilding CXX object lm/filter/CMakeFiles/filter.dir/filter_main.cc.o\u001b[0m\n",
      "[ 70%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/phrase_table_vocab\u001b[0m\n",
      "[ 70%] Built target phrase_table_vocab\n",
      "\u001b[35m\u001b[1mScanning dependencies of target kenlm_interpolate\u001b[0m\n",
      "[ 71%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/backoff_reunification.cc.o\u001b[0m\n",
      "[ 72%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/corpus_count.cc.o\u001b[0m\n",
      "[ 73%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/bounded_sequence_encoding.cc.o\u001b[0m\n",
      "[ 75%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/merge_probabilities.cc.o\u001b[0m\n",
      "[ 76%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/initial_probabilities.cc.o\u001b[0m\n",
      "[ 77%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/merge_vocab.cc.o\u001b[0m\n",
      "[ 78%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/normalize.cc.o\u001b[0m\n",
      "[ 79%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/interpolate.cc.o\u001b[0m\n",
      "[ 80%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/output.cc.o\u001b[0m\n",
      "[ 81%] \u001b[32m\u001b[1mLinking CXX executable ../bin/kenlm_benchmark\u001b[0m\n",
      "[ 81%] Built target kenlm_benchmark\n",
      "[ 82%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/kenlm_builder.dir/pipeline.cc.o\u001b[0m\n",
      "[ 83%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/pipeline.cc.o\u001b[0m\n",
      "[ 84%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/filter\u001b[0m\n",
      "[ 84%] Built target filter\n",
      "[ 85%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/split_worker.cc.o\u001b[0m\n",
      "[ 86%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/tune_derivatives.cc.o\u001b[0m\n",
      "[ 87%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/tune_instances.cc.o\u001b[0m\n",
      "[ 88%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/tune_weights.cc.o\u001b[0m\n",
      "[ 89%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/kenlm_interpolate.dir/universal_vocab.cc.o\u001b[0m\n",
      "[ 90%] \u001b[32m\u001b[1mLinking CXX static library ../../lib/libkenlm_builder.a\u001b[0m\n",
      "[ 90%] Built target kenlm_builder\n",
      "\u001b[35m\u001b[1mScanning dependencies of target lmplz\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target count_ngrams\u001b[0m\n",
      "[ 91%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/lmplz.dir/lmplz_main.cc.o\u001b[0m\n",
      "[ 92%] \u001b[32mBuilding CXX object lm/builder/CMakeFiles/count_ngrams.dir/count_ngrams_main.cc.o\u001b[0m\n",
      "[ 93%] \u001b[32m\u001b[1mLinking CXX static library ../../lib/libkenlm_interpolate.a\u001b[0m\n",
      "[ 93%] Built target kenlm_interpolate\n",
      "\u001b[35m\u001b[1mScanning dependencies of target interpolate\u001b[0m\n",
      "\u001b[35m\u001b[1mScanning dependencies of target streaming_example\u001b[0m\n",
      "[ 94%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/streaming_example.dir/streaming_example_main.cc.o\u001b[0m\n",
      "[ 95%] \u001b[32mBuilding CXX object lm/interpolate/CMakeFiles/interpolate.dir/interpolate_main.cc.o\u001b[0m\n",
      "[ 96%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/lmplz\u001b[0m\n",
      "[ 96%] Built target lmplz\n",
      "[ 97%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/count_ngrams\u001b[0m\n",
      "[ 97%] Built target count_ngrams\n",
      "[ 98%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/interpolate\u001b[0m\n",
      "[ 98%] Built target interpolate\n",
      "[100%] \u001b[32m\u001b[1mLinking CXX executable ../../bin/streaming_example\u001b[0m\n",
      "[100%] Built target streaming_example\n"
     ]
    }
   ],
   "source": [
    "mkdir build\n",
    "cd build\n",
    "cmake ..\n",
    "make -j4\n",
    "# make install to install kenlm system-wide"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous commands built the KenLM binaries inside the bin folder. Let's copy it in a more accessible directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 192\n",
      "drwxr-xr-x 7 geopar geopar  4096 Oct 17 20:41 \u001b[0m\u001b[01;34mbuild\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar   696 Oct 17 20:40 BUILDING\n",
      "-rwxr-xr-x 1 geopar geopar    81 Oct 17 20:40 \u001b[01;32mclean_query_only.sh\u001b[0m\n",
      "drwxr-xr-x 3 geopar geopar  4096 Oct 17 20:40 \u001b[01;34mcmake\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar  3689 Oct 17 20:40 CMakeLists.txt\n",
      "-rwxr-xr-x 1 geopar geopar  1154 Oct 17 20:40 \u001b[01;32mcompile_query_only.sh\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar 26530 Oct 17 20:40 COPYING\n",
      "-rw-r--r-- 1 geopar geopar 35147 Oct 17 20:40 COPYING.3\n",
      "-rw-r--r-- 1 geopar geopar  7637 Oct 17 20:40 COPYING.LESSER.3\n",
      "-rw-r--r-- 1 geopar geopar 63537 Oct 17 20:40 Doxyfile\n",
      "-rw-r--r-- 1 geopar geopar  1150 Oct 17 20:40 LICENSE\n",
      "drwxr-xr-x 7 geopar geopar  4096 Oct 17 20:40 \u001b[01;34mlm\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar   220 Oct 17 20:40 MANIFEST.in\n",
      "drwxr-xr-x 2 geopar geopar  4096 Oct 17 20:40 \u001b[01;34mpython\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar  5394 Oct 17 20:40 README.md\n",
      "-rw-r--r-- 1 geopar geopar  1937 Oct 17 20:40 setup.py\n",
      "drwxr-xr-x 4 geopar geopar  4096 Oct 17 20:40 \u001b[01;34mutil\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "ls -l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cp: cannot stat 'bin': No such file or directory\n",
      "total 56K\n",
      "drwxrwxrwx 13 geopar geopar 4.0K Oct 17 19:45 \u001b[0m\u001b[34;42m.\u001b[0m\n",
      "drwxr-xr-x 24 geopar geopar 4.0K Oct 17 20:53 \u001b[01;34m..\u001b[0m\n",
      "drwxr-xr-x  6 geopar geopar 4.0K Oct 18 06:48 \u001b[01;34mbash_tutorial\u001b[0m\n",
      "drwxr-xr-x  4 geopar geopar 4.0K Sep 22 16:06 \u001b[01;34mexam-generator\u001b[0m\n",
      "drwxr-xr-x  3 geopar geopar 4.0K Oct  7 20:20 \u001b[01;34mfsspell\u001b[0m\n",
      "-rw-r--r--  1 geopar geopar  174 Oct  8 18:25 mispel.py\n",
      "drwxr-xr-x  3 geopar geopar 4.0K Oct  8 18:20 \u001b[01;34m.mypy_cache\u001b[0m\n",
      "drwxr-xr-x  4 geopar geopar 4.0K Sep  3 13:23 \u001b[01;34mpython-lab\u001b[0m\n",
      "drwxrwxrwx 17 geopar geopar 4.0K Sep  7 06:21 \u001b[34;42mslp\u001b[0m\n",
      "drwxr-xr-x 14 geopar geopar 4.0K Sep 28 18:57 \u001b[01;34mslp_daptmlm\u001b[0m\n",
      "drwxr-xr-x 10 geopar geopar 4.0K Sep  6 14:21 \u001b[01;34mspace-vim\u001b[0m\n",
      "drwxr-xr-x  2 geopar geopar 4.0K Oct  7 18:19 \u001b[01;34mspellweb\u001b[0m\n",
      "drwxr-xr-x 14 geopar geopar 4.0K Sep  7 08:09 \u001b[01;34msplchk\u001b[0m\n",
      "drwxr-xr-x 15 geopar geopar 4.0K Sep 15 19:20 \u001b[01;34mtransformers\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "cp -r bin ../../  # 2 directories up\n",
    "cd ../../\n",
    "ls -lah"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and preprocessing training corpus\n",
    "\n",
    "Let's get a book from project gutenberg and clean it up using bash"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-10-17 20:54:39--  http://www.gutenberg.org/cache/epub/345/pg345.txt\n",
      "Resolving www.gutenberg.org (www.gutenberg.org)... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47\n",
      "Connecting to www.gutenberg.org (www.gutenberg.org)|152.19.134.47|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 883160 (862K) [text/plain]\n",
      "Saving to: ‘data/dracula.txt’\n",
      "\n",
      "data/dracula.txt    100%[===================>] 862.46K   272KB/s    in 3.2s    \n",
      "\n",
      "2020-10-17 20:54:43 (272 KB/s) - ‘data/dracula.txt’ saved [883160/883160]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mkdir data\n",
    "wget -O data/dracula.txt http://www.gutenberg.org/cache/epub/345/pg345.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 1.7M\n",
      "drwxr-xr-x 2 geopar geopar 4.0K Oct 18 06:42 \u001b[0m\u001b[01;34m.\u001b[0m\n",
      "drwxr-xr-x 6 geopar geopar 4.0K Oct 18 06:42 \u001b[01;34m..\u001b[0m\n",
      "-rw-r--r-- 1 geopar geopar 859K Oct 18 06:42 dracula1.txt\n",
      "-rw-r--r-- 1 geopar geopar 863K Oct  1 11:00 dracula.txt\n"
     ]
    }
   ],
   "source": [
    "ls -lah data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Count the number of lines, words and characters using wc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15973 data/dracula.txt\n"
     ]
    }
   ],
   "source": [
    "wc -l data/dracula.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can format column printing using awk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/dracula.txt contains 15973 lines\n"
     ]
    }
   ],
   "source": [
    "wc -l data/dracula.txt | awk '{printf \"%s contains %s lines\\n\", $2, $1}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/dracula.txt contains 164424 words\n"
     ]
    }
   ],
   "source": [
    "wc -w data/dracula.txt | awk '{printf \"%s contains %s words\\n\", $2, $1}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/dracula.txt contains 883160 characters\n"
     ]
    }
   ],
   "source": [
    "wc -c data/dracula.txt | awk '{printf \"%s contains %s characters\\n\", $2, $1}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the first 250 lines using head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "﻿The Project Gutenberg EBook of Dracula, by Bram Stoker\n",
      "\n",
      "This eBook is for the use of anyone anywhere at no cost and with\n",
      "almost no restrictions whatsoever.  You may copy it, give it away or\n",
      "re-use it under the terms of the Project Gutenberg License included\n",
      "with this eBook or online at www.gutenberg.org/license\n",
      "\n",
      "\n",
      "Title: Dracula\n",
      "\n",
      "Author: Bram Stoker\n",
      "\n",
      "Release Date: August 16, 2013 [EBook #345]\n",
      "\n",
      "Language: English\n",
      "\n",
      "\n",
      "*** START OF THIS PROJECT GUTENBERG EBOOK DRACULA ***\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Produced by Chuck Greif and the Online Distributed\n",
      "Proofreading Team at http://www.pgdp.net (This file was\n",
      "produced from images generously made available by The\n",
      "Internet Archive)\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "                                DRACULA\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "                                DRACULA\n",
      "\n",
      "                                  _by_\n",
      "\n",
      "                              Bram Stoker\n",
      "\n",
      "                        [Illustration: colophon]\n",
      "\n",
      "                                NEW YORK\n",
      "\n",
      "                            GROSSET & DUNLAP\n",
      "\n",
      "                              _Publishers_\n",
      "\n",
      "      Copyright, 1897, in the United States of America, according\n",
      "                   to Act of Congress, by Bram Stoker\n",
      "\n",
      "                        [_All rights reserved._]\n",
      "\n",
      "                      PRINTED IN THE UNITED STATES\n",
      "                                   AT\n",
      "               THE COUNTRY LIFE PRESS, GARDEN CITY, N.Y.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "                                   TO\n",
      "\n",
      "                             MY DEAR FRIEND\n",
      "\n",
      "                               HOMMY-BEG\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "CONTENTS\n",
      "\n",
      "\n",
      "CHAPTER I\n",
      "                                                                    Page\n",
      "\n",
      "Jonathan Harker's Journal                                              1\n",
      "\n",
      "CHAPTER II\n",
      "\n",
      "Jonathan Harker's Journal                                             14\n",
      "\n",
      "CHAPTER III\n",
      "\n",
      "Jonathan Harker's Journal                                             26\n",
      "\n",
      "CHAPTER IV\n",
      "\n",
      "Jonathan Harker's Journal                                             38\n",
      "\n",
      "CHAPTER V\n",
      "\n",
      "Letters--Lucy and Mina                                                51\n",
      "\n",
      "CHAPTER VI\n",
      "\n",
      "Mina Murray's Journal                                                 59\n",
      "\n",
      "CHAPTER VII\n",
      "\n",
      "Cutting from \"The Dailygraph,\" 8 August                               71\n",
      "\n",
      "CHAPTER VIII\n",
      "\n",
      "Mina Murray's Journal                                                 84\n",
      "\n",
      "CHAPTER IX\n",
      "\n",
      "Mina Murray's Journal                                                 98\n",
      "\n",
      "CHAPTER X\n",
      "\n",
      "Mina Murray's Journal                                                111\n",
      "\n",
      "CHAPTER XI\n",
      "\n",
      "Lucy Westenra's Diary                                                124\n",
      "\n",
      "CHAPTER XII\n",
      "\n",
      "Dr. Seward's Diary                                                   136\n",
      "\n",
      "CHAPTER XIII\n",
      "\n",
      "Dr. Seward's Diary                                                   152\n",
      "\n",
      "CHAPTER XIV\n",
      "\n",
      "Mina Harker's Journal                                                167\n",
      "\n",
      "CHAPTER XV\n",
      "\n",
      "Dr. Seward's Diary                                                   181\n",
      "\n",
      "CHAPTER XVI\n",
      "\n",
      "Dr. Seward's Diary                                                   194\n",
      "\n",
      "CHAPTER XVII\n",
      "\n",
      "Dr. Seward's Diary                                                   204\n",
      "\n",
      "CHAPTER XVIII\n",
      "\n",
      "Dr. Seward's Diary                                                   216\n",
      "\n",
      "CHAPTER XIX\n",
      "\n",
      "Jonathan Harker's Journal                                            231\n",
      "\n",
      "CHAPTER XX\n",
      "\n",
      "Jonathan Harker's Journal                                            243\n",
      "\n",
      "CHAPTER XXI\n",
      "\n",
      "Dr. Seward's Diary                                                   256\n",
      "\n",
      "CHAPTER XXII\n",
      "\n",
      "Jonathan Harker's Journal                                            269\n",
      "\n",
      "CHAPTER XXIII\n",
      "\n",
      "Dr. Seward's Diary                                                   281\n",
      "\n",
      "CHAPTER XXIV\n",
      "\n",
      "Dr. Seward's Phonograph Diary, spoken by Van Helsing                 294\n",
      "\n",
      "CHAPTER XXV\n",
      "\n",
      "Dr. Seward's Diary                                                   308\n",
      "\n",
      "CHAPTER XXVI\n",
      "\n",
      "Dr. Seward's Diary                                                   322\n",
      "\n",
      "CHAPTER XXVII\n",
      "\n",
      "Mina Harker's Journal                                                338\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "DRACULA\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "CHAPTER I\n",
      "\n",
      "JONATHAN HARKER'S JOURNAL\n",
      "\n",
      "(_Kept in shorthand._)\n",
      "\n",
      "\n",
      "_3 May. Bistritz._--Left Munich at 8:35 P. M., on 1st May, arriving at\n",
      "Vienna early next morning; should have arrived at 6:46, but train was an\n",
      "hour late. Buda-Pesth seems a wonderful place, from the glimpse which I\n",
      "got of it from the train and the little I could walk through the\n",
      "streets. I feared to go very far from the station, as we had arrived\n",
      "late and would start as near the correct time as possible. The\n",
      "impression I had was that we were leaving the West and entering the\n",
      "East; the most western of splendid bridges over the Danube, which is\n",
      "here of noble width and depth, took us among the traditions of Turkish\n",
      "rule.\n",
      "\n",
      "We left in pretty good time, and came after nightfall to Klausenburgh.\n",
      "Here I stopped for the night at the Hotel Royale. I had for dinner, or\n",
      "rather supper, a chicken done up some way with red pepper, which was\n",
      "very good but thirsty. (_Mem._, get recipe for Mina.) I asked the\n",
      "waiter, and he said it was called \"paprika hendl,\" and that, as it was a\n",
      "national dish, I should be able to get it anywhere along the\n",
      "Carpathians. I found my smattering of German very useful here; indeed, I\n",
      "don't know how I should be able to get on without it.\n",
      "\n",
      "Having had some time at my disposal when in London, I had visited the\n",
      "British Museum, and made search among the books and maps in the library\n",
      "regarding Transylvania; it had struck me that some foreknowledge of the\n",
      "country could hardly fail to have some importance in dealing with a\n",
      "nobleman of that country. I find that the district he named is in the\n",
      "extreme east of the country, just on the borders of three states,\n",
      "Transylvania, Moldavia and Bukovina, in the midst of the Carpathian\n",
      "mountains; one of the wildest and least known portions of Europe. I was\n",
      "not able to light on any map or work giving the exact locality of the\n",
      "Castle Dracula, as there are no maps of this country as yet to compare\n",
      "with our own Ordnance Survey maps; but I found that Bistritz, the post\n",
      "town named by Count Dracula, is a fairly well-known place. I shall enter\n",
      "here some of my notes, as they may refresh my memory when I talk over my\n",
      "travels with Mina.\n",
      "\n",
      "In the population of Transylvania there are four distinct nationalities:\n",
      "Saxons in the South, and mixed with them the Wallachs, who are the\n",
      "descendants of the Dacians; Magyars in the West, and Szekelys in the\n",
      "East and North. I am going among the latter, who claim to be descended\n",
      "from Attila and the Huns. This may be so, for when the Magyars conquered\n",
      "the country in the eleventh century they found the Huns settled in it. I\n",
      "read that every known superstition in the world is gathered into the\n",
      "horseshoe of the Carpathians, as if it were the centre of some sort of\n",
      "imaginative whirlpool; if so my stay may be very interesting. (_Mem._, I\n",
      "must ask the Count all about them.)\n",
      "\n",
      "I did not sleep well, though my bed was comfortable enough, for I had\n",
      "all sorts of queer dreams. There was a dog howling all night under my\n",
      "window, which may have had something to do with it; or it may have been\n"
     ]
    }
   ],
   "source": [
    "head -250 data/dracula.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the first 200 lines contain project gutenberg specific text and the tableof contents. We can remove these using sed. Then we inspect the new file using head and wc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "_3 May. Bistritz._--Left Munich at 8:35 P. M., on 1st May, arriving at\n",
      "Vienna early next morning; should have arrived at 6:46, but train was an\n",
      "hour late. Buda-Pesth seems a wonderful place, from the glimpse which I\n",
      "got of it from the train and the little I could walk through the\n",
      "streets. I feared to go very far from the station, as we had arrived\n",
      "late and would start as near the correct time as possible. The\n",
      "impression I had was that we were leaving the West and entering the\n",
      "East; the most western of splendid bridges over the Danube, which is\n",
      "here of noble width and depth, took us among the traditions of Turkish\n"
     ]
    }
   ],
   "source": [
    "sed -e \"1,200d\" data/dracula.txt > data/dracula1.txt\n",
    "head data/dracula1.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/dracula1.txt contains 15773 lines, 164092 words and 878987 characters\n"
     ]
    }
   ],
   "source": [
    "wc data/dracula1.txt | \\\n",
    "    awk '{\n",
    "    printf \"%s contains %s lines, %s words and %s characters\\n\", $4, $1, $2, $3\n",
    "    }'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also remove all empty lines using sed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/dracula2.txt contains 13323 lines, 164092 words and 874087 characters\n"
     ]
    }
   ],
   "source": [
    "sed -r '/^\\s*$/d' data/dracula1.txt > data/dracula2.txt\n",
    "\n",
    "wc data/dracula2.txt | \\\n",
    "    awk '{\n",
    "    printf \"%s contains %s lines, %s words and %s characters\\n\", $4, $1, $2, $3\n",
    "    }'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert all characters to lowercase using tr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_3 may. bistritz._--left munich at 8:35 p. m., on 1st may, arriving at\n",
      "vienna early next morning; should have arrived at 6:46, but train was an\n",
      "hour late. buda-pesth seems a wonderful place, from the glimpse which i\n",
      "got of it from the train and the little i could walk through the\n",
      "streets. i feared to go very far from the station, as we had arrived\n",
      "late and would start as near the correct time as possible. the\n",
      "impression i had was that we were leaving the west and entering the\n",
      "east; the most western of splendid bridges over the danube, which is\n",
      "here of noble width and depth, took us among the traditions of turkish\n",
      "rule.\n"
     ]
    }
   ],
   "source": [
    "tr A-Z a-z <data/dracula2.txt >data/dracula3.txt\n",
    "head data/dracula3.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And remove punctuation and numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " may bistritzleft munich at  p m on st may arriving at\n",
      "vienna early next morning should have arrived at  but train was an\n",
      "hour late budapesth seems a wonderful place from the glimpse which i\n",
      "got of it from the train and the little i could walk through the\n",
      "streets i feared to go very far from the station as we had arrived\n",
      "late and would start as near the correct time as possible the\n",
      "impression i had was that we were leaving the west and entering the\n",
      "east the most western of splendid bridges over the danube which is\n",
      "here of noble width and depth took us among the traditions of turkish\n",
      "rule\n"
     ]
    }
   ],
   "source": [
    "cat data/dracula3.txt | tr -d [:punct:] | tr -d [:digit:] > data/dracula4.txt\n",
    "head data/dracula4.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can perform a word frequency analysis using uniq and sort.\n",
    "First we need to substitute spaces with newlines and then sort them to group the same words together. Uniq then will count consecutive lines that are the same and print word frequencies. We reverse sort the result to print most frequent words first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sed -r 's/\\s+/\\n/g' data/dracula4.txt | \\  # Replace spaces with new lines\n",
    "#     awk 'NF' | \\  # another way to remove empty lines\n",
    "#     sort | \\  # alphabetical sort\n",
    "#     uniq -c | \\ # word count \n",
    "#     sort -nr | \\ # reverse numeric sort\n",
    "#     awk '{$1=$1; print}' | \\  # strip leading and trailing whitespace\n",
    "#     awk 'BEGIN { OFS=\"\\t\" } {print $2,$1}' > data/wordcount.txt  # reverse columns\n",
    "    \n",
    "    \n",
    "sed -r 's/\\s+/\\n/g' data/dracula4.txt | \\\n",
    "    awk 'NF' | \\\n",
    "    sort | \\\n",
    "    uniq -c | \\\n",
    "    sort -nr | \\\n",
    "    awk '{$1=$1; print}' | \\\n",
    "    awk 'BEGIN { OFS=\" \"  } {print $2,$1}' > data/wordcount.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the 8027\n",
      "and 5893\n",
      "i 4710\n",
      "to 4538\n",
      "of 3732\n",
      "a 2962\n",
      "in 2555\n",
      "he 2543\n",
      "that 2455\n",
      "it 2138\n"
     ]
    }
   ],
   "source": [
    "head data/wordcount.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even create a histogram of word counts using a simple python script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "function histogram {\n",
    "python3 -c 'import sys\n",
    "for line in sys.stdin:\n",
    "  data, width = line.split()\n",
    "  print(\"{:<15}{:=<{width}}\".format(data, \"\", width=int(int(width) / 75)))' # each = corresponds to a count of 75\n",
    "\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat data/wordcount.txt  | histogram > data/histogram.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the            ===========================================================================================================\n",
      "and            ==============================================================================\n",
      "i              ==============================================================\n",
      "to             ============================================================\n",
      "of             =================================================\n",
      "a              =======================================\n",
      "in             ==================================\n",
      "he             =================================\n",
      "that           ================================\n",
      "it             ============================\n",
      "was            =========================\n",
      "as             =====================\n",
      "we             ====================\n",
      "for            ====================\n",
      "is             ====================\n",
      "his            ===================\n",
      "you            ===================\n",
      "me             ===================\n",
      "not            ==================\n",
      "with           =================\n",
      "my             ================\n",
      "all            ===============\n",
      "be             ===============\n",
      "so             ==============\n",
      "at             ==============\n",
      "on             ==============\n",
      "but            ==============\n",
      "have           ==============\n",
      "her            ==============\n",
      "had            =============\n",
      "him            ============\n",
      "she            ==========\n",
      "when           ==========\n",
      "there          ==========\n",
      "this           ========\n",
      "which          ========\n",
      "if             ========\n",
      "from           ========\n",
      "are            ========\n",
      "said           =======\n",
      "were           =======\n",
      "then           =======\n",
      "by             =======\n",
      "or             ======\n",
      "could          ======\n",
      "one            ======\n",
      "do             ======\n",
      "no             ======\n",
      "they           ======\n",
      "them           ======\n",
      "what           ======\n",
      "us             ======\n",
      "will           ======\n",
      "must           ======\n",
      "up             =====\n",
      "some           =====\n",
      "out            =====\n",
      "would          =====\n",
      "shall          =====\n",
      "may            =====\n",
      "our            =====\n",
      "now            =====\n",
      "see            =====\n",
      "been           =====\n",
      "know           =====\n",
      "can            =====\n",
      "more           ====\n",
      "time           ====\n",
      "an             ====\n",
      "has            ====\n",
      "come           ====\n",
      "am             ====\n",
      "over           ====\n",
      "any            ====\n",
      "van            ====\n",
      "your           ====\n",
      "came           ====\n",
      "helsing        ===\n",
      "went           ===\n",
      "into           ===\n",
      "only           ===\n",
      "who            ===\n",
      "go             ===\n",
      "very           ===\n",
      "did            ===\n",
      "before         ===\n",
      "like           ===\n",
      "here           ===\n",
      "back           ===\n",
      "down           ===\n",
      "again          ===\n",
      "seemed         ===\n",
      "about          ===\n",
      "well           ===\n",
      "even           ===\n",
      "such           ===\n",
      "way            ===\n",
      "took           ==\n",
      "lucy           ==\n",
      "than           ==\n",
      "good           ==\n",
      "dear           ==\n",
      "think          ==\n",
      "their          ==\n",
      "much           ==\n",
      "where          ==\n",
      "saw            ==\n",
      "how            ==\n",
      "though         ==\n",
      "man            ==\n",
      "through        ==\n",
      "mina           ==\n",
      "too            ==\n",
      "night          ==\n",
      "hand           ==\n",
      "after          ==\n",
      "room           ==\n",
      "face           ==\n",
      "should         ==\n",
      "door           ==\n",
      "made           ==\n",
      "tell           ==\n",
      "poor           ==\n",
      "old            ==\n",
      "own            ==\n",
      "other          ==\n",
      "eyes           ==\n",
      "away           ==\n",
      "looked         ==\n",
      "work           ==\n",
      "great          ==\n",
      "friend         ==\n",
      "sleep          ==\n",
      "once           ==\n",
      "jonathan       ==\n",
      "dr             ==\n",
      "things         ==\n",
      "get            ==\n",
      "look           ==\n",
      "little         ==\n",
      "make           ==\n",
      "just           ==\n",
      "might          ==\n",
      "got            ==\n",
      "day            ==\n",
      "professor      ==\n",
      "its            ==\n",
      "found          ==\n",
      "yet            ==\n",
      "count          ==\n",
      "off            =\n",
      "god            =\n",
      "take           =\n",
      "long           =\n",
      "say            =\n",
      "thought        =\n",
      "told           =\n",
      "men            =\n",
      "let            =\n",
      "life           =\n",
      "asked          =\n",
      "without        =\n",
      "something      =\n",
      "last           =\n",
      "till           =\n",
      "place          =\n",
      "oh             =\n",
      "myself         =\n",
      "first          =\n",
      "fear           =\n",
      "arthur         =\n",
      "ever           =\n",
      "house          =\n",
      "heart          =\n",
      "two            =\n",
      "never          =\n",
      "knew           =\n",
      "done           =\n",
      "himself        =\n",
      "these          =\n",
      "quite          =\n",
      "same           =\n",
      "want           =\n",
      "find           =\n",
      "still          =\n",
      "harker         =\n",
      "began          =\n",
      "nothing        =\n",
      "coming         =\n",
      "window         =\n",
      "round          =\n",
      "put            =\n",
      "head           =\n",
      "many           =\n",
      "hands          =\n",
      "however        =\n",
      "help           =\n",
      "right          =\n",
      "mr             =\n",
      "hear           =\n",
      "blood          =\n",
      "whilst         =\n",
      "mind           =\n",
      "white          =\n",
      "open           =\n",
      "full           =\n",
      "moment         =\n",
      "anything       =\n",
      "keep           =\n",
      "thing          =\n",
      "terrible       =\n",
      "morning        =\n",
      "left           =\n",
      "rest           =\n",
      "seen           =\n",
      "diary          =\n",
      "madam          =\n",
      "heard          =\n",
      "far            =\n",
      "upon           =\n",
      "why            =\n",
      "mrs            =\n",
      "strange        =\n",
      "cannot         =\n",
      "bed            =\n",
      "felt           =\n",
      "each           =\n",
      "few            =\n",
      "both           =\n",
      "tonight        =\n",
      "since          =\n",
      "project        =\n",
      "godalming      =\n",
      "every          =\n",
      "dont           =\n",
      "turned         =\n",
      "read           =\n",
      "those          =\n",
      "seward         =\n",
      "light          =\n",
      "being          =\n",
      "others         =\n",
      "give           =\n",
      "alone          =\n",
      "sort           =\n",
      "quincey        =\n",
      "opened         =\n",
      "love           =\n",
      "another        =\n",
      "stood          =\n"
     ]
    }
   ],
   "source": [
    "head -250 data/histogram.txt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training an n-gram language model\n",
    "\n",
    "Now we can use KenLM to train a 3-gram Language model on our preprocessed corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Builds unpruned language models with modified Kneser-Ney smoothing.\n",
      "\n",
      "Please cite:\n",
      "@inproceedings{Heafield-estimate,\n",
      "  author = {Kenneth Heafield and Ivan Pouzyrevsky and Jonathan H. Clark and Philipp Koehn},\n",
      "  title = {Scalable Modified {Kneser-Ney} Language Model Estimation},\n",
      "  year = {2013},\n",
      "  month = {8},\n",
      "  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics},\n",
      "  address = {Sofia, Bulgaria},\n",
      "  url = {http://kheafield.com/professional/edinburgh/estimate\\_paper.pdf},\n",
      "}\n",
      "\n",
      "Provide the corpus on stdin.  The ARPA file will be written to stdout.  Order of\n",
      "the model (-o) is the only mandatory option.  As this is an on-disk program,\n",
      "setting the temporary file location (-T) and sorting memory (-S) is recommended.\n",
      "\n",
      "Memory sizes are specified like GNU sort: a number followed by a unit character.\n",
      "Valid units are % for percentage of memory (supported platforms only) and (in\n",
      "increasing powers of 1024): b, K, M, G, T, P, E, Z, Y.  Default is K (*1024).\n",
      "This machine has 13238644736 bytes of memory.\n",
      "\n",
      "Language model building options:\n",
      "  -h [ --help ]                         Show this help message\n",
      "  -o [ --order ] arg                    Order of the model\n",
      "  --interpolate_unigrams [=arg(=1)] (=1)\n",
      "                                        Interpolate the unigrams (default) as \n",
      "                                        opposed to giving lots of mass to <unk>\n",
      "                                        like SRI.  If you want SRI's behavior \n",
      "                                        with a large <unk> and the old lmplz \n",
      "                                        default, use --interpolate_unigrams 0.\n",
      "  --skip_symbols                        Treat <s>, </s>, and <unk> as \n",
      "                                        whitespace instead of throwing an \n",
      "                                        exception\n",
      "  -T [ --temp_prefix ] arg (=/tmp/)     Temporary file prefix\n",
      "  -S [ --memory ] arg (=80%)            Sorting memory\n",
      "  --minimum_block arg (=8K)             Minimum block size to allow\n",
      "  --sort_block arg (=64M)               Size of IO operations for sort \n",
      "                                        (determines arity)\n",
      "  --block_count arg (=2)                Block count (per order)\n",
      "  --vocab_estimate arg (=1000000)       Assume this vocabulary size for \n",
      "                                        purposes of calculating memory in step \n",
      "                                        1 (corpus count) and pre-sizing the \n",
      "                                        hash table\n",
      "  --vocab_pad arg (=0)                  If the vocabulary is smaller than this \n",
      "                                        value, pad with <unk> to reach this \n",
      "                                        size. Requires --interpolate_unigrams\n",
      "  --verbose_header                      Add a verbose header to the ARPA file \n",
      "                                        that includes information such as token\n",
      "                                        count, smoothing type, etc.\n",
      "  --text arg                            Read text from a file instead of stdin\n",
      "  --arpa arg                            Write ARPA to a file instead of stdout\n",
      "  --intermediate arg                    Write ngrams to intermediate files.  \n",
      "                                        Turns off ARPA output (which can be \n",
      "                                        reactivated by --arpa file).  Forces \n",
      "                                        --renumber on.\n",
      "  --renumber                            Renumber the vocabulary identifiers so \n",
      "                                        that they are monotone with the hash of\n",
      "                                        each string.  This is consistent with \n",
      "                                        the ordering used by the trie data \n",
      "                                        structure.\n",
      "  --collapse_values                     Collapse probability and backoff into a\n",
      "                                        single value, q that yields the same \n",
      "                                        sentence-level probabilities.  See \n",
      "                                        http://kheafield.com/professional/edinb\n",
      "                                        urgh/rest_paper.pdf for more details, \n",
      "                                        including a proof.\n",
      "  --prune arg                           Prune n-grams with count less than or \n",
      "                                        equal to the given threshold.  Specify \n",
      "                                        one value for each order i.e. 0 0 1 to \n",
      "                                        prune singleton trigrams and above.  \n",
      "                                        The sequence of values must be \n",
      "                                        non-decreasing and the last value \n",
      "                                        applies to any remaining orders. \n",
      "                                        Default is to not prune, which is \n",
      "                                        equivalent to --prune 0.\n",
      "  --limit_vocab_file arg                Read allowed vocabulary separated by \n",
      "                                        whitespace. N-grams that contain \n",
      "                                        vocabulary items not in this list will \n",
      "                                        be pruned. Can be combined with --prune\n",
      "                                        arg\n",
      "  --discount_fallback [=arg(=0.5 1 1.5)]\n",
      "                                        The closed-form estimate for Kneser-Ney\n",
      "                                        discounts does not work without \n",
      "                                        singletons or doubletons.  It can also \n",
      "                                        fail if these values are out of range. \n",
      "                                        This option falls back to \n",
      "                                        user-specified discounts when the \n",
      "                                        closed-form estimate fails.  Note that \n",
      "                                        this option is generally a bad idea: \n",
      "                                        you should deduplicate your corpus \n",
      "                                        instead.  However, class-based models \n",
      "                                        need custom discounts because they lack\n",
      "                                        singleton unigrams.  Provide up to \n",
      "                                        three discounts (for adjusted counts 1,\n",
      "                                        2, and 3+), which will be applied to \n",
      "                                        all orders where the closed-form \n",
      "                                        estimates fail.\n",
      "\n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "./bin/lmplz --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== 1/5 Counting and sorting n-grams ===\n",
      "Reading /home/geopar/projects/bash_tutorial/data/dracula4.txt\n",
      "----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\n",
      "****************************************************************************************************\n",
      "Unigram tokens 163116 types 10719\n",
      "=== 2/5 Calculating and sorting adjusted counts ===\n",
      "Chain sizes: 1:128628 2:3683752192 3:6907035648\n",
      "Statistics:\n",
      "1 10719 D1=0.644236 D2=1.00333 D3+=1.39803\n",
      "2 72604 D1=0.771329 D2=1.15115 D3+=1.31645\n",
      "3 133828 D1=0.882174 D2=1.25019 D3+=1.4389\n",
      "Memory estimate for binary LM:\n",
      "type      kB\n",
      "probing 4326 assuming -p 1.5\n",
      "probing 4793 assuming -r models -p 1.5\n",
      "trie    1828 without quantization\n",
      "trie    1039 assuming -q 8 -b 8 quantization \n",
      "trie    1738 assuming -a 22 array pointer compression\n",
      "trie     949 assuming -a 22 -q 8 -b 8 array pointer compression and quantization\n",
      "=== 3/5 Calculating and sorting initial probabilities ===\n",
      "Chain sizes: 1:128628 2:1161664 3:2676560\n",
      "----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\n",
      "####################################################################################################\n",
      "=== 4/5 Calculating and writing order-interpolated probabilities ===\n",
      "Chain sizes: 1:128628 2:1161664 3:2676560\n",
      "----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\n",
      "####################################################################################################\n",
      "=== 5/5 Writing ARPA model ===\n",
      "----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\n",
      "****************************************************************************************************\n",
      "Name:lmplz\tVmPeak:10491128 kB\tVmRSS:10520 kB\tRSSMax:2417444 kB\tuser:0.256158\tsys:0.52217\tCPU:0.778345\treal:0.757482\n"
     ]
    }
   ],
   "source": [
    "./bin/lmplz -o 3 <data/dracula4.txt > data/dracula.lm.arpa "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see some 1-gram, 2-gram and 3-gram scores using grep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\\\u001b[01;31m\u001b[K1-grams\u001b[m\u001b[K:\n",
      "-4.890992\t<unk>\t0\n",
      "0\t<s>\t-0.61173564\n",
      "-1.4264659\t</s>\t0\n",
      "-2.7945037\tmay\t-0.3863618\n",
      "-4.7507243\tbistritzleft\t-0.11276052\n",
      "-4.7507243\tmunich\t-0.11276052\n",
      "-2.1378868\tat\t-0.7028289\n",
      "-3.8816328\tp\t-0.14545205\n",
      "-3.9838684\tm\t-0.19899167\n",
      "-2.1765325\ton\t-0.64976496\n",
      "\u001b[36m\u001b[K--\u001b[m\u001b[K\n",
      "\\\u001b[01;31m\u001b[K2-grams\u001b[m\u001b[K:\n",
      "-1.7202902\t<s> </s>\t0\n",
      "-1.1438557\tmay </s>\t0\n",
      "-1.0149596\tat </s>\t0\n",
      "-1.0583433\tp </s>\t0\n",
      "-0.9642732\tm </s>\t0\n",
      "-1.2002825\ton </s>\t0\n",
      "-1.1483785\tearly </s>\t0\n",
      "-0.9711424\tnext </s>\t0\n",
      "-1.4812524\tmorning </s>\t0\n",
      "-1.1897001\tshould </s>\t0\n",
      "\u001b[36m\u001b[K--\u001b[m\u001b[K\n",
      "\\\u001b[01;31m\u001b[K3-grams\u001b[m\u001b[K:\n",
      "-0.8619659\t<s> may </s>\n",
      "-1.1233317\ti may </s>\n",
      "-0.7419162\tof may </s>\n",
      "-0.9894832\tit may </s>\n",
      "-1.0609393\tand may </s>\n",
      "-1.0421566\twe may </s>\n",
      "-1.3042028\the may </s>\n",
      "-1.8012102\tthere may </s>\n",
      "-0.9887751\tthis may </s>\n",
      "-1.196441\tyou may </s>\n"
     ]
    }
   ],
   "source": [
    "cat data/dracula.lm.arpa | egrep \"1-grams|2-grams|3-grams\" -A10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use query to use the trained language model to score the perplexity of a sentence.\n",
    "Lower perplexity indicates a more probable sentence.\n",
    "\n",
    "Let's have the model score two possible endings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./bin/query: invalid option -- 'h'\n",
      "KenLM was compiled with maximum order 6.\n",
      "Usage: ./bin/query [-b] [-n] [-w] [-s] lm_file\n",
      "-b: Do not buffer output.\n",
      "-n: Do not wrap the input in <s> and </s>.\n",
      "-v summary|sentence|word: Print statistics at this level.\n",
      "   Can be used multiple times: -v summary -v sentence -v word\n",
      "-l lazy|populate|read|parallel: Load lazily, with populate, or malloc+read\n",
      "The default loading method is populate on Linux and read on others.\n",
      "\n",
      "Each word in the output is formatted as:\n",
      "  word=vocab_id ngram_length log10(p(word|context))\n",
      "where ngram_length is the length of n-gram matched.  A vocab_id of 0 indicates\n",
      "the unknown word. Sentence-level output includes log10 probability of the\n",
      "sentence and OOV count.\n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "./bin/query -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo \"harker and mina die a horrible death\" > data/bad_ending\n",
    "echo \"harker and mina live happily ever after\" > data/good_ending"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "harker and mina die a horrible death\n",
      "Perplexity including OOVs:\t456.47716780020465\n"
     ]
    }
   ],
   "source": [
    "cat data/bad_ending\n",
    "./bin/query data/dracula.lm.arpa < data/bad_ending 2>&1| grep \"Perplexity\" | head -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "harker and mina live happily ever after\n",
      "Perplexity including OOVs:\t916.2937036462345\n"
     ]
    }
   ],
   "source": [
    "cat data/good_ending\n",
    "./bin/query data/dracula.lm.arpa < data/good_ending 2>&1| grep \"Perplexity\" | head -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}