NLP_Week4_복습과제.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **2 Coding: Implementing word2vec (18 points)**\n",
        "In this part you will implement the word2vec model and train your own word vectors with stochastic gradient descent (SGD). Before you begin, first run the following commands within the assignment directory in order to create the appropriate conda virtual environment. This guarantees that you have all the necessary packages to complete the assignment. Also note that you probably want to finish the previous math section before writing the code since you will be asked to implement the math functions in Python. You want to implement and test the following subsections in order since they are accumulative.\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "conda env create -f env.yml\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "conda activate a2\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "Once you are done with the assignment you can deactivate this environment by running:"
      ],
      "metadata": {
        "id": "mIMvghVZPz7M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "os.chdir(\"/content/drive/MyDrive/유런/a2/a2\")"
      ],
      "metadata": {
        "id": "18VDLf0OvoRc"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install ipynb-py-convert"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tAsR1nfvtFxx",
        "outputId": "72ac6f37-b15c-400d-c852-db5f4c8cc921"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting ipynb-py-convert\n",
            "  Downloading ipynb-py-convert-0.4.6.tar.gz (3.9 kB)\n",
            "Building wheels for collected packages: ipynb-py-convert\n",
            "  Building wheel for ipynb-py-convert (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ipynb-py-convert: filename=ipynb_py_convert-0.4.6-py3-none-any.whl size=4640 sha256=5c7d62e0d619a9a6c4f82a12323083bf16855799222e8c51b2d6e0d31050e03a\n",
            "  Stored in directory: /root/.cache/pip/wheels/af/31/a9/761b134adbbca3c92d491eff3bad785c0a0c0079695d6f0504\n",
            "Successfully built ipynb-py-convert\n",
            "Installing collected packages: ipynb-py-convert\n",
            "Successfully installed ipynb-py-convert-0.4.6\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!ipynb-py-convert word2vec.py word2vec.ipynb"
      ],
      "metadata": {
        "id": "PIPEQxWNtPxi"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "*conda deactivate*\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "For each of the methods you need to implement, we included approximately how many lines of code our\n",
        "solution has in the code comments. These numbers are included to guide you. You don’t have to stick to\n",
        "them, you can write shorter or longer code as you wish. If you think your implementation is significantly\n",
        "longer than ours, it is a signal that there are some numpy methods you could utilize to make your code both\n",
        "shorter and faster. for loops in Python take a long time to complete when used over large arrays, so we\n",
        "expect you to utilize numpy methods. We will be checking the efficiency of your code. You will be able to\n",
        "see the results of the autograder when you submit your code to Gradescope, we recommend submitting\n",
        "early and often."
      ],
      "metadata": {
        "id": "16UYFj10QDhq"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m0OpXHfVyoDz"
      },
      "source": [
        "#!/usr/bin/env python\n",
        "\n",
        "import argparse\n",
        "import numpy as np\n",
        "import random\n",
        "\n",
        "from utils.gradcheck import gradcheck_naive, grad_tests_softmax, grad_tests_negsamp\n",
        "from utils.utils import normalizeRows, softmax\n",
        "\n",
        "\n",
        "def sigmoid(x):\n",
        "    \"\"\"\n",
        "    Compute the sigmoid function for the input here.\n",
        "    Arguments:\n",
        "    x -- A scalar or numpy array.\n",
        "    Return:\n",
        "    s -- sigmoid(x)\n",
        "    \"\"\"\n",
        "\n",
        "    ### YOUR CODE HERE (~1 Line)\n",
        "    s= 1/(np.exp(-x)+1)\n",
        "\n",
        "    ### END YOUR CODE\n",
        "\n",
        "    return s\n",
        "\n",
        "\n",
        "def naiveSoftmaxLossAndGradient(\n",
        "    centerWordVec,\n",
        "    outsideWordIdx,\n",
        "    outsideVectors,\n",
        "    dataset\n",
        "):\n",
        "    \"\"\" Naive Softmax loss & gradient function for word2vec models\n",
        "\n",
        "    Implement the naive softmax loss and gradients between a center word's \n",
        "    embedding and an outside word's embedding. This will be the building block\n",
        "    for our word2vec models. For those unfamiliar with numpy notation, note \n",
        "    that a numpy ndarray with a shape of (x, ) is a one-dimensional array, which\n",
        "    you can effectively treat as a vector with length x.\n",
        "\n",
        "    Arguments:\n",
        "    centerWordVec -- numpy ndarray, center word's embedding\n",
        "                    in shape (word vector length, )\n",
        "                    (v_c in the pdf handout)\n",
        "    outsideWordIdx -- integer, the index of the outside word\n",
        "                    (o of u_o in the pdf handout)\n",
        "    outsideVectors -- outside vectors is\n",
        "                    in shape (num words in vocab, word vector length) \n",
        "                    for all words in vocab (tranpose of U in the pdf handout)\n",
        "    dataset -- needed for negative sampling, unused here.\n",
        "\n",
        "    Return:\n",
        "    loss -- naive softmax loss\n",
        "    gradCenterVec -- the gradient with respect to the center word vector\n",
        "                     in shape (word vector length, )\n",
        "                     (dJ / dv_c in the pdf handout)\n",
        "    gradOutsideVecs -- the gradient with respect to all the outside word vectors\n",
        "                    in shape (num words in vocab, word vector length) \n",
        "                    (dJ / dU)\n",
        "    \"\"\"\n",
        "\n",
        "    ### YOUR CODE HERE (~6-8 Lines)\n",
        "\n",
        "    ### Please use the provided softmax function (imported earlier in this file)\n",
        "    ### This numerically stable implementation helps you avoid issues pertaining\n",
        "    ### to integer overflow. \n",
        "\n",
        "    U, vc, o = outsideVectors, centerWordVec, outsideWordIdx\n",
        "    yhat = softmax(np.dot(U, vc))\n",
        "    loss = -np.log(yhat[o])\n",
        "    yhat[o]-= 1\n",
        "    gradCenterVec=np.dot(U.T,yhat)\n",
        "    gradOutsideVecs = np.dot(yhat[:,np.newaxis], vc[:, np.newaxis].T)\n",
        "\n",
        "\n",
        "    ### END YOUR CODE\n",
        "\n",
        "    return loss, gradCenterVec, gradOutsideVecs\n",
        "\n",
        "\n",
        "def getNegativeSamples(outsideWordIdx, dataset, K):\n",
        "    \"\"\" Samples K indexes which are not the outsideWordIdx \"\"\"\n",
        "\n",
        "    negSampleWordIndices = [None] * K\n",
        "    for k in range(K):\n",
        "        newidx = dataset.sampleTokenIdx()\n",
        "        while newidx == outsideWordIdx:\n",
        "            newidx = dataset.sampleTokenIdx()\n",
        "        negSampleWordIndices[k] = newidx\n",
        "    return negSampleWordIndices\n",
        "\n",
        "\n",
        "def negSamplingLossAndGradient(\n",
        "    centerWordVec,\n",
        "    outsideWordIdx,\n",
        "    outsideVectors,\n",
        "    dataset,\n",
        "    K=10\n",
        "):\n",
        "    \"\"\" Negative sampling loss function for word2vec models\n",
        "\n",
        "    Implement the negative sampling loss and gradients for a centerWordVec\n",
        "    and a outsideWordIdx word vector as a building block for word2vec\n",
        "    models. K is the number of negative samples to take.\n",
        "\n",
        "    Note: The same word may be negatively sampled multiple times. For\n",
        "    example if an outside word is sampled twice, you shall have to\n",
        "    double count the gradient with respect to this word. Thrice if\n",
        "    it was sampled three times, and so forth.\n",
        "\n",
        "    Arguments/Return Specifications: same as naiveSoftmaxLossAndGradient\n",
        "    \"\"\"\n",
        "\n",
        "    # Negative sampling of words is done for you. Do not modify this if you\n",
        "    # wish to match the autograder and receive points!\n",
        "    negSampleWordIndices = getNegativeSamples(outsideWordIdx, dataset, K)\n",
        "    indices = [outsideWordIdx] + negSampleWordIndices\n",
        "\n",
        "    ### YOUR CODE HERE (~10 Lines)\n",
        "    U, vc, o = outsideVectors, centerWordVec, outsideWordIdx\n",
        "    gradCenterVec, gradOutsideVecs = np.zeros(vc.shape), np.zeros(U.shape)\n",
        "\n",
        "    ### Please use your implementation of sigmoid in here.\n",
        "    yhat = sigmoid(np.dot(U,vc))\n",
        "    loss = -np.log(yhat[o])\n",
        "    yhat[o] -= 1\n",
        "    gradCenterVec += U[o]*yhat[o]\n",
        "    gradOutsideVecs[o] +=vc*yhat[o]\n",
        "    for k in range(len(indices)):\n",
        "      if indices[k] !=o:\n",
        "        neg = indices\n",
        "        negsig = sigmoid(-np.dot(U[neg], vc))\n",
        "        loss -= np.log(negsig)\n",
        "        gradCenterVec += np.multipy(U[neg], 1-negsig)\n",
        "        gradOutsideVecs[neg] += np.multiply(vc, 1-negsig)\n",
        "\n",
        "    ### END YOUR CODE\n",
        "\n",
        "    return loss, gradCenterVec, gradOutsideVecs\n",
        "\n",
        "\n",
        "def skipgram(currentCenterWord, windowSize, outsideWords, word2Ind,\n",
        "             centerWordVectors, outsideVectors, dataset,\n",
        "             word2vecLossAndGradient=naiveSoftmaxLossAndGradient):\n",
        "    \"\"\" Skip-gram model in word2vec\n",
        "\n",
        "    Implement the skip-gram model in this function.\n",
        "\n",
        "    Arguments:\n",
        "    currentCenterWord -- a string of the current center word\n",
        "    windowSize -- integer, context window size\n",
        "    outsideWords -- list of no more than 2*windowSize strings, the outside words\n",
        "    word2Ind -- a dictionary that maps words to their indices in\n",
        "              the word vector list\n",
        "    centerWordVectors -- center word vectors (as rows) is in shape \n",
        "                        (num words in vocab, word vector length) \n",
        "                        for all words in vocab (V in pdf handout)\n",
        "    outsideVectors -- outside vectors is in shape \n",
        "                        (num words in vocab, word vector length) \n",
        "                        for all words in vocab (transpose of U in the pdf handout)\n",
        "    word2vecLossAndGradient -- the loss and gradient function for\n",
        "                               a prediction vector given the outsideWordIdx\n",
        "                               word vectors, could be one of the two\n",
        "                               loss functions you implemented above.\n",
        "\n",
        "    Return:\n",
        "    loss -- the loss function value for the skip-gram model\n",
        "            (J in the pdf handout)\n",
        "    gradCenterVec -- the gradient with respect to the center word vector\n",
        "                     in shape (word vector length, )\n",
        "                     (dJ / dv_c in the pdf handout)\n",
        "    gradOutsideVecs -- the gradient with respect to all the outside word vectors\n",
        "                    in shape (num words in vocab, word vector length) \n",
        "                    (dJ / dU)\n",
        "    \"\"\"\n",
        "\n",
        "    loss = 0.0\n",
        "    gradCenterVecs = np.zeros(centerWordVectors.shape)\n",
        "    gradOutsideVectors = np.zeros(outsideVectors.shape)\n",
        "\n",
        "    ### YOUR CODE HERE (~8 Lines)\n",
        "    U,vc = outsideVectors, centerWordVectors\n",
        "    cur = vc[word2Ind[currentCenterWord]]\n",
        "    for i in outsideWords:\n",
        "      outside_i, cur_i = word2Ind[i], word2Ind[currentCenterWord]\n",
        "      loss_i, gradVC_i, gradU_i = word2vecLossAndGradient(cur, outside_i, U,dataset)\n",
        "      loss += loss_i\n",
        "      gradCenterVecs[cur_i] += gradVC_i\n",
        "      gradOutsideVectors += gradU_i\n",
        "\n",
        "\n",
        "    ### END YOUR CODE\n",
        "    \n",
        "    return loss, gradCenterVecs, gradOutsideVectors\n",
        "\n",
        "\n",
        "#############################################\n",
        "# Testing functions below. DO NOT MODIFY!   #\n",
        "#############################################\n",
        "\n",
        "def word2vec_sgd_wrapper(word2vecModel, word2Ind, wordVectors, dataset,\n",
        "                         windowSize,\n",
        "                         word2vecLossAndGradient=naiveSoftmaxLossAndGradient):\n",
        "    batchsize = 50\n",
        "    loss = 0.0\n",
        "    grad = np.zeros(wordVectors.shape)\n",
        "    N = wordVectors.shape[0]\n",
        "    centerWordVectors = wordVectors[:int(N/2),:]\n",
        "    outsideVectors = wordVectors[int(N/2):,:]\n",
        "    for i in range(batchsize):\n",
        "        windowSize1 = random.randint(1, windowSize)\n",
        "        centerWord, context = dataset.getRandomContext(windowSize1)\n",
        "\n",
        "        c, gin, gout = word2vecModel(\n",
        "            centerWord, windowSize1, context, word2Ind, centerWordVectors,\n",
        "            outsideVectors, dataset, word2vecLossAndGradient\n",
        "        )\n",
        "        loss += c / batchsize\n",
        "        grad[:int(N/2), :] += gin / batchsize\n",
        "        grad[int(N/2):, :] += gout / batchsize\n",
        "\n",
        "    return loss, grad\n",
        "\n",
        "def test_sigmoid():\n",
        "    \"\"\" Test sigmoid function \"\"\"\n",
        "    print(\"=== Sanity check for sigmoid ===\")\n",
        "    assert sigmoid(0) == 0.5\n",
        "    assert np.allclose(sigmoid(np.array([0])), np.array([0.5]))\n",
        "    assert np.allclose(sigmoid(np.array([1,2,3])), np.array([0.73105858, 0.88079708, 0.95257413]))\n",
        "    print(\"Tests for sigmoid passed!\")\n",
        "\n",
        "def getDummyObjects():\n",
        "    \"\"\" Helper method for naiveSoftmaxLossAndGradient and negSamplingLossAndGradient tests \"\"\"\n",
        "\n",
        "    def dummySampleTokenIdx():\n",
        "        return random.randint(0, 4)\n",
        "\n",
        "    def getRandomContext(C):\n",
        "        tokens = [\"a\", \"b\", \"c\", \"d\", \"e\"]\n",
        "        return tokens[random.randint(0,4)], \\\n",
        "            [tokens[random.randint(0,4)] for i in range(2*C)]\n",
        "\n",
        "    dataset = type('dummy', (), {})()\n",
        "    dataset.sampleTokenIdx = dummySampleTokenIdx\n",
        "    dataset.getRandomContext = getRandomContext\n",
        "\n",
        "    random.seed(31415)\n",
        "    np.random.seed(9265)\n",
        "    dummy_vectors = normalizeRows(np.random.randn(10,3))\n",
        "    dummy_tokens = dict([(\"a\",0), (\"b\",1), (\"c\",2),(\"d\",3),(\"e\",4)])\n",
        "\n",
        "    return dataset, dummy_vectors, dummy_tokens\n",
        "\n",
        "def test_naiveSoftmaxLossAndGradient():\n",
        "    \"\"\" Test naiveSoftmaxLossAndGradient \"\"\"\n",
        "    dataset, dummy_vectors, dummy_tokens = getDummyObjects()\n",
        "\n",
        "    print(\"==== Gradient check for naiveSoftmaxLossAndGradient ====\")\n",
        "    def temp(vec):\n",
        "        loss, gradCenterVec, gradOutsideVecs = naiveSoftmaxLossAndGradient(vec, 1, dummy_vectors, dataset)\n",
        "        return loss, gradCenterVec\n",
        "    gradcheck_naive(temp, np.random.randn(3), \"naiveSoftmaxLossAndGradient gradCenterVec\")\n",
        "\n",
        "    centerVec = np.random.randn(3)\n",
        "    def temp(vec):\n",
        "        loss, gradCenterVec, gradOutsideVecs = naiveSoftmaxLossAndGradient(centerVec, 1, vec, dataset)\n",
        "        return loss, gradOutsideVecs\n",
        "    gradcheck_naive(temp, dummy_vectors, \"naiveSoftmaxLossAndGradient gradOutsideVecs\")\n",
        "\n",
        "def test_negSamplingLossAndGradient():\n",
        "    \"\"\" Test negSamplingLossAndGradient \"\"\"\n",
        "    dataset, dummy_vectors, dummy_tokens = getDummyObjects()\n",
        "\n",
        "    print(\"==== Gradient check for negSamplingLossAndGradient ====\")\n",
        "    def temp(vec):\n",
        "        loss, gradCenterVec, gradOutsideVecs = negSamplingLossAndGradient(vec, 1, dummy_vectors, dataset)\n",
        "        return loss, gradCenterVec\n",
        "    gradcheck_naive(temp, np.random.randn(3), \"negSamplingLossAndGradient gradCenterVec\")\n",
        "\n",
        "    centerVec = np.random.randn(3)\n",
        "    def temp(vec):\n",
        "        loss, gradCenterVec, gradOutsideVecs = negSamplingLossAndGradient(centerVec, 1, vec, dataset)\n",
        "        return loss, gradOutsideVecs\n",
        "    gradcheck_naive(temp, dummy_vectors, \"negSamplingLossAndGradient gradOutsideVecs\")\n",
        "\n",
        "def test_skipgram():\n",
        "    \"\"\" Test skip-gram with naiveSoftmaxLossAndGradient \"\"\"\n",
        "    dataset, dummy_vectors, dummy_tokens = getDummyObjects()\n",
        "\n",
        "    print(\"==== Gradient check for skip-gram with naiveSoftmaxLossAndGradient ====\")\n",
        "    gradcheck_naive(lambda vec: word2vec_sgd_wrapper(\n",
        "        skipgram, dummy_tokens, vec, dataset, 5, naiveSoftmaxLossAndGradient),\n",
        "        dummy_vectors, \"naiveSoftmaxLossAndGradient Gradient\")\n",
        "    grad_tests_softmax(skipgram, dummy_tokens, dummy_vectors, dataset)\n",
        "\n",
        "    print(\"==== Gradient check for skip-gram with negSamplingLossAndGradient ====\")\n",
        "    gradcheck_naive(lambda vec: word2vec_sgd_wrapper(\n",
        "        skipgram, dummy_tokens, vec, dataset, 5, negSamplingLossAndGradient),\n",
        "        dummy_vectors, \"negSamplingLossAndGradient Gradient\")\n",
        "    grad_tests_negsamp(skipgram, dummy_tokens, dummy_vectors, dataset, negSamplingLossAndGradient)\n",
        "\n",
        "def test_word2vec():\n",
        "    \"\"\" Test the two word2vec implementations, before running on Stanford Sentiment Treebank \"\"\"\n",
        "    test_sigmoid()\n",
        "    test_naiveSoftmaxLossAndGradient()\n",
        "    test_negSamplingLossAndGradient()\n",
        "    test_skipgram()\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    parser = argparse.ArgumentParser(description='Test your implementations.')\n",
        "    parser.add_argument('function', nargs='?', type=str, default='all',\n",
        "                        help='Name of the function you would like to test.')\n",
        "\n",
        "    args = parser.parse_args()\n",
        "    if args.function == 'sigmoid':\n",
        "        test_sigmoid()\n",
        "    elif args.function == 'naiveSoftmaxLossAndGradient':\n",
        "        test_naiveSoftmaxLossAndGradient()\n",
        "    elif args.function == 'negSamplingLossAndGradient':\n",
        "        test_negSamplingLossAndGradient()\n",
        "    elif args.function == 'skipgram':\n",
        "        test_skipgram()\n",
        "    elif args.function == 'all':\n",
        "        test_word2vec()\n"
      ],
      "outputs": [],
      "execution_count": null
    }
  ]
}