textanalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This project focus on extracting topic from the given paragraph "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "gensim is a lib used for topic ectraction "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "from gensim import corpora\n",
    "from gensim.corpora.dictionary import Dictionary "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nltk is used various purposes like stopwords, lemmitiazation, etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.corpus import wordnet\n",
    "from nltk.tokenize import SpaceTokenizer \n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.tokenize import word_tokenize \n",
    "\n",
    "from collections import Counter \n",
    "\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "lem = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "tk = SpaceTokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "para = \"\"\"Which things that you find exotic and intoxicating , in a good way? A great conversation with a stranger?  A good partern or a relation ? A good novel or movie?\n",
    "You know what I find intoxicating. Two things 1. A worthy rival and second being a mentor. I request you all the retrospect your life and I am pretty sure that at one point you met someone who shared similarities with your persona was facing same problem that you have faced weather first speech or handle a particular situation and you must of thought , I should help him.. Finding someone to mentor or nurture is one of the best thing that can happen to you, the more worthy mentee  more enlighting you feel. \n",
    "India  is a weird country, we produce one of the best student but we fail dramatically at producing good teachers. If someone says he is a teacher we make fun of him or her, we think that he became teacher because he was not good at anything other or he wanted a boring life, and the frightening thing is this the HARD TRUTH. People become teacher because they think of this as a last resort, they are not passionate about teaching and it is quite depressing.\n",
    ".Ok, I am not here to talk about teachers or how they are being mistreated in society.  I am here for the art of teaching , and I will demonstrate it right now. \n",
    "Tell me, how many of you are from science background, please raise your hands. Ok how many are from physics, chemistry or engineering ?? wow so many, I feel like I am in a conference. So tell me do you know what an entropy is ? and do you know the answer of an entropy of an isolated system always_______? Don’t say it out loud. I want to explain it. \n",
    "So the classic definition of entropy is “order of randomness” or “disorder”  and it is true, but it does not explain why exactly we need this. So the simple definition of the entropy is “spread of the energy” or how much the energy has been spread. Sun is ultimate source of energy , belive it or not almost all of our energy we receive from sun including water dams , winds and fossile fuels. We use those fuel to do work. Suppose  1litter of petrol has become some amount of work, some of it converted in heat and some might get lost in friction. So can we say that we took compact energy and spread is across of a system.?? And it will safe to say that we cannot convert that friction of heat into petrol, but here comes the shocking news, “IT is not impossible” on microscopic it is not impossible to atoms from atmosphere come together and get converted into petrol.  But the chances of it are very low, while the chances of thermal equillbrium is very high. \n",
    "So answer me this, Entropy is the spread of energy, the wider the spread , higher the entropy. The sun has high amount of concentrated energy, we consume that energy in the form of food or sunlight and as I wave my hand around like this, it gets converted into work,heat and some amount of friction in air. And all I am doing is spreading the energy, it is still energy but this one is hard to harness. And tell me now, our universe is system and there are millions of stars out there,each one emitting energy which is used to in some other work thus converting the pure high grade energy into low grade energy, So tell me , if I asked you a question that “Is the entropy of universe increasing or decreasing”. Pause\n",
    "And this the point I want to make, a good teacher should not just tell facts to student nor put his opinions on them but to educate enough that they will be independent  to make their own opinions or come up with the conclusion. This does not mean my point should only be used by teachers, it can be also used by you. When you kid , brother , friend ask you what is 2+2 instead of saying 4. Tell them to count fingers. Because the job of teacher is not to just bombard student with facts and make them walk a trodden path but to enlighten them enough so that they can choose their own path.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = nltk.sent_tokenize(para)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#approach no.1 \n",
    "\n",
    "for i in range(len(sentences)):\n",
    "    words= nltk.word_tokenize(sentences[i])\n",
    "    words = [lem.lemmatize(word) for word in words if word not in set(stopwords.words('english'))]\n",
    "    sentences[i] =' '.join(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['which', 'things', 'find', 'exotic', 'intoxicating', 'good', 'way', 'great', 'conversation', 'stranger', 'good', 'partern', 'relation', 'good', 'novel', 'movie', 'you', 'know', 'find', 'intoxicating', 'two', 'things', 'worthy', 'rival', 'second', 'mentor', 'request', 'retrospect', 'life', 'pretty', 'sure', 'one', 'point', 'met', 'someone', 'shared', 'similarities', 'persona', 'facing', 'problem', 'faced', 'weather', 'first', 'speech', 'handle', 'particular', 'situation', 'must', 'thought', 'help', 'him..', 'finding', 'someone', 'mentor', 'nurture', 'one', 'best', 'thing', 'happen', 'worthy', 'mentee', 'enlighting', 'feel', 'india', 'weird', 'country', 'produce', 'one', 'best', 'student', 'fail', 'dramatically', 'producing', 'good', 'teachers', 'if', 'someone', 'says', 'teacher', 'make', 'fun', 'think', 'became', 'teacher', 'good', 'anything', 'wanted', 'boring', 'life', 'frightening', 'thing', 'hard', 'truth', 'people', 'become', 'teacher', 'think', 'last', 'resort', 'passionate', 'teaching', 'quite', 'depressing', '.ok', 'talk', 'teachers', 'mistreated', 'society', 'art', 'teaching', 'demonstrate', 'right', 'tell', 'many', 'science', 'background', 'please', 'raise', 'hands', 'ok', 'many', 'physics', 'chemistry', 'engineering', 'wow', 'many', 'feel', 'like', 'conference', 'so', 'tell', 'know', 'entropy', 'know', 'answer', 'entropy', 'isolated', 'system', 'always_______', 'don', 'say', 'loud', 'want', 'explain', 'so', 'classic', 'definition', 'entropy', 'order', 'randomness', 'disorder', 'true', 'explain', 'exactly', 'need', 'so', 'simple', 'definition', 'entropy', 'spread', 'energy', 'much', 'energy', 'spread', 'sun', 'ultimate', 'source', 'energy', 'belive', 'almost', 'energy', 'receive', 'sun', 'including', 'water', 'dams', 'winds', 'fossile', 'fuels', 'we', 'use', 'fuel', 'work', 'suppose', '1litter', 'petrol', 'become', 'amount', 'work', 'converted', 'heat', 'might', 'get', 'lost', 'friction', 'so', 'say', 'took', 'compact', 'energy', 'spread', 'across', 'system.', 'and', 'safe', 'say', 'convert', 'friction', 'heat', 'petrol', 'comes', 'shocking', 'news', 'it', 'impossible', 'microscopic', 'impossible', 'atoms', 'atmosphere', 'come', 'together', 'get', 'converted', 'petrol', 'but', 'chances', 'low', 'chances', 'thermal', 'equillbrium', 'high', 'so', 'answer', 'entropy', 'spread', 'energy', 'wider', 'spread', 'higher', 'entropy', 'the', 'sun', 'high', 'amount', 'concentrated', 'energy', 'consume', 'energy', 'form', 'food', 'sunlight', 'wave', 'hand', 'around', 'like', 'gets', 'converted', 'work', 'heat', 'amount', 'friction', 'air', 'and', 'spreading', 'energy', 'still', 'energy', 'one', 'hard', 'harness', 'and', 'tell', 'universe', 'system', 'millions', 'stars', 'one', 'emitting', 'energy', 'used', 'work', 'thus', 'converting', 'pure', 'high', 'grade', 'energy', 'low', 'grade', 'energy', 'so', 'tell', 'asked', 'question', 'is', 'entropy', 'universe', 'increasing', 'decreasing', 'pause', 'and', 'point', 'want', 'make', 'good', 'teacher', 'tell', 'facts', 'student', 'put', 'opinions', 'educate', 'enough', 'independent', 'make', 'opinions', 'come', 'conclusion', 'this', 'mean', 'point', 'used', 'teachers', 'also', 'used', 'when', 'kid', 'brother', 'friend', 'ask', '2+2', 'instead', 'saying', 'tell', 'count', 'fingers', 'because', 'job', 'teacher', 'bombard', 'student', 'facts', 'make', 'walk', 'trodden', 'path', 'enlighten', 'enough', 'choose', 'path']\n"
     ]
    }
   ],
   "source": [
    "#approach no.2 (better)\n",
    "\n",
    "\n",
    "stop_words =  set(stopwords.words('english'))\n",
    "word_token = word_tokenize(para)\n",
    "actual_list = [w for w in word_token if not w in stop_words]\n",
    "\n",
    "actual_list = []\n",
    "\n",
    "for w in word_token:\n",
    "    lem.lemmatize(w)\n",
    "    if w not in stop_words:\n",
    "        if len(w)>1:\n",
    "            actual_list.append(w)\n",
    "            actual_list = [w.lower() for w in actual_list]\n",
    "\n",
    "print(actual_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Which thing find exotic intoxicating , good way ?',\n",
       " 'A great conversation stranger ?',\n",
       " 'A good partern relation ?',\n",
       " 'A good novel movie ?',\n",
       " 'You know I find intoxicating .',\n",
       " 'Two thing 1 .',\n",
       " 'A worthy rival second mentor .',\n",
       " 'I request retrospect life I pretty sure one point met someone shared similarity persona facing problem faced weather first speech handle particular situation must thought , I help him.. Finding someone mentor nurture one best thing happen , worthy mentee enlighting feel .',\n",
       " 'India weird country , produce one best student fail dramatically producing good teacher .',\n",
       " 'If someone say teacher make fun , think became teacher good anything wanted boring life , frightening thing HARD TRUTH .',\n",
       " 'People become teacher think last resort , passionate teaching quite depressing .',\n",
       " '.Ok , I talk teacher mistreated society .',\n",
       " 'I art teaching , I demonstrate right .',\n",
       " 'Tell , many science background , please raise hand .',\n",
       " 'Ok many physic , chemistry engineering ? ?',\n",
       " 'wow many , I feel like I conference .',\n",
       " 'So tell know entropy ?',\n",
       " 'know answer entropy isolated system always_______ ?',\n",
       " 'Don ’ say loud .',\n",
       " 'I want explain .',\n",
       " 'So classic definition entropy “ order randomness ” “ disorder ” true , explain exactly need .',\n",
       " 'So simple definition entropy “ spread energy ” much energy spread .',\n",
       " 'Sun ultimate source energy , belive almost energy receive sun including water dam , wind fossile fuel .',\n",
       " 'We use fuel work .',\n",
       " 'Suppose 1litter petrol become amount work , converted heat might get lost friction .',\n",
       " 'So say took compact energy spread across system. ? ?',\n",
       " 'And safe say convert friction heat petrol , come shocking news , “ IT impossible ” microscopic impossible atom atmosphere come together get converted petrol .',\n",
       " 'But chance low , chance thermal equillbrium high .',\n",
       " 'So answer , Entropy spread energy , wider spread , higher entropy .',\n",
       " 'The sun high amount concentrated energy , consume energy form food sunlight I wave hand around like , get converted work , heat amount friction air .',\n",
       " 'And I spreading energy , still energy one hard harness .',\n",
       " 'And tell , universe system million star , one emitting energy used work thus converting pure high grade energy low grade energy , So tell , I asked question “ Is entropy universe increasing decreasing ” .',\n",
       " 'Pause And point I want make , good teacher tell fact student put opinion educate enough independent make opinion come conclusion .',\n",
       " 'This mean point used teacher , also used .',\n",
       " 'When kid , brother , friend ask 2+2 instead saying 4 .',\n",
       " 'Tell count finger .',\n",
       " 'Because job teacher bombard student fact make walk trodden path enlighten enough choose path .']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Because',\n",
       " 'job',\n",
       " 'teacher',\n",
       " 'bombard',\n",
       " 'student',\n",
       " 'fact',\n",
       " 'make',\n",
       " 'walk',\n",
       " 'trodden',\n",
       " 'path',\n",
       " 'enlighten',\n",
       " 'enough',\n",
       " 'choose',\n",
       " 'path',\n",
       " '.']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'Because': 1,\n",
       "         'job': 1,\n",
       "         'teacher': 1,\n",
       "         'bombard': 1,\n",
       "         'student': 1,\n",
       "         'fact': 1,\n",
       "         'make': 1,\n",
       "         'walk': 1,\n",
       "         'trodden': 1,\n",
       "         'path': 2,\n",
       "         'enlighten': 1,\n",
       "         'enough': 1,\n",
       "         'choose': 1,\n",
       "         '.': 1})"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bag_words = Counter(words)\n",
    "bag_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('path', 2), ('Because', 1), ('job', 1), ('teacher', 1), ('bombard', 1), ('student', 1), ('fact', 1), ('make', 1), ('walk', 1), ('trodden', 1), ('enlighten', 1), ('enough', 1), ('choose', 1), ('.', 1)]\n"
     ]
    }
   ],
   "source": [
    "print(bag_words.most_common(14))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#approach 1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary([i.split(',') for i in words])\n",
    "DT_matrix = [dictionary.doc2bow(doc.split()) for doc in words] \n",
    "Lda_object = gensim.models.ldamodel.LdaModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, '0.104*\"path\" + 0.096*\".\" + 0.090*\"bombard\" + 0.082*\"fact\" + 0.082*\"choose\"'), (1, '0.103*\"path\" + 0.090*\"enlighten\" + 0.089*\"make\" + 0.086*\"walk\" + 0.082*\"teacher\"')]\n"
     ]
    }
   ],
   "source": [
    "lda_model = Lda_object(DT_matrix, 2, id2word=dictionary)\n",
    "print(lda_model.print_topics(2, num_words=5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "above line represents a topic with individual weight\n",
    "topic 1 here 0, seems to be more about path to a dream job, which has hurdles and enlightenment\n",
    "topic 2 here 1, seems to be more about teachers and facts, etc \n",
    "\n",
    "now according to these topic we will see if the speaker has answered according to the theme of the question \n",
    "and how well has he/she answered it \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#approach 2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary([i.split(',') for i in actual_list])\n",
    "DT_matrix = [dictionary.doc2bow(doc.split()) for doc in actual_list] \n",
    "Lda_object = gensim.models.ldamodel.LdaModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0, '0.052*\"energy\" + 0.028*\"entropy\" + 0.025*\"tell\" + 0.025*\"so\" + 0.019*\"and\" + 0.014*\"converted\" + 0.013*\"amount\" + 0.013*\"used\" + 0.013*\"high\" + 0.012*\"make\" + 0.012*\"heat\" + 0.010*\"become\" + 0.010*\"things\" + 0.010*\"facts\" + 0.010*\"answer\" + 0.010*\"worthy\" + 0.009*\"grade\" + 0.009*\"definition\" + 0.009*\"intoxicating\" + 0.009*\"mentor\" + 0.009*\"like\" + 0.009*\"system\" + 0.009*\"opinions\" + 0.007*\"petrol\" + 0.006*\"conclusion\" + 0.006*\"thought\" + 0.006*\"mentee\" + 0.006*\"country\" + 0.006*\"retrospect\" + 0.006*\"is\" + 0.006*\"higher\" + 0.006*\"two\" + 0.006*\"exotic\" + 0.006*\"disorder\" + 0.006*\"mean\" + 0.006*\"we\" + 0.006*\"handle\" + 0.006*\"pretty\" + 0.006*\"kid\" + 0.006*\"great\" + 0.006*\"producing\" + 0.006*\"became\" + 0.006*\"air\" + 0.006*\"1litter\" + 0.006*\"loud\" + 0.006*\"simple\" + 0.006*\"ok\" + 0.006*\"india\" + 0.005*\"nurture\" + 0.005*\"job\"'), (1, '0.024*\"spread\" + 0.023*\"teacher\" + 0.015*\"know\" + 0.015*\"point\" + 0.014*\"many\" + 0.014*\"say\" + 0.014*\"student\" + 0.014*\"teachers\" + 0.014*\"someone\" + 0.014*\"friction\" + 0.011*\"think\" + 0.011*\"enough\" + 0.011*\"come\" + 0.011*\"best\" + 0.010*\"explain\" + 0.010*\"feel\" + 0.010*\"find\" + 0.010*\"impossible\" + 0.009*\"get\" + 0.009*\"low\" + 0.009*\"petrol\" + 0.007*\"energy\" + 0.007*\"make\" + 0.006*\"way\" + 0.006*\"this\" + 0.006*\"art\" + 0.006*\"gets\" + 0.006*\"help\" + 0.006*\"him..\" + 0.006*\"shared\" + 0.006*\"met\" + 0.006*\"good\" + 0.006*\"particular\" + 0.006*\"sure\" + 0.006*\"people\" + 0.006*\"stars\" + 0.006*\"emitting\" + 0.006*\"educate\" + 0.006*\"receive\" + 0.006*\"converting\" + 0.006*\"wider\" + 0.006*\"concentrated\" + 0.006*\"ask\" + 0.006*\"similarities\" + 0.006*\"system.\" + 0.006*\"choose\" + 0.006*\"finding\" + 0.006*\"boring\" + 0.006*\"isolated\" + 0.006*\"chemistry\"'), (2, '0.028*\"good\" + 0.025*\"one\" + 0.019*\"work\" + 0.016*\"sun\" + 0.011*\"thing\" + 0.011*\"hard\" + 0.011*\"teaching\" + 0.010*\"chances\" + 0.010*\"life\" + 0.010*\"universe\" + 0.009*\"energy\" + 0.008*\"path\" + 0.007*\"randomness\" + 0.007*\"independent\" + 0.007*\"talk\" + 0.007*\"happen\" + 0.007*\"2+2\" + 0.007*\"trodden\" + 0.007*\"says\" + 0.007*\"wave\" + 0.007*\"want\" + 0.007*\"mistreated\" + 0.007*\"around\" + 0.007*\"took\" + 0.007*\"water\" + 0.007*\"ultimate\" + 0.007*\"but\" + 0.006*\"lost\" + 0.006*\"hands\" + 0.006*\"source\" + 0.006*\"quite\" + 0.006*\"dramatically\" + 0.006*\"right\" + 0.006*\"facing\" + 0.006*\"the\" + 0.006*\"fossile\" + 0.006*\"truth\" + 0.006*\"walk\" + 0.006*\"atoms\" + 0.006*\"because\" + 0.006*\"wow\" + 0.006*\"use\" + 0.006*\"situation\" + 0.006*\"speech\" + 0.006*\"rival\" + 0.006*\"classic\" + 0.006*\"together\" + 0.006*\"almost\" + 0.006*\"when\" + 0.006*\"conference\"')]\n"
     ]
    }
   ],
   "source": [
    "lda_model = Lda_object(DT_matrix, 3, id2word=dictionary)\n",
    "print(lda_model.print_topics(5, num_words=50)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "theme = energy conversion, fuel, enviroment, job, best answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_list = ['']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "write words related to the theme of the para"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting target.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile target.txt\n",
    "\n",
    "influence exotic rival mentor understanding learning lessons lesson failure background inspiring profound \n",
    "alive memorable captivate unpopular opinon latest effortless rules work energy fuel fossil waste overuse \n",
    "depletion wind solar water entropy enviroment humans toxic think produce careful conscious true waste \n",
    "exotic deplet carbon dioxide oxygen mother nature equilibrium producing thinking safe job mentor \n",
    "oppurtunity rival system education study university school teachers teacher impossible possible approach\n",
    "instead susbstitute replace renew recycle things materials food clothes facts news survey emitting\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_list =[]\n",
    "\n",
    "with open('target.txt', \"r\", encoding='utf-8') as ft:\n",
    "    for line in ft:\n",
    "        target_list.extend(line.split())\n",
    "        \n",
    "#target_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "common words :  ['energy', 'impossible', 'water', 'facts', 'teachers', 'entropy', 'safe', 'background', 'mentor', 'job', 'emitting', 'teacher', 'exotic', 'instead', 'producing', 'rival', 'think', 'produce', 'system', 'true', 'news', 'work', 'food', 'fuel', 'things']\n",
      "\n",
      "number of common words 25\n",
      "\n",
      "You are on level :  E\n"
     ]
    }
   ],
   "source": [
    "class cm():\n",
    "        \n",
    "    \n",
    "        def common(a, b):\n",
    "            seta = set(a)\n",
    "            setb = set(b)\n",
    "\n",
    "            if (seta & setb ):\n",
    "                global c\n",
    "                c = seta & setb\n",
    "                c=list(c)\n",
    "\n",
    "                global counter \n",
    "                counter = len(c)\n",
    "\n",
    "                print(f'common words : ', c)\n",
    "                print(f'\\nnumber of common words',counter)\n",
    "\n",
    "            else:\n",
    "\n",
    "                print('\\nno words common')\n",
    "\n",
    "        common(actual_list, target_list)\n",
    "        \n",
    "        #---------------------------------------\n",
    "        \n",
    "        def acc():\n",
    "            \n",
    "            \n",
    "            if counter in range(1,10):\n",
    "                level = 'B'\n",
    "                print('\\nYou are on level : ', level)\n",
    "                \n",
    "            if counter in range(10,20):\n",
    "                level = 'I'\n",
    "                print('\\nYou are on level : ', level)\n",
    "                \n",
    "            if counter in range(20,30):\n",
    "                level = 'E'\n",
    "                print('\\nYou are on level : ', level)             \n",
    "           \n",
    "        acc()\n",
    "            \n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "on the words matching in both the files, we are measuring the level of the speaker "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "synset names :  rival.n.01\n",
      "\n",
      "synset defination :  the contestant you hope to defeat\n",
      "\n",
      "synset example :  ['he had respect for his rivals', 'he wanted to know what the competition was doing']\n",
      "\n",
      "synset tag :  n\n"
     ]
    }
   ],
   "source": [
    "#word \n",
    "sy = wordnet.synsets(\"rival\")[0]\n",
    "mul = wordnet.synsets(\"rival\")\n",
    "\n",
    "print('synset names : ', sy.name())\n",
    "print('\\nsynset defination : ', sy.definition())\n",
    "print('\\nsynset example : ', sy.examples())\n",
    "print('\\nsynset tag : ', sy.pos())\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "synonyms :  ['good', 'good', 'goodness', 'good', 'goodness', 'commodity', 'trade_good', 'good', 'good', 'full', 'good', 'good', 'estimable', 'good', 'honorable', 'respectable', 'beneficial', 'good', 'good', 'good', 'just', 'upright', 'adept', 'expert', 'good', 'practiced', 'proficient', 'skillful', 'skilful', 'good', 'dear', 'good', 'near', 'dependable', 'good', 'safe', 'secure', 'good', 'right', 'ripe', 'good', 'well', 'effective', 'good', 'in_effect', 'in_force', 'good', 'good', 'serious', 'good', 'sound', 'good', 'salutary', 'good', 'honest', 'good', 'undecomposed', 'unspoiled', 'unspoilt', 'good', 'well', 'good', 'thoroughly', 'soundly', 'good']\n",
      "\n",
      "antonyms :  ['evil', 'evilness', 'bad', 'badness', 'bad', 'evil', 'ill']\n"
     ]
    }
   ],
   "source": [
    "#word\n",
    "synonyms = []\n",
    "antonyms = []\n",
    "for syn in wordnet.synsets(\"good\"):\n",
    "    for l in syn.lemmas():\n",
    "        synonyms.append(l.name())\n",
    "        if l.antonyms():\n",
    "            antonyms.append(l.antonyms()[0].name())\n",
    "            \n",
    "print('synonyms : ', synonyms)\n",
    "print('\\nantonyms : ', antonyms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "synset names :  energy.n.01\n",
      "\n",
      "synset defination :  (physics) a thermodynamic quantity equivalent to the capacity of a physical system to do work; the units of energy are joules or ergs\n",
      "\n",
      "synset example :  ['energy can take a wide variety of forms']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  impossible.n.01\n",
      "\n",
      "synset defination :  something that cannot be done\n",
      "\n",
      "synset example :  ['his assignment verged on the impossible']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  water.n.01\n",
      "\n",
      "synset defination :  binary compound that occurs at room temperature as a clear colorless odorless tasteless liquid; freezes into ice below 0 degrees centigrade and boils above 100 degrees centigrade; widely used as a solvent\n",
      "\n",
      "synset example :  []\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  fact.n.01\n",
      "\n",
      "synset defination :  a piece of information about circumstances that exist or events that have occurred\n",
      "\n",
      "synset example :  ['first you must collect all the facts of the case']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  teacher.n.01\n",
      "\n",
      "synset defination :  a person whose occupation is teaching\n",
      "\n",
      "synset example :  []\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  information.n.05\n",
      "\n",
      "synset defination :  (communication theory) a numerical measure of the uncertainty of an outcome\n",
      "\n",
      "synset example :  ['the signal contained thousands of bits of information']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  safe.n.01\n",
      "\n",
      "synset defination :  strongbox where valuables can be safely kept\n",
      "\n",
      "synset example :  []\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  background.n.01\n",
      "\n",
      "synset defination :  a person's social heritage: previous experience or training\n",
      "\n",
      "synset example :  ['he is a lawyer with a sports background']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  mentor.n.01\n",
      "\n",
      "synset defination :  a wise and trusted guide and advisor\n",
      "\n",
      "synset example :  []\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  occupation.n.01\n",
      "\n",
      "synset defination :  the principal activity in your life that you do to earn money\n",
      "\n",
      "synset example :  [\"he's not in my line of business\"]\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  emit.v.01\n",
      "\n",
      "synset defination :  expel (gases or odors)\n",
      "\n",
      "synset example :  []\n",
      "\n",
      "synset tag :  v\n",
      "-----------------------\n",
      "synset names :  teacher.n.01\n",
      "\n",
      "synset defination :  a person whose occupation is teaching\n",
      "\n",
      "synset example :  []\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  alien.s.02\n",
      "\n",
      "synset defination :  being or from or characteristic of another place or part of the world\n",
      "\n",
      "synset example :  ['alien customs', 'exotic plants in a greenhouse', 'exotic cuisine']\n",
      "\n",
      "synset tag :  s\n",
      "-----------------------\n",
      "synset names :  alternatively.r.01\n",
      "\n",
      "synset defination :  in place of, or as an alternative to\n",
      "\n",
      "synset example :  ['Felix became a herpetologist instead', 'alternatively we could buy a used car']\n",
      "\n",
      "synset tag :  r\n",
      "-----------------------\n",
      "synset names :  produce.v.01\n",
      "\n",
      "synset defination :  bring forth or yield\n",
      "\n",
      "synset example :  ['The tree would not produce fruit']\n",
      "\n",
      "synset tag :  v\n",
      "-----------------------\n",
      "synset names :  rival.n.01\n",
      "\n",
      "synset defination :  the contestant you hope to defeat\n",
      "\n",
      "synset example :  ['he had respect for his rivals', 'he wanted to know what the competition was doing']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  think.n.01\n",
      "\n",
      "synset defination :  an instance of deliberate thinking\n",
      "\n",
      "synset example :  ['I need to give it a good think']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  produce.n.01\n",
      "\n",
      "synset defination :  fresh fruits and vegetable grown for the market\n",
      "\n",
      "synset example :  []\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  system.n.01\n",
      "\n",
      "synset defination :  instrumentality that combines interrelated interacting artifacts designed to work as a coherent entity\n",
      "\n",
      "synset example :  ['he bought a new stereo system', 'the system consists of a motor and a small computer']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  true.n.01\n",
      "\n",
      "synset defination :  proper alignment; the property possessed by something that is in correct or proper alignment\n",
      "\n",
      "synset example :  ['out of true']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  news.n.01\n",
      "\n",
      "synset defination :  information about recent and important events\n",
      "\n",
      "synset example :  ['they awaited news of the outcome']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  work.n.01\n",
      "\n",
      "synset defination :  activity directed toward making or doing something\n",
      "\n",
      "synset example :  ['she checked several points needing further work']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  food.n.01\n",
      "\n",
      "synset defination :  any substance that can be metabolized by an animal to give energy and build tissue\n",
      "\n",
      "synset example :  []\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  fuel.n.01\n",
      "\n",
      "synset defination :  a substance that can be consumed to produce energy\n",
      "\n",
      "synset example :  ['more fuel is needed during the winter months', 'they developed alternative fuels for aircraft']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n",
      "synset names :  things.n.01\n",
      "\n",
      "synset defination :  any movable possession (especially articles of clothing)\n",
      "\n",
      "synset example :  ['she packed her things and left']\n",
      "\n",
      "synset tag :  n\n",
      "-----------------------\n"
     ]
    }
   ],
   "source": [
    "#list\n",
    "for var in c:\n",
    "        #c is list of common words\n",
    "\n",
    "        sy = wordnet.synsets(var)[0]\n",
    "        mul = wordnet.synsets(var)\n",
    "\n",
    "        print('synset names : ', sy.name())\n",
    "        print('\\nsynset defination : ', sy.definition())\n",
    "        print('\\nsynset example : ', sy.examples())\n",
    "        print('\\nsynset tag : ', sy.pos())\n",
    "        print('-----------------------')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\n\\nsynonyms = []\\nantonyms = []\\n\\nfor var in range(len(c)):\\n        \\n        for s in wordnet.synsets(var):\\n            synonyms.append(s.name())\\n            if s.antonyms():\\n                antonyms.append(s.antonyms()[0].name())\\n\\nprint('synonyms : ', synonyms[1:15])\\nprint('\\nantonyms : ', antonyms)\\nprint('\\n')\\n\\n\""
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#list ->not working \n",
    "\n",
    "'''\n",
    "\n",
    "synonyms = []\n",
    "antonyms = []\n",
    "\n",
    "for var in range(len(c)):\n",
    "        \n",
    "        for s in wordnet.synsets(var):\n",
    "            synonyms.append(s.name())\n",
    "            if s.antonyms():\n",
    "                antonyms.append(s.antonyms()[0].name())\n",
    "\n",
    "print('synonyms : ', synonyms[1:15])\n",
    "print('\\nantonyms : ', antonyms)\n",
    "print('\\n')\n",
    "\n",
    "'''\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}