Untitled1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "715e467e",
   "metadata": {},
   "source": [
    "# Clustring Articles  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47764c95",
   "metadata": {},
   "source": [
    "The problem that is being addressed here is how to group similar articles\n",
    "in each category into same group without the need of a human reading\n",
    "them, this problem could be classified as an unsupervised problem and\n",
    "also could be classified as an NLP problem because the data that's being\n",
    "used is text data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fffddf82",
   "metadata": {},
   "source": [
    "<h3>Importing the libs and the data </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "e67f0dad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 2461 entries, 0 to 2480\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   body      2461 non-null   object\n",
      " 1   title     2461 non-null   object\n",
      " 2   category  2461 non-null   object\n",
      "dtypes: object(3)\n",
      "memory usage: 76.9+ KB\n"
     ]
    }
   ],
   "source": [
    "#os and random help you define a random seed to make the code deterministically reproducible.\n",
    "import os\n",
    "import random\n",
    "# provides you with linear algebra utilities you'll use to evaluate results. Also, it's used for setting a random seed to make the code deterministically reproducible.\n",
    "import numpy as np\n",
    "#pandas helps you read the data.\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from itertools import cycle\n",
    "import math\n",
    "from time import time\n",
    "from IPython.display import display # Allows the use of display() for DataFrames\n",
    "import nltk\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk.stem import  WordNetLemmatizer  \n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "import string\n",
    "from sklearn.preprocessing import normalize\n",
    "from sklearn.decomposition import TruncatedSVD,PCA\n",
    "from sklearn.cluster import Birch\n",
    "from sklearn.cluster import KMeans,AffinityPropagation,AgglomerativeClustering\n",
    "from sklearn.metrics import silhouette_score\n",
    "import json\n",
    "SEED = 42\n",
    "random.seed(SEED)\n",
    "os.environ[\"PYTHONHASHSEED\"] = str(SEED)\n",
    "np.random.seed(SEED)\n",
    "# Pretty display for notebooks\n",
    "%matplotlib inline\n",
    "# Load the dataset\n",
    "data = pd.read_json(\"data.json\")\n",
    "data.drop_duplicates(inplace=True)\n",
    "data.head(n=5)\n",
    "#2461 articles \n",
    "data.info()\n",
    "#there are only three main categories\n",
    "#print(len(data[data.category == 'Engineering'])+len(data[data.category == 'Startups & Business'])+len(data[data.category == 'Product & Design']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c239b76c",
   "metadata": {},
   "source": [
    "<h3>Dividing the data</h3>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7221920a",
   "metadata": {},
   "source": [
    "as seen from above thre are three main categories in that dataset and the task is to divide each of them to different clusters, and to make it easier for the models to do the clustring: I divide the dataset into three different data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e06ba714",
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped = data.groupby(data.category)\n",
    "eng_df=grouped.get_group(\"Engineering\")\n",
    "sb_df=grouped.get_group(\"Startups & Business\")\n",
    "pd_df=grouped.get_group(\"Product & Design\")\n",
    "#print(len(eng_df))\n",
    "df_list=[eng_df,sb_df,pd_df]\n",
    "#print(len(df_list[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfd0d0b5",
   "metadata": {},
   "source": [
    "# Processing The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e4640cd",
   "metadata": {},
   "source": [
    "Because the data that's being used in this problem is unstructured text\n",
    "data, the Pre-processing step was the most important and the biggest\n",
    "step. <br>\n",
    "In that step I lowered all the letters, deleted all the digits and the stop\n",
    "words from the text, I also lemmatized all the words in the text, then\n",
    "after that the text was fed into a Tf-idf Vectorizer to vectorize the text\n",
    "and to calculate the overall texts weightage of the words, and finally the\n",
    "output of the Tf-idf vectorizer will be fed to a dimensionality reduction\n",
    "model called Truncated Singular Value Decomposition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8c529679",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    " def proc(data):\n",
    "    articles = data['body'].values.astype(str)\n",
    "    #tokenize every article\n",
    "    #tokens: list of lists of strings\n",
    "    tokens = [word_tokenize(article) for article in articles]\n",
    "    #print(tokens[0])\n",
    "    lower_tokens = []\n",
    "    for token in tokens:\n",
    "        lower_tokens .append([t.lower() for t in token])\n",
    "    # Retain alphabetic words: alpha_only\n",
    "    alpha_only=[]\n",
    "    for lower_token in lower_tokens:\n",
    "        alpha_only .append( [t for t in lower_token if t.isalpha()])\n",
    "\n",
    "    # Remove all stop words: no_stops\n",
    "    stops=stopwords.words('english')\n",
    "    #Remove all the punctuations\n",
    "    stops+=list(string.punctuation)\n",
    "    #more words that doesn't matter to the clustring process\n",
    "    stops+=['also','tls','one','two','three','four','five','six','seven','eight','nine','ten','first','second','third','fourth']\n",
    "    no_stops=[]\n",
    "    for a in alpha_only:\n",
    "        no_stops .append( [t for t in a if t not in stops])\n",
    "\n",
    "    # Instantiate the WordNetLemmatizer\n",
    "    wordnet_lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "    # Lemmatize all tokens into a new list: lemmatized\n",
    "    lemmatized=[]\n",
    "    for n in no_stops:\n",
    "         lemmatized .append([wordnet_lemmatizer.lemmatize(t) for t in n])\n",
    "    #because vectorizer.fit_transform takes only a list of strings not and lemmatized is a list of lists of strings, we need to transform it by joining the words inside each list into one string\n",
    "    new_doc=[]\n",
    "    for text in lemmatized:\n",
    "        new_doc.append(' '.join(text))\n",
    "    vectorizer = TfidfVectorizer()\n",
    "    features = vectorizer.fit_transform(new_doc)\n",
    "    #tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2, max_features=1000)\n",
    "    #tf = tf_vectorizer.fit_transform(features)   \n",
    "    new_features = normalize(features)\n",
    "    svd_model = TruncatedSVD(n_components=50, algorithm='randomized',random_state=42)\n",
    "    svd_features = svd_model.fit_transform(new_features)\n",
    "    return svd_features \n",
    "features_list=[]\n",
    "for df in df_list:\n",
    "    #print(len(df))\n",
    "    features_list.append(proc(df))\n",
    "#print(features_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87da7776",
   "metadata": {},
   "source": [
    "# Clustring"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a9e2311",
   "metadata": {},
   "source": [
    "In this stage we're going to use the output from the last stage to be an input to three clustring algorithems and compare thier output to define the best one to be used. \n",
    "\n",
    "useful links:\n",
    "\n",
    "https://scikit-learn.org/stable/auto_examples/cluster/plot_cluster_comparison.html#sphx-glr-auto-examples-cluster-plot-cluster-comparison-py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "172e94dd",
   "metadata": {},
   "source": [
    "<h2>K-means</h2>\n",
    "\n",
    "K-means clustering is one of the simplest and popular unsupervised machine learning algorithms, And it is an extensively used technique for data cluster analysis.<br>\n",
    "The K-means algorithm identifies k number of centroids, and then allocates every data point to the nearest cluster, while keeping the centroids as small as possible.<br>\n",
    "\n",
    "For more check out that link: <a>https://towardsdatascience.com/understanding-k-means-clustering-in-machine-learning-6a6e67336aa1 </a> <br>\n",
    "\n",
    "since we don't know what is the best K value to be used, I used I function to exprement with multible values for K and choose the best one accourding to thier Silhouette Score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "d6745139",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_4212/1012318830.py:14: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['KMeans_cluster'] =model.labels_\n",
      "/tmp/ipykernel_4212/1012318830.py:14: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['KMeans_cluster'] =model.labels_\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[88, 91, 149] [0.18666937548032633, 0.1432119056139904, 0.11811449353291605]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_4212/1012318830.py:14: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['KMeans_cluster'] =model.labels_\n"
     ]
    }
   ],
   "source": [
    "def best_k(features,max_k,df):\n",
    "    best_score=-math.inf\n",
    "    bk=2\n",
    "    for k in range(2,max_k+1):\n",
    "        model = KMeans(n_clusters=k, init='k-means++', max_iter=1000000, n_init=1,random_state = 42)#AgglomerativeClustering(n_clusters=k, affinity = \"euclidean\")#Birch(n_clusters = k)#\n",
    "        model.fit(features )\n",
    "        label=model.labels_\n",
    "        s=silhouette_score(features, label)\n",
    "        if s>best_score:\n",
    "            best_score=s\n",
    "            bk=k\n",
    "    model = KMeans(n_clusters=bk, init='k-means++', max_iter=1000000, n_init=1,random_state = 42)\n",
    "    model.fit(features)\n",
    "    df['KMeans_cluster'] =model.labels_\n",
    "    return bk,best_score\n",
    "bk=[0,0,0]\n",
    "best_kmeans_score=[-1,-1,-1]\n",
    "\n",
    "bk[0],best_kmeans_score[0]=best_k(features_list[0],150,eng_df)\n",
    "\n",
    "\"\"\"df_list[0]['KMeans_cluster'] =model.labels_\n",
    "p=df_list[0]['KMeans_cluster'].value_counts().idxmax()\n",
    "print(len(df_list[0][df_list[0].KMeans_cluster == p]))\n",
    "\"\"\"\n",
    "\n",
    "bk[1],best_kmeans_score[1]=best_k(features_list[1],150,sb_df)\n",
    "\n",
    "bk[2],best_kmeans_score[2]=best_k(features_list[2],150,pd_df)\n",
    "\n",
    "print(bk,best_kmeans_score)\n",
    "#eng_df.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "98c5d9cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'#Load Data\\n\\npca = PCA(2)\\n \\n#Transform the data\\nX = pca.fit_transform(features_list[0])\\n\\n\\nms = KMeans(n_clusters=bk[0], init=\\'k-means++\\', max_iter=1000000, n_init=1,random_state = 42)\\nms.fit(X)\\nlabels = ms.labels_\\ncluster_centers = ms.cluster_centers_\\n\\nlabels_unique = np.unique(labels)\\nn_clusters_ = len(labels_unique)\\n\\ndf_list[0][\\'KMeans_cluster\\'] =ms.labels_\\np=df_list[0][\\'KMeans_cluster\\'].value_counts().idxmax()\\nprint(len(df_list[0][df_list[0].KMeans_cluster == p]))\\n\\nprint(\"number of estimated clusters : %d\" % n_clusters_)\\n\\n\\nplt.figure(1)\\nplt.clf()\\n\\ncolors = cycle(\"bgrcmykbgrcmykbgrcmykbgrcmyk\")\\nfor k, col in zip(range(n_clusters_), colors):\\n    my_members = labels == k\\n    cluster_center = cluster_centers[k]\\n    plt.plot(X[my_members, 0], X[my_members, 1], col + \".\")\\n    plt.plot(\\n        cluster_center[0],\\n        cluster_center[1],\\n        \"o\",\\n        markerfacecolor=col,\\n        markeredgecolor=\"k\",\\n        markersize=4,\\n    )\\nplt.title(\"Estimated number of clusters: %d\" % n_clusters_)\\nplt.show()'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"#Load Data\n",
    "\n",
    "pca = PCA(2)\n",
    " \n",
    "#Transform the data\n",
    "X = pca.fit_transform(features_list[0])\n",
    "\n",
    "\n",
    "ms = KMeans(n_clusters=bk[0], init='k-means++', max_iter=1000000, n_init=1,random_state = 42)\n",
    "ms.fit(X)\n",
    "labels = ms.labels_\n",
    "cluster_centers = ms.cluster_centers_\n",
    "\n",
    "labels_unique = np.unique(labels)\n",
    "n_clusters_ = len(labels_unique)\n",
    "\n",
    "df_list[0]['KMeans_cluster'] =ms.labels_\n",
    "p=df_list[0]['KMeans_cluster'].value_counts().idxmax()\n",
    "print(len(df_list[0][df_list[0].KMeans_cluster == p]))\n",
    "\n",
    "print(\"number of estimated clusters : %d\" % n_clusters_)\n",
    "\n",
    "\n",
    "plt.figure(1)\n",
    "plt.clf()\n",
    "\n",
    "colors = cycle(\"bgrcmykbgrcmykbgrcmykbgrcmyk\")\n",
    "for k, col in zip(range(n_clusters_), colors):\n",
    "    my_members = labels == k\n",
    "    cluster_center = cluster_centers[k]\n",
    "    plt.plot(X[my_members, 0], X[my_members, 1], col + \".\")\n",
    "    plt.plot(\n",
    "        cluster_center[0],\n",
    "        cluster_center[1],\n",
    "        \"o\",\n",
    "        markerfacecolor=col,\n",
    "        markeredgecolor=\"k\",\n",
    "        markersize=4,\n",
    "    )\n",
    "plt.title(\"Estimated number of clusters: %d\" % n_clusters_)\n",
    "plt.show()\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a3f92a5",
   "metadata": {},
   "source": [
    "# Affinity Propagation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0190c677",
   "metadata": {},
   "source": [
    "Affinity Propagation creates clusters by sending messages between data points until convergence. Unlike clustering algorithms such as k-means or k-medoids, affinity propagation does not require the number of clusters to be determined or estimated before running the algorithm, for this purpose the two important parameters are the preference, which controls how many exemplars (or prototypes) are used, and the damping factor which damps the responsibility and availability of messages to avoid numerical oscillations when updating these messages.\n",
    "\n",
    "check out:<br>\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AffinityPropagation.html <br>\n",
    " https://scikit-learn.org/stable/modules/clustering.html#affinity-propagation\n",
    " <br>\n",
    " https://www.geeksforgeeks.org/affinity-propagation-in-ml-to-find-the-number-of-clusters/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "fcb03996",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_4212/1282043271.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['AffinityPropagation_cluster'] =labels\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.19105164617974554, 71, 59)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/aml/.local/lib/python3.8/site-packages/sklearn/cluster/_affinity_propagation.py:236: ConvergenceWarning: Affinity propagation did not converge, this model may return degenerate cluster centers and labels.\n",
      "  warnings.warn(\n",
      "/tmp/ipykernel_4212/1282043271.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['AffinityPropagation_cluster'] =labels\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.15462605145758643, 164, 53)\n",
      "(0.12499697511852463, 103, 32)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/aml/.local/lib/python3.8/site-packages/sklearn/cluster/_affinity_propagation.py:236: ConvergenceWarning: Affinity propagation did not converge, this model may return degenerate cluster centers and labels.\n",
      "  warnings.warn(\n",
      "/tmp/ipykernel_4212/1282043271.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['AffinityPropagation_cluster'] =labels\n"
     ]
    }
   ],
   "source": [
    "def AP(X,df,preference):\n",
    "    model =AffinityPropagation(preference =preference,random_state=42)\n",
    "    model.fit(X)\n",
    "    labels = model.labels_\n",
    "    s=silhouette_score(X, labels)\n",
    "    #print(s)\n",
    "    labels_unique = np.unique(labels)\n",
    "    n_clusters_ = len(labels_unique)\n",
    "    df['AffinityPropagation_cluster'] =labels\n",
    "    biggest_cluster=df['AffinityPropagation_cluster'].value_counts().idxmax()\n",
    "    #print(n_clusters_,len(df[df.AffinityPropagation_cluster == biggest_cluster]))\n",
    "    return s,n_clusters_,len(df[df.AffinityPropagation_cluster == biggest_cluster])\n",
    "#finding the best preference to be used\n",
    "\n",
    "#best preference for df_list[0]=-.4\n",
    "#best preference for df_list[1]=-.2\n",
    "#best preference for df_list[2]=-.3\n",
    "\"\"\"bp=0\n",
    "nc=0\n",
    "bs=0\n",
    "bb=0\n",
    "for i in range(1,100):\n",
    "    i=(i*-1)/10\n",
    "    s,c,b=AP(features_list[0],df_list[0],i)\n",
    "    if s>bs:\n",
    "        bs=s\n",
    "        bp=i\n",
    "        nc=c\n",
    "        bb=b\n",
    "print(bs,bp,nc,bb)\"\"\"\n",
    "\"\"\"pca = PCA(2)\n",
    " \n",
    "#Transform the data\n",
    "X = pca.fit_transform(features_list[0])\n",
    "\n",
    "\n",
    "ms = AffinityPropagation(preference =-.1,random_state=42)\n",
    "ms.fit(X)\n",
    "labels = ms.labels_\n",
    "\n",
    "labels_unique = np.unique(labels)\n",
    "n_clusters_ = len(labels_unique)\n",
    "\n",
    "df_list[0]['AffinityPropagation_cluster'] =ms.labels_\n",
    "p=df_list[0]['AffinityPropagation_cluster'].value_counts().idxmax()\n",
    "print(len(df_list[0][df_list[0].AffinityPropagation_cluster == p]))\n",
    "\n",
    "print(\"number of estimated clusters : %d\" % n_clusters_)\n",
    "\n",
    "\n",
    "plt.figure(1)\n",
    "plt.clf()\n",
    "\n",
    "colors = cycle(\"bgrcmykbgrcmykbgrcmykbgrcmyk\")\n",
    "for k, col in zip(range(n_clusters_), colors):\n",
    "    my_members = labels == k\n",
    "    plt.plot(X[my_members, 0], X[my_members, 1],col + \".\")\n",
    "\n",
    "plt.title(\"Estimated number of clusters: %d\" % n_clusters_)\n",
    "plt.show()\"\"\"\n",
    "\n",
    "\n",
    "print(AP(features_list[0],df_list[0],-.41))\n",
    "print(AP(features_list[1],df_list[1],-.22))\n",
    "print(AP(features_list[2],df_list[2],-.3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "959c420b",
   "metadata": {},
   "source": [
    "# Agglomerative Clustering.\n",
    "\n",
    "Agglomerative Clustering is the most common type of hierarchical clustering used to group objects in clusters based on their similarity.This algorithm starts by treating each object as a singleton cluster. Next, pairs of clusters are successively merged until all clusters have been merged into one big cluster containing all objects. The result is a tree-based representation of the objects\n",
    "<br>\n",
    "check out:<br>\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering<br>\n",
    "https://www.datanovia.com/en/lessons/agglomerative-hierarchical-clustering/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "19804c98",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_4212/1491966588.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df[\"Agglomerative_clusters\"] =model.labels_\n",
      "/tmp/ipykernel_4212/1491966588.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df[\"Agglomerative_clusters\"] =model.labels_\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.16517158636304818\n",
      "0.13342300808901814\n",
      "0.11921261313009135\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_4212/1491966588.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df[\"Agglomerative_clusters\"] =model.labels_\n"
     ]
    }
   ],
   "source": [
    "def Agg(X,df, n_clusters):\n",
    "    model=  AgglomerativeClustering( n_clusters= n_clusters, linkage=\"ward\")\n",
    "    model.fit(X)\n",
    "    labels = model.labels_\n",
    "    s=silhouette_score(X, labels)\n",
    "    #print(s)\n",
    "    labels_unique = np.unique(labels)\n",
    "    n_clusters_ = len(labels_unique)\n",
    "    df[\"Agglomerative_clusters\"] =model.labels_\n",
    "    biggest_cluster=df[\"Agglomerative_clusters\"].value_counts().idxmax()\n",
    "    p=len(df[df.Agglomerative_clusters == biggest_cluster])\n",
    "    #print(\"number of estimated clusters : %d\" % n_clusters_)\n",
    "    return s#,p, n_clusters_\n",
    "#finding the best n_clusters\n",
    "\"\"\"\n",
    "best_score=0\n",
    "best_n_clusters=0\n",
    "bp=0\n",
    "bn=0\n",
    "#for eng 45\n",
    "#for bs 81\n",
    "#for pd 73\n",
    "for i in range(5,100):\n",
    "    s,p,n=Agg(features_list[0],df_list[0],i)\n",
    "    if (s-best_score)>=.01:\n",
    "        best_score=s\n",
    "        best_n_clusters=i\n",
    "        bp=p\n",
    "        bn=n\n",
    "print(best_n_clusters,best_score,bp)\n",
    "\"\"\"\"\"\"pca = PCA(4)\n",
    " \n",
    "#Transform the data\n",
    "X = pca.fit_transform(features_list[0])\n",
    "\n",
    "\n",
    "ms = cluster.AgglomerativeClustering( n_clusters= 30, linkage=\"ward\")\n",
    "ms.fit(X)\n",
    "labels = ms.labels_\n",
    "\n",
    "labels_unique = np.unique(labels)\n",
    "n_clusters_ = len(labels_unique)\n",
    "\n",
    "df_list[0]['KMeans_cluster'] =ms.labels_\n",
    "p=df_list[0]['KMeans_cluster'].value_counts().idxmax()\n",
    "print(len(df_list[0][df_list[0].KMeans_cluster == p]))\n",
    "\n",
    "print(\"number of estimated clusters : %d\" % n_clusters_)\n",
    "\n",
    "\n",
    "plt.figure(1)\n",
    "plt.clf()\n",
    "\n",
    "colors = cycle(\"bgrcmykbgrcmykbgrcmykbgrcmyk\")\n",
    "for k, col in zip(range(n_clusters_), colors):\n",
    "    my_members = labels == k\n",
    "    plt.plot(X[my_members, 0], X[my_members, 1] ,col + \".\")\n",
    "\n",
    "plt.title(\"Estimated number of clusters: %d\" % n_clusters_)\n",
    "plt.show()\"\"\"\n",
    "\n",
    "print(Agg(features_list[0],df_list[0],37))\n",
    "print(Agg(features_list[1],df_list[1],84))\n",
    "print(Agg(features_list[2],df_list[2],72))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04ccbdc7",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "\n",
    "\n",
    "The metric that I chose to evaluate the three modelling Algorithms is silhouette score,  It's one of the most popular metrics to calculate the goodness of a clustering technique. Its value ranges from -1 to 1\n",
    "\n",
    "1: This Means clusters are well apart from each other and clearly distinguished.\n",
    "\n",
    "0: This means clusters are indifferent, or we can say that the distance between clusters is not significant.\n",
    "\n",
    "-1: This means clusters are assigned in the wrong way.\n",
    "\n",
    "Silhouette Score = (b-a)/max(a,b)\n",
    "\n",
    "where\n",
    "\n",
    "a= average intra-cluster distance i.e the average distance between each point within a cluster.\n",
    "\n",
    "b= average inter-cluster distance i.e the average distance between all clusters.\n",
    "\n",
    "--> I have calculated the silhouette score and came to the conclusion that the best algorithm to cluster the three categories is Affinity Propagation.<--\n",
    "\n",
    "\n",
    "for more check out those links:\n",
    "<br>\n",
    "https://towardsdatascience.com/silhouette-coefficient-validating-clustering-techniques-e976bb81d10c\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "71fa3da0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#writing in the Json file\n",
    "#writing in the Json file\n",
    "def wd(df,cluster,category):\n",
    "        d=dict()\n",
    "        d[\"Category\"]=category\n",
    "        for c,body in zip(df[cluster],df.body):\n",
    "            s=\"Group \"+str(c)\n",
    "            if s in d:\n",
    "                d[s].append(body)\n",
    "            else:\n",
    "                d[s]=[]\n",
    "                d[s].append(body)\n",
    "        return d\n",
    "d1=wd(df_list[0],\"AffinityPropagation_cluster\",\"Engineering\")\n",
    "d2=wd(df_list[0],\"AffinityPropagation_cluster\",\"Startups & Business\")\n",
    "d3=wd(df_list[0],\"AffinityPropagation_cluster\",\"Product & Design\")\n",
    "\n",
    "l=[d1,d2,d3]\n",
    "jsonString = json.dumps(l)\n",
    "jsonFile = open(\"output\", \"w\")\n",
    "jsonFile.write(jsonString)\n",
    "jsonFile.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}