14.MachineLearning_2z5_Regresja.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warsztaty Python w Data Science\n",
    "\n",
    "---\n",
    "\n",
    "## Machine Learning - część 2 z 5. Walidacja krzyżowa. Regresja  \n",
    "\n",
    "- ### Balansowanie próby\n",
    "- ### Walidacja Krzyżowa\n",
    "- ### Inżynieria wymiarów\n",
    "- ### Regresja\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Proces nauczania w Machine Learning\n",
    "\n",
    "1. Przygotowanie danych\n",
    "2. Podział danych\n",
    "3. Budowanie modelu\n",
    "4. Test dokładności"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.kaggle.com/uciml/sms-spam-collection-dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv('data/spam.csv', encoding='ISO-8859-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.rename(columns = {'v1':'class_label', 'v2':'message'}, inplace = True)\n",
    "df.drop(['Unnamed: 2', 'Unnamed: 3', 'Unnamed: 4'], axis = 1, inplace = True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['class_label'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as ab\n",
    "import numpy as np\n",
    "ab.style.use(\"dark_background\")\n",
    "labels = ['ham', 'spam']\n",
    "counts = [4825, 747]\n",
    "ypos = np.arange(len(labels)) #converting text labels to numberic value, 0 and 1\n",
    "ypos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ab.xticks(ypos, labels)\n",
    "ab.xlabel(\"class label\")\n",
    "ab.ylabel(\"Frequency\")\n",
    "ab.title(\"# of spam and ham in dataset\")\n",
    "ab.bar(ypos, counts);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['class_label'] = df['class_label'].apply(lambda x: 1 if x == 'spam' else 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_class_0 = df[df['class_label']==0]\n",
    "df_class_1 = df[df['class_label']==1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_class_0.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_class_1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Robimy OVERSAMPLING (dokładamy do mniejszej klasy powielone wartości)\n",
    "#### moglibyśmy zrobić UNDERSAMPLING (usuwamy z większej klasy)\n",
    "#### albo wogóle dołożyć SYNTENTYCZNE dane (\"sztuczne\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_class_1_over = df_class_1.sample(df_class_0.shape[0], replace=True)\n",
    "df_test_over = pd.concat([df_class_0, df_class_1_over], axis=0)\n",
    "\n",
    "print('Random over-sampling:')\n",
    "print(df_test_over.class_label.value_counts())\n",
    "\n",
    "df_test_over.class_label.value_counts().plot(kind='bar', title=\"# of spam and ham in dataset\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Klasyfikacja niezbalansowanych zbiorów"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(df['message'], df['class_label'], test_size = 0.3, random_state = 0)\n",
    "print('rows in test set: ' + str(x_test.shape))\n",
    "print('rows in train set: ' + str(x_train.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "data = x_train.tolist()\n",
    "\n",
    "vectorizer = TfidfVectorizer(\n",
    "input= data ,  \n",
    "lowercase=True,      \n",
    "stop_words='english' \n",
    ")\n",
    "features_train_transformed = vectorizer.fit_transform(data)  \n",
    "features_test_transformed  = vectorizer.transform(x_test) \n",
    "df_vectorized = pd.DataFrame(features_train_transformed.toarray(), columns = vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "# train the model\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(features_train_transformed, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"classifier accuracy {:.2f}%\".format(classifier.score(features_test_transformed, y_test) * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = classifier.predict(features_test_transformed)\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "actual = y_test.tolist()\n",
    "predicted = labels\n",
    "results = confusion_matrix(actual, predicted)\n",
    "print('Confusion Matrix :')\n",
    "print(results)\n",
    "print ('Accuracy Score :',accuracy_score(actual, predicted))\n",
    "print ('Report : ')\n",
    "print (classification_report(actual, predicted) )\n",
    "score_2 = f1_score(actual, predicted, average = 'binary')\n",
    "print('F-Measure: %.3f' % score_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "group_names = ['True Neg','False Pos','False Neg','True Pos']\n",
    "group_counts = [\"{0:0.0f}\".format(value) for value in\n",
    "                results.flatten()]\n",
    "group_percentages = [\"{0:.2%}\".format(value) for value in\n",
    "                     results.flatten()/np.sum(results)]\n",
    "labels = [f\"{v1}\\n{v2}\\n{v3}\" for v1, v2, v3 in\n",
    "          zip(group_names,group_counts,group_percentages)]\n",
    "labels = np.asarray(labels).reshape(2,2)\n",
    "sns.heatmap(results, annot=labels, fmt='', cmap='Reds');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Klasyfikacja zbalansowanych zbiorów"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(df_test_over['message'], df_test_over['class_label'], test_size = 0.3, random_state = 0)\n",
    "print('rows in test set: ' + str(x_test.shape))\n",
    "print('rows in train set: ' + str(x_train.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "data = x_train.tolist()\n",
    "\n",
    "vectorizer = TfidfVectorizer(\n",
    "input= data ,  \n",
    "lowercase=True,      \n",
    "stop_words='english' \n",
    ")\n",
    "features_train_transformed = vectorizer.fit_transform(data)  \n",
    "features_test_transformed  = vectorizer.transform(x_test) \n",
    "df_vectorized = pd.DataFrame(features_train_transformed.toarray(), columns = vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "# train the model\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(features_train_transformed, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"classifier accuracy {:.2f}%\".format(classifier.score(features_test_transformed, y_test) * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = classifier.predict(features_test_transformed)\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "actual = y_test.tolist()\n",
    "predicted = labels\n",
    "results = confusion_matrix(actual, predicted)\n",
    "print('Confusion Matrix :')\n",
    "print(results)\n",
    "print ('Accuracy Score :',accuracy_score(actual, predicted))\n",
    "print ('Report : ')\n",
    "print (classification_report(actual, predicted) )\n",
    "score_2 = f1_score(actual, predicted, average = 'binary')\n",
    "print('F-Measure: %.3f' % score_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "group_names = ['True Neg','False Pos','False Neg','True Pos']\n",
    "group_counts = [\"{0:0.0f}\".format(value) for value in\n",
    "                results.flatten()]\n",
    "group_percentages = [\"{0:.2%}\".format(value) for value in\n",
    "                     results.flatten()/np.sum(results)]\n",
    "labels = [f\"{v1}\\n{v2}\\n{v3}\" for v1, v2, v3 in\n",
    "          zip(group_names,group_counts,group_percentages)]\n",
    "labels = np.asarray(labels).reshape(2,2)\n",
    "sns.heatmap(results, annot=labels, fmt='', cmap='Reds');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Walidacja krzyżowa\n",
    "\n",
    "![Walidacja krzyżowa](img\\xvi.png)\n",
    "\n",
    "https://scikit-learn.org/stable/modules/cross_validation.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(df_test_over['message'], df_test_over['class_label'], test_size = 0.1, random_state = 0)\n",
    "print('rows in test set: ' + str(x_test.shape))\n",
    "print('rows in train set: ' + str(x_train.shape))\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "data = x_train.tolist()\n",
    "\n",
    "vectorizer = TfidfVectorizer(\n",
    "input= data ,  \n",
    "lowercase=True,      \n",
    "stop_words='english' \n",
    ")\n",
    "features_train_transformed = vectorizer.fit_transform(data)  \n",
    "features_test_transformed  = vectorizer.transform(x_test) \n",
    "df_vectorized = pd.DataFrame(features_train_transformed.toarray(), columns = vectorizer.get_feature_names())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cross_val_score(MultinomialNB(), features_train_transformed, y_train, cv=5)\n",
    "print(list(scores))\n",
    "print()\n",
    "print(\"Accuracy: %0.2f (+/- %0.2f)\" % (scores.mean(), scores.std() * 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Dobór estymatorów\n",
    "\n",
    "![Dobór estymatorów](img\\ml_map.png)\n",
    "\n",
    "https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('data/adverts_29_04.csv', sep=';')\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Feature Engineering\n",
    "\n",
    "## The features you use influence more than everything else the result. \n",
    "## No algorithm alone, to my knowledge, can supplement the information gain given by correct feature engineering.\n",
    "## <div style=\"text-align: right\">— Luca Massaron Autor, Kaggle master</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Coming up with features is difficult, time-consuming, requires expert knowledge.\n",
    "## \"_*Applied machine learning*_\" is basically feature engineering.\n",
    "## <div style=\"text-align: right\">— Andrew Ng</div>\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['cena_za_metr'] = data['Cena'] / data['Wielkość (m2)']\n",
    "data = data.dropna(subset=['cena_za_metr'])\n",
    "df = data.drop(['Cena', 'Data dodania'], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zmienne kategoryczne na indeksy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "labelencoder = LabelEncoder()\n",
    "\n",
    "label_encoded = df\n",
    "\n",
    "label_encoded['Lokalizacja_Cat'] = labelencoder.fit_transform(label_encoded['Lokalizacja'])\n",
    "label_encoded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __*One-hot encoding*__ zmiennych kategorycznych"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "enc = OneHotEncoder(handle_unknown='ignore')\n",
    "\n",
    "enc_df = pd.DataFrame(enc.fit_transform(label_encoded[['Lokalizacja_Cat']]).toarray())\n",
    "\n",
    "one_hot_data = label_encoded.join(enc_df)\n",
    "one_hot_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dum_df = pd.get_dummies(df, columns=['Lokalizacja'])\n",
    "dum_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from numpy import log2\n",
    "\n",
    "data = pd.read_csv('data/adverts_29_04.csv', sep=';')\n",
    "data['cena_za_metr'] = data['Cena'] / data['Wielkość (m2)']\n",
    "data[\"log\"] = data['Wielkość (m2)'].apply(lambda x: log2(x))\n",
    "data[\"clog\"] = data['Cena'].apply(lambda x: log2(x))\n",
    "data = data.dropna(subset=['cena_za_metr'])\n",
    "df = data.drop(['Cena', 'Data dodania'], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dum_df = pd.get_dummies(data, columns=['Lokalizacja', 'Na sprzedaż przez', 'Rodzaj nieruchomości', 'Liczba pokoi', 'Liczba łazienek', 'Parking'])\n",
    "dum_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dum_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dum_df.corr()['cena_za_metr']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.dates as mdates\n",
    "\n",
    "plt.figure(figsize=(20,15))\n",
    "plt.style.use(\"dark_background\")\n",
    "\n",
    "sns.heatmap(dum_df.corr(), cmap=\"seismic\", annot=True, vmin=-1, vmax=1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Regresja"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "y = dum_df['cena_za_metr']\n",
    "X = dum_df.drop(['opis', 'Data dodania', 'Cena', 'cena_za_metr'], axis=1)\n",
    "\n",
    "reg = LinearRegression().fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg.score(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\n",
    "R^2\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://en.wikipedia.org/wiki/Coefficient_of_determination\n",
    "\n",
    "Współczynnik determinacji - Jaka część wariancji zmiennej objaśnianej jest pochodzi od zmiennych tłumaczących\n",
    "\n",
    "- 1.0 - Idealnie dopasowania\n",
    "- 0.0 - Funkcja stała\n",
    "- ... ale może być i ujemna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error, make_scorer\n",
    "\n",
    "\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=17)\n",
    "scores = cross_val_score(LinearRegression(), X_train, y_train, cv=5)\n",
    "print(list(scores))\n",
    "print()\n",
    "print(\"Mean square error: %0.2f (+/- %0.2f)\" % (scores.mean(), scores.std() * 2))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}