Modified Sample Models for Reference with Blockchain.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lkKG1lECduU2"
   },
   "source": [
    "# Outline of Sample Models with Blockchain\n",
    "\n",
    "In this project, some sample AI models and blockchain algorithms will be displayed to guide further research and modeling work. Given the two goals we had: find the source of financial and medical resources waste & build AI models to reduce financial and medcial waste in order to reduce the healthcare cost as well as the insurance premium cost, We have already figured out the source of financial and medical resources waste:\n",
    "\n",
    "- **Insured persons' part**\n",
    "  - ***Medical waste actions***.  Given the insurance company will pay part or even most of bills, insured people will unavoidably intend to look for unnecessarily more expensive treatment or even just unnecessary treatment to take advantages of insurance company, which will cause the waste of financial and medical resources.\n",
    "  - ***Fraud actions***. A more serious situation is that some insured persons will fake medical records to defraud insurance companies. Because all insured people share the same insurance fund, no matter if they are honest or not, and the high usage of the fund will lead to high insurance premium cost, so the fraud actions have to be controled to stop this kind of financial waste.\n",
    "  - ***Impersonation actions***. Not all people are insured, but all people will take medical cares to some extent. Thus, there are some people will take impersonation actions to lower their medical cost. This behavior will waste insurance funds that do not originally intend to be used on them, and increase the insurance premium the policyholders paid.\n",
    "\n",
    "- **Doctors' part**\n",
    "  - ***Substance abuse***. When cooperating with insurance company, patients' bills are paid by both patient and insurance company. Both of them are responsible for part of the bills, and the whole budget will be increased in a certain degree, so a few doctors will intend to do subtance abuse to increase their income. However, even if there's no insurance company's involvement, the substance abuse is inevitable sometimes. The substance abuse is one of the source of financial and medical resources waste which makes the insurance premium cost high.\n",
    "\n",
    "\n",
    "- **Insurance companies' part**\n",
    "  - ***Inaccurate premium pricing model***. Insurance companies have their own pricing models to decide the insurance premium cost. These models are designed to make sure there's profit for insurance company after paying all medical fees for their insured clients. In order to ensure that, there will be set as much margin space as possible between the premium collected from policyholders and the payment to doctors. More accurate pricing models will make the margin smaller, which won't collect financail resources wastely anymore, and makes the financial resources freely flow to whole society where needs it more.\n",
    "\n",
    "\n",
    "And have already listed all data, algorithms, inouts and outputs shown below:\n",
    "\n",
    " - ***A. AI models to monitor medical waste actions***\n",
    "    - **a. Data needed**: \\\n",
    "      Insured people's basic numeric information collected when signing insurance contract like age(0~150,Integer), income(Numeric, Unit:k), insured people size(Integer), medical test scores(Numeric), credit scores(Numeric), etc.\n",
    "    - **b. AI algorithm candidates**:\\\n",
    "      Principal component analysis(PCA), KMeans, Support Vector Classifier (SVC), K-Nearest Neighbors(KNN), Naive Bayes, XGBoost, RandomForest\n",
    "    - **c. Input**: \\\n",
    "      Insured people's basic numeric information collected when signing insurance contract (Numeric)\n",
    "    - **d. Output**:\\\n",
    "      0~1, the probability of taking medical waste action\n",
    "   \n",
    "      \n",
    "  - ***B. AI models to monitor fraud actions***\n",
    "    - **a. Data needed**:\\\n",
    "      Doctors' or experts' tagged honest and fraud medical records' database (data series with tag 0 or 1)\n",
    "    - **b. AI algorithm candidates**:\\\n",
    "      Bidirectional Encoder Representations from Transformers (BERT), Convolutional Neural Network(CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM)\n",
    "    - **c. Input**:\\\n",
    "      Insured people's historical medical records and new medical records (Images, text, and numbers data series without tag)\n",
    "    - **d. Output**:\\\n",
    "      0~1, the probability of defrauding\\\n",
    "\n",
    "\n",
    "  - ***C. AI models to monitor impersonation actions***\n",
    "    - **a. Data needed**:\\\n",
    "      Insured people's bio information such as photo or fingerprint(Image), and database that includes large amount comparable bio photos marked 0 and 1 (Image data with tag 0 or 1).\n",
    "    - **b. AI algorithm candidates**:\\\n",
    "      Principal component analysis(PCA), Convolutional Neural Network(CNN)\n",
    "    - **c. Input**:\\\n",
    "      Real-time face recognition video taken when entering insurance information (Image)\n",
    "    - **d. Output**:\\\n",
    "      0 or 1, stands for it's the exact insured person or not\n",
    "\n",
    "  - ***D. AI models to monitor substance abuse***\n",
    "    - **a. Data needed**: \\\n",
    "      Doctors' or experts' tagged abuse or not medical records' database (text data series with tag 0 or 1).\n",
    "    - **b. AI algorithm candidates**:\\\n",
    "      Bidirectional Encoder Representations from Transformers (BERT), Convolutional Neural Network(CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM)\n",
    "    - **c. Input**:\\\n",
    "      Doctors' medical records on one certain insured person (text data series without tag)\n",
    "    - **d. Output**:\\\n",
    "      0~1, the probability of substance abusing\n",
    "\n",
    "\n",
    "  - ***E. AI models to build more accurate premium pricing model***\n",
    "    - **a. Data needed**:\\\n",
    "      History database of insured people's basic information(Yearly, Numeric), all scores above AI models generated(Numeric), usage of insurance fund(Yearly, Numeric), and economic data like inflation rate etc. (Numeric).\n",
    "    - **b. AI algorithm candidates**:\\\n",
    "      Principal component analysis(PCA), K-Nearest Neighbors(KNN), Support Vector Regression (SVR), RidgeRegression, XGBoost, RandomForest\n",
    "    - **c. Input**:\\\n",
    "      Insured people's basic information (Numeric), scores generated from above AI models (Numeric), and economic data (Numeric)\n",
    "    - **d. Output**:\\\n",
    "      Expectaion of the cost of certain type insured people (Yearly, Numeric)\n",
    "      \n",
    "      \n",
    "And we also stressed the importance of Blockchain's application in medical industry, which could give patients and other data subjects full ownership and use rights of data, greatly improving the privacy and security of data, and at the same time providing safe and unimpeded communication between data owners and data recipients, which is beneficial to all participants.\n",
    "\n",
    "\n",
    "This file will show details of the sample AI models and blockchain algorithms to give a hint for all other participants."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nXXJn7-mfk0a"
   },
   "source": [
    "## AI models to monitor medical waste actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nl9c0QQ4fqhP"
   },
   "source": [
    "### Data Generated\n",
    "\n",
    "Insured people's basic numeric information collected when signing insurance contract like age(0~150,Integer), income(Numeric, Unit:k), insured people size(Integer), medical test scores(Numeric), credit scores(Numeric), etc. Given all data are numeric, and the ranges of different data are different, use the pipeline function to scale all data to make algorithms working better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SdsZeg83ffQs"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# Split Data\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "# X is the data of all insured people's numeric information we collected\n",
    "# Y is the target which stands for whether the insured people has had medical waste acions before\n",
    "# Which is shown by 0 and 1\n",
    "\n",
    "# Scale Data\n",
    "\n",
    "pipe = Pipeline([('scaler', StandardScaler()),...])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_IgQZD-ViyXR"
   },
   "source": [
    "After doing the Scaler pipeline, there are too many parameters in our models given many X variables, thus, we could use Principal component analysis(PCA) algorithms to make the variables more concise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3D6rrgENjKgK"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Fit PCA\n",
    "\n",
    "pipe = Pipeline([('scaler', StandardScaler()),('PCA',PCA(n_components=10, svd_solver='full'))])\n",
    "\n",
    "# Where n_components could be any numbers we want to keep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_XxtV1tViWvV"
   },
   "source": [
    "By now, the data has been splited and processed well to be fitted in proper algorithms, with the suitable range and suitable dimensions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7IyQOUHAigR-"
   },
   "source": [
    "### Algorithms 1 - KMeans\n",
    "\n",
    "The target Y is not always easy to get, and the values the insurance company collected is not always accurate. Thus, we could use the KMeans algorithms first, to observe the insured people roughly, to check if the target Y is accurate enough. If the results shows most of main clusters' components have the same target value, the data we collected and the target we tagged are accurate and credible to do the following models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jg9qvbI1l2sc"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "import numpy as np\n",
    "\n",
    "# Fit KMeans\n",
    "\n",
    "kmeans = KMeans(n_clusters=2, random_state=0, n_init=\"auto\").fit(X)\n",
    "# There's only 0 and 1 in Y.\n",
    "# Set 2 clusters to fit.\n",
    "\n",
    "# Check if the data and target values are accurate\n",
    "\n",
    "Y_bar = kmeans.labels_\n",
    "Y_test = Y - Y_bar\n",
    "p = (Y_test == 0).sum()/len(Y_test)\n",
    "# Higher p means more accurate data.\n",
    "# If the p is higher than 0.8, we could do the following modeling work.\n",
    "# Otherwise, we should improve the accuracy of data and target."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F6QtfDT_np02"
   },
   "source": [
    "### Alogorithms 2 - Support Vector Classifier (SVC)\n",
    "\n",
    "After feature engineering, and accuracy check, we could do the machine learning, or to say AI work, to predict wheather the insured peoson will do medical waste actions or not.\n",
    "\n",
    "Support Vector Classifier (SVC) is an advanced model to do the classification work. Samples are data points allocate in the P-dimension space, each axis represents one feature. The idea of the Support Vector Classifier is to find the \"hyperplane\" to separate samples into 2 classes in this P-dimension space. In a 2-D space, when we only have 2 features, SVC is to find the straight line that can separate samples.\n",
    "\n",
    "Of course, different people will draw different straight lines. However, no matter how you draw the line between these 2 classes, you will always find a closest sample from each class to your straight line. That is the SUPPORT VECTOR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6dkHjkHHoOsT"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "pipe_1 = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                  ('svc', SVC(gamma='auto'))])\n",
    "pipe_1.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "pipe_1_gridsearch=GridSearchCV(estimator = pipe_1,\n",
    "                        param_grid = {'C': [1, 10], 'kernel': ('linear', 'rbf')},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "pipe_1_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Which should produce higher test score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(pipe_1_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5wE_EMFYsN6P"
   },
   "source": [
    "### Algorithms 3 - K-Nearest Neighbors(KNN)\n",
    "\n",
    "Besides SVC, KNN also falls in the supervised learning algorithms. In the classification problem, the K-nearest neighbor algorithm essentially said that for a given value of K algorithm will find the K nearest neighbor of unseen data point and then it will assign the class to unseen data point by having the class which has the highest number of data points out of all classes of K neighbors. For distance metrics, we will use the Euclidean metric. Finally, the input x gets assigned to the class with the largest probability.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fOhPqMoztUra"
   },
   "outputs": [],
   "source": [
    "# Standard KNeighborsClassifier Import\n",
    "\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "pipe_2 = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                  ('KNN', KNeighborsClassifier(n_neighbors=3))])\n",
    "pipe_2.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "pipe_2_gridsearch=GridSearchCV(estimator = pipe_2,\n",
    "                        param_grid = {'n_neighbors': [1, 10],\n",
    "                        'algorithm': ('auto','ball_tree', 'kd_tree', 'brute')},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "pipe_2_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Which should produce higher test score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "scores = cross_val_score(pipe_2_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2u46gW2NuUZB"
   },
   "source": [
    "### Algorithms 4 - Naive Bayes\n",
    "\n",
    "Naive Bayes is another classification technique that is based on Bayes’ Theorem with an assumption that all the features that predicts the target value are independent of each other. It calculates the probability of each class and then pick the one with the highest probability.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vzgtndWEusNi"
   },
   "outputs": [],
   "source": [
    "# Standard Naive Bayes Classifier Import\n",
    "\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "pipe_3 = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                   ('NB', GaussianNB())])\n",
    "pipe_3.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "scores = cross_val_score(pipe_3, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Algorithms 5 - XGBoost\n",
    "\n",
    "XGBoost algorithms could perform extreme level improvements on multiple weak learners and integrate them into a strong learner which could run at high speed and low memory without overfitting and under-fitting problems. At the same time, the algorithm is stable and has high accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard XGBoost Import\n",
    "\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "pipe_4 = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                   ('XGBoost', GradientBoostingClassifier(n_estimators=100, learning_rate=1.0,\n",
    "     max_depth=1, random_state=0))])\n",
    "\n",
    "pipe_4.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "pipe_4_gridsearch=GridSearchCV(estimator = pipe_4,\n",
    "                        param_grid = {'n_estimators': [100,200,300,500,1000],\n",
    "                        'max_depth': [1,3,4,5,10,15,20],\n",
    "                        'learning_rate':[0.01,0.05,0.1,0.3,0.5,1.0,5.0]},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "pipe_4_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Which should produce higher test score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(pipe_4_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Algorithms 6 - RandomForest\n",
    "\n",
    "RandomForest algorithm is a tree algorithms structure, which is a collection of different decision trees obtained from the sample. Yes or no judgment is made on each node by Gini Index. After continuous iteration, the leaf nodes are the final decision results. This algotihm can explain the process from sample data to results, which can be used to explain patients’ behavior. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard RandomForest Classifier Import\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "pipe_5 = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                   ('RandomForest', RandomForestClassifier(max_depth=10, random_state=0))])\n",
    "\n",
    "pipe_5.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# RandomForest algorithm has already did 'cross validation' inside, no more needed additional test\n",
    "# RandomForest algorithm is time-consuming, if the test score is much lower than XGBoost, then stop fine tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iaATrZ8-gozt"
   },
   "source": [
    "## AI models to monitor fraud actions\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HROO1qQ2gyaF"
   },
   "source": [
    "### Data Generated\n",
    "\n",
    "As discussed above, in some more serious situation, some insured persons will fake medical records to defraud insurance companies to get more money from insurance companies which are not belong to them and not in the insurance company's budget. So we should have a data base which includes doctors' or experts' tagged honest and fraud medical records' database (data series with tag 0 or 1). The data from data base is images, text, or numbers data with tags, which could teach AI models to classify data without any tag.\n",
    "\n",
    "But before we do any modeling things, we should firstly unify data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ro2IxoSmh-AU"
   },
   "source": [
    "#### Numeric Data Engineering\n",
    "\n",
    "This processing is very similar as previous numeric data engineering, which only need to be scaled,or sometimes with PCA processing to reduce the data's dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A8nC1D64ikPs"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Split Data\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "# X is the data of all numeric data we created in data base\n",
    "# Y is the target which doctors and experts tagged whether it's fraud information or not\n",
    "# Which is shown by 0 and 1\n",
    "\n",
    "# Scale Data\n",
    "\n",
    "pipe = Pipeline([('scaler', StandardScaler()),...])\n",
    "\n",
    "# Fit PCA\n",
    "\n",
    "pipe = Pipeline([('scaler', StandardScaler()),('PCA',PCA(n_components=10, svd_solver='full'))])\n",
    "# Where n_components could be any numbers we want to keep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-xagCaNwjlmo"
   },
   "source": [
    "#### Image Data Engineering\n",
    "\n",
    "The image data is fundamentally numeric data, which is a vector data having three dimensions - height, width, and depth. So we can use image data directly if we like. But in order to make it easier to fit models, we should also scale it in same distribution and pca it to compress the iamge data, which is predictablly very big."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2vlrOPntkeV4"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Scale data\n",
    "\n",
    "# X: the array of features\n",
    "# y: the array of labels\n",
    "X, y = np.array([]), np.array([])\n",
    "X = data/255 y = labels\n",
    "\n",
    "# Split Data\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "# X is the data of all image data we created in data base\n",
    "# Y is the target which doctors and experts tagged whether it's fraud information or not\n",
    "# Which is shown by 0 and 1\n",
    "\n",
    "# Compress Image Data\n",
    "\n",
    "pipe = Pipeline([('PCA', PCA(n_components=20)),...])\n",
    "# Where n_components could be any numbers we want to keep\n",
    "# Smaller the n_components, blurry the images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KVyc7HSMl9ze"
   },
   "source": [
    "#### Text Data Engineering\n",
    "\n",
    "Text data is the most difficult one to handle, because text data is fundamentally not numeric data, and has to be transferred by many advanced algorithms to AI's 'readable' numeric data. Fortunately we have the BERT model and pretrained data base, which could help us transfer text data fast and accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Cm6qPLmqmceN"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import tensorflow_text as text\n",
    "\n",
    "# Install Tensorflow Packages\n",
    "\n",
    "pip install -U \"tensorflow-text==2.13.*\"\n",
    "pip install \"tf-models-official==2.13.*\"\n",
    "\n",
    "# Choose Bert Model\n",
    "\n",
    "bert_model_name = 'small_bert/bert_en_uncased_L-4_H-512_A-8'\n",
    "\n",
    "map_name_to_handle = {\n",
    "    'bert_en_uncased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3',\n",
    "    'bert_en_cased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_cased_L-12_H-768_A-12/3',\n",
    "    'bert_multi_cased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_multi_cased_L-12_H-768_A-12/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-2_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-2_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-2_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-4_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-4_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-4_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-4_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-6_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-6_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-6_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-6_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-6_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-8_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-8_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-8_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-8_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-8_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-10_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-10_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-10_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-10_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-10_H-768_A-12/1',\n",
    "    'small_bert/bert_en_uncased_L-12_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-128_A-2/1',\n",
    "    'small_bert/bert_en_uncased_L-12_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-256_A-4/1',\n",
    "    'small_bert/bert_en_uncased_L-12_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-512_A-8/1',\n",
    "    'small_bert/bert_en_uncased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-12_H-768_A-12/1',\n",
    "    'albert_en_base':\n",
    "        'https://tfhub.dev/tensorflow/albert_en_base/2',\n",
    "    'electra_small':\n",
    "        'https://tfhub.dev/google/electra_small/2',\n",
    "    'electra_base':\n",
    "        'https://tfhub.dev/google/electra_base/2',\n",
    "    'experts_pubmed':\n",
    "        'https://tfhub.dev/google/experts/bert/pubmed/2',\n",
    "    'experts_wiki_books':\n",
    "        'https://tfhub.dev/google/experts/bert/wiki_books/2',\n",
    "    'talking-heads_base':\n",
    "        'https://tfhub.dev/tensorflow/talkheads_ggelu_bert_en_base/1',\n",
    "}\n",
    "\n",
    "map_model_to_preprocess = {\n",
    "    'bert_en_uncased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'bert_en_cased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_cased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-2_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-4_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-4_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-4_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-4_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-6_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-6_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-6_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-6_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-8_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-8_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-8_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-8_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-10_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-10_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-10_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-10_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-12_H-128_A-2':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-12_H-256_A-4':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-12_H-512_A-8':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'small_bert/bert_en_uncased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'bert_multi_cased_L-12_H-768_A-12':\n",
    "        'https://tfhub.dev/tensorflow/bert_multi_cased_preprocess/3',\n",
    "    'albert_en_base':\n",
    "        'https://tfhub.dev/tensorflow/albert_en_preprocess/3',\n",
    "    'electra_small':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'electra_base':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'experts_pubmed':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'experts_wiki_books':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "    'talking-heads_base':\n",
    "        'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',\n",
    "}\n",
    "\n",
    "tfhub_handle_encoder = map_name_to_handle[bert_model_name]\n",
    "tfhub_handle_preprocess = map_model_to_preprocess[bert_model_name]\n",
    "\n",
    "# Text Data Engineering\n",
    "\n",
    "bert_preprocess_model = hub.KerasLayer(tfhub_handle_preprocess)\n",
    "X = bert_preprocess_model(text_preprocessed)\n",
    "# So far, the text data has been tranformed to numeric data\n",
    "# And ready to do following AI modeling work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tolmYr0yrxnR"
   },
   "source": [
    "### Algorithms for Originally Numeric Data\n",
    "\n",
    "This part is similar as AI models to monitor medical waste actions, and can use the KMeans, Support Vector Classifier (SVC), K-Nearest Neighbors(KNN), Naive Bayes, XGBoost and RandomForest algorithms directly. Thus, this sample won’t go into details agian here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ayPso4UQsdis"
   },
   "source": [
    "### Algorithms for Originally Image Data\n",
    "\n",
    "Convolutional Neural Network(CNN) is the best algorithms to handle image data. In a CNN model, the input data is passed through a series of layers that are designed to extract increasingly abstract features and then do the classification. The basic building blocks of a CNN are convolutional layers, which use filters to extract features from the input data, and pooling layers, which down sample the output of the convolutional layers to reduce the dimensionality of the data. After passing through several convolutional and pooling layers, the output is then flattened and fed into a series of fully connected layers, which perform classification or regression on the extracted features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "P6-3EYtWtHve"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import datasets, layers, models\n",
    "\n",
    "# Creat Models\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(layers.MaxPooling2D((2, 2)))\n",
    "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
    "# Convolute and Pooling Images\n",
    "model.add(layers.Flatten())\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(10))\n",
    "# Transfering iamges\n",
    "model.add(Dropout(0.2))\n",
    "# Prevent overfitting\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "              metrics=['accuracy'])\n",
    "# Add layers to construct the Neural Networks\n",
    "\n",
    "# Fit Pipeline and Model\n",
    "\n",
    "model_pip = Pipeline([('PCA',PCA(n_components=30)),('cnn', model)])\n",
    "model_pip.fit(X_train, y_train).score(X_test, y_test)\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "model_pip_gridsearch=GridSearchCV(estimator = model_pip,\n",
    "                        param_grid = {'optimizer': ('adam','SGD','RMSprop'),\n",
    "                                      'loss': ('squared_hinge', 'CategoricalHinge'),\n",
    "                                      'metrics':('accuracy','binary_accuracy','CategoricalAccuracy')},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "model_pip_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Try Best to find parameters contribute to higher score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(model_pip_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference\n",
    "# And prevent overfitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5odtHBpTwRTx"
   },
   "source": [
    "### Algorithms for Originally Text Data\n",
    "\n",
    "After preprocessing by Bidirectional Encoder Representations from Transformers (BERT) model, the Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) will be the best algorithms for the transformed text data to fit, which are good at temporal and sequential data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bBQfqR2Bw56R"
   },
   "outputs": [],
   "source": [
    "# Fit BERT Model\n",
    "\n",
    "def build_classifier_model():\n",
    "  text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')\n",
    "  preprocessing_layer = hub.KerasLayer(tfhub_handle_preprocess, name='preprocessing')\n",
    "  encoder_inputs = preprocessing_layer(text_input)\n",
    "  encoder = hub.KerasLayer(tfhub_handle_encoder, trainable=True, name='BERT_encoder')\n",
    "  outputs = encoder(encoder_inputs)\n",
    "  net = outputs['pooled_output']\n",
    "  net = tf.keras.layers.Dropout(0.1)(net)\n",
    "  net = tf.keras.layers.Dense(1, activation=None, name='classifier')(net)\n",
    "  return tf.keras.Model(text_input, net)\n",
    "# Define model builder function\n",
    "\n",
    "model_1 = build_classifier_model()\n",
    "result = model_1(tf.constant(X))\n",
    "# X is the original and raw text data\n",
    "# Next two models use the BERT-transformed X data gained above\n",
    "\n",
    "# Fit RNN & LSTM Model\n",
    "# The LSTM model is one kind of agvanced RNN model, which add LSTM layers\n",
    "# In order to avoid duplication, here's only sample of LSTM model\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from keras.models import Model\n",
    "from keras.layers import LSTM, Activation, Dense, Dropout, Input, Embedding\n",
    "from keras.utils import to_categorical\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "def LSTM():\n",
    "    inputs = Input(name='inputs',shape=[max_len])\n",
    "    layer = Embedding(max_words,50,input_length=max_len)(inputs)\n",
    "    layer = LSTM(64)(layer)\n",
    "    layer = Dense(256,name='FC1')(layer)\n",
    "    layer = Activation('relu')(layer)\n",
    "    layer = Dropout(0.5)(layer)\n",
    "    layer = Dense(1,name='out_layer')(layer)\n",
    "    layer = Activation('sigmoid')(layer)\n",
    "    model = Model(inputs=inputs,outputs=layer)\n",
    "    return model\n",
    "\n",
    "model_2 = LSTM()\n",
    "model_2.compile(loss='binary_crossentropy',optimizer=RMSprop(),metrics=['accuracy'])\n",
    "\n",
    "# Fit Pipeline and Model\n",
    "\n",
    "model_2_pip = Pipeline([('PCA',PCA(n_components=30)),('lstm', model_2)])\n",
    "model_2_pip.fit(X_train, y_train).score(X_test, y_test)\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "model_2_pip_gridsearch=GridSearchCV(estimator = model_2_pip,\n",
    "                        param_grid = {'optimizer': ('adam','SGD','RMSprop'),\n",
    "                                      'loss': ('squared_hinge', 'CategoricalHinge'),\n",
    "                                      'metrics':('accuracy','binary_accuracy','CategoricalAccuracy')},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "model_2_pip_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Try Best to find parameters contribute to higher score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(model_2_pip_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference\n",
    "# And prevent overfitting\n",
    "\n",
    "model_2_pip_gridsearch.fit(X_train,Y_train,batch_size=128,epochs=10,\n",
    "          callbacks=[EarlyStopping(monitor='val_loss',min_delta=0.0001)])\n",
    "# Using EarlyStopping to save time when there's little earnings continuing modeling\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uyby9yiR16dn"
   },
   "source": [
    "## AI models to monitor impersonation actions\n",
    "\n",
    "In order to avoid impersonation actions, comparing insured people's bio information during medical care is necessary. No matter it's video or fingerprint, it's all image data, which could be trained by Convolutional Neural Network(CNN) to decide whether the collected new image is belong to certain insured person or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fPixcA1s5r8w"
   },
   "source": [
    "### Data Colletected\n",
    "\n",
    "The data base we should use is the insured people's comparable bio information, like video of insured person's face or several different positions of fingerprint. And we do scale and dimension reduction before feed the data to algorithms to shorten the processing time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aPkOlG7O5rMV"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Scale data\n",
    "\n",
    "# X: the array of features\n",
    "# y: the array of labels\n",
    "X, y = np.array([]), np.array([])\n",
    "X = data/255 y = labels\n",
    "\n",
    "# Split Data\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "# X is the data of all image data we created in data base\n",
    "# Y is the target which the image data belongs to\n",
    "\n",
    "# Compress Image Data\n",
    "\n",
    "pipe = Pipeline([('PCA', PCA(n_components=20)),...])\n",
    "# Where n_components could be any numbers we want to keep\n",
    "# Smaller the n_components, blurry the images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MDM-_J407tGj"
   },
   "source": [
    "### Convolutional Neural Network(CNN) Algorithm\n",
    "\n",
    "Using CNN algorithms to learn the path to decide whether the image comes from one insured person or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AO2DDigE7sR0"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras import datasets, layers, models\n",
    "\n",
    "# Creat Models\n",
    "\n",
    "cnn_model = models.Sequential([\n",
    "    layers.Conv2D(filters=25, kernel_size=(3, 3), activation='relu', input_shape=(28,28,1)),\n",
    "    layers.MaxPooling2D((2, 2)),\n",
    "    layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'),\n",
    "    layers.MaxPooling2D((2, 2)),\n",
    "    layers.Conv2D(filters=64, kernel_size=(3, 3), activation='relu'),\n",
    "    layers.MaxPooling2D((2, 2)),\n",
    "    # Convolute and Pooling Images\n",
    "    layers.Flatten(),\n",
    "    layers.Dense(64, activation='relu'),\n",
    "    layers.Dense(10, activation='softmax'),\n",
    "    # Transfering iamges\n",
    "    layers.Dropout(0.2)\n",
    "    # Prevent overfitting\n",
    "])\n",
    "\n",
    "cnn_model.compile(optimizer='adam',\n",
    "              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "              metrics=['accuracy'])\n",
    "# compile model\n",
    "\n",
    "# Fit Pipeline and Model\n",
    "\n",
    "cnn_model_pip = Pipeline([('PCA',PCA(n_components=30)),('cnn', cnn_model)])\n",
    "cnn_model_pip.fit(X_train, y_train).score(X_test, y_test)\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "cnn_model_pip_gridsearch=GridSearchCV(estimator = cnn_model_pip,\n",
    "                        param_grid = {'optimizer': ('adam','SGD','RMSprop'),\n",
    "                                      'loss': ('squared_hinge', 'CategoricalHinge'),\n",
    "                                      'metrics':('accuracy','binary_accuracy','CategoricalAccuracy')},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "cnn_model_pip_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Try Best to find parameters contribute to higher score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(cnn_model_pip_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference\n",
    "\n",
    "# After all modeling work, the algorithm has learned how to recognize image data\n",
    "# and compare it to existed insured people's bio info data base"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jN3ncaLE9jvR"
   },
   "source": [
    "## AI models to monitor substance abuse\n",
    "\n",
    "This part is quite similar as previous AI models to monitor fraud actions, but only focus on text data part, which stressed on prescriptions the doctors wrote."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xkVLuS0l-RcA"
   },
   "source": [
    "### Data Generated\n",
    "\n",
    "Doctors' or experts' tagged abuse or not medical records' database (text data series with tag 0 or 1). Considering it's text data, BERT algorithm could be used to preprocess the data to transform it to numeric data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "a2wMk1LeBMSE"
   },
   "outputs": [],
   "source": [
    "# Given that the model map has been created above\n",
    "# Index the model name to use BERT preprocessed model directly\n",
    "\n",
    "bert_model_name = 'small_bert/bert_en_uncased_L-4_H-512_A-8'\n",
    "\n",
    "tfhub_handle_encoder = map_name_to_handle[bert_model_name]\n",
    "tfhub_handle_preprocess = map_model_to_preprocess[bert_model_name]\n",
    "\n",
    "# Text Data Engineering\n",
    "\n",
    "bert_preprocess_model = hub.KerasLayer(tfhub_handle_preprocess)\n",
    "X = bert_preprocess_model(text_preprocessed)\n",
    "# X is the prescription data base\n",
    "# So far, the text data has been tranformed to numeric data\n",
    "# And ready to do following AI modeling work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sPUM9PbvBLr0"
   },
   "source": [
    "### Long Short-Term Memory (LSTM) Algorithms\n",
    "\n",
    "After preprocessing by Bidirectional Encoder Representations from Transformers (BERT) model, the Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) will be the best algorithms for the transformed text data to fit, which are good at temporal and sequential data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ukaPdW8bCPE4"
   },
   "outputs": [],
   "source": [
    "# Following code is quite similar as AI models to monitor fraud actions's text data part\n",
    "# The different aspects are the data base and exact parameters when fine tune\n",
    "\n",
    "# Fit BERT Model\n",
    "\n",
    "def build_classifier_model():\n",
    "  text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')\n",
    "  preprocessing_layer = hub.KerasLayer(tfhub_handle_preprocess, name='preprocessing')\n",
    "  encoder_inputs = preprocessing_layer(text_input)\n",
    "  encoder = hub.KerasLayer(tfhub_handle_encoder, trainable=True, name='BERT_encoder')\n",
    "  outputs = encoder(encoder_inputs)\n",
    "  net = outputs['pooled_output']\n",
    "  net = tf.keras.layers.Dropout(0.1)(net)\n",
    "  net = tf.keras.layers.Dense(1, activation=None, name='classifier')(net)\n",
    "  return tf.keras.Model(text_input, net)\n",
    "# Define model builder function\n",
    "\n",
    "model_bert = build_classifier_model()\n",
    "result = model_bert(tf.constant(X))\n",
    "# X is the original and raw text data\n",
    "# Next model use the BERT-transformed X data gained above\n",
    "\n",
    "# Fit RNN & LSTM Model\n",
    "# The LSTM model is one kind of agvanced RNN model, which add LSTM layers\n",
    "# In order to avoid duplication, here's only sample of LSTM model\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from keras.models import Model\n",
    "from keras.layers import LSTM, Activation, Dense, Dropout, Input, Embedding\n",
    "from keras.utils import to_categorical\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "def LSTM():\n",
    "    inputs = Input(name='inputs',shape=[max_len])\n",
    "    layer = Embedding(max_words,50,input_length=max_len)(inputs)\n",
    "    layer = LSTM(64)(layer)\n",
    "    layer = Dense(256,name='FC1')(layer)\n",
    "    layer = Activation('relu')(layer)\n",
    "    layer = Dropout(0.5)(layer)\n",
    "    layer = Dense(1,name='out_layer')(layer)\n",
    "    layer = Activation('sigmoid')(layer)\n",
    "    model = Model(inputs=inputs,outputs=layer)\n",
    "    return model\n",
    "\n",
    "model_lstm = LSTM()\n",
    "model_lstm.compile(loss='binary_crossentropy',optimizer=RMSprop(),metrics=['accuracy'])\n",
    "\n",
    "# Fit Pipeline and Model\n",
    "\n",
    "model_lstm_pip = Pipeline([('PCA',PCA(n_components=30)),('lstm', model_lstm)])\n",
    "model_lstm_pip.fit(X_train, y_train).score(X_test, y_test)\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "model_lstm_pip_gridsearch=GridSearchCV(estimator = model_lstm_pip,\n",
    "                        param_grid = {'optimizer': ('adam','SGD','RMSprop'),\n",
    "                                      'loss': ('squared_hinge', 'CategoricalHinge'),\n",
    "                                      'metrics':('accuracy','binary_accuracy','CategoricalAccuracy')},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "model_lstm_pip_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Try Best to find parameters contribute to higher score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(model_lstm_pip_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference\n",
    "# And prevent overfitting\n",
    "\n",
    "model_lstm_pip_gridsearch.fit(X_train,Y_train,batch_size=128,epochs=10,\n",
    "          callbacks=[EarlyStopping(monitor='val_loss',min_delta=0.0001)])\n",
    "# Using EarlyStopping to save time when there's little earnings continuing modeling\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KjPNvxFBDmw8"
   },
   "source": [
    "## AI models to build more accurate premium pricing model\n",
    "\n",
    "More accurate the premium pricing model is, lower the risk the insurance company faces. In financial world, lower risk means lower cost, which leads to lower premium cost for insured people. AI models are the best choice to do this work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CKJICc9NEW-L"
   },
   "source": [
    "### Data Generated\n",
    "\n",
    "History database of insured people's basic information(Yearly, Numeric), all scores above AI models generated(Numeric), usage of insurance fund(Yearly, Numeric), and economic data like inflation rate etc. (Numeric). The results we get above, like probability of defrauding or wasting, and probability of abusing have high value to be considered when pricing insurance products. And because all data are numeric, what we should do is just scale and PCA to reduce dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EMwzO_ccFD2b"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Split Data\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n",
    "# X is the data of all insured people's numeric information we collected\n",
    "# Y is the yearly medical cost the insured person spent\n",
    "\n",
    "# Scale Data\n",
    "\n",
    "pipe = Pipeline([('scaler', StandardScaler()),...])\n",
    "\n",
    "# Fit PCA to Reduce Dimensions\n",
    "\n",
    "pipe = Pipeline([('scaler', StandardScaler()),('PCA',PCA(n_components=20, svd_solver='full'))])\n",
    "\n",
    "# Where n_components could be any numbers we want to keep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-jFlD7oJFi_O"
   },
   "source": [
    "### Grouping Insured People\n",
    "\n",
    "Different insurance company has different insured people's pool, which has many layers and groups. There are different pricing models for different groups. However, the groups which are created based on simple background information is not accurate. Thus, the KMeans algorithm should be used to group insured people."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ulByfaZdHE4G"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "# Fit KMeans to Group People\n",
    "\n",
    "kmeans = KMeans(n_clusters=5, random_state=0, n_init=\"auto\").fit(X)\n",
    "# X is the data base of insured people's information\n",
    "# Set 5 clusters to fit in this sample model\n",
    "# In different insurance company, the n_cluster could be changed to the exact numer\n",
    "# of their settled groups."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3QYYrDcyI8hK"
   },
   "source": [
    "### Algorithms 1 - RidgeRegression(RR) for Different Groups\n",
    "\n",
    "Our goal is the predict every insured people's yearly medical cost through all factors we discussed above. More factors we considered, higher probability of multicollinearity. RidgeRegression is the best algorithm to handle multicollinearity, and it can reduce model complexity and prevents overfitting as well, with lower variance. Thus, we should try RidgeRegression to pricing cost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hgdhwqvXKR_4"
   },
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "pipe_RR = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                  ('ridge', Ridge(alpha=1.0))])\n",
    "pipe_RR.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "pipe_RR_gridsearch=GridSearchCV(estimator = pipe_RR,\n",
    "                        param_grid = {'alpha': [0, 1, 5, 10],\n",
    "                        'solver': ('auto', 'svd', 'cholesky', 'lsqr', 'saga', 'lbfgs')},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "pipe_RR_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Which should produce higher test score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(pipe_RR_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ImqrR6KoHlqG"
   },
   "source": [
    "### Algorithms 2 - Support Vector Regression (SVR) for Different Groups\n",
    "\n",
    "Similar as Support Vector Classifier (SVC), which draw a line to make all points' distance to the line smallest, SVR gives us the flexibility to define how much error is acceptable in our model and will find an appropriate line (or hyperplane in higher dimensions) to fit the data and produce real numbers to be results.\n",
    "\n",
    "In contrast to OLS, the objective function of SVR is to minimize the coefficients — more specifically, the l2-norm of the coefficient vector — not the squared error. The error term is instead handled in the constraints, where we set the absolute error less than or equal to a specified margin, called the maximum error, ϵ (epsilon)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lxeX09fpI37z"
   },
   "outputs": [],
   "source": [
    "# Standard SVR Classifier\n",
    "\n",
    "from sklearn.svm import SVR\n",
    "\n",
    "pipe_SVR = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                  ('svr', SVR(C=1.0, epsilon=0.2))])\n",
    "pipe_SVR.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "pipe_SVR_gridsearch=GridSearchCV(estimator = pipe_SVR,\n",
    "                        param_grid = {'C': [1,3,5,10],\n",
    "                        'kernel': ('linear', 'poly', 'rbf', 'sigmoid', 'precomputed')},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "pipe_SVR_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Which should produce higher test score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(pipe_SVR_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Algorithms 3 - XGBoost for Different Groups\n",
    "\n",
    "XGBoost algorithms could perform extreme level improvements on multiple weak learners and integrate them into a strong learner which could run at high speed and low memory without overfitting and under-fitting problems. At the same time, the algorithm is stable and has high accuracy. This part is quite similar to XGBoost mentioned above, but it's applied on regression task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "\n",
    "from sklearn import ensemble\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# Preset Parameters to build model\n",
    "\n",
    "params = {\n",
    "    \"n_estimators\": 500,\n",
    "    \"max_depth\": 4,\n",
    "    \"learning_rate\": 0.01,\n",
    "    \"loss\": \"squared_error\",\n",
    "}\n",
    "\n",
    "# Generate XGBoost regressioon Model\n",
    "\n",
    "pipe_XG = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                  ('XG', ensemble.GradientBoostingRegressor(**params))])\n",
    "pipe_XG.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# Using GridSearchCV to Find the Best Parameters\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "pipe_XG_gridsearch=GridSearchCV(estimator = pipe_XG,\n",
    "                        param_grid = {'n_estimators': [100,200,300,500,1000],\n",
    "                        'max_depth': [3,4,5,10,15,20],\n",
    "                        'learning_rate':[0.01,0.05,0.1,0.3]},\n",
    "                        scoring='accuracy',\n",
    "                        cv=10,\n",
    "                        n_jobs=-1)\n",
    "pipe_XG_gridsearch.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Which should produce higher test score\n",
    "\n",
    "#Using Cross Validation to test Classifier More Accurate\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "scores = cross_val_score(pipe_XG_gridsearch, X, y, cv=5)\n",
    "# Which should give a more comprehensive scores for reference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Algorithms 4 - RandomForest for Different Groups\n",
    "\n",
    "RandomForest algorithm is a tree algorithms structure, which is a collection of different decision trees obtained from the sample. Yes or no judgment is made on each node by Gini Index. After continuous iteration, the leaf nodes are the final decision results. This algotihm can explain the process from sample data to results, which can be used to explain patients’ behavior. This part is quite similar to RandomForest mentioned above, but it's applied on regression task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Standard Import\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "\n",
    "pipe_RF = Pipeline([('scaler', StandardScaler()),\n",
    "                  ('PCA',PCA(n_components=10, svd_solver='full')),\n",
    "                   ('RandomForest', RandomForestClassifier(max_depth=10, random_state=0))])\n",
    "\n",
    "pipe_RF.fit(X_train, y_train).score(X_test, y_test)\n",
    "# Test the accuracy score to see if the model works well\n",
    "\n",
    "# RandomForest algorithm has already did 'cross validation' inside, no more needed additional test\n",
    "# RandomForest algorithm is time-consuming, if the test score is much lower than XGBoost, then stop fine tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Blockchain Interior Codes\n",
    "\n",
    "Blockchain's interior structure includes three parts: input the data uploaded to the blockchain, read the Hash value of the previous blockchain, and generate a new Hash value of the current blockchain to pass to the next blockchain. After these three parts, the data could be permanently and tamper-resistantly saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python program to create Blockchain\n",
    "# This code's reference:https://www.geeksforgeeks.org/create-simple-blockchain-using-python/\n",
    "\n",
    "# For timestamp\n",
    "import datetime\n",
    "\n",
    "# Calculating the hash\n",
    "# in order to add digital\n",
    "# fingerprints to the blocks\n",
    "import hashlib\n",
    "\n",
    "# To store data\n",
    "# in our blockchain\n",
    "import json\n",
    "\n",
    "\n",
    "class Blockchain:\n",
    "\n",
    "    # This function is created\n",
    "    # to create the very first\n",
    "    # block and set its hash to \"0\"\n",
    "    def __init__(self):\n",
    "        self.chain = []\n",
    "        self.create_block(proof=1, previous_hash='0')\n",
    "\n",
    "    # This function is created\n",
    "    # to add further blocks\n",
    "    # into the chain\n",
    "    def create_block(self, proof, previous_hash):\n",
    "        block = {'index': len(self.chain) + 1,\n",
    "                'timestamp': str(datetime.datetime.now()),\n",
    "                'proof': proof,\n",
    "                'previous_hash': previous_hash}\n",
    "        self.chain.append(block)\n",
    "        return block\n",
    "\n",
    "    # This function is created\n",
    "    # to display the previous block\n",
    "    def print_previous_block(self):\n",
    "        return self.chain[-1]\n",
    "\n",
    "    # This is the function for proof of work\n",
    "    # and used to successfully mine the block\n",
    "    def proof_of_work(self, previous_proof):\n",
    "        new_proof = 1\n",
    "        check_proof = False\n",
    "\n",
    "        while check_proof is False:\n",
    "            hash_operation = hashlib.sha256(\n",
    "                str(new_proof**2 - previous_proof**2).encode()).hexdigest()\n",
    "            if hash_operation[:5] == '00000':\n",
    "                check_proof = True\n",
    "            else:\n",
    "                new_proof += 1\n",
    "\n",
    "        return new_proof\n",
    "\n",
    "    def hash(self, block):\n",
    "        encoded_block = json.dumps(block, sort_keys=True).encode()\n",
    "        return hashlib.sha256(encoded_block).hexdigest()\n",
    "\n",
    "    def chain_valid(self, chain):\n",
    "        previous_block = chain[0]\n",
    "        block_index = 1\n",
    "\n",
    "    while block_index < len(chain):\n",
    "            block = chain[block_index]\n",
    "            if block['previous_hash'] != self.hash(previous_block):\n",
    "                return False\n",
    "\n",
    "            previous_proof = previous_block['proof']\n",
    "            proof = block['proof']\n",
    "            hash_operation = hashlib.sha256(\n",
    "                str(proof**2 - previous_proof**2).encode()).hexdigest()\n",
    "\n",
    "            if hash_operation[:5] != '00000':\n",
    "                return False\n",
    "            previous_block = block\n",
    "            block_index += 1\n",
    "\n",
    "        return True"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}