A-B Testing with Facebook's PlanOut.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A/B Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will start with a quote:\n",
    "    \n",
    "    \"Insanity is doing the same thing over and over again and expecting different results\" - Albert Einstein"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Considering that most businesses give their customers their first impression via a Website or a mobile app now, I decided to hit that popular problem space: What site/mobile app design will cause the most engagement with users?. Thus, I decided to do an A/B test set-up using Facebook's PlanOut. \n",
    "\n",
    "The reason why I went with PlanOut was that it has production ready ports. A competitor, Wasabi, is an API-driven testing framework. Downside to API-driven run is the latency it adds to each experiment decision. The other widely used framework is Optimizely. It is a server-side JS or REACT SDK for testing. Using SDK's means that the decision can take place in memory, which means that there is no latency. Despite the other options, Facebook's Prophet was my best bet since I wanted production ready ports.\n",
    "\n",
    "Facebook's set up is extremely easy:\n",
    "Straight from its homepage-\n",
    "\n",
    "1. Define the experiment\n",
    "2. Get a random assignment\n",
    "3. Logging treatment assignment\n",
    "4. Logging an action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The way I will go about this is actually use someone else's Facebook PlanOut set up and gathered data. And I will independently do the statistical analysis on it using Hypothesis testing in order to determine which website design attracted most users/incurred most engagement. So let's start!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start using Facebook's PlanOut:\n",
    "\n",
    "pip install planout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from planout.namespace import SimpleNamespace\n",
    "from planout.experiment import SimpleExperiment, DefaultExperiment\n",
    "from planout.ops.random import *\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Define Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SignupExperiment(SimpleExperiment):\n",
    "    def assign(self, params, cookieid):\n",
    "        params.button_color = UniformChoice(\n",
    "        choices = [\"#ff0000\", \"#00ff00\"],\n",
    "                   unit = cookieid)\n",
    "        params.button_text = UniformChoice(\n",
    "        choices = [\"Join Now\", \"Sign Me Up\"],\n",
    "            unit = cookieid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Weighted Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# class SignupExperiment2(SimpleExperiment):\n",
    "#     def assign(self, params, cookieid):\n",
    "#         parms.button_color = UniformChoice(\n",
    "#             choices = [\"#ff0000\", \"00ff00\"],\n",
    "#             unit = cookieid)\n",
    "#         params.button_text = WeightedChoice(\n",
    "#             choices = [\"Join Now!\", \"Sign Me Up!\"],\n",
    "#         weights = [8,2],\n",
    "#         unit = cookieid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Get Random Assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SearchRankingExperiment(SimpleExperiment):\n",
    "    def assign(self, params, userid, sessionid):\n",
    "        params. candidate_model = UniformChoice(choices = 'v100', 'v101', 'v102'], unit = userid)\n",
    "        params.ranking_model = UniformChoice(choices = ['v0', params.candidate_model], unit=[userid, sessionid])\n",
    "        \n",
    "for s in xrange(5):\n",
    "    print SearchRankingExperiment(userid=6m sessionid=s).get('ranking model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output from previous code block's run:\n",
    "# v100\n",
    "# v100\n",
    "# v100\n",
    "# v0\n",
    "# v0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Likert(SimpleExperiment):\n",
    "    \"\"\"\n",
    "    A randomization of the survey scale for robustness of the politics measure.\n",
    "    \"\"\"\n",
    "    def setup(self):\n",
    "        self.set_log_file('logs/survey.log')\n",
    "\n",
    "    def assign(self, params, userid):\n",
    "        params.reversed_scale = UniformChoice(choices=[0,1], unit=userid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class CueExperiment(SimpleExperiment):\n",
    "    \"\"\"\n",
    "    Randomly reorder the stories and choose from available cues.\n",
    "    \"\"\"\n",
    "    def setup(self):\n",
    "        self.set_log_file('logs/experiment.log')\n",
    "\n",
    "    def assign(self, params, userid, story_keys):\n",
    "        params.story_keys = Sample(choices=story_keys, unit=userid)\n",
    "        balanced_sources = list(islice(cycle([\n",
    "            'msnbc',\n",
    "            'cnn',\n",
    "            'foxnews',\n",
    "        ]), len(story_keys)))\n",
    "        params.sources = Sample(choices=balanced_sources, unit=userid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sim_users = [SignupExperiment(cookieid=i).get_params() for i in xrange(1000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments[:10]\n",
    "assignments.groupby(['button_text', 'button_color']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Output from above block (to analyze):\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "button_color button_text\n",
    "0      #00ff00  Sign Me Up\n",
    "1      #00ff00    Join Now\n",
    "2      #ff0000  Sign Me Up\n",
    "3      #00ff00  Sign Me Up\n",
    "4      #ff0000  Sign Me Up\n",
    "5      #00ff00    Join Now\n",
    "6      #ff0000    Join Now\n",
    "7      #00ff00    Join Now\n",
    "8      #ff0000    Join Now\n",
    "9      #ff0000    Join Now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "button_text  button_color\n",
    "Join Now     #00ff00         234\n",
    "             #ff0000         247\n",
    "Sign Me Up   #00ff00         246\n",
    "             #ff0000         273\n",
    "dtype: int64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing Significance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If divide all numbers from above exposure results by 1000 and then multiple by 100%, then will see that highest conversion rate was from the last group. That means the button text, 'Sign Me Up' with #ff0000 button color. That means the last design increased engagement by 27% and was 17% better than first the first design. \n",
    "\n",
    "I am 98% certain that the changes in the last experiment set up will improve your conversion rate.\n",
    "\n",
    "My Multivariate test is statistically significant!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}