bootstrap_analysis_project_notebook.ipynb

{"cells":[{"metadata":{"dc":{"key":"3"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 1. Of cats and cookies\n<p><a href=\"https://www.facebook.com/cookiecatsgame\">Cookie Cats</a> is a hugely popular mobile puzzle game developed by <a href=\"http://tactile.dk\">Tactile Entertainment</a>. It's a classic \"connect three\"-style puzzle game where the player must connect tiles of the same color to clear the board and win the level. It also features singing cats. We're not kidding! Check out this short demo:</p>\n<p><a href=\"https://youtu.be/GaP5f0jVTWE\"><img src=\"https://assets.datacamp.com/production/project_184/img/cookie_cats_video.jpeg\" style=\"width: 500px\"></a></p>\n<p>As players progress through the levels of the game, they will occasionally encounter gates that force them to wait a non-trivial amount of time or make an in-app purchase to progress. In addition to driving in-app purchases, these gates serve the important purpose of giving players an enforced break from playing the game, hopefully resulting in that the player's enjoyment of the game being increased and prolonged.</p>\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/cc_gates.png\" alt></p>\n<p>But where should the gates be placed? Initially the first gate was placed at level 30, but in this notebook we're going to analyze an AB-test where we moved the first gate in Cookie Cats from level 30 to level 40. In particular, we will look at the impact on player retention. But before we get to that, a key step before undertaking any analysis is understanding the data. So let's load it in and take a look!</p>"},{"metadata":{"dc":{"key":"3"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Importing pandas\nimport pandas as pd\n\n# Reading in the data\ndf = pd.read_csv(\"datasets/cookie_cats.csv\")\n\n# Showing the first few rows\ndf.head()","execution_count":75,"outputs":[{"output_type":"execute_result","execution_count":75,"data":{"text/plain":"   userid  version  sum_gamerounds  retention_1  retention_7\n0     116  gate_30               3        False        False\n1     337  gate_30              38         True        False\n2     377  gate_40             165         True        False\n3     483  gate_40               1        False        False\n4     488  gate_40             179         True         True","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>userid</th>\n      <th>version</th>\n      <th>sum_gamerounds</th>\n      <th>retention_1</th>\n      <th>retention_7</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>116</td>\n      <td>gate_30</td>\n      <td>3</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>337</td>\n      <td>gate_30</td>\n      <td>38</td>\n      <td>True</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>377</td>\n      <td>gate_40</td>\n      <td>165</td>\n      <td>True</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>483</td>\n      <td>gate_40</td>\n      <td>1</td>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>488</td>\n      <td>gate_40</td>\n      <td>179</td>\n      <td>True</td>\n      <td>True</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"dc":{"key":"10"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 2. The AB-test data\n<p>The data we have is from 90,189 players that installed the game while the AB-test was running. The variables are:</p>\n<ul>\n<li><code>userid</code> - a unique number that identifies each player.</li>\n<li><code>version</code> - whether the player was put in the control group (<code>gate_30</code> - a gate at level 30) or the group with the moved gate (<code>gate_40</code> - a gate at level 40).</li>\n<li><code>sum_gamerounds</code> - the number of game rounds played by the player during the first 14 days after install.</li>\n<li><code>retention_1</code> - did the player come back and play <strong>1 day</strong> after installing?</li>\n<li><code>retention_7</code> - did the player come back and play <strong>7 days</strong> after installing?</li>\n</ul>\n<p>When a player installed the game, he or she was randomly assigned to either <code>gate_30</code> or <code>gate_40</code>. As a sanity check, let's see if there are roughly the same number of players in each AB group. </p>"},{"metadata":{"dc":{"key":"10"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Counting the number of players in each AB group.\ndf['version'].value_counts()\ndf.groupby(by = 'version').count()","execution_count":77,"outputs":[{"output_type":"execute_result","execution_count":77,"data":{"text/plain":"         userid  sum_gamerounds  retention_1  retention_7\nversion                                                  \ngate_30   44700           44700        44700        44700\ngate_40   45489           45489        45489        45489","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>userid</th>\n      <th>sum_gamerounds</th>\n      <th>retention_1</th>\n      <th>retention_7</th>\n    </tr>\n    <tr>\n      <th>version</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>gate_30</th>\n      <td>44700</td>\n      <td>44700</td>\n      <td>44700</td>\n      <td>44700</td>\n    </tr>\n    <tr>\n      <th>gate_40</th>\n      <td>45489</td>\n      <td>45489</td>\n      <td>45489</td>\n      <td>45489</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"dc":{"key":"17"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 3. The distribution of game rounds\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/mr_waffles_smiling.png\" style=\"width:200px; float:left\"> </p>\n<p>It looks like there is roughly the same number of players in each group, nice!</p>\n<p>The focus of this analysis will be on how the gate placement affects player retention, but just for fun: Let's plot the distribution of the number of game rounds players played during their first week playing the game.</p>"},{"metadata":{"dc":{"key":"17"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# This command makes plots appear in the notebook\n%matplotlib inline\n\n# Counting the number of players for each number of gamerounds \nplot_df = df.groupby('sum_gamerounds')['userid'].count()\nprint(plot_df)\n# Plotting the distribution of players that played 0 to 100 game rounds\nax = plot_df.head(n=100).plot(x = 'sum_gamerounds', y = 'userid', kind = 'hist')\nax.set_xlabel(\"userid\")\nax.set_ylabel(\"gamerounds\")","execution_count":79,"outputs":[{"output_type":"stream","text":"sum_gamerounds\n0        3994\n1        5538\n2        4606\n3        3958\n4        3629\n5        2992\n6        2861\n7        2379\n8        2267\n9        2013\n10       1752\n11       1654\n12       1570\n13       1594\n14       1519\n15       1446\n16       1342\n17       1269\n18       1228\n19       1158\n20       1033\n21       1032\n22        934\n23        879\n24        794\n25        792\n26        689\n27        723\n28        662\n29        613\n         ... \n1457        1\n1462        1\n1469        1\n1490        1\n1491        1\n1522        1\n1528        1\n1529        1\n1544        1\n1558        1\n1559        1\n1573        1\n1643        2\n1667        1\n1687        1\n1697        1\n1705        1\n1714        1\n1816        1\n1906        1\n2015        1\n2063        1\n2124        1\n2156        1\n2251        1\n2294        1\n2438        1\n2640        1\n2961        1\n49854       1\nName: userid, Length: 942, dtype: int64\n","name":"stdout"},{"output_type":"execute_result","execution_count":79,"data":{"text/plain":"Text(0,0.5,'gamerounds')"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<matplotlib.figure.Figure at 0x7f5413e94b00>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAFVJJREFUeJzt3W2wZVV95/HvzwaEEYUG2x5GcBoM0WEsRHJHcXQoIz5jhFKL4DhOR6nq0slktJxUbCfJmKcXaFWMmkpp9YhMO6KoRIsOlpq2xXFSlUFvAyIPmkbECgzQ1weCTyWi/3mxV+ul7aY32Pvede/5fqpOnb3X2fuc/zp1b/96r73v2qkqJEnqzcOWuwBJkvbFgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR1yYCSJHXJgJIkdcmAkiR16ZDlLmCMRz/60bVhw4blLkOSdBDs3Lnzm1W17kDbrYiA2rBhA/Pz88tdhiTpIEjyjTHbOcQnSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnq0oqYSeJg2LD5E8tdAgC3Xnj2cpcgSSuCR1CSpC4ZUJKkLk0WUEmekOTaRY97krwhyTFJtifZ1Z7XTlWDJGnlmiygquqrVXVaVZ0G/BrwA+DjwGZgR1WdDOxo65Ik3c9SDfGdBXytqr4BnANsbe1bgXOXqAZJ0gqyVAF1PvChtry+qu5oy3cC6/e1Q5JNSeaTzC8sLCxFjZKkjkweUEkOA14CfHTv16qqgNrXflW1parmqmpu3boD3nhRkrTKLMUR1AuBq6vqrrZ+V5LjANrz7iWoQZK0wixFQL2Cnw/vAWwDNrbljcDlS1CDJGmFmTSgkjwCeC7wsUXNFwLPTbILeE5blyTpfiad6qiqvg8cu1fbtxiu6pMkab+cSUKS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1CUDSpLUJQNKktQlA0qS1KVJAyrJ0UkuS/KVJDcleXqSY5JsT7KrPa+dsgZJ0so09RHUO4FPVdUTgScDNwGbgR1VdTKwo61LknQ/kwVUkqOAM4GLAKrq3qq6GzgH2No22wqcO1UNkqSVa8ojqBOBBeDiJNckeW+SRwDrq+qOts2dwPp97ZxkU5L5JPMLCwsTlilJ6tGUAXUIcDrw7qp6CvB99hrOq6oCal87V9WWqpqrqrl169ZNWKYkqUdTBtRtwG1VdVVbv4whsO5KchxAe949YQ2SpBVqsoCqqjuBf0zyhNZ0FnAjsA3Y2No2ApdPVYMkaeU6ZOL3/x3gkiSHAbcAr2YIxY8kuQD4BnDexDVIklagSQOqqq4F5vbx0llTfq4kaeVzJglJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDpnyzZPcCnwX+AlwX1XNJTkG+DCwAbgVOK+qvjNlHZKklWcpjqB+vapOq6q5tr4Z2FFVJwM72rokSfezHEN85wBb2/JW4NxlqEGS1LmpA6qAv02yM8mm1ra+qu5oy3cC6/e1Y5JNSeaTzC8sLExcpiSpN5OegwKeWVW3J3kMsD3JVxa/WFWVpPa1Y1VtAbYAzM3N7XMbSdLqNekRVFXd3p53Ax8HngrcleQ4gPa8e8oaJEkr02QBleQRSR65Zxl4HnA9sA3Y2DbbCFw+VQ2SpJVryiG+9cDHk+z5nA9W1aeSfBH4SJILgG8A501YgyRphZosoKrqFuDJ+2j/FnDWVJ8rSVodnElCktQlA0qS1CUDSpLUpQcdUEnWJjl1imIkSdpjVEAl+VySR7WJXq8G/keSt09bmiRplo09gjqqqu4BXgq8v6qeBjxnurIkSbNubEAd0mZ9OA+4YsJ6JEkCxgfUnwCfBm6uqi8mOQnYNV1ZkqRZN+oPdavqo8BHF63fArxsqqIkSXrAgErylwy3zNinqvovB70iSZI48BDfPLATOBw4nWFYbxdwGnDYtKVJkmbZAx5BVdVWgCSvY7i3031t/T3A/5m+PEnSrBp7kcRa4FGL1o9sbZIkTWLsbOYXAtckuRIIcCbwR1MVJUnS2Kv4Lk7ySeBprelNVXXndGVJkmbdg5mLbw2wAHwH+NUkZ05TkiRJI4+gkrwV+E3gBuCnrbmAz09UlyRpxo09B3Uu8ISq+tGUxUiStMfYIb5bgEOnLESSpMXGHkH9ALg2yQ7gZ0dRziQhSZrK2IDa1h6SJC2JsZeZb526EEmSFht7Fd/X2ceksVV10oh91zDM6Xd7Vb04yYnApcCxDPP8vaqq7n1QVUuSVr2xF0nMAf+mPf4d8C7gAyP3fT1w06L1twJ/UVW/wvA3VReMfB9J0gwZFVBV9a1Fj9ur6h3A2QfaL8nxbbv3tvUAzwYua5tsZbiEXZKk+xk7xHf6otWHMRxRjdn3HcDvAY9s68cCd++ZFR24DXjsfj5zE7AJ4HGPe9yYMiVJq8jYq/j+fNHyfcCtwHkPtEOSFwO7q2pnkmc92MKqaguwBWBubm6/N02UJK1OY6/i+/WH8N7PAF6S5EUMNzx8FPBO4Ogkh7SjqOOB2x/Ce0uSVrlR56CSHJXk7Unm2+PPkxz1QPtU1Zur6viq2gCcD3y2ql4JXAm8vG22Ebj8l6hfkrRKjb2K733AdxmG9c4D7gEufoif+SbgjUluZjgnddFDfB9J0io29hzU46vqZYvW/zjJtWM/pKo+B3yuLd8CPHXsvpKk2TT2COqHSZ65ZyXJM4AfTlOSJEnjj6BeC7x/0Xmn7zCcP5IkaRIHDKgkD2O4F9STkzwKoKrumbwySdJMO+AQX1X9lOGPbamqewwnSdJSGHsO6jNJfjfJCUmO2fOYtDJJ0kwbew7qN9vzby9qK+CAs5lLkvRQjJ1J4sSpC5EkabGxM0n8syR/kGRLWz+5zbUnSdIkxp6Duhi4F/i3bf124M8mqUiSJMYH1OOr6m3AjwGq6gdAJqtKkjTzxgbUvUmOoN32PcnjgR9NVpUkaeaNvYrvLcCngBOSXMJwK43fmqooSZLGXsW3PcnVwBkMQ3uvr6pvTlqZJGmmjR3ig+HW7GuAw4Azk7x0mpIkSRp5BJXkfcCpwA3AT1tzAR+bqC5J0owbew7qjKo6ZdJKJElaZOwQ398nMaAkSUtm7BHU+xlC6k6Gy8sDVFWdOlllkqSZNjagLgJeBXyZn5+DkiRpMmMDaqGqtk1aiSRJi4wNqGuSfBD4GxbNIFFVXsUnSZrE2IA6giGYnreozcvMJUmTGTuTxKsf7BsnORz4PPDw9jmXVdVbkpwIXAocC+wEXlVV9z7Y95ckrW5j/1D3cOAC4F8Dh+9pr6rXPMBuPwKeXVXfS3Io8HdJPgm8EfiLqro0yXva+777oXZAkrQ6jf07qP8F/HPg+cD/Bo4HvvtAO9Tge2310PYo4NnAZa19K3Dug6xZkjQDxgbUr1TVHwLfr6qtwNnA0w60U5I1Sa4FdgPbga8Bd1fVfW2T2xjm+NvXvpuSzCeZX1hYGFmmJGm1GBtQP27Pdyd5EnAU8JgD7VRVP6mq0xiOuJ4KPHFsYVW1parmqmpu3bp1Y3eTJK0SY6/i25JkLfAHwDbgSOAPx35IVd2d5Erg6cDRSQ5pR1HHM9w+XpKk+xl7BHUU8GpgDvgr4K3AfUlO298OSdYlObotHwE8F7gJuBJ4edtsI3D5QytdkrSajT2C+jWGcPqbtv5i4DrgtUk+WlVv28c+xwFbk6xhCMKPVNUVSW4ELk3yZ8A1DNMoSZJ0P2MD6njg9D1X5SV5C/AJ4EyGv2X6hYCqquuAp+yj/RaG81GSJO3X2CG+x7BoiiOGiybWV9UP92qXJOmgGHsEdQlwVZI954t+A/hgkkcAN05SmSRppo2d6uhP2ywQz2hNr62q+bb8ykkqkyTNtLFHULRAmj/ghpIkHQRjz0FJkrSkDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcMKElSlwwoSVKXDChJUpcmC6gkJyS5MsmNSW5I8vrWfkyS7Ul2tee1U9UgSVq5pjyCug/4r1V1CnAG8NtJTgE2Azuq6mRgR1uXJOl+Jguoqrqjqq5uy98FbgIeC5wDbG2bbQXOnaoGSdLKtSTnoJJsAJ4CXAWsr6o72kt3Auv3s8+mJPNJ5hcWFpaiTElSRyYPqCRHAn8NvKGq7ln8WlUVUPvar6q2VNVcVc2tW7du6jIlSZ2ZNKCSHMoQTpdU1cda811JjmuvHwfsnrIGSdLKNOVVfAEuAm6qqrcvemkbsLEtbwQun6oGSdLKdciE7/0M4FXAl5Nc29r+G3Ah8JEkFwDfAM6bsAZJ0go1WUBV1d8B2c/LZ031uZKk1cGZJCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXZosoJK8L8nuJNcvajsmyfYku9rz2qk+X5K0sk15BPU/gRfs1bYZ2FFVJwM72rokSb9gsoCqqs8D396r+Rxga1veCpw71edLkla2pT4Htb6q7mjLdwLr97dhkk1J5pPMLywsLE11kqRuLNtFElVVQD3A61uqaq6q5tatW7eElUmSerDUAXVXkuMA2vPuJf58SdIKccgSf942YCNwYXu+fIk/f9lt2PyJ5S4BgFsvPHu5S5CkBzTlZeYfAv4eeEKS25JcwBBMz02yC3hOW5ck6RdMdgRVVa/Yz0tnTfWZkqTVY6mH+NQJhxol9c6pjiRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV0yoCRJXTKgJEldMqAkSV3yflBaVt6XStL+eAQlSeqSASVJ6pJDfFInHO68P78PeQQlSeqSASVJ6lKqauk/NHkB8E5gDfDeqrrwgbafm5ur+fn5X+ozexkukKSV7GAMeSbZWVVzB9puyY+gkqwB/gp4IXAK8Iokpyx1HZKkvi3HEN9TgZur6paquhe4FDhnGeqQJHVsOa7ieyzwj4vWbwOetvdGSTYBm9rq95J89QDv+2jgmwelwpVnVvs+q/0G+z6Lfe+i33nrQXmbfzlmo24vM6+qLcCWsdsnmR8zprkazWrfZ7XfYN9nse+z2O/lGOK7HThh0frxrU2SpJ9ZjoD6InBykhOTHAacD2xbhjokSR1b8iG+qrovyX8GPs1wmfn7quqGg/DWo4cDV6FZ7fus9hvs+yyauX4vy99BSZJ0IM4kIUnqkgElSerSig+oJC9I8tUkNyfZvNz1HAxJ3pdkd5LrF7Udk2R7kl3teW1rT5J3tf5fl+T0RftsbNvvSrJxOfryYCQ5IcmVSW5MckOS17f2Wej74Um+kORLre9/3NpPTHJV6+OH24VFJHl4W7+5vb5h0Xu9ubV/Ncnzl6dHD16SNUmuSXJFW5+Jvie5NcmXk1ybZL61rfqf+VGqasU+GC6y+BpwEnAY8CXglOWu6yD060zgdOD6RW1vAza35c3AW9vyi4BPAgHOAK5q7ccAt7TntW157XL37QD9Pg44vS0/EvgHhumwZqHvAY5sy4cCV7U+fQQ4v7W/B3hdW/5PwHva8vnAh9vyKe334OHAie33Y81y92/kd/BG4IPAFW19JvoO3Ao8eq+2Vf8zP+ax0o+gVuW0SVX1eeDbezWfA2xty1uBcxe1v78G/xc4OslxwPOB7VX17ar6DrAdeMH01T90VXVHVV3dlr8L3MQw88gs9L2q6ntt9dD2KODZwGWtfe++7/lOLgPOSpLWfmlV/aiqvg7czPB70rUkxwNnA+9t62FG+r4fq/5nfoyVHlD7mjbpsctUy9TWV9UdbflOYH1b3t93sKK/mzZs8xSGI4mZ6Hsb4roW2M3wD8zXgLur6r62yeJ+/KyP7fV/Ao5lhfYdeAfwe8BP2/qxzE7fC/jbJDszTPEGM/IzfyDdTnWk/auqSrJq/z4gyZHAXwNvqKp7hv8cD1Zz36vqJ8BpSY4GPg48cZlLWhJJXgzsrqqdSZ613PUsg2dW1e1JHgNsT/KVxS+u5p/5A1npR1CzNG3SXe1Qnva8u7Xv7ztYkd9NkkMZwumSqvpYa56Jvu9RVXcDVwJPZxjC2fMfycX9+Fkf2+tHAd9iZfb9GcBLktzKMEz/bIb7xc1C36mq29vzbob/mDyVGfuZ35+VHlCzNG3SNmDPlTkbgcsXtf/HdnXPGcA/taGBTwPPS7K2XQH0vNbWrXYe4SLgpqp6+6KXZqHv69qRE0mOAJ7LcA7uSuDlbbO9+77nO3k58NkazpZvA85vV7qdCJwMfGFpevHQVNWbq+r4qtrA8Dv82ap6JTPQ9ySPSPLIPcsMP6vXMwM/86Ms91Uav+yD4aqWf2AYr//95a7nIPXpQ8AdwI8ZxpIvYBhj3wHsAj4DHNO2DcMNIL8GfBmYW/Q+r2E4UXwz8Orl7teIfj+TYTz+OuDa9njRjPT9VOCa1vfrgf/e2k9i+Ef2ZuCjwMNb++Ft/eb2+kmL3uv323fyVeCFy923B/k9PIufX8W36vve+vil9rhhz79hs/AzP+bhVEeSpC6t9CE+SdIqZUBJkrpkQEmSumRASZK6ZEBJkrpkQEkrQJJ/keSy/bz2uSRzS12TNDWnOpI6l+SQqvp//PyPVqWZYEBJB1mb6PaKqnpSW/9d4EiGGepfC9wH3FhV57fZA/4SeBLDDOZ/VFWXJ/kt4KVtvzXt/j5XVNWT2kwTFwNPBr4CHLGE3ZOWjAElLZ3NwIlV9aM90xoxzHzw2ap6TWv7QpLPtNdOB06tqm8vvikf8DrgB1X1r5KcCly9RPVLS8pzUNLSuQ64JMl/YDiKgmHOtM3tNhufY5jG53Htte1Vtfd9wWC4oeUHAKrquva+0qpjQEkH333c/3fr8PZ8NsM8aqcDX2wzcQd4WVWd1h6Pq6qb2vbfX7KKpQ4ZUNLBdxfwmCTHJnk48GKG37UTqupK4E0Mt4g4kmHG6d9pM7mT5Ckj3v/zwL9v2z+JYaJZadXxHJR0kFXVj5P8CcNM27czXMiwBvhAkqMYjpreVVV3J/lThrvJXpfkYcDXGQLtgbwbuDjJTQy35Ng5UVekZeVs5pKkLjnEJ0nqkgElSeqSASVJ6pIBJUnqkgElSeqSASVJ6pIBJUnq0v8H4vrertyCrXsAAAAASUVORK5CYII=\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"dc":{"key":"24"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 4. Overall 1-day retention\n<p>In the plot above we can see that some players install the game but then never play it (0 game rounds), some players just play a couple of game rounds in their first week, and some get really hooked!</p>\n<p>What we want is for players to like the game and to get hooked. A common metric in the video gaming industry for how fun and engaging a game is <em>1-day retention</em>: The percentage of players that comes back and plays the game <em>one day</em> after they have installed it.  The higher 1-day retention is, the easier it is to retain players and build a large player base. </p>\n<p>As a first step, let's look at what 1-day retention is overall.</p>"},{"metadata":{"dc":{"key":"24"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# The % of users that came back the day after they installed\nretention_1_perc = round(df['retention_1'].sum()/df['retention_1'].count() *100,2)\n\nprint(\"percentage of players that comes back and plays the game one day after installation: \"\n      + str(retention_1_perc) + \"%\")\n","execution_count":81,"outputs":[{"output_type":"stream","text":"percentage of players that comes back and plays the game one day after installation: 44.52%\n","name":"stdout"}]},{"metadata":{"dc":{"key":"31"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 5. 1-day retention by AB-group\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/belle_cookie.png\" style=\"width:200px; float:right\"> </p>\n<p>So, a little less than half of the players come back one day after installing the game. Now that we have a benchmark, let's look at how 1-day retention differs between the two AB-groups.</p>"},{"metadata":{"dc":{"key":"31"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Calculating 1-day retention for each AB-group\ndf.groupby('version')['retention_1'].mean()","execution_count":83,"outputs":[{"output_type":"execute_result","execution_count":83,"data":{"text/plain":"version\ngate_30    0.448188\ngate_40    0.442283\nName: retention_1, dtype: float64"},"metadata":{}}]},{"metadata":{"dc":{"key":"38"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 6. Should we be confident in the difference?\n<p>It appears that there was a slight decrease in 1-day retention when the gate was moved to level 40 (44.2%) compared to the control when it was at level 30 (44.8%). It's a small change, but even small changes in retention can have a large impact. But while we are certain of the difference in the data, how certain should we be that a gate at level 40 will be worse in the future?</p>\n<p>There are a couple of ways we can get at the certainty of these retention numbers. Here we will use bootstrapping: We will repeatedly re-sample our dataset (with replacement) and calculate 1-day retention for those samples. The variation in 1-day retention will give us an indication of how uncertain the retention numbers are.</p>"},{"metadata":{"dc":{"key":"38"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Creating an list with bootstrapped means for each AB-group\nboot_1d = []\niterations = 500\nfor i in range(iterations):\n    boot_mean = df.sample(frac = 1, replace = True).groupby('version')['retention_1'].mean()\n    boot_1d.append(boot_mean)\n   \n# Transforming the list to a DataFrame\nboot_1d = pd.DataFrame(boot_1d)\nboot_1d\n# A Kernel Density Estimate plot of the bootstrap distributions\nboot_1d.plot.kde()","execution_count":85,"outputs":[{"output_type":"execute_result","execution_count":85,"data":{"text/plain":"<matplotlib.axes._subplots.AxesSubplot at 0x7f53d9432080>"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<matplotlib.figure.Figure at 0x7f53d911e160>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xd8XNWZ+P/PozqyumQ1S7Ysuci9m2bTEmoAEyAEyCbBaaQQNlk22ZCEBDYhS34JgW82ZMPChkAKgYTea0JvLtiWuyVZttWsLqtLozm/P+6MLNuyrTJ37p3x83695nU1d+7c+2hAfuac89xzxBiDUkop5TZRTgeglFJKDUcTlFJKKVfSBKWUUsqVNEEppZRyJU1QSimlXEkTlFJKKVfSBKWUUsqVNEEppZRyJU1QSimlXCnG6QDGY+LEiWbq1KlOh6GUUmoU1q1b12iMyTrecWGdoKZOncratWudDkMppdQoiMiekRynXXxKKaVcybYEJSL3i0i9iGwesu8REdngf1SKyAb//qki0j3ktXvsiksppVR4sLOL7wHgbuCPgR3GmKsCP4vIr4C2IceXG2MW2RiPUkqpMGJbgjLGvCkiU4d7TUQE+DTwMbuur5RSdujv76eqqoqenh6nQ3E9j8dDQUEBsbGxY3q/U0USpwP7jTG7huwrEpGPgAPAzcaYt4Z7o4hcB1wHMGXKFNsDVUqpoaqqqkhOTmbq1KlY37XVcIwxNDU1UVVVRVFR0ZjO4VSRxDXAX4c8rwWmGGMWAzcCD4lIynBvNMbca4xZZoxZlpV13CpFpZQKqp6eHjIzMzU5HYeIkJmZOa6WZsgTlIjEAJcDjwT2GWN6jTFN/p/XAeXAzFDHppRSI6HJaWTG+zk50YI6B9hujKkK7BCRLBGJ9v9cDMwAKhyITSmllEvYWWb+V+A9oEREqkTkS/6XrubQ7j2AM4BN/rLzR4GvGWOa7YpNRRhvL7z1K3jyG1D5jtPRKOW4H//4x7z66qtOhzFudlbxXXOU/auH2fcY8JhdsagI5vPB31fDjuchPgU2/hWueQRmnud0ZErZyhiDMYaoqCPbGT/5yU8ciCj4dCYJFd42PWwlp/P/C27cBtlz4YnroLvF6ciUGpGbbrqJ3/72t4PPb731Vu644w5++ctfsnz5chYsWMAtt9wCQGVlJSUlJXz+859n3rx57Nu3j9WrVzNv3jzmz5/PXXfdBcDq1at59NFHAXjttddYvHgx8+fP54tf/CK9vb2ANVXcLbfcwpIlS5g/fz7bt28P8W9+fJqgVPjyDcA/b4f8pXDy1yE+CS77HXS3wtt3OR2dUiNy1VVX8be//W3w+d/+9jeysrLYtWsXH374IRs2bGDdunW8+eabAOzatYtvfOMbbNmyhcbGRqqrq9m8eTOlpaV84QtfOOTcPT09rF69mkceeYTS0lK8Xi+/+93vBl+fOHEi69ev5+tf/zp33HFHaH7hUdAEpcLXjhegbS+s+DYEujly58Pcy2DtA9DX6Wh4So3E4sWLqa+vp6amho0bN5Kenk5paSkvv/wyixcvZsmSJWzfvp1du6zbRgsLCznllFMAKC4upqKightuuIEXX3yRlJRD787ZsWMHRUVFzJxpFUVfe+21g4kO4PLLLwdg6dKlVFZWhuC3HR1NUCp8bXoEknKg5BOH7j/pOuhtg01/G/59SrnMlVdeyaOPPsojjzzCVVddhTGG73//+2zYsIENGzZQVlbGl75k1ZklJiYOvi89PZ2NGzdy1llncc899/DlL395VNeNj48HIDo6Gq/XG7xfKEg0Qanw1NcFZa/C7Esg+rBanymnWGNRH/3ZmdiUGqWrrrqKhx9+mEcffZQrr7yS888/n/vvv5+Ojg4Aqqurqa+vP+J9jY2N+Hw+rrjiCm677TbWr19/yOslJSVUVlZSVlYGwJ/+9CfOPPNM+3+hIAnr9aDUCaz8NejvshLU4URg/qfgtf+E1r2QplNiKXebO3cu7e3t5Ofnk5eXR15eHtu2bePUU08FICkpiT//+c9ER0cf8r7q6mq+8IUv4PP5ALj99tsPed3j8fCHP/yBK6+8Eq/Xy/Lly/na174Wml8qCMQY43QMY7Zs2TKjCxaeoJ6+AbY+Bd+tOLIFBdBcAf+9GM67DU67IfTxqYi1bds2Zs+e7XQYYWO4z0tE1hljlh3vvdrFp8LT7jehcOXwyQkgoxjyFsKWJ0Ibl1IqaDRBqfDTuhdaKqHo9GMfN/sSqF4H7ftDEpZSKrg0Qanws9u/EkvRGcc+bvq51rb8NXvjUUrZQhOUCj/7PgBPGmQdZxwgd4FVhr7rldDEpZQKKk1QKvxUr7dmjxhmDrJDREXB9HOg/B8w4L57PJRSx6YJSoWXvk6o3wr5S0Z2/PRzoKfVGotSSoUVTVAqvNRuAjNgtaBGYtrZIFFQFjndfFUtXdz8ZCnf+ftGyurbnQ5HKdtoglLhpcZ/p/ykEbagEtKhYLnVzRcBalq7ufx/3uVva6t4vrSWK373Hrsbdc5BdWxPPvkkW7duHdN777nnHubPn8+iRYtYuXLlIee5/fbbmT59OiUlJbz00kvBCneQziShwkv1OkidDMk5I39P8dnw5i+sJTgS0u2LzWbGGH7wRCkdvV6e+eZKEmKjufg3b/Hjpzbzxy+epMuQO+A/n9nC1poDQT3nnEkp3HLJ3KCe88knn+Tiiy9mzpw5o37vZz7zmcHZJ55++mluvPFGXnzxRbZu3crDDz/Mli1bqKmp4ZxzzmHnzp1HzHYxHtqCUuGldpN1A+5oFJ8FxgeVb9sRUci8uq2e13c08N3zSyjJTWZK5gT+9eMzeGtXI+v3tjodngqxn/70p5SUlLBy5UquueYa7rjjDu677z6WL1/OwoULueKKK+jq6uLdd9/l6aef5rvf/S6LFi2ivLyc8vJyLrjgApYuXcrpp59+zLWghs6Q3tnZOfhF6KmnnuLqq68mPj6eoqIipk+fzocffhjU31FbUCp89HdDcznMu3x07ytYBnFJUP7P4efuCwPGGO7+ZxmTMxL43CmFg/uvOWkKv351F39+fw9LC8O3dRiugt3SGak1a9bw2GOPsXHjRvr7+1myZAlLly7l8ssv5ytf+QoAN998M7///e+54YYbWLVqFRdffDGf+tSnAPj4xz/OPffcw4wZM/jggw/4xje+wT/+cfRu8N/+9rfceeed9PX1DR5XXV09uOwHQEFBAdXV1UH9PTVBqfDRsN1qCWWPspsiOhamroSK120JKxQ+3N3Mxn2t/OyyecREH+z4SIyP4eKFeTy9oYae/gE8scHrXlHu9c4773DppZfi8XjweDxccon1xWvz5s3cfPPNtLa20tHRwfnnn3/Eezs6Onj33Xe58sorB/cFVtk9muuvv57rr7+ehx56iNtuu40HH3wwuL/QUWgXnwof+/2DsznzRv/e4rOs1lfr3mBGFDJ//XAvyZ4YLl9ccMRrF8zLo7NvgLd2NToQmXKT1atXc/fdd1NaWsott9xCT0/PEcf4fD7S0tIG15rasGED27ZtG9H5r776ap588kkA8vPz2bdv3+BrVVVV5OfnB+cX8dMEpcLH/i0QkwAZRaN/b/FZ1jYMW1FtXf08v7mOTy7KJyHuyBbSqcWZpHhieHFznQPRKSesWLGCZ555hp6eHjo6Onj22WcBaG9vJy8vj/7+fv7yl78MHp+cnEx7u3VLQkpKCkVFRfz9738HrO7jjRs3HvVagZV8AZ577jlmzJgBwKpVq3j44Yfp7e1l9+7d7Nq1i5NOOimov6cmKBU+6rdA9iyIGkM3VtYsSMoNywT11MZq+rw+rlo+edjX42KiOGNmFm+XNRDOy+eokVu+fDmrVq1iwYIFXHjhhcyfP5/U1FR++tOfcvLJJ7NixQpmzZo1ePzVV1/NL3/5SxYvXkx5eTl/+ctf+P3vf8/ChQuZO3cuTz311FGvdffddzN37lwWLVrEnXfeOdi9N3fuXD796U8zZ84cLrjgAn77298GtYIPdD0oFU5+OR1mnA+f/O3Y3v/4V61VeL+z6/jTJLnIFb97l85eLy9+++iT4z70wV5+8EQp//j3MynOSgphdCcet6wH1dHRQVJSEl1dXZxxxhnce++9LFkywvsDQ0jXg1KRr6MeOhsgZxxVU8VnQVej1RILEzWt3azb08LFC/KOedxp0zIBeLe8KRRhKRe47rrrWLRoEUuWLOGKK65wZXIaL9uq+ETkfuBioN4YM8+/71bgK0CD/7AfGGOe97/2feBLwADwr8aY4N+WrMJXg/8+jexxfHMtPtPalv8TcuePP6YQeL60FoCLFkw65nGFmROYlOrhvYomPjukDF1Froceeiio5/vZz342OC4VcOWVV/LDH/4wqNcZDTvLzB8A7gb+eNj+u4wxdwzdISJzgKuBucAk4FURmWmMGbAxPhVOmsqs7cQZYz9HyiRrLKridVjxr0EJy27PbKpl7qQUiiYmHvM4EWFJYTof6Q27aox++MMfOpqMhmNbF58x5k2geYSHXwo8bIzpNcbsBsqA4JaDqPDWVA4xHkg+dkviuIrPgj3vgvfY9324wb7mLjbua+Xi47SeAhZNTqO6tZuGdvf/bkqNhBNjUN8UkU0icr+IBG59zwf2DTmmyr/vCCJynYisFZG1DQ0Nwx2iIlFTOWQUj7+4ofgs8HZbix663HP+7r3jjT8FLJqcBsCGfdqKUpEh1Anqd8A0YBFQC/xqtCcwxtxrjFlmjFmWlZUV7PiUWzWXQ+a08Z9n6kqQ6LAoN392Uw0LC1KZnDFhRMfPnZRKdJSwUROUihAhTVDGmP3GmAFjjA+4j4PdeNXA0Js8Cvz7lLJWw23eDRlBSFDxydbyGy5PUHubuthcfYBPzB9Z6wkgIS6aWbnJ2oJSESOkCUpEhv61XQZs9v/8NHC1iMSLSBEwAwjutLgqfLXtA18/ZE4PzvmKz4Kaj6zlN1zqhc1W995oEhTAwslpbKxq1Rt21SHGsx5UwGOPPYaIMPTe07BdD0pE/gqcBUwUkSrgFuAsEVkEGKAS+CqAMWaLiPwN2Ap4geu1gk8Naiq3tsHo4gMrQb3xc9j9FsxZFZxzBtnzpbXMzx95917AvEmpPPTBXqpaukf9XjUGL9wEdaXBPWfufLjw50E95XjWgwJrCqVf//rXnHzyyYP7wno9KGPMNcaYPGNMrDGmwBjze2PM54wx840xC4wxq4wxtUOO/5kxZpoxpsQY84Jdcakw1OxPUMHo4oODy2+4tJuvqqWLjVVto249AczOSwZga21wF9FT7hOq9aAAfvSjH/G9730Pj8czuE/Xg1IKrHug4pIhKTs453P58hsvlFqTvn5ifu6o31uSm4wIbKs9wPlzR/9+NUpBbumMVCjXg1q/fj379u3joosu4pe//OXgfl0PSimwuvgyiyGYS5oXnwU7X7SW30ibErzzBsHzm62bcwszj31z7nAmxMVQlJnINm1BRbRQrQfl8/m48cYbeeCBB2z5PY5HE5Ryv+ZyyF8a3HMWn21tK16HJZ8P7rnHoaa1m4/2tvLd80vGfI7ZeSmUVrcFMSoVLlavXs2TTz7JwoULeeCBB3j99dePOGboelDH097ezubNmznrrLMAqKurY9WqVTz99NO6HpRSePusVk6wxp8CskpcufzGC5sD3XujH38KmJ2XzN7mLtp7+oMVlnKZUK0HlZqaSmNjI5WVlVRWVnLKKafw9NNPs2zZMl0PSilaKq1l3oNVwRcgYnXzVbwBPl9wzz0OL5TWMjvv+HPvHcvsvBQAdtS1Byss5TKhXA/qaEKxHhTGmLB9LF261KgIt+05Y25JMWbfmuCf+6OHrHPXbAz+ucdgX3OnKfzes+Y3r+0c13mqW7pM4feeNX98d3dwAlOH2Lp1q9MhGGOMaW9vN8YY09nZaZYuXWrWrVvncETDG+7zAtaaEfwbr2NQyt0GS8yLg3/u4rOsbcXrkLcg+OcfpSfWWxVQly4aXz9+XqqH1IRYttZqCyqSXXfddWzdupWenh6uvfZaXQ9KqZBrKoeEDJiQEfxzp+S5ZvkNYwyPra/i1OLMcd9gKyKU5Cazo04r+SKZrgellNOayoI//jRU8Vmw7kHo74FYz/GOts26PS1UNnXxzY+NY72rIWblJvP4+mqMMUgwy/MVQER+rnasB2XGOeWWFkkod2uuCN4cfMMpPttafqPK2akfH1tfxYS4aC6cF5yba2flptDR66WqpTso51MHeTwempqadL7D4zDG0NTUdMjsE6OlLSjlXn1dcKA6+CXmQ01dcXD5jaIz7LvOMfT0D/DsxlounJdHYnxw/iRLcq0pj7bXteucfEFWUFBAVVUVuh7d8Xk8HgoKCsb8fk1Qyr2aK6xtpg0FEgFDl9/4+I/tu84xvLSljvZeL1csDd5NjoEEtaPuAOfOyQnaeRXExsZSVFTkdBgnBO3iU+4VqOCzs4sPHF9+49F1VeSnJXBKUWbQzpkUH0NBegLb9V4oFcY0QSn3aiqztnaUmA81dYV1M3DVOnuvM4y6th7eKWvkiiX5REUFd9B9Vm6y3qyrwpomKOVeTRXWdETxyfZeZ9JikCioWmPvdYbxxEfV+AxcvmTs/fRHU5KbTEVjJ71eXVpNhSdNUMq97C4xD4hPhuw5Ia/kM8bw6Lp9LCtMZ+o4pjY6mpLcFAZ8hvL6zqCfW6lQ0ASl3Ku53P7uvYCC5VYXXwjn5dtY1UZ5QydXLA1+6wmsLj6AHfv1hl0VnjRBKXfqaYPOBvsLJAIKlkNvGzTtCs31gMfWVREfE8VFC8Y+c/mxFE1MJDZatFBChS1NUMqdmgIVfCHo4gMrQQHsC003X693gKc31nD+3FxSPLG2XCM2Oorp2VooocKXJijlToP3QIWoBZU5HTxpISuUeG1bPW3d/bZ17wXMyk1mu04aq8KUJijlTk1lgEB6iG6IjIqCgmVQtTYkl3t0XRU5KfGsnD7R1uuU5CZTd6CHti5dvFCFH01Qyp2ayiG1ILQTuE5aDA3bod/e+esa2nt5Y2cDly0uIDrI9z4d7uCUR1ooocKPJijlTs3loRt/CshdAGYA9m+19TLPbKxhwGe4YknwpjY6moOVfNrNp8KPbQlKRO4XkXoR2Txk3y9FZLuIbBKRJ0Qkzb9/qoh0i8gG/+Meu+JSYcAYq4vPzklihxNYtLBuo62XeWpDNXPyUpiRY/MNyEBuiocUT4xW8qmwZGcL6gHggsP2vQLMM8YsAHYC3x/yWrkxZpH/8TUb41Ju19VslZmHqkAiIK0QPKlQa1+CqmjoYGNVG5cttr/1BNbihbNyU7SST4Ul2xKUMeZNoPmwfS8bY7z+p+8D9pYwqfAUmIMv1F18IlY3X+0m2y7x5IYaROCShZNsu8bhSnKT2VnXrusXqbDj5BjUF4EXhjwvEpGPROQNETn9aG8SketEZK2IrNX1WCJUYBbzUHfxAeQthP1bYCD4VW/GGJ7aUM1p0zLJTQ1d8UdJbjLtvV6qW3XxQhVeHElQIvJDwAv8xb+rFphijFkM3Ag8JCIpw73XGHOvMWaZMWZZVlZWaAJWodVUZi0imF4Y+mvnLYSBXmjcGfRTb6k5wJ6mLlaFsPUEQwoltJtPhZmQJygRWQ1cDPyL8fc5GGN6jTFN/p/XAeXAzFDHplyiqdxKTtH2zLBwTLn+Qgkbuvle3rqfKIFzZod2AcGZQ1bXVSqchDRBicgFwH8Aq4wxXUP2Z4lItP/nYmAGUBHK2JSLNJeHvkAiYOIMiEmwpVDi5S11LJuaQWZSfNDPfSwpnljy03TxQhV+7Cwz/yvwHlAiIlUi8iXgbiAZeOWwcvIzgE0isgF4FPiaMaZ52BOryGaMtQ6UE+NPAFHRkD0b6rcE9bT7mrvYXtfOeQ4tv24tXqg366rwEmPXiY0x1wyz+/dHOfYx4DG7YlFhpL0O+jtDX8E3VM4c2PlSUE/58tb9AJzrUIIqyU3mjZ0N9Hl9xMXo/fkqPOj/qcpdnCoxHyp7jrXUR0fwqkRf27afmTlJFGYGf2HCkSjJTcbrM5Q3dDhyfaXGQhOUchcnS8wDsudY2yB183X3DbC2soUzZzpXdTor1yqK1Uo+FU40QSl3aSqH6Hhrolin5My1tkGak29NZTN9Az5W2Dxz+bEUZ+nihSr8aIJS7tJUDhlFVrGCU5KyYcJEqA9OgnqnrJHYaOGkooygnG8sYqOjmJaVpIUSKqxoglLu0lzubPdeQPbsoCWot8saWTIlnQlxttUkjUhJrq6uq8KLJijlHr4BayVdJwskAnLmQv128PnGdZrmzj621Bzg9BnOde8FzMpNoaath7ZuXbxQhQdNUMo92qpgoM8dCSp7jlXu3rpnXKd5v6IJgNMcHH8K0CmPVLjRBKXcI1DB59QsEkMFCiXG2c23prIZT2wU8/NTgxDU+JQMJigdh1LhQROUco8mF5SYB2SVWNtxVvKt39PCwoI0YqOd/1PLS/WQrIsXqjDi/F+NUgFN5RCbCMm5TkcC8cnWAobjaEF19w2wpeYASwvTgxjY2IkIs3NT2FqrLSgVHjRBKfdoKoPMYmvhQDfImTuuBLWxqhWvz7gmQQEsnJzKlpoD9HnHV/yhVChoglLu4ZYS84DsOdC4C7y9Y3r7uj0tACyZ4p4EtWhyOn1eH9u0FaXCgCYo5Q4D/dCyxx0FEgHZs8EMjHnxwnV7WpiWlUh6YlyQAxu7RVPSANiwr9XhSJQ6Pk1Qyh1a9ljJwA0l5gHjmPLIGMP6vS2u6t4DmJTqISs5XhOUCguaoJQ7uKnEPCBzOkTFjmnS2KqWblq7+llQkGZDYGMnIiyanKYJSoUFTVDKHQLLbLhpDCo61io3H0MLanN1G4Ar7n863OIpaexu7KS1q8/pUJQ6Jk1Qyh2aysGTChOcm1B1WNlzxlTJV1rdRkyUDN4c6yaLJus4lAoPmqCUOzSXW11qbikxD8iZAweqobtlVG8rrW5jZk4ynlgHZ2U/igUFaYhoglLupwlKuUOTy0rMA7IDUx5tH/FbjDFsqTnAvPwUm4Ian6T4GEpykgfL4JVyK01Qynn93dZEsW4qkAjIGf3qujVtPTR39rly/CngpKIM1u1poX9Ab9hV7qUJSjmveTdg3FViHpCSD/GpoyqUKK2yCiTmuThBnVKcSVffAKX+Yg6l3EgTlHJeoMQ8o9jZOIYjMurFC7fUtBEdJczOc2cXHzC4uu8HFc0OR6LU0WmCUs4LlJi7sQUFVjff/q1gzIgOL61uY0Z2kisLJAImJsUzIztpcL0qpdxIE5RyXlM5JGZZZeZulD0Hetusar4R2FJzgLmTXPq7DHFKcSZrK5vx6jiUcilbE5SI3C8i9SKyeci+DBF5RUR2+bfp/v0iIv8tImUisklEltgZm3KR5gp3FkgEjGLKo+bOPhrae5md5777nw53cnEGnX0DbK7RiWOVO9ndgnoAuOCwfTcBrxljZgCv+Z8DXAjM8D+uA35nc2zKLZrK3FliHpA929qOoJJvu3+1WjfeoHu4k4syAbSbT7mWrQnKGPMmcPgo7KXAg/6fHwQ+OWT/H43lfSBNRPLsjE+5QG87dOy31oFyq4R0q5pvBC2oHf7Vakty3J+gspKtcah3yhqdDkWpYTkxBpVjjKn1/1wH5Ph/zgf2DTmuyr/vECJynYisFZG1DQ0N9kaq7NfkwklihzPCKY927m8nfUIsWcnxIQhq/M6YmcUHu5vp7htwOhSljuBokYQxxgAjK406+J57jTHLjDHLsrKybIpMhcxgibmLu/jAquRr2GGtW3UM2+vaKclNRtw2ZdNRnDEziz6vjw92azefch8nEtT+QNedf1vv318NTB5yXIF/n4pkTRXW1o33QA2VPRd8/QdL4ofh8xl21rUzK9e99z8d7uSiDOJjonhjp/ZGKPcZUYISkcdF5CIRCUZCexq41v/ztcBTQ/Z/3l/NdwrQNqQrUEWqpjJrfCdugtORHFtgyqO6zUc9pLq1m86+AWaGwfhTgCc2mpOLM3lTE5RyoZEmnP8BPgPsEpGfi0jJSN4kIn8F3gNKRKRKRL4E/Bw4V0R2Aef4nwM8D1QAZcB9wDdG/muosNVc7v7WE0DWLIiOg7qNRz1ke6BAIgwq+IY6c2YW5Q2d7GvucjoUpQ4RM5KDjDGvAq+KSCpwjf/nfViJ5M/GmGE75o0x1xzllB8f5lgDXD+iqFXkaCqDOZ88/nFOi4617oeqPXqC2rnfSlAzc5JCFVVQnDkzi58Cb+5q4F9OLnQ6HKUGjbjLTkQygdXAl4GPgF8DS4BXbIlMRb6uZmudJbdOcXS4vIVWgjrKlEfb69rJT0sg2RMb4sDGZ1pWIvlpCdrNp1xnpGNQTwBvAROAS4wxq4wxjxhjbgDC6+uico9mf4GE20vMA/IWQU8btO4Z9uUddQeYFWbdewAiwhkzs3inrEmX31CuMtIW1H3GmDnGmNsDhQsiEg9gjFlmW3QqsgUq4txeYh6Qt9DaDtPN1+f1UdHQGXbjTwFnzpxIR6+X9bqIoXKRkSao24bZ914wA1EnoKZykChIn+p0JCOTPQeiYoZNULsbO/H6TNgmqNOmTyQ6Snhzl3bzKfc4ZoISkVwRWQokiMhiEVnif5yF1d2n1Ng1lUHaFIiJczqSkYn1QNbsYRNUeUMHANOywrPHO8UTy+LJaby5U6c9Uu5xvCq+87EKIwqAO4fsbwd+YFNM6kTRXB4+3XsBeQth54tWocSQ2SLK660EVZyV6FRk43bGzCzuenUnTR29ZCaFx1RNKrIdswVljHnQGHM2sNoYc/aQxypjzOMhilFFImOsWSTCpUAiIG8hdDXCgZpDdpc3dJCflsCEuBHdueFKZ8zMwhh4WyePVS5xzL8mEfmsMebPwFQRufHw140xdw7zNqWOr6Me+trDp8Q8YNIia1vzEaQenMu4vKEzrFtPAPPzU0mbEMsbOxu4dNER8zQrFXLHK5II/MUlAcnDPJQam8AkseGWoHLnW4US1WsHdxljqGjoCNvxp4DoKGHl9Im8tasRM8Ll7ZWy0zFbUMaY//Vv/zM04agTRriVmAfEJlhJqupggtp/oJfOr2ozAAAgAElEQVTOvgGmZYd3ggKrm+/ZTbVsq21nzqTwmfRWRaaR3qj7CxFJEZFYEXlNRBpE5LN2B6ciWFM5RMVC6uTjH+s2Bcuhej34rDWUDlbwhXcXH8AZM6wlbLTcXLnBSO+DOs8YcwC4GKgEpgPftSsodQJoKoOMIogOw6KCguXQ3wn124DwLzEfKjfVQ0lOMm9pglIuMNIEFfhX5CLg78aYNpviUSeK5orw694LKPBPnlK1BrBKzJPiY8gOk1V0j+fUaZms29NCn1enPVLOGmmCelZEtgNLgddEJAvosS8sFdF8PitBhVuBREB6EUzIHByHKm/oZFpWYtisons8pxRn0NPvo7S61elQ1AluRAnKGHMTcBqwzL+0RidwqZ2BqQh2oBq8PeGboESsbr5ACyoCKviGOqkoE4D3K5odjkSd6EazQu4s4CoR+TzwKeA8e0JSES9QYh6uXXxgdfM17qCjrYnatp6IqOALyEiMY2ZOEh/s1gSlnDWiEWoR+RMwDdgADPh3G+CPNsWlIlmgxDzcZpEYqmA5AA3b3gWiIqKCb6iTizJ5fH0V3gEfMdGj+R6rVPCMtIRqGTDH6N17KhiaKiAmAZLznI5k7CYtAYSeyveB0yKqiw/gpKIM/vT+HrbUHGDh5DSnw1EnqJF+NdoM5NoZiDqBNJdb409RYfzN3JMCOXNJqF1DdJQwJTOyJvc/uSgDgDWV2s2nnDPSFtREYKuIfAj0BnYaY1bZEpWKbE1l1tpK4W7KqeSu/TNT0+OJj4l2Opqgyk7xkJ+WwEf7tJJPOWekCepWO4NQJ5ABL7RUwuxLnI5k/ApPxbPmPs5KrXU6ElssmpzGhr2aoJRzRlpm/gbWDBKx/p/XAOttjEtFqtY94POGd4GE38DkUwE4NWanw5HYY/GUNKpbu6lv11selTNGOhffV4BHgf/178oHnrQrKBXBmiusbTiXmPtVe9PY48tmVu9mp0OxxSJ/cYS2opRTRjpKfT2wAjgAYIzZBWTbFZSKYE2BZTbCvwVV3tDBWlNCTtsGawHGCDMvP5WYKGGDjkMph4w0QfUaY/oCT0QkBus+qFETkRIR2TDkcUBEvi0it4pI9ZD9nxjL+ZXLNZVBfAokTnQ6knErb+jgQ98sYnuaoHGX0+EEnSc2mll5yZqglGNGmqDeEJEfAAkici7wd+CZsVzQGLPDGLPIGLMIa26/LuAJ/8t3BV4zxjw/lvMrl2suh4xia7qgMFfe0MGu+HnWk73vOhuMTRZNTmNTVRs+X+S1EJX7jTRB3QQ0AKXAV4HngZuDcP2PA+XGmD1BOJcKB01lEdG9B1Be30lM1kxIzIK97zsdji0W5KfR0eulsqnT6VDUCWikVXw+rKKIbxhjPmWMuS9Is0pcDfx1yPNvisgmEblfRNKHe4OIXCcia0VkbUODrlkTVry90FYVvpPEHqa8oYPi7CSYcgrsicwWVGBV3S01BxyORJ2IjpmgxHKriDQCO4Ad/tV0fzzeC4tIHLAKq7sQ4HdY8/0tAmqBXw33PmPMvcaYZcaYZVlZWeMNQ4VSSyUYX0S0oFo6+2jq7LOmOJpymlU+f6DG6bCCbkZOEjFRwtZaTVAq9I7Xgvo3rOq95caYDGNMBnAysEJE/m2c174QWG+M2Q9gjNlvjBnwt9buA04a5/mV2wQmiY2AEvOKRv8qutmJUGjdDxWJraj4mGhm5CRrC0o54ngJ6nPANcaY3YEdxpgK4LPA58d57WsY0r0nIkNnDr0Ma/4/FUkGS8yLnY0jCMrrrTGZaVlJkDMf4pJg73sOR2WPuZNS2KoJSjngeAkq1hjTePhOY0wDEDvWi4pIInAu8PiQ3b8QkVIR2QScjdV6U5GkqcxaiTZh2OHFsFLe0EFcdBQF6RMgOgYmnxSRLSiAOXkpNHb0Un9AZ5RQoXW8ufj6xvjaMRljOoHMw/Z9bqznU2GiuSIiuvfASlBFExOJjvKXyxeugH/8FLqaYUKGs8EF2dwhhRLZKR6Ho1EnkuO1oBb6b6Q9/NEOzA9FgCqCNJVHRIEEQHlDpzX+FFC4wtpGYCtqtj9BaaGECrVjJihjTLQxJmWYR7IxZsxdfOoE1NcJ7TURMf7U5/Wxt7nr0EUK85dAjCciE1SKJ5YpGRPYUtPmdCjqBBPGK8apsBJBk8Tube5kwGcOTVAx8dYy8Hvedi4wG82dlKKVfCrkNEGp0AiUmE+c4WwcQVA2tIJvqMIVUFcKPZHX0pidl8Kepi66+rxOh6JOIJqgVGg0Bu6BCv8uvvIG6x6o4qzEQ1+YusK6EXnvBw5EZa+ZOckA7Nrf4XAk6kSiCUqFRlMZpORDXOLxj3W58oYO8lI9JMYfVgSbvwyiYiOym29WrpWgdtS1OxyJOpFoglKh0VQWQXPwdR7ZvQcQNwHyl0ZkocTkjAl4YqPYsV8TlAodTVDKfsZA066IKDE3xlBR38G0w7v3AgpPg5qPrKrFCBIdJczITmanJigVQpqglP26mq3CgczwL5BoaO+lvdfLtOxhWlBgjUP5vLAvMsehtItPhZImKGW/QAVfBLSgygIFEhOPkqAmnwwSHZHdfCW5SdS399LSOeZJZJQaFU1Qyn6DCSr8x6DK660ENf1oLaj4ZMhbCJXvhDCq0AhU8uk4lAoVTVDKfk1lEBUDaYVORzJu5Q2dJMXHkJMSf/SDpq6A6rXQH1mTq87KtaY80nEoFSqaoJT9msogvcia9TvMlfkLJETk6AcVroCBPitJRZCclHhSPDE6DqVCRhOUsl9ETRLbMXyJ+VBTTgEk4rr5RISSXK3kU6GjCUrZy+eD5vKIGH/q6PVS29Zz9Aq+gIR0yJkHeyIrQYE1DrW9rh1jjNOhqBOAJihlrwPV4O2JiBZUhb+C77gtKLDGofZ9CN7IqngryU2mvcdLnS5eqEJAE5SyVySVmB+vgm+owhXg7YbaDTZHFVqDlXw6DqVCQBOUslcEzWJe3tBBTJRQmDnh+AcXnmZtKyNrXr4Sf4LScSgVCpqglL2ayiAuCZJynI5k3MrqOyjMnEBs9Aj+bBInQtasiBuHSk+MIzs5nh11Oqu5sp8mKGWvwCSxxyrLDhNHnST2aApPs5beGIisNZRKcpPZsV8XL1T20wSl7NVUFhHjT/0DPiobO0c2/hRQuAL62qFuk32BOWBmTjK79ncw4NNKPmUvTVDKPt5eaN0bEQlqb3MX3sOXeT+ewhXWNsK6+WblJtPr9VHZFFkztiv30QSl7NNcYa0wGwEJalQVfAEpedYKwhE2cWxgyiOt5FN2cyxBiUiliJSKyAYRWevflyEir4jILv823an4VBA07LC2WbOcjSMIjrrM+/EUrrASlM9nQ1TOmJGTRJTAdk1QymZOt6DONsYsMsYs8z+/CXjNGDMDeM3/XIWrhh2ARESJeVl9B7kpHpI9saN7Y+EK6GmF+q32BOYAT2w0Uycmsr1WCyWUvZxOUIe7FHjQ//ODwCcdjEWNV8N2SC+E2ASnIxm38oZOpmWPsvUE1owSEJHjULrshrKbkwnKAC+LyDoRuc6/L8cYU+v/uQ444uYZEblORNaKyNqGhoZQxarGomFHRHTv+XyGsv3tzMhOHv2b06ZA6pQIvGE3hb3NXXT1RVYJvXIXJxPUSmPMEuBC4HoROWPoi8aajfKIOlZjzL3GmGXGmGVZWVkhClWN2oAXmnZBVonTkYxbdWs3nX0DlOSOIUGBdT/UnnchgiZYLclNxhjYuV9v2FX2cSxBGWOq/dt64AngJGC/iOQB+Lf1TsWnxqml0loTKQJaUIFigDEnqKkroKsRGncGMSpnzc4LzMmn41DKPo4kKBFJFJHkwM/AecBm4GngWv9h1wJPORGfCoKG7dY2AlpQgX+EAxOljloE3g81OX0CE+Ki2Var41DKPk61oHKAt0VkI/Ah8Jwx5kXg58C5IrILOMf/XIWjRn+J+cSZzsYRBNvr2pmckUBS/BhXBM4ohqTciFrAMCpKmJGTrPdCKVs5sga3MaYCWDjM/ibg46GPSAVdww5IKYD4MbY6XGRHXTslOSljP4GI1c235x1rHCoC5iUEmJWTzCvb9mOMQSLkd1Lu4rYycxUpGrZHRPder3eAisZOZo11/Cmg8DRor4WW3cEJzAVm5SXT3NlHQ0ev06GoCKUJSgWfzwcNOyOiQKK8vpMBnxl7gURA4UprG0HdfIHPRLv5lF00Qanga620VpONgBZUYFmJcSeorBKYkBlR8/IF5uTbpjNKKJtoglLBV7fZ2ubOczaOINhe105stFA0cQyzSAwl4r8fKnJu2M1IjCM/LYHSak1Qyh6aoFTw1ZWCREH2HKcjGbcdde1My0oa2Sq6x1O40lp+pHXf+M/lEvPyU9hc3eZ0GCpCaYJSwbd/M2TOiIg5+LbWHGBO3jgq+IYqPM3aRlA33/z8VHY3dnKgp9/pUFQE0gSlgq+uFHLnOx3FuO0/0EN9ey/zC1KDc8KcueBJjahuvvkFaQDailK20ASlgqu7Bdr2RcT406Yq6x/dBcFKUFHRMOXUiGtBgSYoZQ9NUCq4AgUSOeHfgiqtaiVKYE5ekBIUWNMeNZVBe13wzumgQKFEIJkrFUyaoFRw7Q9U8IV/gtpU3cbMnGQS4qKDd9IIXB9qfn6qtqCULTRBqeCq3QiJ2ZB8xFJeYcUYQ2lV22AXVtDkLoS4pMjq5itIpbKpi7ZuLZRQwaUJSgVX1VrIX+p0FONW09ZDU2df8MafAqJjYPLJETWjRCCJb9FWlAoyTVAqeLpbrEUKC8I/QZX6x1QCVWpBVXgaNGyDzqbgn9sBgST+0b5WhyNRkUYTlAqe6vXWNn+Zs3EEwfq9LcRFR41/ktjhTPXPy7c3Mrr50ibEMSM7iTWVzU6HoiKMJigVPNXrAIH8JU5HMm4f7m5mQUEqntggFkgETFoCMZ6I6uZbNjWDdXtaGPBFzrL2ynmaoFTwVK21Fij0BHncJsS6+wbYXN3G8qIMey4QEwcFyyPqht2TitJp7/Gyc7/ObK6CRxOUCg5joHotFIR/995H+1rw+gwnTbUpQQEUnWnNuNFRb981QmhZofVZaTefCiZNUCo4msqhq8lqGYS5NbtbEIElhen2XWTGuda27DX7rhFCBekJ5KZ4WFPZ4nQoKoJoglLBUfmmtZ16urNxBMEHu5soyUkmNSHWvovkLoCkHNj1sn3XCCERYdnUdNbsbsYYHYdSwaEJSgXH7rcgOQ8ypzkdybh09npZW9nCGTOz7L1QVBRMPxfKX4MBr73XCpFTijOpO9BDRWOn06GoCKEJSo2fMVD5ttV6EnE6mnF5r7yJvgEfZ9mdoMDq5utpg6o19l8rBM70f2Zv7GhwOBIVKTRBqfFr2AGd9VAU/t17b+xsYEJcNEun2jj+FDDtbIiKiZhuvskZEyjOSuSNnZqgVHBoglLjV+4f6C8609k4xskYwxs7Gzht2kTiY2y4/+lwnlRr+Y1dr9h/rRA5c2YW71c00dM/4HQoKgJoglLjt/NFyJoN6YVORzIu22rb2dvcxcdmZYfuojPOhf2l0FYVumva6IyZWfR6fXywW8vN1fiFPEGJyGQR+aeIbBWRLSLyLf/+W0WkWkQ2+B+fCHVsagx62qyZuUsucDqScXtmUw3RUcIF83JDd9GSi6zttmdDd00bnVKUSVxMFP/cHhn3dylnOdGC8gL/boyZA5wCXC8ic/yv3WWMWeR/PO9AbGq0yl4FnxdmhneCMsbwzMYaVkyfSEZiXOguPHE6ZM+BrU+F7po2SoiL5owZE3lpSx0+nfZIjVPIE5QxptYYs97/czuwDcgPdRwqSLY9AxMmhv0Nuh/ta6WqpZtLFuSF/uJzLoW970H7/tBf2wYXLcijtq2Hj/bpTbtqfBwdgxKRqcBi4AP/rm+KyCYRuV9Ehi2jEpHrRGStiKxtaNBqIUf1HIAdL8DcyyAqBEUFNnrog71MiIvm/FB27wXMuRQwsP2Z0F/bBufMziEuJopnN9U6HYoKc44lKBFJAh4Dvm2MOQD8DpgGLAJqgV8N9z5jzL3GmGXGmGVZWSG4V0Ud3bZnwNsDC65yOpJxaens45mNNVy2OJ8Uj42zRxxN1ixrkt0I6eZL9sRy5swsni+t1W4+NS6OJCgRicVKTn8xxjwOYIzZb4wZMMb4gPuAk5yITY3CpochvSjsJ4j929p99Hp9fPYUh6oQRaxWVOXb0BEZvQIXL8hj/4FereZT4+JEFZ8Avwe2GWPuHLJ/aOf/ZcDmUMemRqFhB+x+ExZ/Nqxnj+juG+C+t3Zz2rRMZuelOBfIvCvA+GDzo87FEETnzckl2RPDI2v2Oh2KCmNOtKBWAJ8DPnZYSfkvRKRURDYBZwP/5kBsaqQ+vBei42HpaqcjGZc/v7+Hxo5evn3OTGcDyZ4NeYtgw0POxhEkCXHRXLY4n+c319HS2ed0OCpMOVHF97YxRowxC4aWlBtjPmeMme/fv8oYoyOsbtXdChv+CvM/BYkTnY5mzDp7vdzzRjkrp0/kJLsWJxyNRZ+Buk2wf4vTkQTF1cun0Of18fhH1U6HosKUziShRm/NfdDfCSd/1elIxuXuf5bR1NnHjec53HoKmPcpiIqNmFbUnEkpLJqcxl8+2KPFEmpMNEGp0elpg3d/AzMvhLyFTkczZuUNHfzfWxVcsaSAJVNCMDHsSCRmwszzYdMj4O11Opqg+MKKqVQ0dPIPnVlCjYEmKDU67/2PlaTO/r7TkYyZMYZbn96CJzaamy6c5XQ4h1r2RehsiJiS80/MzyM/LYF736xwOhQVhjRBqZE7UGO1nmZfEtatpxc31/HWrkb+/dyZZCXHOx3OoYrPhszp8MH/Oh1JUMRGR/GllUV8WNnM+r06s4QaHU1QauRe/pE17965P3U6kjHr7PXyk2e3Mjsvxbn7no4lKgpOug6q10L1OqejCYqrlk8mNSGWe14vdzoUFWY0QamRqXzHukdnxbcgo8jpaMbs16/torath9s+OY+YaJf+77/wGohLgvd/53QkQZEYH8Pq06by8tb9bK5uczocFUZc+heqXMXbC8/dCKmTYWX43p62ve4Av397N9ecNJmlhS4pjBiOJwWWfQE2PwZNkdHq+NLpRaQmxPKrl3c4HYoKI5qg1PG9fjs0bIeL74K4CU5HMyY+n+HmJzaTmhDLf5zvssKI4Zx6A0THwdt3Hv/YMJDiieVrZ07jnzsaWLdHpz9SI6MJSh3bvjXwzq9h8ees1V/D1KPrqli7p4WbLpxFeijXexqr5BxYci1sfBhaI2O6oGtPK2RiUjx3vLQTY/S+KHV8mqDU0fW0weNfgeRJcP5/OR3NmLV09nH7C9tYPjWdTy0pcDqckVvxLZBo+MfPnI4kKCbExfDNs6fxXkUTr++IjElxlb00QanhGQNPXW99e7/i/6xxkTD18xe2097j5bZPzicqKowmtk3Nh1Ovt2aNj5CKvs+cXEjxxER++txW+gd8ToejXE4TlBree3db6z2dcysUnup0NGO2bk8zj6zdx5dWFlGSm+x0OKO38t8gMQte+qH1pSHMxcVEcfPFs6lo6ORP7+1xOhzlcpqg1JG2PWPd8zTrYjjtBqejGTPvgI8fPrGZSake/vXjM5wOZ2w8KfCxm60l4df/0eloguLskmxOnzGR//fqTp3pXB2TJih1qH0fwmNfhvylcPl9Yb3W0wPvVrK9rp1bVs0lMT7G6XDGbvHnYerpViuqdZ/T0YybiPCji+fQ0evlDi07V8egCUodtO9D+POnIDkPPvNI2JaUA9S2dXPXKzv52KxszpuT43Q44xMVBZfebS1o+OTXYcDrdETjNjMnmWtPm8pfPtjL2kotO1fD0wSlLJVvw58us2bUXv1sWK/zBPCTZ7YyYAz/uWouEsatwEHpU+ETv4DKt+DVW5yOJii+c14J+WkJ3PR4Kb3eAafDUS6kCUrBR3+GP34SUvLhCy9AahiVYg/j5S11vLC5jhs+NoPJGeHbCjzC4s/C8q9YBSwf3ud0NOOWGB/DbZfNo6y+g//5Z2TMmKGCSxPUiWzAa41rPHU9TF0JX3oJknOdjmpcGjt6+f7jpczJS+Erpxc7HU7wXXA7lFwEz38H3r077Cv7zi7J5tJFk/jtP8t0tnN1BE1QJ6q2anjwEuvb+ElfhX95FBJcPD/dCBhj+OETpbT3eLnrqkXExUTg/97RsXDlAzB7Fbz8Q+tG6u5Wp6Mal59cOo/cVA83PPQRbV39ToejXCQC/4LVce18Ge5ZCbUb4bJ7rbGN6DCucvP7wzuVvLRlP/9+3szwvOdppGLi4MoHrfLzzY/Bb5bA+/dAb4fTkY1JakIsv7lmMfsP9PCdRzcyoMvDKz9NUCeSgX54+WZ46EprvOmrb8LCq5yOKijeKWvkZ89v49w5OZHZtXe4qCg447tw3RuQNRte/B7cOQeevdFaGsUXXrM0LJ6Szg8+MZtXtu7nZ89tczoc5RLh/7VZjUz9NqtEueYjWP5lOO9nEOtxOqqg2LCvla/9aR3TshK566pF4TWd0XjlLYAvPGdN6vvBPbDhIVj7e2v+xHmXw4KrrGPCwBdWTGVvcxf3v7OblIQYvvXxGZFRganGTBNUpBvot8aZ/vlfEJ8Mn/4jzLnU6aiC5u1djXz9L+tIT4zjwS+eRFI435A7HpOXW4/eDtj5Imx+3Fo2/r27IWeetQjigk9DUrbTkR5V4Abe9h4v/+/VXbR09nHzxXOIdevCksp24rZp70XkAuDXQDTwf8aYnx/t2GXLlpm1a9eGLLawYgzseB5e+TE0lcHsS+CiuyApy+nIgqLP6+N/Xi/jN/8oY3pWEvd/YTn5aQlOh+UuXc3WGNWGh6BmvTUz+vSPw/xPw6xPQFyi0xEOy+cz/Nfz2/i/t3ezsCCVO65cyIycCB5TPAGJyDpjzLLjHuemBCUi0cBO4FygClgDXGOM2Trc8ZqghtFWDdufgzX/B407YOJMOO82mHFeSKYtMsbQ2tVPVUs31a1dVLV0s/9AD71eH/0DPmKjo0ibEEf6hFhyUjzkpHjIS/WQnRw/oiXYmzp6eXZTLfe9VUFVSzeXLprEzy6bf+K2nEaqYYeVqEofhQNVEJsIsy6y1vgqOtNaf8plni+t5fuPl9Le08+qhZO4+qQpLJ+aQfQIunC9Az72t/dS09pNTWs31a3d1LX10D9gEIHEuGgmJsWTlRxPbqqH/LQEclM9xMdEh+A3U+GaoE4FbjXGnO9//n0AY8ztwx0/7gTVXgf93f4n/s9h6Ofh/7lvYICmjl4wZshCawePD+w75DVjAHPIwmzGgGCsKWv8x1kvmyHXPbjP+N8kQ/f5t2IM0t9BTHcT0V31eFp24mnaiqfFmtusO3MejfO+SOu0SzFRMUf+aoMxmWH2Df2QzBH7+rw+Wrv7aevup7Wrn/r2HvY1d1PV0sW+5i46+w6dFcATG0VCbDQx0VH0D/ho6+4/4vadKIGJSfHkpQ5JWikejDF09Q1Q09rNrvoOttYewBhYMiWNb50zkzNnRkaLMGR8Ptj7Lmz6G2x7Grr99x6lFUL2HMiaaU11lZgFEzIgxgPR8VblYHQ8yJAvEcN94Tlknxxl/8i1dPbx0Jp9PLG+mh7vACnxMZTkJpOX5iHVE0d0FAz4oKPXS1NnL82dfTS099LQ3osP6Dcx7CcDgBRPDHEx0YChvcdLr/fIQpKJSfHkp3nIS01gUloCk9I8TEpLIDUhlriYKOKio4iLiSI2OmrwVwr8ZoHxskM+gcFjZDwfg+vkpHjGdRtHuCaoTwEXGGO+7H/+OeBkY8w3hzt+3AnqD5+APe+M/f0uUmsy2OabwlpfCS/5llFuJnHon4p9EuOimZwxgYL0BArSre3kjAnkpyUwOX0CKQkxhwx2D/gMbd397D/QQ11bD3UHeqht62F/Ww+1B/zbtm4O9FhzzsVECTkpHoqzElkyJZ1z5+Qwd1KKDqCPl28A6jZBxRvWLQf126zuYF/k3IvUkTaLus+8Sl5qwiETBhtj6OwboN7//2B1aze1bT1Wiyuwbe2mq0+nYBrOC986ndl5Y18jbqQJKuz6RUTkOuA6gClTpozvZCtvtKaPCfxDPvgP3tDnQle/l9LqAwAYZPDb0MGvR3LwLf7KfYP4X5bBlglivdfIwXOYwdMcfA3/e83BKx16LayHN2YC/Z5M+jwZ+GKs8YRFwOJjfZMb+qWWw74CDvlx6D/+B/dZ29joKFITYkmbEEtaQhye2KhRJYvoKCEjMY6MxLhj/k/e0z9AdJToILldoqJh0mLrEeDzQU8rdDZYY1jeHhjoA2+vtR2mB+GgI3sfjtgfRMaY4/5/l+RJY3r2keNXIkJSfAxJWUkUZyUd9fwHur1Ut3bT3tNP34CPPq//cdhii4Ff12CO3Df4WuTISw1NBbDbElQ1MHnI8wL/vkHGmHuBe8FqQY3rajPOGdFhE4CTl47rSmoMPLE6HhByUVFW196EDKcjOS67288iQuqEWFInxNp8JXU0bvtqugaYISJFIhIHXA087XBMSimlHOCqFpQxxisi3wRewiozv98Ys8XhsJRSSjnAVQkKwBjzPPC803EopZRyltu6+JRSSilAE5RSSimX0gSllFLKlTRBKaWUciVNUEoppVxJE5RSSilXctVcfKMlIg3AnhBfdiLQGOJrjofGay+N114ar72cirfQGHPcmZ7DOkE5QUTWjmSSQ7fQeO2l8dpL47WX2+PVLj6llFKupAlKKaWUK2mCGr17nQ5glDRee2m89tJ47eXqeHUMSimllCtpC0oppZQraYJSSinlSid8ghKRC0Rkh4iUichNxzjuChExIrLM//wkEdngf2wUkctGe04XxVspIqX+19a6Id4h+6eISIeIfGe053RJrK77bEVkqoh0D/n/4Z4hxy71x1smIv8tx1tT3fl4X/efMz58+q8AAAPMSURBVPBattPx+vctEJH3RGSL//P0+Pe77vM9Try2fb4jYow5YR9YiyKWA8VAHLARmDPMccnAm8D7wDL/vglAjP/nPKAea32tEZ3TLfH6n1cCE930+Q557VHg78B3RnNON8Tq1s8WmApsPsp5PwROwVpR/QXgQpfH+/rh/x1cEG8MsAlY6H+eCUS7+PM9Vry2fL4jfZzoLaiTgDJjTIUxpg94GLh0mON+Cvx/QE9ghzGmyxjj9T/1AIFqk5Ge0y3x2mnM8QKIyCeB3cDQVZXt+nztiNVO44p3OCKSB6QYY9431r9OfwQ+6dZ4bTaeeM8DNhljNgIYY5qMMQMu/nyHjTdIcY3LiZ6g8oF9Q55X+fcNEpElwGRjzHOHv1lEThaRLUAp8DV/AjjuOV0WL1jJ6mURWSci1wUp1nHFKyJJwPeA/xztOV0UK7jws/UrEpGPROQNETl9yDmrjnVOl8Ub8Ad/99OPgthlNp54ZwJGRF4SkfUi8h9DzunGz/do8QbY8fmOiOuWfHcTEYkC7gRWD/e6MeYDYK6IzAYeFJEXQhjeEcYSrzGmB1hpjKn29y+/IiLbjTFvOhzvrcBdxpiOEP9NDGscsbrxs60FphhjmkRkKfCkiMy1O6ZjGUu8xpgDwL/4P99k4DHgc1gtEyfjjQFWAsuBLuA1EVkHtNkd19GMJV5jzGs49PkGnOgtqGpg8pDnBf59AcnAPOB1EanE6jt+Wg4bHDfGbAM6/Mce75xuixdjTLV/Ww88gdVd4HS8JwO/8O//NvADEfnmCM7pplhd+dkaY3qNMU3+uNZhjV3M9L+/4BjndFu8Qz/fduAhXPD5YrVe3jTGNBpjuoDngSW49PM9Rrx2fr4j49TglxseWN8cKoAiDg4szj3G8a9zcGCxiINFBoVADdbMwKM6pwviTQSS/fsTgXeBC5yO97D9t3KwSMKWz9emWF352QJZHBwEL8b6hyzD//zwQfxPuDVe/zkn+vfHYhWpfM0F8aYD6/EXJgGvAhe5+PMdNl47P9+RPk7oLj5jjNf/TfclrCqY+40xW0TkJ8BaY8zTx3j7SuAmEekHfMA3jDGNAMOd063xikgx8IS/ayoGeMgY86IL4h3VOd0YK5CDOz/bM4CfDPl/4WvGmGb/a98AHgASsP4BDUq3tR3xikgi8JKIxPrP+Spwn9PxGmNaROROYA3WGOTz5uC4j+s+36PFa+fnO1I61ZFSSilXOtHHoJRSSrmUJiillFKupAlKKaWUK2mCUkop5UqaoJRSSrmSJiillFKupAlKKaWUK/3/hEBM8gwGvi0AAAAASUVORK5CYII=\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"dc":{"key":"46"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 7. Zooming in on the difference\n<p>These two distributions above represent the bootstrap uncertainty over what the underlying 1-day retention could be for the two AB-groups. Just eyeballing this plot, we can see that there seems to be some evidence of a difference, albeit small. Let's zoom in on the difference in 1-day retention</p>\n<p>(<em>Note that in this notebook we have limited the number of bootstrap replication to 500 to keep the calculations quick. In \"production\" we would likely increase this to a much larger number, say, 10 000.</em>)</p>"},{"metadata":{"dc":{"key":"46"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Adding a column with the % difference between the two AB-groups\nboot_1d['diff'] = (boot_1d['gate_30'] - boot_1d['gate_40'])/boot_1d['gate_40']*100.0\n\n# Ploting the bootstrap % difference\nax = boot_1d['diff'].plot.kde()\nax.set_xlabel(\"difference in 1-day retention\")","execution_count":87,"outputs":[{"output_type":"execute_result","execution_count":87,"data":{"text/plain":"Text(0.5,0,'difference in 1-day retention')"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<matplotlib.figure.Figure at 0x7f53d9147400>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl0XOWZ5/Hvo122Fi+SJVmyLWO8SNgGbJnNQAiGBEIwECAJgWw9PUy6A0k6nZ6mu3MYmu4+0x26M51JIAlDyELIkIV0YmjAELaACXgBr5Jt5F22JMubFtuyLOmZP6rECEeyyrZu3arS73OODnVv3br3qcKqn+573/u+5u6IiIgkmrSwCxARERmIAkpERBKSAkpERBKSAkpERBKSAkpERBKSAkpERBKSAkpERBKSAkpERBKSAkpERBJSRtgFnKqioiKvrKwMuwwRETlNq1at2ufuxUNtF2hAmdk1wLeAdOARd//nE57/HPAAsDu66jvu/sjJ9llZWcnKlSsDqFZEROLBzHbEsl1gAWVm6cCDwNVAA7DCzJa4e+0Jm/7c3e8Kqg4REUlOQV6DugCod/et7t4FPAHcEODxREQkhQQZUOXArn7LDdF1J7rZzNaa2a/MbFKA9YiISBIJuxffU0Clu88FXgB+PNBGZnanma00s5UtLS1xLVBERMIRZEDtBvqfEVXw/ztDAODu+939WHTxEWD+QDty94fdvcbda4qLh+z4ISIiKSDIgFoBTDezqWaWBXwSWNJ/AzMr67e4GKgLsB4REUkigfXic/duM7sLWEqkm/mj7r7BzO4HVrr7EuBLZrYY6AYOAJ8Lqh4REUkulmxTvtfU1Ljug5JU5e7sPHCEhoNHOd7TS1FeNjNK8snKCPtyscjwMbNV7l4z1HZJN5KESCrqONbNj9/YzhMrdrLrwNH3PZeVkcZVVRP4k4VTqakcF1KFIvGngBIJ2YrtB7j7Z+/Q1NbJZdOLuPPyaUyfkEdmehpNrZ0s37afJWv28My6Jj52fjn33zibvGz96krq079ykRA9vXYPX35iNZPG5vLkn13C/Clj/2ib6+aWcc+1VTz4cj3ffXULtY1t/PRPL6QoLzuEikXiRw3bIiF5obaZLz+xmvmTx/LU3ZcOGE59crPS+dqHZ/LDzy1g+/7DfOYHyznS1R3HakXiTwElEoKtLR38xc9XM3tiAY9+fgH5OZkxve7yGcV89/b51DW18fXfrA+4SpFwKaBE4qyru5c/f/xtsjLS+O4d80/5etIHZ03g7iun8+u3d/NCbXNAVYqETwElEmcP/34LG5va+Zeb5zJxTO5p7ePuK89mRkke9y3ZQOfxnmGuUCQxKKBE4mjH/sP875fquW5OGVdXl5z2fjLT07hv8TnsPnSUx9/aOYwViiQOBZRIHP3b85tJN+Pe66vPeF+XTCvikmnj+e4r9Rzt0lmUpB4FlEicbNjTypI1e/iTSyspKcgZln3efeV09nV08dTaPcOyP5FEooASiZN/XbqJwtxM7rx82rDt86KzxjGjJI/H/rCDZBu2TGQoCiiROKjd08bLm1q48/KzKMyNrUt5LMyMT180hXW7W1nT0Dps+xVJBAookTj4wevbGJWVzh0XThn2fd80r4JRWen8YuWuoTcWSSIKKJGANbd1smTNbj5eM4nCUcN39tQnLzuDRVUlPLe+ieM9vcO+f5GwKKBEAvbjN7bT3et8fmFlYMe4fm4ZBw538caW/YEdQyTeFFAiAeo83sPPlu/kQ9UlTBk/OrDjfGBmMfk5GSxZrd58kjoUUCIBem59E4eOHOczF1cGepzsjHSurirhxY3N9PSqN5+kBgWUSICeWLGTyeNGcfFZ4wM/1pVVEzh05Djv7DwY+LFE4kEBJRKQbfsO8+bWA3xiwSTS0izw4102vZj0NOPlTXsDP5ZIPCigRALyxIqdpKcZt86viMvxCnMzqZkylpc2tsTleCJBU0CJBKC7p5cnV+1m0awJTBimYY1iceWsCdQ1ttHU2hm3Y4oERQElEoDX6vexr+MYt8Tp7KnPZdOLAXhjy764HlckCAookQD89p3dFOZmcsXMCXE97qzSfApzM3lzq+6HkuSngBIZZke6unm+tpmPzCkjKyO+v2JpacaFU8fxBwWUpAAFlMgwe6G2mSNdPdx43sRQjn/xtPHsOnCUhoNHQjm+yHBRQIkMs9+8s5uJhTksqBwXyvEvit5z9ebWA6EcX2S4KKBEhtH+jmP8/t19LD6vPC73Pg1kZkk+Y0fpOpQkPwWUyDB6em0jPb3OTeeXh1ZD5DrUeAWUJD0FlMgwemrNHmaV5jOzND/UOuZPGUvDwaPsbdf9UJK8FFAiw6SptZOVOw5y3ZyysEth3pQxALyz81DIlYicPgWUyDBZuqEJgGsTIKDOmVhIZrrxtgaOlSSmgBIZJs+sa2RGSR5nT8gLuxRyMtOpnljIOzt0BiXJSwElMgz2tneyfPsBrp0d/tlTn3mTx7B29yFNAy9JSwElMgyWbmjGHT6SAM17feZNHkvn8V42NraHXYrIaVFAiQyDZ9c1Mq14NDNKwm/e6zNvylgAXYeSpBVoQJnZNWa2yczqzeyek2x3s5m5mdUEWY9IEPZ3HOPNrfv5yJwyzMK5OXcgEwtzKM7PZs0uXYeS5BRYQJlZOvAgcC1QDdxmZtUDbJcPfBl4K6haRIK0dEMzvU5CXX8CMDPmlBeybndr2KWInJYgz6AuAOrdfau7dwFPADcMsN0/AP8C6I5CSUrPrm+kcvwoqsrCvTl3ILPLC9nS0sGRru6wSxE5ZUEGVDmwq99yQ3Tde8xsHjDJ3f/zZDsyszvNbKWZrWxp0XTWkjgOHu7ijS2J17zXZ055Ib0OtXvawi5F5JSF1knCzNKAbwJ/OdS27v6wu9e4e01xcXHwxYnE6IW6Znp6PeGa9/rMKS8EUDOfJKUgA2o3MKnfckV0XZ98YDbwipltBy4ClqijhCST5zc0Uz4ml9nlBWGXMqCSgmyK8rIVUJKUggyoFcB0M5tqZlnAJ4ElfU+6e6u7F7l7pbtXAm8Ci919ZYA1iQybI13dvPZuC1dXlyRk8x70dZQoYL0CSpJQYAHl7t3AXcBSoA74hbtvMLP7zWxxUMcViZfX3t3Hse5ePlRdEnYpJzWnvJD6veooIcknI8idu/szwDMnrLt3kG2vCLIWkeH2/IZmCnIyWDA1nJlzYzU72lGirrGN+VMSu1aR/jSShMhp6O7p5aWNzSyqKiEzPbF/jeZURDtKNKiZT5JLYv9miSSolTsOcvDIca5O8OY9gNKCHIrysli3W13NJbkooEROw/MbmsnKSOPyGYl/24OZMbu8UB0lJOkooEROkbvzQl0TC6eNJy870Mu4w2b2xELqWzroPN4TdikiMVNAiZyijU3t7DpwlA+dUxp2KTGbVZZPT69Tv7cj7FJEYqaAEjlFL9Q2YwaLqiaEXUrMqsoiNxLXNuo6lCQPBZTIKfpdXTPnTRrDhPycsEuJWeX40eRkplGngJIkooASOQUt7cdY29DKolnJc/YEkJ5mzCwtUEBJUlFAiZyCVzdHRtO/YmZyBRRAdVk+G5vacfewSxGJiQJK5BS8vGkvE/KzOWdiYg4OezJVZQUcOnKcpjZNvSbJQQElEqPunl5+v7mFK2YWJ+zgsCfT11FCzXySLBRQIjF6e+ch2ju7+WASNu8BzCqNzPhb19geciUisVFAicTo5U17yUgzFk4vCruU05Kfk8mkcbnqai5JQwElEqOXN+6lpnIsBTmZYZdy2qrUk0+SiAJKJAaNrUfZ2NSetM17farKCti+7zBHuzTkkSQ+BZRIDF7dlLzdy/urKiug12FTs65DSeJTQInE4PX6fZQUZDOjJC/sUs5ItXrySRJRQIkMobfXeWPLfhZOK0rK7uX9VYzNJS87QwElSUEBJTKEjU3tHDjcxSVnJ2fvvf7S0oyZpfkKKEkKCiiRIbyxZR8AC88eH3Ilw6OqLJ+NjRrySBKfAkpkCMvq93FW8WjKCnPDLmVYVJUV0H6sm4aDR8MuReSkFFAiJ9HV3ctb2w6wcFryN+/1mVUa6SixsUk9+SSxKaBETmJNwyGOdPWkTPMe9B/ySNehJLEpoEROYln9PszgorNSJ6BGZ2cwZfwoNjYpoCSxKaBETuKNLfuZPbGQMaOywi5lWM0qjXSUEElkCiiRQRzr7mH1rkNcOHVc2KUMu6qyArbtP8yRru6wSxEZlAJKZBDrd7fS1d3LghQMqFmlBbjD5uaOsEsRGZQCSmQQy7cdBKBmytiQKxl+fUMebVRHCUlgCiiRQazcfoCzikczPi877FKGXcXYXEZnpasnnyQ0BZTIAHp7nZU7DrJgSuo170G/IY90L5QkMAWUyADqWzpoPXqcmsrUa97rU1VWwMbGNg15JAlLASUygBXbDwCwoDI1z6AAZpUV0NbZzZ7WzrBLERmQAkpkACu3H6Q4P5sp40eFXUpgqqIjSqijhCQqBZTIAJZvO8CCyrFJP//TyczsCyhdh5IEFWhAmdk1ZrbJzOrN7J4Bnv+Cma0zs9Vm9rqZVQdZj0gsmlo72X3oKPNTtINEn/ycTCaNy6VWZ1CSoAILKDNLBx4ErgWqgdsGCKCfufscdz8P+AbwzaDqEYnV6l2R+5/mTR4TciXBm1VaoCY+SVhBnkFdANS7+1Z37wKeAG7ov4G79//NGA2oO5GEbvWuVjLTjarozayprKqsgG37DtN5vCfsUkT+SJABVQ7s6rfcEF33Pmb2RTPbQuQM6ksD7cjM7jSzlWa2sqWlJZBiRfqs2XWIqrICcjLTwy4lcFWl+fQ6bG7WdShJPKF3knD3B919GvDXwNcH2eZhd69x95ri4uL4FigjSk+vs253K+dWpH7zHkS6mgMa2VwSUpABtRuY1G+5IrpuME8ANwZYj8iQtrZ00HGsm3MnjYyAmjJuFLmZ6dRpbihJQEEG1ApguplNNbMs4JPAkv4bmNn0fovXAe8GWI/IkFbvOgTAeZMKQ64kPt4b8kgdJSQBZQS1Y3fvNrO7gKVAOvCou28ws/uBle6+BLjLzK4CjgMHgc8GVY9ILNY0HCIvO4OzivLCLiVuqsryeXZ9E+6e0vd9SfIJLKAA3P0Z4JkT1t3b7/GXgzy+yKlas6uVuRWFpKWNnC/qqrIC/u/yXTS3HaO0MCfsckTeE3onCZFE0Xm8h7rGthFz/anPrNJIRwk180miUUCJRNU2ttHd6yOmB1+fWWWRIY/UUUISjQJKJGrNex0kRlZAFeRkUj4mV13NJeHEFFBm9mszu87MFGiSsjbsaaMoL2tEXoepKlNPPkk8sQbOQ8CngHfN7J/NbGaANYmEonZP24gY3mggVWUFbNWQR5JgYgood/+du98OzAO2A78zszfM7PNmlhlkgSLxcLynl/q9HVSP0ICaVVpAT69Tv7cj7FJE3hNzk52ZjQc+B/wp8A7wLSKB9UIglYnE0ZaWDrp6eqmeOEIDqq+jhJr5JIHEdB+Umf0HMBN4DLje3RujT/3czFYGVZxIvPR9MY/UJr7K8aPJyUzT5IWSUGK9Uff/RG+6fY+ZZbv7MXevCaAukbiq3dNGVkYaZxWNDruUUKSnGTNL8tmoruaSQGJt4vvHAdb9YTgLEQlTXWM7M0vyyUgfuR1VZ5UWUNfYjrumZZPEcNLfRjMrNbP5QK6ZnW9m86I/VwCj4lKhSMDcnbrGNqqi12FGqqqyfA4c7qKl/VjYpYgAQzfxfZhIx4gK3j8dezvwtwHVJBJXe9uPsf9w14i9/tSnb26ouqZ2JhSMvHvBJPGcNKDc/cfAj83sZnd/Mk41icRV7QjvINGnqt+YfB+YoYlBJXwnDSgzu8PdfwpUmtlXT3ze3b85wMtEkspI78HXp3BUJhMLc9ioruaSIIZq4uvr0jRyJseREaeusZ3yMbkU5uqe81llBepqLgljqCa+70f/+/fxKUck/mr3tI74s6c+VWX5/H5zC53He8jJTA+7HBnhYh0s9htmVmBmmWb2opm1mNkdQRcnErTO4z1s23d4xI4gcaLZEwvp7nU26SxKEkCsN318yN3bgI8SGYvvbOCvgipKJF42NbXT61A9wruY95lTUQjA2t2tIVciEntA9TUFXgf80t31r1dSgnrwvV/5mFzGjspkfYN+xSV8sQ519LSZbQSOAn9mZsVAZ3BlicRHXWMbedkZTBqr+84BzIzZ5YU6g5KEEOt0G/cAlwA17n4cOAzcEGRhIvFQ19jGrNJ80tIs7FISxtyKQt5tbtfcUBK6WM+gAGYRuR+q/2t+Msz1iMRNb69T19jOTeeXh11KQplTHukoUdfYxvmTx4ZdjoxgsU638RgwDVgN9P1Z5SigJIk1HDxKx7FuXX86wZyKMQCs392qgJJQxXoGVQNUu4Y5lhTS10FCXczfb2JhDuNGZ7FWHSUkZLH24lsPlAZZiEi81TW2kWYws0RdzPszM+aUF7JOHSUkZLGeQRUBtWa2HHhvLH53XxxIVSJxUNfYRmXRaHKzNGLCieaUF/J6/T6NKCGhijWg7guyCJEw1Da2cd6kMWGXkZDmVBTS0+vUNrYxT9ehJCSxdjN/lcgIEpnRxyuAtwOsSyRQbZ3HaTh4VB0kBjGnPDKixDpdh5IQxToW338FfgV8P7qqHPhNUEWJBG1jY2SsuWoF1IDKCnMoylNHCQlXrJ0kvggsBNoA3P1dYEJQRYkErXZP5ItXPfgGZmacWzGG1bsOhl2KjGCxBtQxd+/qW4jerKsu55K06hrbGTc6iwn52WGXkrDmTRnLlpbDHDrSNfTGIgGINaBeNbO/BXLN7Grgl8BTwZUlEqy6pjaqyvIx0xBHgzl/cqQDyTs7D4VciYxUsQbUPUALsA74b8AzwNeDKkokSN09vWxqaqeqVM17J3NuxRjS04y3d6qZT8IRUzdzd+81s98Av3H3loBrEgnUtn2HOdbdq+tPQxidncGs0nwFlITmpGdQFnGfme0DNgGborPp3hvLzs3sGjPbZGb1ZnbPAM9/1cxqzWxtdKbeKaf3NkRipzmgYjdv8lhW7zxET68uOUv8DdXE9xdEeu8tcPdx7j4OuBBYaGZ/cbIXmlk68CBwLVAN3GZm1Sds9g6RKTzmEunG/o3TeA8ip6SusZ3MdGNacV7YpSS8+VPGcrirR1PASyiGCqhPA7e5+7a+Fe6+FbgD+MwQr70AqHf3rdEegE9wwhxS7v6yux+JLr4JVJxK8SKno7axjekT8snKiPUS7MjVN4qEmvkkDEP9hma6+74TV0avQ2UO8dpyYFe/5YbousH8F+DZIfYpcsbqGtvUvBejSeNyKcrLUkBJKIbqJHGyGyCG7eYIM7uDyJQeHxjk+TuBOwEmT548XIeVEWhfxzFa2o9RVaYRzGNhZsybPFZdzSUUQ51BnWtmbQP8tANzhnjtbmBSv+WK6Lr3MbOrgL8DFrv7sROfB3D3h929xt1riouLhzisyODqNAfUKZs/ZSzb9h2mpX3AX0+RwJw0oNw93d0LBvjJd/ehmvhWANPNbKqZZQGfBJb038DMzicyvt9id997Jm9EJBa1e6IBpSa+mF141ngAlm87EHIlMtIEdpXY3buBu4ClQB3wC3ffYGb3m1nfPFIPAHnAL81stZktGWR3IsOirrGNssIcxozKCruUpDF7YgGjs9J5c+v+sEuRESbW+aBOi7s/Q2TUif7r7u33+Kogjy9yorrGdnWQOEUZ6WnUVI7jrW0KKIkv9bOVEaPzeA/1LR1q3jsNF501ns3NHezr0HUoiR8FlIwYm5vb6el1zlEHiVN20VnjAF2HkvhSQMmI8V4HCQXUKZtdXqjrUBJ3CigZMWob28jLzmDS2FFhl5J0MqPXoRRQEk8KKBkxNuxpo7qsgLQ0zQF1OvquQ+1t7wy7FBkhFFAyIvT2OnWNbWreOwOXTS8C4PV3/2j0M5FAKKBkRNi+/zBHunoUUGeguqyA8aOz+P1mTQkn8aGAkhGhbw4odTE/fWlpxmXTi3jt3X30an4oiQMFlIwIG/a0kZFmTC/RHFBn4vIZxew/3PVe4IsESQElI0Ltnjaml+STnZEedilJ7bLpkcGaX1Uzn8SBAkpGhNrGNjXvDYPi/GzOmVig61ASFwooSXl72ztpaT+mESSGyeUzilm14yDtncfDLkVSnAJKUp5GkBheV86aQHevq5lPAqeAkpS3IRpQGsV8eMybPJbxo7N4fkNz2KVIilNAScrbsKeVSeNyKcwdao5NiUV6mrGoagIvb9rL8Z7esMuRFKaAkpS3ZlcrcyvGhF1GSrm6upT2zm7e2qrRzSU4CihJafs7jrH70FHOrSgMu5SUcunZReRkpvF8bVPYpUgKU0BJSlvb0AqgM6hhlpuVzuXTi3mhthl3jSohwVBASUpb03AIs8h8RjK8rq4uobG1k/W7NaqEBEMBJSltbUMrZxfnkZedEXYpKWdRVQnpacaz6xvDLkVSlAJKUpa7s7bhkJr3AjJudBaXTBvP02sb1cwngVBAScrafego+zq6OHeSmveCcv3ciew8cIR1u1vDLkVSkAJKUlZfB4lzdQYVmA+fU0pmuvHUmj1hlyIpSAElKWtNwyEy041ZZflhl5KyCkdl8oEZxTy9tlFzRMmwU0BJynpnxyGqJxZqio2AfXTuRBpbO1m182DYpUiKUUBJSurq7mVNwyFqpowNu5SUd1V1CdkZaWrmk2GngJKUtH5PK8e6exVQcZCXncFVVSU8vbaRrm6NzSfDRwElKWnV9khz0/xKBVQ83Dy/nAOHu3hl096wS5EUooCSlLRyxwEmjxvFhPycsEsZES6fXkxRXja/WtUQdimSQhRQknLcnVU7Dqp5L44y0tP42LxyXtq4l/0dx8IuR1KEAkpSzo79R9jX0aXmvTi7eV4F3b3Ob1ers4QMDwWUpJzl2yJzFC2oHBdyJSPLzNJ85pQXqplPho0CSlLOG1v2UZSXzfQJeWGXMuLcMr+C2sY2avdohHM5cwooSSnuzrIt+7lk2njMLOxyRpzF504kM9148m2dRcmZU0BJSqnf20FL+zEWnj0+7FJGpLGjs7iqqoT/eGc3x7p7wi5HklygAWVm15jZJjOrN7N7Bnj+cjN728y6zeyWIGuRkeGNLfsBuGRaUciVjFyfunAyBw538dx6TQcvZyawgDKzdOBB4FqgGrjNzKpP2Gwn8DngZ0HVISPLsvp9TBqXy6Rxo8IuZcRaOK2IKeNH8fibO8MuRZJckGdQFwD17r7V3buAJ4Ab+m/g7tvdfS2g8VHkjHX39PLm1v0s1NlTqNLSjE9dMJnl2w+wubk97HIkiQUZUOXArn7LDdF1p8zM7jSzlWa2sqWlZViKk9SzasdB2jq7+cCM4rBLGfFumV9BVnoaP3tLZ1Fy+pKik4S7P+zuNe5eU1ysLx8Z2Isb95KVnsZlCqjQjc/L5to5pTz5dgNHurrDLkeSVJABtRuY1G+5IrpOJBC/q2vmwrPGkZedEXYpAtx+4RTaO7t5ek1j2KVIkgoyoFYA081sqpllAZ8ElgR4PBnBtu07zNaWwyyaNSHsUiRqQeVYZpTk8dibO3DXbLty6gILKHfvBu4ClgJ1wC/cfYOZ3W9miwHMbIGZNQC3At83sw1B1SOp7cW6ZgAWVZWEXIn0MTM+c3El63a3snKHZtuVUxfoNSh3f8bdZ7j7NHf/p+i6e919SfTxCnevcPfR7j7e3c8Jsh5JXUs3NDGrNF/dyxPMzfMqGDMqk0de2xp2KZKEkqKThMjJNBw8wortB7n+3IlhlyInyM1K544Lp/B8bTM79h8OuxxJMgooSXpPRS/CL1ZAJaTPXDyFjDTjh8u2h12KJBkFlCS9367ezfmTx6h5L0FNKMhh8bnl/GLlLlqPHA+7HEkiCihJapub29nY1M4NOntKaH962VSOdPXw6LJtYZciSUQBJUntieW7yEgzrpurgEpkVWUFfKi6hEeXbaOtU2dREhsFlCStI13d/HLVLq6ZXUpxfnbY5cgQvrRoOu2d3fxI16IkRgooSVpPrdlDe2c3n75oStilSAxmlxdyVdUEfvC6zqIkNgooSUruzmNv7mBGSR4XTB0XdjkSoy8vmkHr0eN875UtYZciSUABJUnpD1v2s353G5++uFJTuyeRORWF3HR+OY+8vo1dB46EXY4kOAWUJKVvv1TPhPxsbp1fEXYpcor+6sMzSTP4xtJNYZciCU4BJUln5fYD/GHrfu68/CxyMtPDLkdO0cQxudx52Vk8tWYPb23dH3Y5ksAUUJJU3J3/9bvNjB+dxe0XqnNEsvrCFdOYNC6Xv35yLUe7esIuRxKUAkqSyksb97Ksfj93X3k2uVk6e0pWo7Iy+Jeb57J9/xH+9Xk19cnAFFCSNI739PJP/1nHtOLR3K6u5UnvkmlF3HHRZH7w+jZe2tgcdjmSgBRQkjQefX0bW/cd5u+uqyIzXf90U8HXr6umuqyArzyxmp371atP3k+/5ZIUtrZ08M0XNvOh6hI+OFOz5qaKnMx0vnfHfAA+/6PlHDjcFXJFkkgUUJLwenudv35yLTmZ6fzjjbN131OKmTx+FA9/poaGg0f57KPLNcqEvEcBJQnvkde3smL7Qe79aDUTCnLCLkcCcNFZ43no9nnUNbbx8e/9gea2zrBLkgSggJKEtmrHQb7x3CY+fE4JH5tXHnY5EqBFVSX88PML2HXgCDc+uIxVOw6EXZKETAElCevg4S7u/tnblI3J4Ru3nKumvRHgsunF/OILF5OZnsbHv/8mD75cT3dPb9hlSUgUUJKQurp7+cJPV7Gvo4uHPjWfwtzMsEuSODlnYiFPf+lSrp1dygNLN7H4O8tYvetQ2GVJCBRQknDcnXueXMtb2w7wwK1zmVNRGHZJEmcFOZl8+7bz+e7t89h/+Bg3PbSMe3+7ntaj6kAxkiigJKG4O//y3CZ+/c5uvnr1DG44T9edRioz49o5Zfzuqx/gsxdX8tM3d7Do317lN+/sxt3DLk/iQAElCcPdeWDpJr736hZuv3Ayd195dtglSQLIz8nkvsXn8NsvXkr5mBy+8vPVfOr/vEX93vbtvphGAAANeklEQVSwS5OAKaAkIfT0Ovc/XctDr2zhtgsm8w836H4neb85FYX8+s8X8o83zmbDnlau/dZrfOO5jRpsNoUpoCR0Hce6ufMnK/nhsu38ycKp/NONs0lLUzjJH0tPM+64aAovfe0KFp9bzkOvbOGj336NdQ2tYZcmAVBASahW7TjIdf/7NV7Z3MI/3Dibe6+vVjjJkIrysvm3j5/LT//LhRw+1sNNDy3j2y++qy7pKUYBJaHY297J3/x6Lbd+7w26e5z/+18v4tMaoVxO0aXTi3juK5dxzexS/u2FzXz8+39gx/7DYZclw8SSrTdMTU2Nr1y5Muwy5DRtaengJ29s55erGujq7uXTF0/hq1fPID9H9znJmfnt6t18/Tfr6e117r2+mo/XTNJ1zARlZqvcvWao7TLiUYyMbLsOHGHphiaWbmhixfaDZKYb18+dyN2LpjO1aHTY5UmKuOG8cmoqx/G1X6zhr59cx4t1e/mfH5vD+LzssEuT06QzKAnEu83tPLe+iec2NLFhTxsAVWUFXDenlE8smExxvr40JBi9vc4PXt/GA0s3UZCbyQO3zOWDszRFSyKJ9QxKASXDwt3ZsKeNZ9c38tz6Jra0RK4DzJs8hmtml/Lhc0qZMl5nSxI/dY1tfOWJ1WxqbueW+RXcc+0sinQ2lRAUUBK43l7nnV2HeG59I8+ub6Lh4FHS04wLp47j2tmlfOicUko0PYaEqPN4D//+u3d55LWtjMpK52sfnsltF0zWjMwhU0BJIHp7nbd3HuSZdU08u76RxtZOMtONS88u4trZZVxVXcK40VlhlynyPvV72/kfSzawrH4/FWNz+eIHz+Zj88rJzkgPu7QRKSECysyuAb4FpAOPuPs/n/B8NvATYD6wH/iEu28/2T4VUPF36EgXy+r389q7LbyyqYWmtk6y0tO4fEYx180tZVFVCQXqhScJzt15ZVML//7iu6zZdYixozK56fwKPjavnHMmFqjHXxyFHlBmlg5sBq4GGoAVwG3uXttvmz8H5rr7F8zsk8BN7v6Jk+1XARWc3l5nb/sxdh08wsamdtY3tLJudyt1TW24Q352BgvPLuKa2aUsqpqgruGSlNydZfX7+dnyHbxQ28zxHmdCfjaXTS/mvEmFVJUVMKM0X390BSgRuplfANS7+9ZoQU8ANwC1/ba5Abgv+vhXwHfMzDzA07ofv7GdQ0eO40QO0f9I7z2MruxfRN923m/t/1/3/uX3bXfCNpHt3n/skx1noE+i/8dz4rFjre94Ty/tncdp7+ymvbOb1qPHaWrtpKvfnfhjR2Uyu7yQL105nctnFHNuRSEZaruXJGdmXDq9iEunF3HgcBcv1jXz6uYWXtrYzJNvN7y33eisdEoLcyjKyyYvO4NR2RmMzkonJzOdNDPS0yDNDOv3OLIc4puLk7s+eHZcvguCDKhyYFe/5QbgwsG2cfduM2sFxgP7+m9kZncCdwJMnjz5jIr64bJtbN9/ZMjt+v6R2fvW2QDr+rb74xfYYNsMsP/+zQt2woMBaxjwOIPX17e2bzkjzSjIySQ/J4OivCzOKh5NWWEuFWNzmTRuFGcVjaZibK6aPSSljRudxa01k7i1ZhLuTmNrJ7V72tjS0kFTWydNrZ3s7+iiqa2TI109HOnq5mhXD70Ove709Dre99h9wD8oU9EXPjCNeFy+S4obdd39YeBhiDTxncm+XvzLK/44OPQlLDLimRkTx+QycUwuV1FyWvtItk5niS7IgNoNTOq3XBFdN9A2DWaWARQS6SwRmHQNRCoiAdEfu8MryEbEFcB0M5tqZlnAJ4ElJ2yzBPhs9PEtwEtBXn8SEZHkEdgZVPSa0l3AUiLdzB919w1mdj+w0t2XAD8AHjOzeuAAkRATEREJ9hqUuz8DPHPCunv7Pe4Ebg2yBhERSU7qMywiIglJASUiIglJASUiIglJASUiIglJASUiIgkp6abbMLMWYEfYdSSAIk4YEkreR5/P4PTZDE6fzeCG87OZ4u7FQ22UdAElEWa2MpbRgEcqfT6D02czOH02gwvjs1ETn4iIJCQFlIiIJCQFVPJ6OOwCEpw+n8HpsxmcPpvBxf2z0TUoERFJSDqDEhGRhKSAEhGRhKSASmJm9oCZbTSztWb2H2Y2JuyawmZm15jZJjOrN7N7wq4nUZjZJDN72cxqzWyDmX057JoSjZmlm9k7ZvZ02LUkEjMbY2a/in7X1JnZxfE6tgIqub0AzHb3ucBm4G9CridUZpYOPAhcC1QDt5lZdbhVJYxu4C/dvRq4CPiiPps/8mWgLuwiEtC3gOfcfRZwLnH8jBRQSczdn3f37ujim0BFmPUkgAuAenff6u5dwBPADSHXlBDcvdHd344+bifyJVMeblWJw8wqgOuAR8KuJZGYWSFwOZHJZXH3Lnc/FK/jK6BSx58Az4ZdRMjKgV39lhvQl/AfMbNK4HzgrXArSSj/Dvx3oDfsQhLMVKAF+GG0+fMRMxsdr4MroBKcmf3OzNYP8HNDv23+jkgTzuPhVSrJwMzygCeBr7h7W9j1JAIz+yiw191XhV1LAsoA5gHfdffzgcNA3K7tBjrlu5w5d7/qZM+b2eeAjwKLXDe17QYm9VuuiK4TwMwyiYTT4+7+67DrSSALgcVm9hEgBygws5+6+x0h15UIGoAGd+872/4VcQwonUElMTO7hkizxGJ3PxJ2PQlgBTDdzKaaWRbwSWBJyDUlBDMzItcR6tz9m2HXk0jc/W/cvcLdK4n8m3lJ4RTh7k3ALjObGV21CKiN1/F1BpXcvgNkAy9Evn94092/EG5J4XH3bjO7C1gKpAOPuvuGkMtKFAuBTwPrzGx1dN3fuvszIdYkyeFu4PHoH31bgc/H68Aa6khERBKSmvhERCQhKaBERCQhKaBERCQhKaBERCQhKaBERCQhKaAkdGZ2n5l9Lfr4fjO7Kvr4sujI26vNLDc6evsGM3sg3IoHZ2ZvnOL2t0bfU6+Z1cT4mkozW396FZ4eM/vbM3z9jf0Hp+3//1lkMOpmLqEzs/uADnf/1xPWfw943d1/Gl1uBca5e0+M+83oN5huQjKzKiLjv30f+Jq7r4zhNZXA0+4+exjrOOlnZWYd7p53Bvv/EZGaf3W6+5CRR2dQEgoz+zsz22xmrwMz+63/kZndYmZ/Cnwc+Acze9zMlgB5wCoz+4SZFZvZk2a2IvqzMPr6+8zsMTNbBjwWnePngeg2a83sv0W3u8LMXuk3z83j0dEWMLMFZvaGma0xs+Vmlj/YfgZ4Xx1D7b8/d69z900xfF7zo/WsAb7Yb32lmb1mZm9Hfy6Jrv+Jmd3Yb7vH+4/f2K/G16KfbW103R3R97zazL4ffd//DORG1z0+2HZ979/M/ila65tmVhKtaTHwQHT7aX3/n6OvWWSRgUjXmdmjZpYdXb/dzP4++r7WmdmsoT4nSTHurh/9xPUHmA+sA0YBBUA9kbMHgB8Bt5z4OLrc0e/xz4BLo48nExnCB+A+YBWQG12+E/h69HE2sJLICM1XAK1ExutLA/4AXAr03S2/IPqaAiIjrgy4nwHeW0f0vwPu/ySfyStAzUmeXwtcHn38ALA++ngUkBN9PB1YGX38AeA30ceFwDYg44R9XkFk8M+p0eUq4CkgM7r8EPCZAT77k23nwPXRx9/o95md+P/yR8AtRMa+2wXMiK7/CZGBbAG2A3dHH/858EjY/3b1E98fDXUkYbgM+A+Pjh8Y/Qv+VF0FVPc7KSmwyEjdAEvc/Wj08YeAuX1/rRP5sp4OdAHL3b0hWsNqoJJIqDS6+woAj474bWaD7WfbSWocaP+vn+obtchMyWPc/ffRVY8RmZQRIBP4jpmdB/QAM6J1v2pmD5lZMXAz8KQP3IS33N373sMiIn88rIh+rrnA3gFec7LtuoC+GWlXAVcP8fZmAtvcfXN0+cdEzhD/PbrcN6jtKuBjQ+xLUowCSpJVGnCRu3f2Xxn9wjzcfxWRv8KXnrDdFcCxfqt6OPnvw4D7GcKp7P/9BzP7IZE5m/YAnzrJpn8BNBOZ6TQN6P95/AS4g8gAqIONn3biZ/Vjdx9qZuaTbXfc3fsubJ/Sex5E32c4HPuSJKNrUBKG3wM3WqRnXj5w/Wns43kig1gCED2DGMhS4M8sMtUEZjbDTj7h2iagzMwWRLfPN7OM09jPGXH3z7v7ee7+EY/MYHrIzC6NPn17v00LiZzx9RIZDDa933M/Ar4S3V8sI1C/CNxiZhMAzGycmU2JPne8770Psd1g2oH8AdZvAirN7Ozo8qeBV2OoVUYABZTEnUemHv85sIbILMArTmM3XwJqoh0WaoHBRnF/hEgHgLct0jX7+5zkL3GPTBX/CeDb0Q4JLxC5TnJK+4mVmd1kZg3AxcB/mtlgZ2ifBx6MNhX272zxEPDZaK2z6HdG5O7NRKZ2/2EstURD7OvA82a2lsh7L4s+/TCw1sweH2K7wTwB/FW0M8S0fsfsjL63X5rZOiI9Gr8XS72S+tTNXCRFmdkoIp1R5rl7a9j1iJwqnUGJpCCL3ARbB3xb4STJSmdQIiKSkHQGJSIiCUkBJSIiCUkBJSIiCUkBJSIiCUkBJSIiCen/AfjLqFSOMc33AAAAAElFTkSuQmCC\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"dc":{"key":"53"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 8. The probability of a difference\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/ziggy_smiling.png\" style=\"width:200px; float:left\"> </p>\n<p>From this chart, we can see that the most likely % difference is around 1% - 2%, and that most of the distribution is above 0%, in favor of a gate at level 30. But what is the <em>probability</em> that the difference is above 0%? Let's calculate that as well.</p>"},{"metadata":{"dc":{"key":"53"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Calculating the probability that 1-day retention is greater when the gate is at level 30\nprob = sum(boot_1d['diff']>0)/len(boot_1d)*1.0\nprob\n# Pretty printing the probability\nprint(\"%.2f%%\" % prob)","execution_count":89,"outputs":[{"output_type":"stream","text":"0.97%\n","name":"stdout"}]},{"metadata":{"dc":{"key":"60"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 9. 7-day retention by AB-group\n<p>The bootstrap analysis tells us that there is a high probability that 1-day retention is better when the gate is at level 30. However, since players have only been playing the game for one day, it is likely that most players haven't reached level 30 yet. That is, many players won't have been affected by the gate, even if it's as early as level 30. </p>\n<p>But after having played for a week, more players should have reached level 40, and therefore it makes sense to also look at 7-day retention. That is: What percentage of the people that installed the game also showed up a week later to play the game again.</p>\n<p>Let's start by calculating 7-day retention for the two AB-groups.</p>"},{"metadata":{"dc":{"key":"60"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Calculating 7-day retention for both AB-groups\nretention_7_perc = round(df['retention_7'].sum()/df['retention_7'].count() *100,2)\n\nprint(\"percentage of players that comes back and plays the game 7 days after installation: \"\n      + str(retention_7_perc) + \"%\")\n\ndf.groupby('version')['retention_7'].mean()","execution_count":91,"outputs":[{"output_type":"stream","text":"percentage of players that comes back and plays the game 7 days after installation: 18.61%\n","name":"stdout"},{"output_type":"execute_result","execution_count":91,"data":{"text/plain":"version\ngate_30    0.190201\ngate_40    0.182000\nName: retention_7, dtype: float64"},"metadata":{}}]},{"metadata":{"dc":{"key":"67"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 10. Bootstrapping the difference again\n<p>Like with 1-day retention, we see that 7-day retention is slightly lower (18.2%) when the gate is at level 40 than when the gate is at level 30 (19.0%). This difference is also larger than for 1-day retention, presumably because more players have had time to hit the first gate. We also see that the <em>overall</em> 7-day retention is lower than the <em>overall</em> 1-day retention; fewer people play a game a week after installing than a day after installing.</p>\n<p>But as before, let's use bootstrap analysis to figure out how certain we should be of the difference between the AB-groups.</p>"},{"metadata":{"dc":{"key":"67"},"tags":["sample_code"],"trusted":true},"cell_type":"code","source":"# Creating a list with bootstrapped means for each AB-group\nboot_7d = []\nfor i in range(500):\n    boot_mean = df.sample(frac = 1, replace = True).groupby('version')['retention_7'].mean()\n    boot_7d.append(boot_mean)\n    \n# Transforming the list to a DataFrame\nboot_7d = pd.DataFrame(boot_7d)\n\n# Adding a column with the % difference between the two AB-groups\nboot_7d['diff'] = (boot_7d['gate_30'] - boot_7d['gate_40'])/boot_7d['gate_40']\n\n# Ploting the bootstrap % difference\nax = boot_7d['diff'].plot.kde()\nax.set_xlabel(\"% difference in means\")\n\n# Calculating the probability that 7-day retention is greater when the gate is at level 30\nprob = sum(boot_7d['diff']>0)/len(boot_7d)*1.0\n\n# Pretty printing the probability\nprint(\"%.2f%%\" % prob)","execution_count":93,"outputs":[{"output_type":"stream","text":"1.00%\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"<matplotlib.figure.Figure at 0x7f5413c06cc0>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl0XOV9//H3d7R6kWRJlmwttmXjDdt4lcEJhEBYwtIQSNIszQJpU5KWNgldaZK2/HqaE5LfL2nTpmlDSwJJU5JCSCANaVkKAQcwlo28yIBXeZG8aZcla53n98dcwVhI1tiamXtn5vM6Z47u3LnLV1cz+s6z3Ocx5xwiIiJBE/I7ABERkbEoQYmISCApQYmISCApQYmISCApQYmISCApQYmISCApQYmISCApQYmISCApQYmISCBl+x1ALGbOnOlqamr8DkNEROJgy5YtLc65som2S4kEVVNTQ11dnd9hiIhIHJjZwVi2UxWfiIgEkhKUiIgEkhKUiIgEkhKUiIgEkhKUiIgEkhKUiIgEkhKUiIgEUkrcByWS7po7TvP68W4qi6aweNZ0zMzvkER8pwQl4qP2ngG+9OhOHt9xFOci69bXFPOND65mTslUf4MT8Zmq+ER8cqyzj5u//WuebDjO719xAQ9/5m3c/Z5lvH6smw9950UOt/X6HaKIr5SgRHzQOzDErd99mdZTAzx4+wb+9N1Lqa0p4bZL5/Pg7Rs41T/EH/zHVgaHw36HKuIbJSgRH/zNz3ex+0Q33/7oWtbNKz7jteWVRXzlfSvZdqSTe5/b71OEIv5TghJJsk37W/nR5sN8+vILuHzx2AM637iygmuWzeKfn91HW89AkiMUCQYlKJEkGhoO89ePNVA1Ywqfu2rRWbf9s3cvoXdgiO88ty9J0YkEixKUSBL99JUmXjvWzZduvJApuVln3XbRrAKuX1HBg5sO0TswlKQIRYJDCUokSYaGw/zTM3tZXlnIdStmx7TPJy+toatviEe2NiU4OpHgUYISSZKfb2+msbWXz161KOYbcdfNK2bp7AIe3nIkwdGJBI8SlEiSfO/XjSwsn841F86KeR8z45Y1VdQf7qCxpSeB0YkEjxKUSBJsO9zB9iOdfOJt8wiFzm0Yo5tWV2IGP6tXNZ9kFiUokST4wUsHmZqbxS1rqs5534qiKWyYX8pj9c24kfGQRDKAEpRIgnX0DvDzbc3csqaKgvyc8zrGDRfNZn9LD/tVzScZRAlKJMF+vv0o/UNhPnLx3PM+xru8dqunXz0er7BEAk8JSiTBHqtvYlH5dJZXFp73MapmTGHp7AKefvVEHCMTCTYlKJEEOtLey+bGdm5eUzXpOZ6uurCcuoPtdPYOxik6kWBTghJJoEfrmwG4aVXlpI/1rqWzGA47nt97ctLHEkkFSlAiCfRYfTO184rjMvngquoipudl8+K+1jhEJhJ8SlAiCXKgpYfXj3dz48qKuBwvOyvE+ppiXtyvBCWZIWEJyszmmNkzZrbLzBrM7HPe+rvNrMnM6r3HDYmKQcRPT+46BsA1y2IfOWIib79gJvtP9nC8qy9uxxQJqkSWoIaAP3bOLQM2AHeY2TLvtb9zzq32Ho8nMAYR3zzRcJzllYVUF0++em/E2y4oBVA1n2SEhCUo59xR59xWb7kbeBU499voRVLQye5+thxqj2vpCWBZRSFFU3J4YV9LXI8rEkRJaYMysxpgDbDJW/UHZrbdzL5rZsXj7HO7mdWZWd3Jk+q1JKnl6VeP4xxcuyy2aTViFQoZGxaU8IJKUJIBEp6gzGw68BPg8865LuCfgQuA1cBR4Otj7eecu9c5V+ucqy0rG3tabJGgenLXcapmTOHCioK4H/uS+aUcaT/NsU61Q0l6S2iCMrMcIsnph865RwCcc8edc8POuTDwr8DFiYxBJNlODwzz/N4Wrlk2a9I3545l3bxIpcPWQ+1xP7ZIkCSyF58B9wGvOue+EbU+us/tLcDORMUg4oeXDrQyMBTmyqXlCTn+sspC8nNCbDmoBCXpLTuBx74U+Diww8zqvXVfAD5iZqsBBzQCn05gDCJJ99zuk+Rlh7hkfklCjp+TFWJl1QwlKEl7CUtQzrmNwFj1G+pWLmntud0nuXh+Cfk5WQk7x9p5xdy3cT99g8MJPY+InzSShEgcHWnvZd/JHt65OLEde9bNK2Zw2LGjqTOh5xHxkxKUSBw9tztyf1KiE9TauTMA2KpqPkljSlAicfTc7pNUFOWzsHx6Qs9TOj2P+TOnqR1K0poSlEicDA6H+fXeFt65uCwh3ctHWzu3mK2H2nHOJfxcIn5QghKJk/rDHXT3D3F5gqv3RqyeO4OWUwMcaT+dlPOJJJsSlEic/HpvC2Zw6QUzk3K+VdVFAGw/oo4Skp6UoETi5MV9rSyvLKRoak5Szrd0diG5WSG2H+lIyvlEkk0JSiQO+gaHeeVQB29PUukJIDc7xIUVBWxTgpI0pQQlEgdbDrYzMBx+Y76mZFlZPYOdTV2Ew+ooIelHCUokDl7Y10JWyFhfk5jhjcazsrqIU/1D7G85ldTziiSDEpRIHLywr5VV1UVMz0vk8JZvtWpO5IbdbYfVUULSjxKUyCSd6h9i+5HOpFfvAVxQNp2puVnqKCFpSQlKZJI2H2hjOOyS2kFiRFbIWFFVxDZ1NZc0pAQlMkkv7GshNyv0xkSCybaquohdR7sYGAr7cn6RRFGCEpmkF/e3smbuDN+mvVhZPYOBoTC7j3f7cn6RRFGCEpmEjt4BGpq7fKneG7Gq2usooXYoSTNKUCKTsOlAG87hSweJEXNKplA8NYft6sknaUYJSmQSXtzXSn5OiNVed28/mBkXVc9QCUrSjhKUyCS8tL+V2nkl5Gb7+1FaVV3EnhOnOD0w7GscIvGkBCVynjp6B3jtWDeXzE/u6BFjWVk9g+Gwo6FZ1XySPpSgRM7T5sbIbLYXByBBjUy9ofuhJJ0oQYmcp82NbeRmhd4YbshP5YX5zC7M14gSklaUoETO06YDbaye49/9T6OtmlOkyQslrShBiZyHnv4hdjZ1BqJ6b8TK6hkcaOmhs3fQ71BE4kIJSuQ8bD3UznDYsT5ACWqkq/v2JlXzSXpQghI5Dy8faCNk+Db+3lhWVHkdJQ4rQUl6UIISOQ+bDrSxoir58z+dTdGUHBaUTVNPPkkbSlAi56h/aJj6wx1cnOTZc2OxqnqGSlCSNpSgRM7R9iOdDAyFA9VBYsSq6iJOdPdzrLPP71BEJi1hCcrM5pjZM2a2y8wazOxz3voSM3vSzPZ4P4NTiS8Sg5cPtAGwPoAlqJVeR4l6laIkDSSyBDUE/LFzbhmwAbjDzJYBdwFPO+cWAU97z0VSxqYDbSyZVUDxtFy/Q3mLZRWFZIdMN+xKWkhYgnLOHXXObfWWu4FXgSrgvcAD3mYPADcnKgaReBsaDrOlsY3184NZ8M/PyWJpRYFGNpe0kJQ2KDOrAdYAm4BZzrmj3kvHgFnj7HO7mdWZWd3JkyeTEabIhHYd7aJnYJiL5/s3/9NEVlXPYPuRTsJh53coIpOS8ARlZtOBnwCfd851Rb/mnHPAmJ8i59y9zrla51xtWVlZosMUiclI+1MQe/CNWFU9g+6+IQ609vgdisikJDRBmVkOkeT0Q+fcI97q42ZW4b1eAZxIZAwi8bTpQBvzSqcyuyjf71DGNTJ4rdqhJNUlshefAfcBrzrnvhH10mPArd7yrcCjiYpBJJ7CYUddY1ugS08AC8unMzU3i22aAl5SXCJvg78U+Diww8zqvXVfAO4B/tPMfgc4CHwwgTGIxM3ek6do7x0M1Ph7Y8kKGSuqitRRQlJewhKUc24jYOO8fFWiziuSKFsORiYoDOL9T6Otqi7igRcPMjAU9n06epHzpXeuSIzqGtspnZZLTelUv0OZ0Ko5MxgYCrP7eLffoYicNyUokRhtPdTO2nnFRJpXg21VtUaUkNSnBCUSg5ZT/Rxo6QnU9BpnU108hZJpuerJJylNCUokBlu99qfaFElQZsbK6iKVoCSlKUGJxGDLwXZys0JvTAqYCtbOLWbPiVN09WkKeElNSlAiMdhysJ0VVYXk52T5HUrM1s4txjmoP6RSlKQmJSiRCfQPDbO9qTNl2p9GrJpTRMje7B4vkmqUoEQmsLOpi4GhMOvmBf/+p2gF+TksmV3I1kNKUJKalKBEJrDlYGSA2FQrQQGsnTuD+kMdGtlcUpISlMgEthxsZ17pVMoK8vwO5Zytm1dMd/8Qe06c8jsUkXOmBCVyFs45thzsYN3c1Cs9QaSjBKgdSlKTEpTIWRxq66XlVD9rU7B6D2Be6VRKp+UqQUlKUoISOYuRf+y1NamZoMyMNXOLeUUdJSQFKUGJnMWWg+0U5GWzqLzA71DO27p5xexv6aGtZ8DvUETOiRKUyFlsO9LByjlFZIWCP0DseEZ6H6oUJalGCUpkHH2Dw7x2tPuNkcFT1crqIrJDpnYoSTlKUCLjaGjuYijsWDUntRNUfk4Wyyt1w66kHiUokXFs80YCX5PiCQpgzdxith3uZHA47HcoIjFTghIZR/3hDiqK8ikvzPc7lElbN6+Y016VpUiqUIISGce2Ix0p3/40YuQ+rpFhm0RSgRKUyBjaewY42NrL6rnpkaAqi/KpKMpni6bekBSiBCUyhm3eVOnpUoIyM2prSth8oA3nNHCspAYlKJEx1B/uwAwuqk6dGXQnsr6mmGNdfRxpP+13KCIxUYISGcO2wx0sKp/O9Lxsv0OJm1pvPqs6tUNJiogpQZnZI2Z2o5kpoUnac86x7Ugnq9Oge3m0JbMLKMjPZnOj7oeS1BBrwvk28FvAHjO7x8yWJDAmEV8daT9NW89Ayt+gO1pWyFg3r5jNB1SCktQQU4Jyzj3lnPsosBZoBJ4ysxfM7JNmlpPIAEWS7ZXD6dVBItr6mhL2nDhFuwaOlRQQc5WdmZUCtwGfAl4BvkkkYT2ZkMhEfLLtcAd52SGWzE7dEczHs74m0g6lcfkkFcTaBvVT4HlgKvAe59xNzrkfO+f+EJieyABFkm1nUyfLKgvJyUq/JteV1UXkZoXY3KhqPgm+WD+B/+qcW+ac+4pz7iiAmeUBOOdqx9rBzL5rZifMbGfUurvNrMnM6r3HDZP+DUTiyDnHrqNdLK8s9DuUhMjPyeKi6iIlKEkJsSaovx1j3YsT7HM/cN0Y6//OObfaezwe4/lFkuJw22m6+4ZYXpk+9z+Ntr6mhB1NnfQNDvsdishZnTVBmdlsM1sHTDGzNWa21ntcQaS6b1zOuecAfU2TlLKzuRMgbUtQELlhd3DYUX9Ywx5JsE10F+K7iXSMqAa+EbW+G/jCeZ7zD8zsE0Ad8MfOuTFba83sduB2gLlz557nqUTOTUNzJ1khY/Gs9OsgMWJkht26xjY2LCj1ORqR8Z21BOWce8A5dyVwm3PuyqjHTc65R87jfP8MXACsBo4CXz/Lue91ztU652rLysrO41Qi566huYtF5dPJz8nyO5SEmTE1lyWzCnTDrgTeWUtQZvYx59y/AzVm9kejX3fOfWOM3cblnDsedex/Bf7rXPYXSbSG5i4uX5T+X4hqa4p5rL6Z4bAjK2R+hyMypok6SUzzfk4HCsZ4nBMzq4h6eguwc7xtRZLtRFcfJ7v707r9acT6mhK6+4d47ViX36GIjOusJSjn3He8n//nXA9sZg8CVwAzzewI8NfAFWa2GnBERqT49LkeVyRRGpoj/6wzIUHV1oy0Q7WndY9FSW2x3qj7NTMrNLMcM3vazE6a2cfOto9z7iPOuQrnXI5zrto5d59z7uPOuYuccyu9dqyj8fk1RCavwevBtywDElR18VQqi/J1P5QEWqz3QV3rnOsCfoNIyWch8KeJCkrEDw3NXcwrnUpBfmYML1lbU8LmRk1gKMEVa4IaqQq8EXjIOdeZoHhEfNPQ3MWKDKruWl9TzPGufk1gKIEVa4L6LzN7DVgHPG1mZUBf4sISSa7O04McauvNiOq9EevnRwaOVTWfBFWs023cBbwdqHXODQI9wHsTGZhIMu3KoA4SIxaXF1CYn60EJYF1LvNZLyVyP1T0Pt+Pczwivmh4Y4ijzKniC4WM9TUlbNqvBCXBFFOCMrMfEBkBoh4YGWHSoQQlaWJXcxflBXmUFeT5HUpSbVhQytOvneB4Vx+zCvP9DkfkDLGWoGqBZU7dfSRNNTR3saIqc0pPI0bG4ntpfyvvXV3lczQiZ4q1k8ROYHYiAxHxS9/gMHtPnsqo9qcRyyoLKcjL5iVV80kAxVqCmgnsMrOXgf6Rlc65mxISlUgSvXasm+Gwy8gElRUyLp5fwqb9rX6HIvIWsSaouxMZhIifMrGDRDS1Q0lQxdrN/FdERpDI8ZY3A1sTGJdI0jQ0d1GYn0118RS/Q/FFdDuUSJDEOhbf7wIPA9/xVlUBP0tUUCLJ1NDcxbLKQswyc9qJkXaoTQfUDiXBEmsniTuAS4EuAOfcHqA8UUGJJMvQcJjXjmbWEEejZYWM9fNLVIKSwIk1QfU75wZGnng366rLuaS8/S099A+FWV6VeR0kom1YUML+kz2c6NIIZhIcsSaoX5nZF4ApZnYN8BDw88SFJZIcO5syu4PEiDfaoVTNJwESa4K6CzgJ7CAyyeDjwJcSFZRIsjQ0d5GXHWLBzGkTb5zGllWM3A+laj4Jjpi6mTvnwmb2M+BnzrmTCY5JJGkamjtZWlFIdlas39XSU3ZWSO1QEjhn/VRaxN1m1gK8Drzuzab7V8kJTyRxnHPsau5iRQbeoDuWkXaoY51qh5JgmOhr451Eeu+td86VOOdKgEuAS83szoRHJ5JAR9pP09U3lPHtTyMuXTgTgI17W3yORCRiogT1ceAjzrkDIyucc/uBjwGfSGRgIon2ZgcJlaAALpxdSOm0XDbuUS2+BMNECSrHOfeWr1NeO1ROYkISSY6G5i6yQsaS2QV+hxIIoZBx6cKZbNzbiiYukCCYKEENnOdrIoHX0NzJwrLp5Odk+R1KYFy2aCYtp/p57Vi336GITNiLb5WZdY2x3gCNKikpraG5i8u8dheJeMcirx1qTwsXVqjqU/x11hKUcy7LOVc4xqPAOacqPklZJ7r7ONHdz/IMnKTwbCqKprCwfDrPq6OEBEBm3/whGauhOVIxoA4Sb3XZwpm8fKCVvsFhv0ORDKcEJRlpl5eglilBvcU7Fs2kbzDM1oPtfociGU4JSjJSQ3Mnc0umUpivmurRLllQSnbIVM0nvlOCkozU0Nyl6r1xTM/LZu28Yp7brfuhxF8JS1Bm9l0zO2FmO6PWlZjZk2a2x/tZnKjzi4yn8/QgB1t7WaEOEuO6YkkZDc1dmn5DfJXIEtT9wHWj1t0FPO2cWwQ87T0XSaqR9iclqPFduSQyH+kzr5/wORLJZAlLUM6554DRk8u8F3jAW34AuDlR5xcZT0OzhjiayNLZBVQU5fPMa6rmE/8kuw1qlnPuqLd8DJiV5POLsLOpk4qifGZOz/M7lMAyM65cWs7GvS0MDIX9DkcylG+dJFxksK9xB/wys9vNrM7M6k6e1Lc4iZ+dzV0awTwGVy4p51T/EJsbNcuu+CPZCeq4mVUAeD/HreB2zt3rnKt1ztWWlZUlLUBJb70DQ+w7eYoVVarem8ilC0vJzQ7xv6+pHUr8kewE9Rhwq7d8K/Boks8vGe7Vo104BytUgprQ1NxsNiwoVUcJ8U0iu5k/CLwILDGzI2b2O8A9wDVmtge42nsukjQ7m7whjlSCismVS8rYf7KHg609fociGSiRvfg+4pyrcM7lOOeqnXP3OedanXNXOecWOeeuds6pcluSamdTJ6XTcpldqMH4Y/GupZHu5qrmEz9oJAnJKDubu1heVYSZ+R1KSphXOo0FZdOUoMQXSlCSMfqHhtlzvJsVuv/pnFy1tJyX9rfS3TfodyiSYZSgJGPsPnaKobDTCBLn6JplsxkcdvxKY/NJkilBScbY6Y0goR5852bdvGJKpuXyRMNxv0ORDKMEJRljZ1MnBfnZzCmZ4ncoKSUrZFy1tJxnXj/B4LBGlZDkUYKSjLGzqZPllYXqIHEerlk2i+6+ITbtV8dbSR4lKMkI/UPDvHq0m1VzZvgdSkp6x6Iy8nNCPLHrmN+hSAZRgpKM8OrRbgaGw6yuVoI6H1Nys3jHojKe2nWcyDCaIomnBCUZYdvhDgCVoCbh2mWzaO7so8GbT0sk0ZSgJCNsO9xBeUEeFUUaQeJ8XXXhLEIGTzSomk+SQwlKMkL9kQ5WzZmhDhKTUDItl9qaEp7Ype7mkhxKUJL2OnsH2X+yh9Wq3pu0a5fN4rVj3Rxu6/U7FMkASlCS9rY3ee1P6iAxadcsi0yCrVKUJIMSlKS9kQ4SF1VrBInJmlc6jSWzCnhS3c0lCZSgJO3VH+5kQdk0iqbk+B1KWrhm2SxePtBGe8+A36FImlOCkrTmnKP+cIfuf4qja5fPIuzgqVdVzSeJpQQlaa25s4+WU/26/ymOLqoqomrGFP57p6r5JLGUoCSt1TVGxo5bN6/Y50jSh5lx3YrZPL+nRXNESUIpQUlaq2tsZ1puFktnF/gdSlq5fsVsBobDmmlXEkoJStLa5sY21s4rJjtLb/V4Wju3mPKCPH65Q9V8kjj61Era6jw9yOvHu1lfU+J3KGknFDLevXw2z+4+Qe/AkN/hSJpSgpK0tfVgO85BbY3anxLh+hWz6RsM86vXNRW8JIYSlKStzY1tZIeMNXOUoBLh4vklFE/N4ZfqzScJogQlaauusZ0VVUVMyc3yO5S0lJ0V4tpls/nf107QNzjsdziShpSgJC31Dw1Tf6SD9areS6jrL5rNqf4hNu5p8TsUSUNKUJKWth3uZGAoTK06SCTU2y+YSUF+tqr5JCGUoCQt/XpvCyGDDfNL/Q4lreVmh7jmwlk89epxBofDfocjaUYJStLSr/e2cFH1DIqmaoDYRLtuxWw6Tw/ywr5Wv0ORNKMEJWmnu2+QVw53cNlClZ6S4fLFZRTkZ/NofZPfoUia8SVBmVmjme0ws3ozq/MjBklfm/a3MRx2XLpwpt+hZIT8nCyuXzGb/9l5TL35JK78LEFd6Zxb7Zyr9TEGSUMb97aQnxNi7Vz14EuWm1dX0TMwrCk4JK5UxSdpZ+PeFtbXlJCfo/ufkuWSBaWUF+Txs1ea/Q5F0ohfCcoBT5jZFjO7fawNzOx2M6szs7qTJzWUisTmUGsve0+c4p2Ly/wOJaNkhYybVlXyq90n6OjVTLsSH34lqMucc2uB64E7zOzy0Rs45+51ztU652rLyvTPRmLzpFfFdO2y2T5HknluXlPF4LDjcY1wLnHiS4JyzjV5P08APwUu9iMOST9P7jrGklkFzC2d6ncoGWd5ZSEXlE3jZ+rNJ3GS9ARlZtPMrGBkGbgW2JnsOCT9dPQOsLmxnauXlfsdSkYyM25ZU8XLB9o41NrrdziSBvwoQc0CNprZNuBl4BfOuf/2IQ5JM8+8foLhsOPqC2f5HUrGev+6akIGP9p8yO9QJA1kJ/uEzrn9wKpkn1fS3y93HKO8II9V1TP8DiVjVRRN4V1Ly/nPuiPcec1icjSTsUyC3j2SFjp7B3n29ZP8xspKQiHzO5yM9luXzKXlVD9P654omSQlKEkLj+88ysBwmJvXVPodSsZ75+JyKory+eEmVfPJ5ChBSVr42StNLJg5jYuqivwOJeNlhYwPrZ/D83ta2H/ylN/hSApTgpKUd6Clh00H2rhlTRVmqt4Lgo9eMo/crBD3bTzgdyiSwpSgJOX9+0sHyfa+tUswlBXk8b61VTy85Qitp/r9DkdSlBKUpLTTA8M8VHeY61bMprww3+9wJMqn3jGf/qEw33/xoN+hSIpSgpKU9vCWw3T1DfGJt9X4HYqMsrC8gKsvnMX9LzTSeXrQ73AkBSlBScrqHxrm28/uo3ZeMetrNLVGEN15zSI6Tw9y3/P7/Q5FUpASlKSsh+qOcLSzj89dvUidIwJqeWURN15UwX0bD6gtSs6ZEpSkpM7Tg/z9U7upnVfMZZo5N9DuvGYxpweH+ebTe/wORVKMEpSkpL97cjetPQPcfdNylZ4CbmH5dD6+YR7//tJBGpo7/Q5HUogSlKScusY2vv9iIx+9ZC4rdGNuSvija5ZQPDWXv3q0gXDY+R2OpAglKEkpnacH+dyP6qkunsqfX7fU73AkRkVTc/jz65ey5WA7P3hJ3c4lNkpQkjIGh8Pc8cOtHO/q45sfXk1Bfo7fIck5+M111bxzcRlf+eWr7NMQSBIDJShJCcNhx589vJ2Ne1u45/0rWTNX3cpTjZnxtQ+sJD8niz/6z20MDof9DkkCTglKAm9wOMznfvQKP32liT999xI+sK7a75DkPM0qzOfLN1/EtsMd3PPL1/wORwIu6RMWipyL9p4B7viPrbywr5Uv3LCU2y+/wO+QZJJuXFnB5sYa7tt4gJXVRbx3dZXfIUlAKUFJYL16tIvbf1DH8a5+/t9vrlLJKY188cYL2dXcxZ//ZDuLygtYVlnod0gSQKrik8BxzvHDTQe5+Z9+Tf9gmB/fvkHJKc3kZIX41kfXMGNKLr99/2aaO077HZIEkBKUBEpn7yC//8OtfPGnO7l4fgn/9dnL1CEiTZUX5PO9T66np3+IW7/7Mh29A36HJAGjBCWBUdfYxg3/8DxP7jrOXdcv5YFPXkx5gabQSGcXVhTynU+s42BrL7/7/Tr6Bof9DkkCRAlKfDccdvzj03v40L0vkRUyHv69t/OZd15AKKQhjDLB2y+Yydc/uIq6g+186oE6Tg8oSUmEOkmIr/afPMUfP7SNVw51cNOqSr58ywrdgJuB3rOqkv6hMH/68DZ+54HN3HfreqbkZvkdlvhMCUp8EQ47vvdCI1/779fIz8nimx9ezU2rKjXwawb7wLpqQgZ/8tA2PnbfJu79+DpKp+f5HZb4SAlKkm5nUyd/9ehOth7q4F1Ly/nK+y5ilqZrF+B9a6vJz8nizh/Xc8u3X+C7t61nYfl0v8MSn6gNSpLmZHc/X/zpDt7zrY0cauvl67+5ivturVVykjPccFEFD96+gd6BId7zjxt58OVDOKcR0DM9YqaRAAAORUlEQVSRpcIfvra21tXV1fkdhpynY5193Pvcfn646SBDYccn3jaPz1+9mKIpamuS8R3r7ONPHtrGxr0tXL64jC/deCGLZxX4HZbEgZltcc7VTridEpQkwumBYZ7bc5KH6g7zv6+dwMy4ZU0Vd1y5kPkzp/kdnqSIcNjxwIuNfOPJ3fT0D3HDRRXc+vYa1s0tVi/PFKYEJUnV3jPAjqZOdjR1srmxjRf3tdI/FKasII8PrKvmI+vnMrd0qt9hSopq7xngn3+1jwdfPkR33xDlBXlcdWE5a+YUs7yqkPkzpzE1V03qqSLQCcrMrgO+CWQB/+acu+ds2ytBBUNP/xCNrT0cbO3lQEsPjS09NLb2cKCll5ZT/W9st2DmNN65pIx3LS1nw4JScrLU1Cnx0dM/xP80HOOpV4/z/O4WuvuH3nitZFoulTPyqSiaQmVRPrOLplA5I5/ZhflUzphCeWEeednquh4EgU1QZpYF7AauAY4Am4GPOOd2jbePElRyhMOOlp5+jnf209TRy4GWXhpbejjQGklGJ7r7z9i+rCCP+aXTqJk5lQvKprOiqogVlUUUTVXbkiReOOw42NbLzqZODrX10txx2nv0cbTzNF19Q2/Zp6wgjznFU6gunsqckinMKZ76xnJF0RRys/VlKhliTVB+lIkvBvY65/YDmNmPgPcC4yaoyfrBi4209QzieDMZR+dlN8bK6LQ9snqs/cfa7oxtxzoPvNEracw4JjgnZ2z31pjHjGNUzH0Dw3T3D9HdN8ip/iHaewY50d3H4PCZJ5k5PZea0mlcvriM+TOnUVM6jXmlU6mZOY3peapSEf+EQsb8mdPGbdM81T/Esc5Isjra2cfRjj6aOno50n6aVw6384sdRxkOv/l+DxnMLsynZHou0/OyKcjPoSAvmym5WWSHjFDIzviZZRaX+/biceufkbz2uOn52fzOZfOTci4//sNUAYejnh8BLhm9kZndDtwOMHfu3Emd8P4XGtl3siembaPfLHbGehtj3ch2Y+9ko7Ybve2b+7/1PGesH/OYbz3OuK+PEXN+Tpb3IcymvCCfReUFzC7Kp6LozSqReaVTNaqDpKzpedksLJ8+7n1UQ8NhjnX1caT9NIfbIonrcHsvHb2DnOob4kj7abr7Bjk9MMywcwwPO4adYyjsGPYemaiiKD+tE1RMnHP3AvdCpIpvMsd64s53jp0sNGqBSMbKzgpR7VXxbVhQ6ksM8WhiSYF+bufNjwTVBMyJel7trUuYLHVHFZEACkoVYVD50SK4GVhkZvPNLBf4MPCYD3GIiEiAJb0E5ZwbMrM/AP6HSDfz7zrnGpIdh4iIBJsvbVDOuceBx/04t4iIpAZ1+hcRkUBSghIRkUBSghIRkUBSghIRkUBSghIRkUBKiek2zOwkcHASh5gJtMQpnGRRzMmRajGnWrygmJMh1eKd55wrm2ijlEhQk2VmdbGMnBskijk5Ui3mVIsXFHMypFq8sVIVn4iIBJISlIiIBFKmJKh7/Q7gPCjm5Ei1mFMtXlDMyZBq8cYkI9qgREQk9WRKCUpERFKMEpSIiARSSicoMysxsyfNbI/3s3ic7W71ttljZrdGrX/WzF43s3rvUe6tzzOzH5vZXjPbZGY1QYjZzKaa2S/M7DUzazCze6K2v83MTkb9Lp+aZJzXeddmr5ndNcbr414jM/sLb/3rZvbuWI85Wecbs5ldY2ZbzGyH9/NdUfuM+R4JQMw1ZnY6Kq5/idpnnfe77DWzf7A4Th09iXg/GhVrvZmFzWy195rf1/hyM9tqZkNm9oFRr433vyNh13gyMZvZajN70fv/sN3MPhT12v1mdiDqOq+OZ8wJ4ZxL2QfwNeAub/ku4KtjbFMC7Pd+FnvLxd5rzwK1Y+zz+8C/eMsfBn4chJiBqcCV3ja5wPPA9d7z24BvxSnGLGAfsMA7zzZgWSzXCFjmbZ8HzPeOkxXLMX2MeQ1Q6S2vAJqi9hnzPRKAmGuAneMc92VgA2DAL0feI37GO2qbi4B9AbrGNcBK4PvAB6LWn+1/R0KucRxiXgws8pYrgaPADO/5/dHbpsIjpUtQwHuBB7zlB4Cbx9jm3cCTzrk251w78CRw3Tkc92Hgqjh+QzrvmJ1zvc65ZwCccwPAVqA6TnFFuxjY65zb753nR17c4/0e0dfovcCPnHP9zrkDwF7veLEc05eYnXOvOOeavfUNwBQzy4tjbHGPebwDmlkFUOice8lF/it9n7HfY37G+xFv32SYMGbnXKNzbjsQHrXvmJ/DBF/jScXsnNvtnNvjLTcDJ4AJR2wIqlRPULOcc0e95WPArDG2qQIORz0/4q0b8T2vuPuXUR+kN/Zxzg0BnUBpgGLGzGYA7wGejlr9fq9Y/7CZzZlEjBOen/Gv0Xj7xnLMyZhMzNHeD2x1zvVHrRvrPRKEmOeb2Stm9isze0fU9kcmOKZf8Y74EPDgqHV+XuNz3TeR1/hs5z0nZnYxkRLYvqjVX/b+R/xdkr6ETYovM+qeCzN7Cpg9xktfjH7inHNmdq595j/qnGsyswLgJ8DHiXwbmpQEx4yZZRP5gP+Dc26/t/rnwIPOuX4z+zSRb7HvGu8Y8lZmthz4KnBt1OqEvEfi4Cgw1znXambrgJ958QeamV0C9DrndkatDuo1TlleKe8HwK3OuZFS1l8Q+VKcS+S+qT8H/safCGMT+BKUc+5q59yKMR6PAse9P8TIH+TEGIdoAqJLE9XeOpxzIz+7gf8gUrQ+Yx8vGRQBrUGI2XMvsMc59/dR52yN+tb/b8C6WOM9j/Ofsc2oazTevrEcczImEzNmVg38FPiEc+6Nb5xneY/4GrNXhdrqxbaFyLfkxd720dW+8bzOk7rGng8zqvQUgGt8rvsm8hqf7bwxMbNC4BfAF51zL42sd84ddRH9wPeI73VODL8bwSbzAP4vZ3Y4+NoY25QAB4g0chZ7yyVESo8zvW1yiNSXf8Z7fgdnNvT+ZxBi9l77WyLfMkOj9qmIWr4FeGkSMWYTaRCez5uNtMtHbTPmNQKWc2Ynif1EGn0nPOYkr+tkYp7hbf++MY455nskADGXAVne8gIi/8BG3iOjG/Bv8Dte73nIi3NBkK5x1Lb389ZOEuN9DhNyjeMQcy6Rav/Pj7FthffTgL8H7olXzIl6+B7AJP+Qpd4fYw/wVNSbpxb4t6jtfptIY/1e4JPeumnAFmA7kYbxb0Z94POBh7ztX47+QPkcczXggFeBeu/xKe+1r3i/xzbgGWDpJOO8AdhN5Jv5F711fwPcNNE1IlKVuQ94najeTWMdM87vh/OKGfgS0BN1TeuB8rO9RwIQ8/u9mOqJdJZ5T9Qxa4Gd3jG/hTdijJ/xeq9dwagvTgG5xuuJtPP0ECntNZztc5joazyZmIGPAYOj3survdf+F9jhxf3vwPR4fwbj/dBQRyIiEkiBb4MSEZHMpAQlIiKBpAQlIiKBpAQlIiKBpAQlIiKBpAQlKcPMysxso5ntNLObo9Y/amaVMR7jlPez0swejlr/oDcEzJ1mttQbducVM7sg/r/J5JnZZ8zsE37HIZJI6mYuKcPMPgu0AY8AjzvnrjCz9wDrnHN3x3iMU8656aPWzQY2OucWes/vArKdc397DrFlOeeGY91eRCamEpSkkkEiU47kAcPeUDqfJzKFyZjMbL43P84OM/vbqPU1ZjYyHtwTQJVXavpr75i/Z2bPeNt+zMxe9l7/jplleetPmdnXzWwb8DaLzBH0K4vMKfU/UUNaPWtmX/WOsXtkYFczyzKz/+eVCLeb2R9668c8zqjf624z+5OzHX/U9ld4x3zUzPab2T0WmaPpZe/aXOBtV2ZmPzGzzd7jUm/9xd51fMXMXjCzJd7628zsETP7b4vMmfS1qN/tfu9322Fmd8b6RxZ5g993CuuhR6wPIuO6/QKoA64CPgvcNsE+jxEZXw8iw/Cc8pZr8OZTYtTcSsDdwJ94yxcSGYg3x3v+7ajjOeCD3nIO8AJQ5j3/EPBdb/lZ4Ove8g3AU97y7xEZ2ifbe15ytuOM+r2iYxzz+KO2vwLoACqIJPgm4P94r30O+Htv+T+Ay7zlucCr3nJhVJxXAz/xlm8jMixPEZFRJA4SGUduHZGpKkbOP8Pv948eqfcI/GjmIiOcc53AjQAWmYn4LuAWM/tXImOlfd059+Ko3S4lMiwQREZ3/uo5nvYqIv9sN1tkFogpvDnA7zCRcREBlhCZ7PBJb7ssIiOOj3jE+7mFSEKEyD/6f3GRaSlwzrWZ2YoJjjOesY4/2mbnTfViZvuIlBwhMvzNlVExLbM3Z7woNLPpRBLQA2a2iEhizok67tPe3wYz2wXMIzJs0QIz+0ciXyqeQOQcKUFJqvpL4MtEJr/bSKQk8giRSeZGm0xDqwEPOOf+YozX+tyb7U5GZDy0t41znJGR5oc5++duouOMJ5bjR89xFY56Ho7aJwRscM71nRGU2beAZ5xzt1hkGvdnxznuMJGSVruZrSLy9/gM8EEi49qJxExtUJJyvG/x1c65Z4m0SYWJJKEpY2z+ayKjagN89DxO9zTwATMr985dYmbzxtjudaDMzN7mbZdjE8/P9CTwaa8tDTMrOc/jxNMTwB+OPDGz1d5iEW9O+XDbRAcxs5lERtz/CZHBeNfGN0zJBEpQkoq+zJuTPz5IpC1nM5GRsEf7HHCHme3gPGYldc7tIvIP9gkz204kqbyl04KLTM39AeCrXqeJeuDtExz+34BDwHZvn986z+PE02eBWq/Txi4ipR+IdET5ipm9Qmw1L1XAs2ZWT2Tk7LFKoCJnpW7mIiISSCpBiYhIIClBiYhIIClBiYhIIClBiYhIIClBiYhIIClBiYhIIClBiYhIIP1/BabMFa5Sfd0AAAAASUVORK5CYII=\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"dc":{"key":"74"},"deletable":false,"editable":false,"run_control":{"frozen":true},"tags":["context"]},"cell_type":"markdown","source":"## 11.  The conclusion\n<p>The bootstrap result tells us that there is strong evidence that 7-day retention is higher when the gate is at level 30 than when it is at level 40. The conclusion is: If we want to keep retention high — both 1-day and 7-day retention — we should <strong>not</strong> move the gate from level 30 to level 40. There are, of course, other metrics we could look at, like the number of game rounds played or how much in-game purchases are made by the two AB-groups. But retention <em>is</em> one of the most important metrics. If we don't retain our player base, it doesn't matter how much money they spend in-game.</p>\n<p><img src=\"https://assets.datacamp.com/production/project_184/img/cookie_yellow.png\" style=\"width:100px; float:center\"> </p>\n<p>So, why is retention higher when the gate is positioned earlier? One could expect the opposite: The later the obstacle, the longer people are going to engage with the game. But this is not what the data tells us. The theory of <em>hedonic adaptation</em> can give one explanation for this. In short, hedonic adaptation is the tendency for people to get less and less enjoyment out of a fun activity over time if that activity is undertaken continuously. By forcing players to take a break when they reach a gate, their enjoyment of the game is prolonged. But when the gate is moved to level 40, fewer players make it far enough, and they are more likely to quit the game because they simply got bored of it. </p>"},{"metadata":{"dc":{"key":"74"},"tags":["sample_code"],"trusted":true,"collapsed":true},"cell_type":"code","source":"# So, given the data and the bootstrap analysis\n# Should we move the gate from level 30 to level 40 ?\nmove_to_level_40 = False","execution_count":95,"outputs":[]}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.7","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":2}