notebooks/data-sources/TestGrid/metrics/pct_fixed_each_ts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6dda7368",
   "metadata": {},
   "source": [
    "# Percent of Failing Tests Fixed\n",
    "\n",
    "This notebook is an addition to the series of KPI notebook in which we calculate key performance indicators for CI processes. In this notebook, we will calculate the KPI \"Percent of failing tests fixed in each run/timestamp.\" Essentially, we will determine \n",
    "\n",
    "- percent of tests that were failing and are now fixed\n",
    "\n",
    "For OpenShift managers, this information can potentially help quantify the agility and efficiency of their team. If this number is high, it means they are able to quickly identify the root causes of all failing tests in the previous run and fix them. Conversely if this number is low, it means only a small percent of previously failing tests get fixed in each new run, which in turn implies that their CI process is likely not as efficient as it could be.\n",
    "\n",
    "Related issues: [#149](https://github.com/aicoe-aiops/ocp-ci-analysis/issues/149)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2075a9a6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-08T22:09:14.897443Z",
     "start_time": "2021-02-08T22:09:14.371522Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import gzip\n",
    "import json\n",
    "import datetime\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from ipynb.fs.defs.metric_template import decode_run_length\n",
    "from ipynb.fs.defs.metric_template import testgrid_labelwise_encoding\n",
    "from ipynb.fs.defs.metric_template import CephCommunication\n",
    "from ipynb.fs.defs.metric_template import save_to_disk, read_from_disk\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "\n",
    "load_dotenv(find_dotenv())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d000b0da",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Specify variables\n",
    "\n",
    "METRIC_NAME = \"pct_fixed_each_ts\"\n",
    "\n",
    "# Specify the path for input grid data,\n",
    "INPUT_DATA_PATH = \"../../../../data/raw/testgrid_183.json.gz\"\n",
    "\n",
    "# Specify the path for output metric data\n",
    "OUTPUT_DATA_PATH = f\"../../../../data/processed/metrics/{METRIC_NAME}\"\n",
    "\n",
    "## CEPH Bucket variables\n",
    "## Create a .env file on your local with the correct configs,\n",
    "s3_endpoint_url = os.getenv(\"S3_ENDPOINT\")\n",
    "s3_access_key = os.getenv(\"S3_ACCESS_KEY\")\n",
    "s3_secret_key = os.getenv(\"S3_SECRET_KEY\")\n",
    "s3_bucket = os.getenv(\"S3_BUCKET\")\n",
    "s3_path = os.getenv(\"S3_PROJECT_KEY\", \"ai4ci/testgrid/metrics\")\n",
    "s3_input_data_path = \"raw_data\"\n",
    "AUTOMATION = os.getenv(\"IN_AUTOMATION\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ef6c08d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-08T22:09:19.031967Z",
     "start_time": "2021-02-08T22:09:15.200904Z"
    }
   },
   "outputs": [],
   "source": [
    "## Import data\n",
    "timestamp = datetime.datetime.today()\n",
    "\n",
    "if AUTOMATION:\n",
    "    filename = f\"testgrid_{timestamp.day}{timestamp.month}.json\"\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    s3_object = cc.s3_resource.Object(s3_bucket, f\"{s3_input_data_path}/{filename}\")\n",
    "    file_content = s3_object.get()[\"Body\"].read().decode(\"utf-8\")\n",
    "    testgrid_data = json.loads(file_content)\n",
    "\n",
    "else:\n",
    "    with gzip.open(INPUT_DATA_PATH, \"rb\") as read_file:\n",
    "        testgrid_data = json.load(read_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d40f118",
   "metadata": {},
   "source": [
    "## Calculation\n",
    "\n",
    "To find fixed tests, we modified the [testgrid_labelwise_encoding function](./number_of_flakes.ipynb). The loop is adapted to put a \"True\" if a test was fixed in the current run, and \"False\" otherwise. Basically instead of indicating \"is_flake\" or \"is_pass,\" it indicates \"is passing now but was failing before\" aka \"is_flip.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3b3d3ed1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"redhat-assisted-installer\"\n",
      "\"redhat-openshift-informing\"\n",
      "\"redhat-openshift-ocp-release-4.1-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.1-informing\"\n",
      "\"redhat-openshift-ocp-release-4.2-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.2-informing\"\n",
      "\"redhat-openshift-ocp-release-4.3-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.3-broken\"\n",
      "\"redhat-openshift-ocp-release-4.3-informing\"\n",
      "\"redhat-openshift-ocp-release-4.4-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.4-broken\"\n",
      "\"redhat-openshift-ocp-release-4.4-informing\"\n",
      "\"redhat-openshift-ocp-release-4.5-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.5-broken\"\n",
      "\"redhat-openshift-ocp-release-4.5-informing\"\n",
      "\"redhat-openshift-ocp-release-4.6-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.6-broken\"\n",
      "\"redhat-openshift-ocp-release-4.6-informing\"\n",
      "\"redhat-openshift-ocp-release-4.7-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.7-broken\"\n",
      "\"redhat-openshift-ocp-release-4.7-informing\"\n",
      "\"redhat-openshift-ocp-release-4.8-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.8-informing\"\n",
      "\"redhat-openshift-ocp-release-4.9-blocking\"\n",
      "\"redhat-openshift-ocp-release-4.9-informing\"\n",
      "\"redhat-openshift-okd-release-4.3-informing\"\n",
      "\"redhat-openshift-okd-release-4.4-informing\"\n",
      "\"redhat-openshift-okd-release-4.5-blocking\"\n",
      "\"redhat-openshift-okd-release-4.5-informing\"\n",
      "\"redhat-openshift-okd-release-4.6-blocking\"\n",
      "\"redhat-openshift-okd-release-4.6-informing\"\n",
      "\"redhat-openshift-okd-release-4.7-blocking\"\n",
      "\"redhat-openshift-okd-release-4.7-informing\"\n",
      "\"redhat-openshift-okd-release-4.8-blocking\"\n",
      "\"redhat-openshift-okd-release-4.8-informing\"\n",
      "\"redhat-openshift-okd-release-4.9-informing\"\n",
      "\"redhat-openshift-presubmit-master-gcp\"\n",
      "\"redhat-osd\"\n",
      "\"redhat-single-node\"\n"
     ]
    }
   ],
   "source": [
    "# NOTE: this for loop is a modified version of the testgrid_labelwise_encoding function\n",
    "\n",
    "percent_label_by_grid_csv = []\n",
    "\n",
    "for tab in testgrid_data.keys():\n",
    "    print(tab)\n",
    "\n",
    "    for grid in testgrid_data[tab].keys():\n",
    "        current_grid = testgrid_data[tab][grid]\n",
    "\n",
    "        # get all timestamps for this grid (x-axis of grid)\n",
    "        timestamps = [\n",
    "            datetime.datetime.fromtimestamp(t // 1000)\n",
    "            for t in current_grid[\"timestamps\"]\n",
    "        ]\n",
    "\n",
    "        tests = []\n",
    "        all_tests_did_get_fixed = []\n",
    "\n",
    "        # NOTE: this list of dicts goes from most recent to least recent\n",
    "        for i, current_test in enumerate(current_grid[\"grid\"]):\n",
    "            tests.append(current_test[\"name\"])\n",
    "            statuses_decoded = decode_run_length(current_grid[\"grid\"][i][\"statuses\"])\n",
    "\n",
    "            did_get_fixed = []\n",
    "            for status_i in range(0, len(statuses_decoded) - 1):\n",
    "                did_get_fixed.append(\n",
    "                    statuses_decoded[status_i] == 1\n",
    "                    and statuses_decoded[status_i + 1] == 12\n",
    "                )\n",
    "\n",
    "            # the least recent test cannot have \"True\", assuming it wasnt failing before\n",
    "            did_get_fixed.append(False)\n",
    "\n",
    "            # add results for all timestamps for current test\n",
    "            all_tests_did_get_fixed.append(np.array(did_get_fixed))\n",
    "\n",
    "        all_tests_did_get_fixed = [\n",
    "            list(zip(timestamps, g)) for g in all_tests_did_get_fixed\n",
    "        ]\n",
    "\n",
    "        # add the test, tab and grid name to each entry\n",
    "        # TODO: any ideas for avoiding this quad-loop\n",
    "        for i, d in enumerate(all_tests_did_get_fixed):\n",
    "            for j, k in enumerate(d):\n",
    "                all_tests_did_get_fixed[i][j] = (k[0], tab, grid, tests[i], k[1])\n",
    "\n",
    "        # accumulate the results\n",
    "        percent_label_by_grid_csv.append(all_tests_did_get_fixed)\n",
    "\n",
    "# output above leaves us with a doubly nested list. Flatten\n",
    "flat_list = [item for sublist in percent_label_by_grid_csv for item in sublist]\n",
    "flatter_list = [item for sublist in flat_list for item in sublist]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0ba1989f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(datetime.datetime(2021, 3, 15, 23, 40, 20),\n",
       " '\"redhat-assisted-installer\"',\n",
       " 'periodic-ci-openshift-release-master-nightly-4.6-e2e-metal-assisted',\n",
       " 'Overall',\n",
       " False)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flatter_list[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0cb83fa7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>did_get_fixed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-15 23:40:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-15 00:01:06</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-13 20:51:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-13 07:51:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-13 06:43:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-03-15 23:40:20  \"redhat-assisted-installer\"   \n",
       "1 2021-03-15 00:01:06  \"redhat-assisted-installer\"   \n",
       "2 2021-03-13 20:51:32  \"redhat-assisted-installer\"   \n",
       "3 2021-03-13 07:51:20  \"redhat-assisted-installer\"   \n",
       "4 2021-03-13 06:43:20  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  did_get_fixed  \n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall          False  \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall          False  \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall          False  \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall          False  \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall          False  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this df indicates whether a test was fixed or not at a given timestamp (as compared to previous one)\n",
    "df_csv = pd.DataFrame(\n",
    "    flatter_list, columns=[\"timestamp\", \"tab\", \"grid\", \"test\", \"did_get_fixed\"]\n",
    ")\n",
    "df_csv.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "41a167c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tab                          grid                                                                             timestamp          \n",
       "\"redhat-assisted-installer\"  periodic-ci-openshift-release-master-nightly-4.6-e2e-metal-assisted              2021-03-04 01:01:58    0\n",
       "                                                                                                              2021-03-04 04:21:57    0\n",
       "                                                                                                              2021-03-04 07:22:22    0\n",
       "                                                                                                              2021-03-04 08:47:55    0\n",
       "                                                                                                              2021-03-04 23:12:31    0\n",
       "                                                                                                                                    ..\n",
       "\"redhat-single-node\"         periodic-ci-openshift-release-master-nightly-4.8-e2e-metal-single-node-live-iso  2021-03-14 00:01:00    0\n",
       "                                                                                                              2021-03-15 00:01:09    0\n",
       "                                                                                                              2021-03-16 00:01:15    0\n",
       "                                                                                                              2021-03-17 00:00:40    0\n",
       "                                                                                                              2021-03-18 00:01:25    0\n",
       "Name: did_get_fixed, Length: 37761, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# each element in this multiindexed series tells how many tests got fixed at each run/timestamp\n",
    "num_fixed_per_ts = df_csv.groupby([\"tab\", \"grid\", \"timestamp\"]).did_get_fixed.sum()\n",
    "num_fixed_per_ts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "477bebca",
   "metadata": {},
   "outputs": [],
   "source": [
    "build_failures_list = testgrid_labelwise_encoding(testgrid_data, 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9c69c765",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timestamp</th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>test</th>\n",
       "      <th>test_duration</th>\n",
       "      <th>failure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-03-15 23:40:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>80.283333</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021-03-15 00:01:06</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>92.050000</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2021-03-13 20:51:32</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>80.983333</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021-03-13 07:51:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>117.716667</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2021-03-13 06:43:20</td>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>Overall</td>\n",
       "      <td>108.633333</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            timestamp                          tab  \\\n",
       "0 2021-03-15 23:40:20  \"redhat-assisted-installer\"   \n",
       "1 2021-03-15 00:01:06  \"redhat-assisted-installer\"   \n",
       "2 2021-03-13 20:51:32  \"redhat-assisted-installer\"   \n",
       "3 2021-03-13 07:51:20  \"redhat-assisted-installer\"   \n",
       "4 2021-03-13 06:43:20  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid     test  test_duration  \\\n",
       "0  periodic-ci-openshift-release-master-nightly-4...  Overall      80.283333   \n",
       "1  periodic-ci-openshift-release-master-nightly-4...  Overall      92.050000   \n",
       "2  periodic-ci-openshift-release-master-nightly-4...  Overall      80.983333   \n",
       "3  periodic-ci-openshift-release-master-nightly-4...  Overall     117.716667   \n",
       "4  periodic-ci-openshift-release-master-nightly-4...  Overall     108.633333   \n",
       "\n",
       "   failure  \n",
       "0    False  \n",
       "1    False  \n",
       "2    False  \n",
       "3    False  \n",
       "4    False  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this df indicates whether a test was failing or not at a given timestamp\n",
    "failures_df = pd.DataFrame(\n",
    "    build_failures_list,\n",
    "    columns=[\"timestamp\", \"tab\", \"grid\", \"test\", \"test_duration\", \"failure\"],\n",
    ")\n",
    "failures_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4f9ace01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tab                          grid                                                                             timestamp          \n",
       "\"redhat-assisted-installer\"  periodic-ci-openshift-release-master-nightly-4.6-e2e-metal-assisted              2021-03-04 01:01:58    0\n",
       "                                                                                                              2021-03-04 04:21:57    0\n",
       "                                                                                                              2021-03-04 07:22:22    0\n",
       "                                                                                                              2021-03-04 08:47:55    0\n",
       "                                                                                                              2021-03-04 23:12:31    0\n",
       "                                                                                                                                    ..\n",
       "\"redhat-single-node\"         periodic-ci-openshift-release-master-nightly-4.8-e2e-metal-single-node-live-iso  2021-03-14 00:01:00    0\n",
       "                                                                                                              2021-03-15 00:01:09    0\n",
       "                                                                                                              2021-03-16 00:01:15    0\n",
       "                                                                                                              2021-03-17 00:00:40    0\n",
       "                                                                                                              2021-03-18 00:01:25    0\n",
       "Name: failure, Length: 37761, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# each element in this multiindexed series tells how many tests failed at each run/timestamp\n",
    "num_failures_per_ts = failures_df.groupby([\"tab\", \"grid\", \"timestamp\"]).failure.sum()\n",
    "num_failures_per_ts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "65ce22aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tab                          grid                                                                             timestamp          \n",
       "\"redhat-assisted-installer\"  periodic-ci-openshift-release-master-nightly-4.6-e2e-metal-assisted              2021-03-04 01:01:58    0.0\n",
       "                                                                                                              2021-03-04 04:21:57    0.0\n",
       "                                                                                                              2021-03-04 07:22:22    0.0\n",
       "                                                                                                              2021-03-04 08:47:55    0.0\n",
       "                                                                                                              2021-03-04 23:12:31    0.0\n",
       "                                                                                                                                    ... \n",
       "\"redhat-single-node\"         periodic-ci-openshift-release-master-nightly-4.8-e2e-metal-single-node-live-iso  2021-03-14 00:01:00    0.0\n",
       "                                                                                                              2021-03-15 00:01:09    0.0\n",
       "                                                                                                              2021-03-16 00:01:15    0.0\n",
       "                                                                                                              2021-03-17 00:00:40    0.0\n",
       "                                                                                                              2021-03-18 00:01:25    0.0\n",
       "Length: 37761, dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dividing the above two df's tells us what percent of failing tests got fixed at each timestamp\n",
    "pct_fixed_per_ts = (num_fixed_per_ts / num_failures_per_ts.shift()).fillna(0)\n",
    "pct_fixed_per_ts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2fa5df10",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>pct_fixed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 01:01:58</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 04:21:57</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 07:22:22</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 08:47:55</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 23:12:31</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37756</th>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-14 00:01:00</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37757</th>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-15 00:01:09</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37758</th>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-16 00:01:15</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37759</th>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-17 00:00:40</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37760</th>\n",
       "      <td>\"redhat-single-node\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-18 00:01:25</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>37761 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                               tab  \\\n",
       "0      \"redhat-assisted-installer\"   \n",
       "1      \"redhat-assisted-installer\"   \n",
       "2      \"redhat-assisted-installer\"   \n",
       "3      \"redhat-assisted-installer\"   \n",
       "4      \"redhat-assisted-installer\"   \n",
       "...                            ...   \n",
       "37756         \"redhat-single-node\"   \n",
       "37757         \"redhat-single-node\"   \n",
       "37758         \"redhat-single-node\"   \n",
       "37759         \"redhat-single-node\"   \n",
       "37760         \"redhat-single-node\"   \n",
       "\n",
       "                                                    grid           timestamp  \\\n",
       "0      periodic-ci-openshift-release-master-nightly-4... 2021-03-04 01:01:58   \n",
       "1      periodic-ci-openshift-release-master-nightly-4... 2021-03-04 04:21:57   \n",
       "2      periodic-ci-openshift-release-master-nightly-4... 2021-03-04 07:22:22   \n",
       "3      periodic-ci-openshift-release-master-nightly-4... 2021-03-04 08:47:55   \n",
       "4      periodic-ci-openshift-release-master-nightly-4... 2021-03-04 23:12:31   \n",
       "...                                                  ...                 ...   \n",
       "37756  periodic-ci-openshift-release-master-nightly-4... 2021-03-14 00:01:00   \n",
       "37757  periodic-ci-openshift-release-master-nightly-4... 2021-03-15 00:01:09   \n",
       "37758  periodic-ci-openshift-release-master-nightly-4... 2021-03-16 00:01:15   \n",
       "37759  periodic-ci-openshift-release-master-nightly-4... 2021-03-17 00:00:40   \n",
       "37760  periodic-ci-openshift-release-master-nightly-4... 2021-03-18 00:01:25   \n",
       "\n",
       "       pct_fixed  \n",
       "0            0.0  \n",
       "1            0.0  \n",
       "2            0.0  \n",
       "3            0.0  \n",
       "4            0.0  \n",
       "...          ...  \n",
       "37756        0.0  \n",
       "37757        0.0  \n",
       "37758        0.0  \n",
       "37759        0.0  \n",
       "37760        0.0  \n",
       "\n",
       "[37761 rows x 4 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert to df from multiindex series\n",
    "pct_fixed_per_ts_df = pct_fixed_per_ts.reset_index().rename(columns={0: \"pct_fixed\"})\n",
    "pct_fixed_per_ts_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d819167",
   "metadata": {},
   "source": [
    "## Save to Ceph or local\n",
    "Save the data frame in a parquet format on the Ceph bucket or locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f8225332",
   "metadata": {},
   "outputs": [],
   "source": [
    "save = pct_fixed_per_ts_df\n",
    "\n",
    "if AUTOMATION:\n",
    "    cc = CephCommunication(s3_endpoint_url, s3_access_key, s3_secret_key, s3_bucket)\n",
    "    cc.upload_to_ceph(\n",
    "        save,\n",
    "        s3_path,\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )\n",
    "else:\n",
    "    save_to_disk(\n",
    "        save,\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "640c2612",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tab</th>\n",
       "      <th>grid</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>pct_fixed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 01:01:58</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 04:21:57</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 07:22:22</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 08:47:55</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\"redhat-assisted-installer\"</td>\n",
       "      <td>periodic-ci-openshift-release-master-nightly-4...</td>\n",
       "      <td>2021-03-04 23:12:31</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           tab  \\\n",
       "0  \"redhat-assisted-installer\"   \n",
       "1  \"redhat-assisted-installer\"   \n",
       "2  \"redhat-assisted-installer\"   \n",
       "3  \"redhat-assisted-installer\"   \n",
       "4  \"redhat-assisted-installer\"   \n",
       "\n",
       "                                                grid           timestamp  \\\n",
       "0  periodic-ci-openshift-release-master-nightly-4... 2021-03-04 01:01:58   \n",
       "1  periodic-ci-openshift-release-master-nightly-4... 2021-03-04 04:21:57   \n",
       "2  periodic-ci-openshift-release-master-nightly-4... 2021-03-04 07:22:22   \n",
       "3  periodic-ci-openshift-release-master-nightly-4... 2021-03-04 08:47:55   \n",
       "4  periodic-ci-openshift-release-master-nightly-4... 2021-03-04 23:12:31   \n",
       "\n",
       "   pct_fixed  \n",
       "0        0.0  \n",
       "1        0.0  \n",
       "2        0.0  \n",
       "3        0.0  \n",
       "4        0.0  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Sanity check to see if the dataset is the same\n",
    "if AUTOMATION:\n",
    "    sanity_check = cc.read_from_ceph(\n",
    "        s3_path,\n",
    "        f\"{METRIC_NAME}/{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    ).head()\n",
    "else:\n",
    "    sanity_check = read_from_disk(\n",
    "        OUTPUT_DATA_PATH,\n",
    "        f\"{METRIC_NAME}-{timestamp.year}-{timestamp.month}-{timestamp.day}.parquet\",\n",
    "    ).head()\n",
    "\n",
    "sanity_check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9fc89bd",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "This notebook computed the mean fail length, the mean time to fix failures, pass-to-fail rate, and fail-to-pass rate for tests. The dataframe saved on ceph can be used to generate aggregated views and visualizations on the percent of fixed tests at each timestamp."
   ]
  }
 ],
 "metadata": {
  "finalized": {
   "timestamp": 1612822647266,
   "trusted": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}