aml-pipelines-with-automated-machine-learning-step.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
        "Licensed under the MIT License."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/NotebookVM/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.png)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Azure Machine Learning Pipeline with AutoMLStep (Udacity Course 2)\n",
        "This notebook demonstrates the use of AutoMLStep in Azure Machine Learning Pipeline."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Introduction\n",
        "In this example we showcase how you can use AzureML Dataset to load data for AutoML via AML Pipeline. \n",
        "\n",
        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you have executed the [configuration](https://aka.ms/pl-config) before running this notebook.\n",
        "\n",
        "In this notebook you will learn how to:\n",
        "1. Create an `Experiment` in an existing `Workspace`.\n",
        "2. Create or Attach existing AmlCompute to a workspace.\n",
        "3. Define data loading in a `TabularDataset`.\n",
        "4. Configure AutoML using `AutoMLConfig`.\n",
        "5. Use AutoMLStep\n",
        "6. Train the model using AmlCompute\n",
        "7. Explore the results.\n",
        "8. Test the best fitted model."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Azure Machine Learning and Pipeline SDK-specific imports"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "import os\n",
        "import csv\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn import datasets\n",
        "import pkg_resources\n",
        "\n",
        "import azureml.core\n",
        "from azureml.core.experiment import Experiment\n",
        "from azureml.core.workspace import Workspace\n",
        "from azureml.train.automl import AutoMLConfig\n",
        "from azureml.core.dataset import Dataset\n",
        "\n",
        "from azureml.pipeline.steps import AutoMLStep\n",
        "\n",
        "# Check core SDK version number\n",
        "print(\"SDK version:\", azureml.core.VERSION)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "SDK version: 1.20.0\n"
          ]
        }
      ],
      "execution_count": 1,
      "metadata": {
        "gather": {
          "logged": 1612833689311
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Initialize Workspace\n",
        "Initialize a workspace object from persisted configuration. Make sure the config file is present at .\\config.json"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "ws = Workspace.from_config()\n",
        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Warning: Falling back to use azure cli login credentials.\n",
            "If you run your code in unattended mode, i.e., where you can't give a user input, then we recommend to use ServicePrincipalAuthentication or MsiAuthentication.\n",
            "Please refer to aka.ms/aml-notebook-auth for different authentication mechanisms in azureml-sdk.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ml-experiment-1\n",
            "udacity1st\n",
            "uksouth\n",
            "5cf0bb76-4731-4cb5-ac3d-950c07dda6ef\n"
          ]
        }
      ],
      "execution_count": 2,
      "metadata": {
        "gather": {
          "logged": 1612833693896
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create an Azure ML experiment\n",
        "Let's create an experiment named \"automlstep-classification\" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure.\n",
        "\n",
        "The best practice is to use separate folders for scripts and its dependent files for each step and specify that folder as the `source_directory` for the step. This helps reduce the size of the snapshot created for the step (only the specific folder is snapshotted). Since changes in any files in the `source_directory` would trigger a re-upload of the snapshot, this helps keep the reuse of the step when there are no changes in the `source_directory` of the step.\n",
        "\n",
        "*Udacity Note:* There is no need to create an Azure ML experiment, this needs to re-use the experiment that was already created\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Choose a name for the run history container in the workspace.\n",
        "# NOTE: update these to match your existing experiment name\n",
        "experiment_name = 'ml-experiment-1'\n",
        "project_folder = './pipeline-project'\n",
        "\n",
        "experiment = Experiment(ws, experiment_name)\n",
        "experiment"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 3,
          "data": {
            "text/plain": "Experiment(Name: ml-experiment-1,\nWorkspace: ml-experiment-1)",
            "text/html": "<table style=\"width:100%\"><tr><th>Name</th><th>Workspace</th><th>Report Page</th><th>Docs Page</th></tr><tr><td>ml-experiment-1</td><td>ml-experiment-1</td><td><a href=\"https://ml.azure.com/experiments/ml-experiment-1?wsid=/subscriptions/5cf0bb76-4731-4cb5-ac3d-950c07dda6ef/resourcegroups/udacity1st/workspaces/ml-experiment-1\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment.Experiment?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 3,
      "metadata": {
        "gather": {
          "logged": 1612833701386
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create or Attach an AmlCompute cluster\n",
        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you get the default `AmlCompute` as your training compute resource.\n",
        "\n",
        "**Udacity Note** There is no need to create a new compute target, it can re-use the previous cluster"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.core.compute import AmlCompute\n",
        "from azureml.core.compute import ComputeTarget\n",
        "from azureml.core.compute_target import ComputeTargetException\n",
        "\n",
        "# NOTE: update the cluster name to match the existing cluster\n",
        "# Choose a name for your CPU cluster\n",
        "amlcompute_cluster_name = \"cpu-cluster\" # auto-ml\n",
        "\n",
        "# Verify that cluster does not exist already\n",
        "try:\n",
        "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
        "    print('Found existing cluster, use it.')\n",
        "except ComputeTargetException:\n",
        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',# for GPU, use \"STANDARD_NC6\"\n",
        "                                                           #vm_priority = 'lowpriority', # optional\n",
        "                                                           max_nodes=4)\n",
        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
        "\n",
        "compute_target.wait_for_completion(show_output=True, min_node_count = 1, timeout_in_minutes = 10)\n",
        "# For a more detailed view of current AmlCompute status, use get_status()."
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found existing cluster, use it.\n",
            "Succeeded\n",
            "AmlCompute wait for completion finished\n",
            "\n",
            "Minimum number of nodes requested have been provisioned\n"
          ]
        }
      ],
      "execution_count": 4,
      "metadata": {
        "gather": {
          "logged": 1612833748344
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data\n",
        "\n",
        "**Udacity note:** Make sure the `key` is the same name as the dataset that is uploaded, and that the description matches. If it is hard to find or unknown, loop over the `ws.datasets.keys()` and `print()` them.\n",
        "If it *isn't* found because it was deleted, it can be recreated with the link that has the CSV "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Try to load the dataset from the Workspace. Otherwise, create it from the file\n",
        "# NOTE: update the key to match the dataset name\n",
        "found = False\n",
        "key = \"BankMarketing Dataset\"\n",
        "description_text = \"Bank Marketing DataSet for Udacity Course 2\"\n",
        "\n",
        "if key in ws.datasets.keys(): \n",
        "        found = True\n",
        "        dataset = ws.datasets[key] \n",
        "\n",
        "if not found:\n",
        "        # Create AML Dataset and register it into Workspace\n",
        "        example_data = 'https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv'\n",
        "        dataset = Dataset.Tabular.from_delimited_files(example_data)        \n",
        "        #Register Dataset in Workspace\n",
        "        dataset = dataset.register(workspace=ws,\n",
        "                                   name=key,\n",
        "                                   description=description_text)\n",
        "\n",
        "\n",
        "df = dataset.to_pandas_dataframe()\n",
        "df.describe()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 5,
          "data": {
            "text/plain": "                age      duration      campaign         pdays      previous  \\\ncount  32950.000000  32950.000000  32950.000000  32950.000000  32950.000000   \nmean      40.040212    257.335205      2.561730    962.174780      0.174780   \nstd       10.432313    257.331700      2.763646    187.646785      0.496503   \nmin       17.000000      0.000000      1.000000      0.000000      0.000000   \n25%       32.000000    102.000000      1.000000    999.000000      0.000000   \n50%       38.000000    179.000000      2.000000    999.000000      0.000000   \n75%       47.000000    318.000000      3.000000    999.000000      0.000000   \nmax       98.000000   4918.000000     56.000000    999.000000      7.000000   \n\n       emp.var.rate  cons.price.idx  cons.conf.idx     euribor3m   nr.employed  \ncount  32950.000000    32950.000000   32950.000000  32950.000000  32950.000000  \nmean       0.076228       93.574243     -40.518680      3.615654   5166.859608  \nstd        1.572242        0.578636       4.623004      1.735748     72.208448  \nmin       -3.400000       92.201000     -50.800000      0.634000   4963.600000  \n25%       -1.800000       93.075000     -42.700000      1.344000   5099.100000  \n50%        1.100000       93.749000     -41.800000      4.857000   5191.000000  \n75%        1.400000       93.994000     -36.400000      4.961000   5228.100000  \nmax        1.400000       94.767000     -26.900000      5.045000   5228.100000  ",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>duration</th>\n      <th>campaign</th>\n      <th>pdays</th>\n      <th>previous</th>\n      <th>emp.var.rate</th>\n      <th>cons.price.idx</th>\n      <th>cons.conf.idx</th>\n      <th>euribor3m</th>\n      <th>nr.employed</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>count</th>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n    </tr>\n    <tr>\n      <th>mean</th>\n      <td>40.040212</td>\n      <td>257.335205</td>\n      <td>2.561730</td>\n      <td>962.174780</td>\n      <td>0.174780</td>\n      <td>0.076228</td>\n      <td>93.574243</td>\n      <td>-40.518680</td>\n      <td>3.615654</td>\n      <td>5166.859608</td>\n    </tr>\n    <tr>\n      <th>std</th>\n      <td>10.432313</td>\n      <td>257.331700</td>\n      <td>2.763646</td>\n      <td>187.646785</td>\n      <td>0.496503</td>\n      <td>1.572242</td>\n      <td>0.578636</td>\n      <td>4.623004</td>\n      <td>1.735748</td>\n      <td>72.208448</td>\n    </tr>\n    <tr>\n      <th>min</th>\n      <td>17.000000</td>\n      <td>0.000000</td>\n      <td>1.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>-3.400000</td>\n      <td>92.201000</td>\n      <td>-50.800000</td>\n      <td>0.634000</td>\n      <td>4963.600000</td>\n    </tr>\n    <tr>\n      <th>25%</th>\n      <td>32.000000</td>\n      <td>102.000000</td>\n      <td>1.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>-1.800000</td>\n      <td>93.075000</td>\n      <td>-42.700000</td>\n      <td>1.344000</td>\n      <td>5099.100000</td>\n    </tr>\n    <tr>\n      <th>50%</th>\n      <td>38.000000</td>\n      <td>179.000000</td>\n      <td>2.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>1.100000</td>\n      <td>93.749000</td>\n      <td>-41.800000</td>\n      <td>4.857000</td>\n      <td>5191.000000</td>\n    </tr>\n    <tr>\n      <th>75%</th>\n      <td>47.000000</td>\n      <td>318.000000</td>\n      <td>3.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>1.400000</td>\n      <td>93.994000</td>\n      <td>-36.400000</td>\n      <td>4.961000</td>\n      <td>5228.100000</td>\n    </tr>\n    <tr>\n      <th>max</th>\n      <td>98.000000</td>\n      <td>4918.000000</td>\n      <td>56.000000</td>\n      <td>999.000000</td>\n      <td>7.000000</td>\n      <td>1.400000</td>\n      <td>94.767000</td>\n      <td>-26.900000</td>\n      <td>5.045000</td>\n      <td>5228.100000</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 5,
      "metadata": {
        "gather": {
          "logged": 1612833795309
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Review the Dataset Result\n",
        "\n",
        "You can peek the result of a TabularDataset at any range using `skip(i)` and `take(j).to_pandas_dataframe()`. Doing so evaluates only `j` records for all the steps in the TabularDataset, which makes it fast even against large datasets.\n",
        "\n",
        "`TabularDataset` objects are composed of a list of transformation steps (optional)."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dataset.take(5).to_pandas_dataframe()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 6,
          "data": {
            "text/plain": "   age          job  marital    education  default housing loan    contact  \\\n0   57   technician  married  high.school       no      no  yes   cellular   \n1   55      unknown  married      unknown  unknown     yes   no  telephone   \n2   33  blue-collar  married     basic.9y       no      no   no   cellular   \n3   36       admin.  married  high.school       no      no   no  telephone   \n4   27    housemaid  married  high.school       no     yes   no   cellular   \n\n  month day_of_week  ...  campaign  pdays  previous     poutcome emp.var.rate  \\\n0   may         mon  ...         1    999         1      failure         -1.8   \n1   may         thu  ...         2    999         0  nonexistent          1.1   \n2   may         fri  ...         1    999         1      failure         -1.8   \n3   jun         fri  ...         4    999         0  nonexistent          1.4   \n4   jul         fri  ...         2    999         0  nonexistent          1.4   \n\n   cons.price.idx  cons.conf.idx  euribor3m  nr.employed   y  \n0          92.893          -46.2      1.299       5099.1  no  \n1          93.994          -36.4      4.860       5191.0  no  \n2          92.893          -46.2      1.313       5099.1  no  \n3          94.465          -41.8      4.967       5228.1  no  \n4          93.918          -42.7      4.963       5228.1  no  \n\n[5 rows x 21 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>job</th>\n      <th>marital</th>\n      <th>education</th>\n      <th>default</th>\n      <th>housing</th>\n      <th>loan</th>\n      <th>contact</th>\n      <th>month</th>\n      <th>day_of_week</th>\n      <th>...</th>\n      <th>campaign</th>\n      <th>pdays</th>\n      <th>previous</th>\n      <th>poutcome</th>\n      <th>emp.var.rate</th>\n      <th>cons.price.idx</th>\n      <th>cons.conf.idx</th>\n      <th>euribor3m</th>\n      <th>nr.employed</th>\n      <th>y</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>57</td>\n      <td>technician</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>mon</td>\n      <td>...</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.8</td>\n      <td>92.893</td>\n      <td>-46.2</td>\n      <td>1.299</td>\n      <td>5099.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>55</td>\n      <td>unknown</td>\n      <td>married</td>\n      <td>unknown</td>\n      <td>unknown</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>may</td>\n      <td>thu</td>\n      <td>...</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.1</td>\n      <td>93.994</td>\n      <td>-36.4</td>\n      <td>4.860</td>\n      <td>5191.0</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>33</td>\n      <td>blue-collar</td>\n      <td>married</td>\n      <td>basic.9y</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.8</td>\n      <td>92.893</td>\n      <td>-46.2</td>\n      <td>1.313</td>\n      <td>5099.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>36</td>\n      <td>admin.</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>jun</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>4</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.4</td>\n      <td>94.465</td>\n      <td>-41.8</td>\n      <td>4.967</td>\n      <td>5228.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>27</td>\n      <td>housemaid</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>jul</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.4</td>\n      <td>93.918</td>\n      <td>-42.7</td>\n      <td>4.963</td>\n      <td>5228.1</td>\n      <td>no</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 21 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 6,
      "metadata": {
        "gather": {
          "logged": 1612833839317
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Train\n",
        "This creates a general AutoML settings object.\n",
        "**Udacity notes:** These inputs must match what was used when training in the portal. `label_column_name` has to be `y` for example."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "automl_settings = {\n",
        "    \"experiment_timeout_minutes\": 20,\n",
        "    \"max_concurrent_iterations\": 5,\n",
        "    \"primary_metric\" : 'AUC_weighted'\n",
        "}\n",
        "automl_config = AutoMLConfig(compute_target=compute_target,\n",
        "                             task = \"classification\",\n",
        "                             training_data=dataset,\n",
        "                             label_column_name=\"y\",   \n",
        "                             path = project_folder,\n",
        "                             enable_early_stopping= True,\n",
        "                             featurization= 'auto',\n",
        "                             debug_log = \"automl_errors.log\",\n",
        "                             **automl_settings\n",
        "                            )"
      ],
      "outputs": [],
      "execution_count": 7,
      "metadata": {
        "gather": {
          "logged": 1612833846149
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Create Pipeline and AutoMLStep\n",
        "\n",
        "You can define outputs for the AutoMLStep using TrainingOutput."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core import PipelineData, TrainingOutput\n",
        "\n",
        "ds = ws.get_default_datastore()\n",
        "metrics_output_name = 'metrics_output'\n",
        "best_model_output_name = 'best_model_output'\n",
        "\n",
        "metrics_data = PipelineData(name='metrics_data',\n",
        "                           datastore=ds,\n",
        "                           pipeline_output_name=metrics_output_name,\n",
        "                           training_output=TrainingOutput(type='Metrics'))\n",
        "model_data = PipelineData(name='model_data',\n",
        "                           datastore=ds,\n",
        "                           pipeline_output_name=best_model_output_name,\n",
        "                           training_output=TrainingOutput(type='Model'))"
      ],
      "outputs": [],
      "execution_count": 8,
      "metadata": {
        "gather": {
          "logged": 1612833852203
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create an AutoMLStep."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "automl_step = AutoMLStep(\n",
        "    name='automl_module',\n",
        "    automl_config=automl_config,\n",
        "    outputs=[metrics_data, model_data],\n",
        "    allow_reuse=True)"
      ],
      "outputs": [],
      "execution_count": 16,
      "metadata": {
        "tags": [
          "automlstep-remarks-sample1"
        ],
        "gather": {
          "logged": 1612838177721
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core import Pipeline\n",
        "pipeline = Pipeline(\n",
        "    description=\"pipeline_with_automlstep\",\n",
        "    workspace=ws,    \n",
        "    steps=[automl_step])"
      ],
      "outputs": [],
      "execution_count": 10,
      "metadata": {
        "tags": [
          "automlstep-remarks-sample2"
        ],
        "gather": {
          "logged": 1612833898674
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipeline_run = experiment.submit(pipeline)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Created step automl_module [0e7f3c20][49a44d9f-1cc8-4d4c-9b50-a613e2fc417f], (This step will run and generate new outputs)\n",
            "Submitted PipelineRun ce608786-3c27-466d-bdbd-211a30923007\n",
            "Link to Azure Machine Learning Portal: https://ml.azure.com/experiments/ml-experiment-1/runs/ce608786-3c27-466d-bdbd-211a30923007?wsid=/subscriptions/5cf0bb76-4731-4cb5-ac3d-950c07dda6ef/resourcegroups/udacity1st/workspaces/ml-experiment-1\n"
          ]
        }
      ],
      "execution_count": 11,
      "metadata": {
        "gather": {
          "logged": 1612833907790
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.widgets import RunDetails\n",
        "RunDetails(pipeline_run).show()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …",
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "50e89969793f415f9a6a9da565152883"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/ml-experiment-1/runs/ce608786-3c27-466d-bdbd-211a30923007?wsid=/subscriptions/5cf0bb76-4731-4cb5-ac3d-950c07dda6ef/resourcegroups/udacity1st/workspaces/ml-experiment-1\", \"run_id\": \"ce608786-3c27-466d-bdbd-211a30923007\", \"run_properties\": {\"run_id\": \"ce608786-3c27-466d-bdbd-211a30923007\", \"created_utc\": \"2021-02-09T01:24:04.022229Z\", \"properties\": {\"azureml.runsource\": \"azureml.PipelineRun\", \"runSource\": \"SDK\", \"runType\": \"SDK\", \"azureml.parameters\": \"{}\"}, \"tags\": {\"azureml.pipelineComponent\": \"pipelinerun\"}, \"end_time_utc\": \"2021-02-09T02:05:12.633432Z\", \"status\": \"Completed\", \"log_files\": {\"logs/azureml/executionlogs.txt\": \"https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.ce608786-3c27-466d-bdbd-211a30923007/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=I4QnuE9ddAXQFGUSlX550jGE1Wj1iWW9CH04oL%2BBdq0%3D&st=2021-02-09T05%3A15%3A46Z&se=2021-02-09T13%3A25%3A46Z&sp=r\", \"logs/azureml/stderrlogs.txt\": \"https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.ce608786-3c27-466d-bdbd-211a30923007/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=UToevT403LBVbUadqVuThn%2BkSVN07SdpxxCrabfJG90%3D&st=2021-02-09T05%3A15%3A46Z&se=2021-02-09T13%3A25%3A46Z&sp=r\", \"logs/azureml/stdoutlogs.txt\": \"https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.ce608786-3c27-466d-bdbd-211a30923007/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=PDnho2z%2BJrW1qZgaZyJ3ahK9WmZvHX0YiMF521zRcSQ%3D&st=2021-02-09T05%3A15%3A46Z&se=2021-02-09T13%3A25%3A46Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/executionlogs.txt\", \"logs/azureml/stderrlogs.txt\", \"logs/azureml/stdoutlogs.txt\"]], \"run_duration\": \"0:41:08\"}, \"child_runs\": [{\"run_id\": \"d6700776-5eed-40ba-b978-2a96d733f981\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2021-02-09T01:24:26.620802Z\", \"created_time\": \"2021-02-09T01:24:11.476644Z\", \"end_time\": \"2021-02-09T02:04:36.521772Z\", \"duration\": \"0:40:25\", \"run_number\": 65, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2021-02-09T01:24:11.476644Z\", \"is_reused\": \"\"}], \"children_metrics\": {\"categories\": null, \"series\": null, \"metricName\": null}, \"run_metrics\": [], \"run_logs\": \"[2021-02-09 01:24:11Z] Submitting 1 runs, first five are: 0e7f3c20:d6700776-5eed-40ba-b978-2a96d733f981\\n[2021-02-09 02:05:12Z] Completing processing run id d6700776-5eed-40ba-b978-2a96d733f981.\\n\\nRun is completed.\", \"graph\": {\"datasource_nodes\": {\"c6a22348\": {\"node_id\": \"c6a22348\", \"name\": \"BankMarketing Dataset\"}}, \"module_nodes\": {\"0e7f3c20\": {\"node_id\": \"0e7f3c20\", \"name\": \"automl_module\", \"status\": \"Finished\", \"_is_reused\": false, \"run_id\": \"d6700776-5eed-40ba-b978-2a96d733f981\"}}, \"edges\": [{\"source_node_id\": \"c6a22348\", \"source_node_name\": \"BankMarketing Dataset\", \"source_name\": \"data\", \"target_name\": \"training_data\", \"dst_node_id\": \"0e7f3c20\", \"dst_node_name\": \"automl_module\"}], \"child_runs\": [{\"run_id\": \"d6700776-5eed-40ba-b978-2a96d733f981\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2021-02-09T01:24:26.620802Z\", \"created_time\": \"2021-02-09T01:24:11.476644Z\", \"end_time\": \"2021-02-09T02:04:36.521772Z\", \"duration\": \"0:40:25\", \"run_number\": 65, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2021-02-09T01:24:11.476644Z\", \"is_reused\": \"\"}]}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.20.0\"}, \"loading\": false}"
          },
          "metadata": {}
        }
      ],
      "execution_count": 12,
      "metadata": {
        "gather": {
          "logged": 1612833918787
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipeline_run.wait_for_completion()"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "PipelineRunId: ce608786-3c27-466d-bdbd-211a30923007\n",
            "Link to Azure Machine Learning Portal: https://ml.azure.com/experiments/ml-experiment-1/runs/ce608786-3c27-466d-bdbd-211a30923007?wsid=/subscriptions/5cf0bb76-4731-4cb5-ac3d-950c07dda6ef/resourcegroups/udacity1st/workspaces/ml-experiment-1\n",
            "\n",
            "PipelineRun Execution Summary\n",
            "==============================\n",
            "PipelineRun Status: Finished\n",
            "{'runId': 'ce608786-3c27-466d-bdbd-211a30923007', 'status': 'Completed', 'startTimeUtc': '2021-02-09T01:24:07.05553Z', 'endTimeUtc': '2021-02-09T02:05:12.633432Z', 'properties': {'azureml.runsource': 'azureml.PipelineRun', 'runSource': 'SDK', 'runType': 'SDK', 'azureml.parameters': '{}'}, 'inputDatasets': [], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.ce608786-3c27-466d-bdbd-211a30923007/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=w1k%2BRXjaVLmtyFBqfMqBc3Y0Wda7eRJEZJosc3DpEvo%3D&st=2021-02-09T02%3A15%3A12Z&se=2021-02-09T10%3A25%3A12Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.ce608786-3c27-466d-bdbd-211a30923007/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=R9dAHA%2FyAWEKsNcCxdFQrE2qB0QccT0p%2F1tMOvB8eO0%3D&st=2021-02-09T02%3A15%3A12Z&se=2021-02-09T10%3A25%3A12Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.ce608786-3c27-466d-bdbd-211a30923007/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=88q7KExkYdkVTstlgmn5S32Z8cPqV45a6HKu1gpMpnU%3D&st=2021-02-09T02%3A15%3A12Z&se=2021-02-09T10%3A25%3A12Z&sp=r'}, 'submittedBy': 'A Admin'}\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "execution_count": 14,
          "data": {
            "text/plain": "'Finished'"
          },
          "metadata": {}
        }
      ],
      "execution_count": 14,
      "metadata": {
        "gather": {
          "logged": 1612838162121
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Examine Results\n",
        "\n",
        "### Retrieve the metrics of all child runs\n",
        "Outputs of above run can be used as inputs of other steps in pipeline. In this tutorial, we will examine the outputs by retrieve output data and running some tests."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "metrics_output = pipeline_run.get_pipeline_output(metrics_output_name)\n",
        "num_file_downloaded = metrics_output.download('.', show_progress=True)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading azureml/d6700776-5eed-40ba-b978-2a96d733f981/metrics_data\n",
            "Downloaded azureml/d6700776-5eed-40ba-b978-2a96d733f981/metrics_data, 1 files out of an estimated total of 1\n"
          ]
        }
      ],
      "execution_count": 15,
      "metadata": {
        "gather": {
          "logged": 1612838169511
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "with open(metrics_output._path_on_datastore) as f:\n",
        "    metrics_output_result = f.read()\n",
        "    \n",
        "deserialized_metrics_output = json.loads(metrics_output_result)\n",
        "df = pd.DataFrame(deserialized_metrics_output)\n",
        "df"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 17,
          "data": {
            "text/plain": "                                 d6700776-5eed-40ba-b978-2a96d733f981_14  \\\nAUC_macro                                           [0.9005213514199395]   \nAUC_micro                                           [0.9677083731501034]   \nprecision_score_micro                               [0.8907435508345979]   \nweighted_accuracy                                   [0.9829629587898414]   \nmatthews_correlation                               [0.15296310841947716]   \nAUC_weighted                                        [0.9005213514199394]   \nf1_score_micro                                      [0.8907435508345979]   \naverage_precision_score_macro                        [0.743786592336894]   \nf1_score_macro                                      [0.5094292803970223]   \nrecall_score_macro                                  [0.5192999127530578]   \naverage_precision_score_micro                       [0.9684152996175677]   \nf1_score_weighted                                   [0.8450645952021447]   \nrecall_score_micro                                  [0.8907435508345979]   \nrecall_score_weighted                               [0.8907435508345979]   \nlog_loss                                           [0.25137784415065584]   \naccuracy                                            [0.8907435508345979]   \naverage_precision_score_weighted                    [0.9314791958449989]   \nprecision_score_macro                               [0.8030805480408413]   \nprecision_score_weighted                            [0.8719874949033176]   \nbalanced_accuracy                                   [0.5192999127530578]   \nnorm_macro_recall                                 [0.038599825506115515]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_3  \\\nAUC_macro                                          [0.8875301705853695]   \nAUC_micro                                            [0.87432349101158]   \nprecision_score_micro                               [0.790288315629742]   \nweighted_accuracy                                  [0.7872356595245353]   \nmatthews_correlation                               [0.4245590509208634]   \nAUC_weighted                                       [0.8875301705853695]   \nf1_score_micro                                      [0.790288315629742]   \naverage_precision_score_macro                      [0.7572002627838272]   \nf1_score_macro                                     [0.6679543971328674]   \nrecall_score_macro                                 [0.8025838802475516]   \naverage_precision_score_micro                      [0.8575855082735351]   \nf1_score_weighted                                  [0.8243583119311215]   \nrecall_score_micro                                  [0.790288315629742]   \nrecall_score_weighted                               [0.790288315629742]   \nlog_loss                                           [0.5083664089828028]   \naccuracy                                            [0.790288315629742]   \naverage_precision_score_weighted                   [0.9326583727065825]   \nprecision_score_macro                              [0.6489259668850806]   \nprecision_score_weighted                           [0.8994204407174433]   \nbalanced_accuracy                                  [0.8025838802475516]   \nnorm_macro_recall                                  [0.6051677604951031]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_11  \\\nAUC_macro                                           [0.8925834542009123]   \nAUC_micro                                           [0.8414756344394528]   \nprecision_score_micro                               [0.6622154779969651]   \nweighted_accuracy                                   [0.6279239982077891]   \nmatthews_correlation                               [0.38116521042998763]   \nAUC_weighted                                        [0.8925834542009122]   \nf1_score_micro                                      [0.6622154779969651]   \naverage_precision_score_macro                        [0.734424233215793]   \nf1_score_macro                                      [0.5796955449161503]   \nrecall_score_macro                                  [0.8003355580377403]   \naverage_precision_score_micro                         [0.86189155701341]   \nf1_score_weighted                                   [0.7242184688117966]   \nrecall_score_micro                                  [0.6622154779969651]   \nrecall_score_weighted                               [0.6622154779969651]   \nlog_loss                                             [0.484269722676828]   \naccuracy                                            [0.6622154779969651]   \naverage_precision_score_weighted                    [0.9290347331948864]   \nprecision_score_macro                               [0.6209371599148775]   \nprecision_score_weighted                            [0.9117048164875989]   \nbalanced_accuracy                                   [0.8003355580377403]   \nnorm_macro_recall                                   [0.6006711160754805]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_10  \\\nAUC_macro                                           [0.8381411770371976]   \nAUC_micro                                           [0.8278557892240278]   \nprecision_score_micro                               [0.8227617602427921]   \nweighted_accuracy                                     [0.84175311064373]   \nmatthews_correlation                               [0.38112542853516945]   \nAUC_weighted                                        [0.8381411770371976]   \nf1_score_micro                                      [0.8227617602427921]   \naverage_precision_score_macro                        [0.705522091894682]   \nf1_score_macro                                      [0.6722250838736403]   \nrecall_score_macro                                  [0.7462679240599651]   \naverage_precision_score_micro                       [0.7765509273532566]   \nf1_score_weighted                                   [0.8446040724432422]   \nrecall_score_micro                                  [0.8227617602427921]   \nrecall_score_weighted                               [0.8227617602427921]   \nlog_loss                                            [0.5464585031912554]   \naccuracy                                            [0.8227617602427921]   \naverage_precision_score_weighted                    [0.9134452637942311]   \nprecision_score_macro                               [0.6474578884263742]   \nprecision_score_weighted                            [0.8822677366372598]   \nbalanced_accuracy                                   [0.7462679240599651]   \nnorm_macro_recall                                  [0.49253584811993023]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_16  \\\nAUC_macro                                           [0.8643059978104908]   \nAUC_micro                                            [0.841808045942604]   \nprecision_score_micro                               [0.7559939301972686]   \nweighted_accuracy                                   [0.7565706639630971]   \nmatthews_correlation                                 [0.349078638506184]   \nAUC_weighted                                        [0.8643059978104907]   \nf1_score_micro                                      [0.7559939301972686]   \naverage_precision_score_macro                       [0.7167960153969344]   \nf1_score_macro                                      [0.6271415735498778]   \nrecall_score_macro                                  [0.7536709475092016]   \naverage_precision_score_micro                       [0.8230085213869246]   \nf1_score_weighted                                   [0.7972373277818748]   \nrecall_score_micro                                  [0.7559939301972686]   \nrecall_score_weighted                               [0.7559939301972686]   \nlog_loss                                            [0.5095386749702528]   \naccuracy                                            [0.7559939301972686]   \naverage_precision_score_weighted                    [0.9213485712046892]   \nprecision_score_macro                               [0.6200924830551504]   \nprecision_score_weighted                            [0.8839497346439098]   \nbalanced_accuracy                                   [0.7536709475092016]   \nnorm_macro_recall                                   [0.5073418950184032]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_21  \\\nAUC_macro                                           [0.9446000440865653]   \nAUC_micro                                            [0.979591094245431]   \nprecision_score_micro                                [0.910773899848255]   \nweighted_accuracy                                   [0.9595981069341017]   \nmatthews_correlation                                [0.4965812945659966]   \nAUC_weighted                                        [0.9446000440865653]   \nf1_score_micro                                      [0.9107738998482551]   \naverage_precision_score_macro                       [0.8165951746335667]   \nf1_score_macro                                      [0.7434578073061547]   \nrecall_score_macro                                   [0.714118537289269]   \naverage_precision_score_micro                       [0.9805411439523444]   \nf1_score_weighted                                   [0.9042346763892627]   \nrecall_score_micro                                   [0.910773899848255]   \nrecall_score_weighted                                [0.910773899848255]   \nlog_loss                                             [0.190468543909731]   \naccuracy                                             [0.910773899848255]   \naverage_precision_score_weighted                    [0.9534292590474763]   \nprecision_score_macro                               [0.7879164331527493]   \nprecision_score_weighted                            [0.9015318191709801]   \nbalanced_accuracy                                    [0.714118537289269]   \nnorm_macro_recall                                    [0.428237074578538]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_15  \\\nAUC_macro                                           [0.8729385362889854]   \nAUC_micro                                           [0.9621870632148308]   \nprecision_score_micro                               [0.8880121396054628]   \nweighted_accuracy                                   [0.9843450583187134]   \nmatthews_correlation                                               [0.0]   \nAUC_weighted                                        [0.8729385362889854]   \nf1_score_micro                                      [0.8880121396054628]   \naverage_precision_score_macro                        [0.688261229610788]   \nf1_score_macro                                      [0.4703423886834914]   \nrecall_score_macro                                                 [0.5]   \naverage_precision_score_micro                       [0.9598204816316278]   \nf1_score_weighted                                   [0.8353395018439429]   \nrecall_score_micro                                  [0.8880121396054628]   \nrecall_score_weighted                               [0.8880121396054628]   \nlog_loss                                           [0.26133921308850605]   \naccuracy                                            [0.8880121396054628]   \naverage_precision_score_weighted                    [0.9137884768949512]   \nprecision_score_macro                               [0.4440060698027314]   \nprecision_score_weighted                             [0.788565560086672]   \nbalanced_accuracy                                                  [0.5]   \nnorm_macro_recall                                                  [0.0]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_8  \\\nAUC_macro                                          [0.8933558952814409]   \nAUC_micro                                          [0.9662479362440446]   \nprecision_score_micro                              [0.8880121396054628]   \nweighted_accuracy                                  [0.9843450583187134]   \nmatthews_correlation                                              [0.0]   \nAUC_weighted                                       [0.8933558952814408]   \nf1_score_micro                                     [0.8880121396054628]   \naverage_precision_score_macro                      [0.7446961685919511]   \nf1_score_macro                                     [0.4703423886834914]   \nrecall_score_macro                                                [0.5]   \naverage_precision_score_micro                      [0.9646850975449291]   \nf1_score_weighted                                  [0.8353395018439429]   \nrecall_score_micro                                 [0.8880121396054628]   \nrecall_score_weighted                              [0.8880121396054628]   \nlog_loss                                          [0.26926069134726394]   \naccuracy                                           [0.8880121396054628]   \naverage_precision_score_weighted                   [0.9286681474254169]   \nprecision_score_macro                              [0.4440060698027314]   \nprecision_score_weighted                            [0.788565560086672]   \nbalanced_accuracy                                                 [0.5]   \nnorm_macro_recall                                                 [0.0]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_12  \\\nAUC_macro                                            [0.865037686603797]   \nAUC_micro                                           [0.8462810944987231]   \nprecision_score_micro                                [0.757814871016692]   \nweighted_accuracy                                   [0.7597650948182824]   \nmatthews_correlation                               [0.34538546170550394]   \nAUC_weighted                                        [0.8650376866037969]   \nf1_score_micro                                      [0.7578148710166921]   \naverage_precision_score_macro                       [0.7396190991649754]   \nf1_score_macro                                      [0.6270670510865421]   \nrecall_score_macro                                  [0.7499597108069509]   \naverage_precision_score_micro                       [0.8373579893054997]   \nf1_score_weighted                                   [0.7984264439934582]   \nrecall_score_micro                                   [0.757814871016692]   \nrecall_score_weighted                                [0.757814871016692]   \nlog_loss                                            [0.5933679946976629]   \naccuracy                                             [0.757814871016692]   \naverage_precision_score_weighted                     [0.926379634336469]   \nprecision_score_macro                               [0.6193103448275863]   \nprecision_score_weighted                            [0.8826234106012245]   \nbalanced_accuracy                                   [0.7499597108069509]   \nnorm_macro_recall                                   [0.4999194216139018]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_9  \\\nAUC_macro                                           [0.824191391264562]   \nAUC_micro                                          [0.8213818242105917]   \nprecision_score_micro                               [0.735660091047041]   \nweighted_accuracy                                  [0.7360888940294933]   \nmatthews_correlation                              [0.31729065225060094]   \nAUC_weighted                                        [0.824191391264562]   \nf1_score_micro                                      [0.735660091047041]   \naverage_precision_score_macro                      [0.7122419271618234]   \nf1_score_macro                                     [0.6072640573082512]   \nrecall_score_macro                                 [0.7339329476685061]   \naverage_precision_score_micro                      [0.7987488934446838]   \nf1_score_weighted                                   [0.781525553205324]   \nrecall_score_micro                                  [0.735660091047041]   \nrecall_score_weighted                               [0.735660091047041]   \nlog_loss                                            [0.561554712792468]   \naccuracy                                            [0.735660091047041]   \naverage_precision_score_weighted                   [0.9117912799208243]   \nprecision_score_macro                              [0.6075878355410955]   \nprecision_score_weighted                            [0.878009600469834]   \nbalanced_accuracy                                  [0.7339329476685061]   \nnorm_macro_recall                                  [0.4678658953370123]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_2  \\\nAUC_macro                                          [0.9003606577419159]   \nAUC_micro                                          [0.9677672290521575]   \nprecision_score_micro                              [0.8955993930197269]   \nweighted_accuracy                                  [0.9830537880576069]   \nmatthews_correlation                               [0.2466338262655897]   \nAUC_weighted                                       [0.9003606577419158]   \nf1_score_micro                                     [0.8955993930197269]   \naverage_precision_score_macro                      [0.7506021138019261]   \nf1_score_macro                                     [0.5526560896676928]   \nrecall_score_macro                                 [0.5433483931558386]   \naverage_precision_score_micro                      [0.9678194557998724]   \nf1_score_weighted                                  [0.8566096517720779]   \nrecall_score_micro                                 [0.8955993930197269]   \nrecall_score_weighted                              [0.8955993930197269]   \nlog_loss                                          [0.24073153434773964]   \naccuracy                                           [0.8955993930197269]   \naverage_precision_score_weighted                   [0.9323316000622094]   \nprecision_score_macro                              [0.8508102597928253]   \nprecision_score_weighted                           [0.8864547707362387]   \nbalanced_accuracy                                  [0.5433483931558386]   \nnorm_macro_recall                                 [0.08669678631167721]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_22  \\\nAUC_macro                                           [0.9419057621881755]   \nAUC_micro                                           [0.9790393777300873]   \nprecision_score_micro                               [0.9132018209408195]   \nweighted_accuracy                                    [0.960525485255363]   \nmatthews_correlation                                [0.5127151291642515]   \nAUC_weighted                                        [0.9419057621881755]   \nf1_score_micro                                      [0.9132018209408195]   \naverage_precision_score_macro                       [0.8147202692913998]   \nf1_score_macro                                      [0.7518365313311086]   \nrecall_score_macro                                  [0.7225903820897402]   \naverage_precision_score_micro                       [0.9799780254864652]   \nf1_score_weighted                                   [0.9071286071805743]   \nrecall_score_micro                                  [0.9132018209408195]   \nrecall_score_weighted                               [0.9132018209408195]   \nlog_loss                                           [0.20140669839595468]   \naccuracy                                            [0.9132018209408195]   \naverage_precision_score_weighted                    [0.9526718948733789]   \nprecision_score_macro                               [0.7952472622648324]   \nprecision_score_weighted                            [0.9046450883564172]   \nbalanced_accuracy                                   [0.7225903820897402]   \nnorm_macro_recall                                   [0.4451807641794805]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_4  \\\nAUC_macro                                           [0.889712270328445]   \nAUC_micro                                          [0.9655232441667951]   \nprecision_score_micro                              [0.8880121396054628]   \nweighted_accuracy                                  [0.9843450583187134]   \nmatthews_correlation                                              [0.0]   \nAUC_weighted                                       [0.8897122703284451]   \nf1_score_micro                                     [0.8880121396054628]   \naverage_precision_score_macro                       [0.717721818174303]   \nf1_score_macro                                     [0.4703423886834914]   \nrecall_score_macro                                                [0.5]   \naverage_precision_score_micro                      [0.9662155304239755]   \nf1_score_weighted                                  [0.8353395018439429]   \nrecall_score_micro                                 [0.8880121396054628]   \nrecall_score_weighted                              [0.8880121396054628]   \nlog_loss                                          [0.25875146426964135]   \naccuracy                                           [0.8880121396054628]   \naverage_precision_score_weighted                   [0.9243727763723245]   \nprecision_score_macro                              [0.4440060698027314]   \nprecision_score_weighted                            [0.788565560086672]   \nbalanced_accuracy                                                 [0.5]   \nnorm_macro_recall                                                 [0.0]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_1  \\\nAUC_macro                                          [0.9392346349984347]   \nAUC_micro                                          [0.9781770788959222]   \nprecision_score_micro                              [0.9071320182094081]   \nweighted_accuracy                                  [0.9537972210153172]   \nmatthews_correlation                                [0.488678780261868]   \nAUC_weighted                                       [0.9392346349984347]   \nf1_score_micro                                     [0.9071320182094081]   \naverage_precision_score_macro                      [0.8065229883244922]   \nf1_score_macro                                     [0.7416848907681176]   \nrecall_score_macro                                 [0.7191727470931578]   \naverage_precision_score_micro                      [0.9791945367231853]   \nf1_score_weighted                                  [0.9021127651963996]   \nrecall_score_micro                                 [0.9071320182094081]   \nrecall_score_weighted                              [0.9071320182094081]   \nlog_loss                                           [0.1874363495858499]   \naccuracy                                           [0.9071320182094081]   \naverage_precision_score_weighted                   [0.9505970434373063]   \nprecision_score_macro                              [0.7723958081530135]   \nprecision_score_weighted                           [0.8991976076061607]   \nbalanced_accuracy                                  [0.7191727470931578]   \nnorm_macro_recall                                 [0.43834549418631563]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_5  \\\nAUC_macro                                          [0.8756073480078614]   \nAUC_micro                                           [0.963066954345228]   \nprecision_score_micro                              [0.8977238239757208]   \nweighted_accuracy                                  [0.9733551768141164]   \nmatthews_correlation                              [0.31858669063458983]   \nAUC_weighted                                       [0.8756073480078614]   \nf1_score_micro                                     [0.8977238239757208]   \naverage_precision_score_macro                      [0.7187705891690357]   \nf1_score_macro                                     [0.6249769919873013]   \nrecall_score_macro                                 [0.5930939692172041]   \naverage_precision_score_micro                       [0.963449291026005]   \nf1_score_weighted                                  [0.8731669028038096]   \nrecall_score_micro                                 [0.8977238239757208]   \nrecall_score_weighted                              [0.8977238239757208]   \nlog_loss                                           [0.2575613906785161]   \naccuracy                                           [0.8977238239757208]   \naverage_precision_score_weighted                   [0.9230853367522769]   \nprecision_score_macro                              [0.7725672788016185]   \nprecision_score_weighted                           [0.8770482732548277]   \nbalanced_accuracy                                  [0.5930939692172041]   \nnorm_macro_recall                                 [0.18618793843440828]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_17  \\\nAUC_macro                                           [0.8392299114378703]   \nAUC_micro                                           [0.8376790142787733]   \nprecision_score_micro                               [0.7517450682852808]   \nweighted_accuracy                                   [0.7542127821614079]   \nmatthews_correlation                                [0.3329431800225301]   \nAUC_weighted                                        [0.8392299114378704]   \nf1_score_micro                                      [0.7517450682852809]   \naverage_precision_score_macro                       [0.7178952833685254]   \nf1_score_macro                                      [0.6202346693899543]   \nrecall_score_macro                                  [0.7418055486091429]   \naverage_precision_score_micro                       [0.8152336586494915]   \nf1_score_weighted                                   [0.7936601928429454]   \nrecall_score_micro                                  [0.7517450682852808]   \nrecall_score_weighted                               [0.7517450682852808]   \nlog_loss                                            [0.5615865304579278]   \naccuracy                                            [0.7517450682852808]   \naverage_precision_score_weighted                    [0.9169114563320293]   \nprecision_score_macro                               [0.6146077517256395]   \nprecision_score_weighted                            [0.8800582577385105]   \nbalanced_accuracy                                   [0.7418055486091429]   \nnorm_macro_recall                                   [0.4836110972182859]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_6  \\\nAUC_macro                                          [0.8952647694624588]   \nAUC_micro                                          [0.9670939322696593]   \nprecision_score_micro                              [0.8992412746585736]   \nweighted_accuracy                                  [0.9765071823530921]   \nmatthews_correlation                              [0.32208584960149145]   \nAUC_weighted                                       [0.8952647694624589]   \nf1_score_micro                                     [0.8992412746585736]   \naverage_precision_score_macro                      [0.7352543047729461]   \nf1_score_macro                                     [0.6196843276317618]   \nrecall_score_macro                                 [0.5880277189648178]   \naverage_precision_score_micro                       [0.968573349610069]   \nf1_score_weighted                                  [0.8727204740202779]   \nrecall_score_micro                                 [0.8992412746585736]   \nrecall_score_weighted                              [0.8992412746585736]   \nlog_loss                                           [0.2490233975691646]   \naccuracy                                           [0.8992412746585736]   \naverage_precision_score_weighted                    [0.929779770688238]   \nprecision_score_macro                              [0.7946211026863488]   \nprecision_score_weighted                           [0.8811111704952653]   \nbalanced_accuracy                                  [0.5880277189648178]   \nnorm_macro_recall                                 [0.17605543792963552]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_13  \\\nAUC_macro                                           [0.9308878256246677]   \nAUC_micro                                           [0.9758990146932517]   \nprecision_score_micro                               [0.9062215477996965]   \nweighted_accuracy                                   [0.9766010009385309]   \nmatthews_correlation                                [0.3976739324324451]   \nAUC_weighted                                        [0.9308878256246675]   \nf1_score_micro                                      [0.9062215477996965]   \naverage_precision_score_macro                       [0.7985126174047921]   \nf1_score_macro                                      [0.6650154684677185]   \nrecall_score_macro                                  [0.6227454260188535]   \naverage_precision_score_micro                       [0.9766643355999638]   \nf1_score_weighted                                    [0.885603431576398]   \nrecall_score_micro                                  [0.9062215477996965]   \nrecall_score_weighted                               [0.9062215477996965]   \nlog_loss                                           [0.33655623030329523]   \naccuracy                                            [0.9062215477996965]   \naverage_precision_score_weighted                     [0.947605275820125]   \nprecision_score_macro                                [0.822098675416211]   \nprecision_score_weighted                            [0.8929725418691179]   \nbalanced_accuracy                                   [0.6227454260188535]   \nnorm_macro_recall                                  [0.24549085203770704]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_7  \\\nAUC_macro                                          [0.8579949504211379]   \nAUC_micro                                          [0.9592157151705922]   \nprecision_score_micro                              [0.8883156297420334]   \nweighted_accuracy                                  [0.9843874836349229]   \nmatthews_correlation                             [0.049063912865329386]   \nAUC_weighted                                       [0.8579949504211378]   \nf1_score_micro                                     [0.8883156297420334]   \naverage_precision_score_macro                      [0.7317073954787249]   \nf1_score_macro                                    [0.47312070913357085]   \nrecall_score_macro                                 [0.5013550135501355]   \naverage_precision_score_micro                       [0.960094413862385]   \nf1_score_weighted                                  [0.8360791405851395]   \nrecall_score_micro                                 [0.8883156297420334]   \nrecall_score_weighted                              [0.8883156297420334]   \nlog_loss                                          [0.27374024724306756]   \naccuracy                                           [0.8883156297420334]   \naverage_precision_score_weighted                   [0.9236150003775165]   \nprecision_score_macro                              [0.9441408621736491]   \nprecision_score_weighted                           [0.9007928150046113]   \nbalanced_accuracy                                  [0.5013550135501355]   \nnorm_macro_recall                               [0.0027100271002709064]   \n\n                                 d6700776-5eed-40ba-b978-2a96d733f981_0  \nAUC_macro                                          [0.9450464668693166]  \nAUC_micro                                           [0.979695082216353]  \nprecision_score_micro                              [0.9116843702579667]  \nweighted_accuracy                                  [0.9514937218005303]  \nmatthews_correlation                               [0.5323740218566827]  \nAUC_weighted                                       [0.9450464668693167]  \nf1_score_micro                                     [0.9116843702579667]  \naverage_precision_score_macro                      [0.8151093723721079]  \nf1_score_macro                                     [0.7653697272147331]  \nrecall_score_macro                                 [0.7513392683482543]  \naverage_precision_score_micro                      [0.9806603102489483]  \nf1_score_weighted                                  [0.9091539479147899]  \nrecall_score_micro                                 [0.9116843702579667]  \nrecall_score_weighted                              [0.9116843702579667]  \nlog_loss                                          [0.17775706110025447]  \naccuracy                                           [0.9116843702579667]  \naverage_precision_score_weighted                   [0.9531771295804466]  \nprecision_score_macro                              [0.7819118765348991]  \nprecision_score_weighted                           [0.9072720074188747]  \nbalanced_accuracy                                  [0.7513392683482543]  \nnorm_macro_recall                                  [0.5026785366965085]  ",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_14</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_3</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_11</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_10</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_16</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_21</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_15</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_8</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_12</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_9</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_2</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_22</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_4</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_1</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_5</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_17</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_6</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_13</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_7</th>\n      <th>d6700776-5eed-40ba-b978-2a96d733f981_0</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>AUC_macro</th>\n      <td>[0.9005213514199395]</td>\n      <td>[0.8875301705853695]</td>\n      <td>[0.8925834542009123]</td>\n      <td>[0.8381411770371976]</td>\n      <td>[0.8643059978104908]</td>\n      <td>[0.9446000440865653]</td>\n      <td>[0.8729385362889854]</td>\n      <td>[0.8933558952814409]</td>\n      <td>[0.865037686603797]</td>\n      <td>[0.824191391264562]</td>\n      <td>[0.9003606577419159]</td>\n      <td>[0.9419057621881755]</td>\n      <td>[0.889712270328445]</td>\n      <td>[0.9392346349984347]</td>\n      <td>[0.8756073480078614]</td>\n      <td>[0.8392299114378703]</td>\n      <td>[0.8952647694624588]</td>\n      <td>[0.9308878256246677]</td>\n      <td>[0.8579949504211379]</td>\n      <td>[0.9450464668693166]</td>\n    </tr>\n    <tr>\n      <th>AUC_micro</th>\n      <td>[0.9677083731501034]</td>\n      <td>[0.87432349101158]</td>\n      <td>[0.8414756344394528]</td>\n      <td>[0.8278557892240278]</td>\n      <td>[0.841808045942604]</td>\n      <td>[0.979591094245431]</td>\n      <td>[0.9621870632148308]</td>\n      <td>[0.9662479362440446]</td>\n      <td>[0.8462810944987231]</td>\n      <td>[0.8213818242105917]</td>\n      <td>[0.9677672290521575]</td>\n      <td>[0.9790393777300873]</td>\n      <td>[0.9655232441667951]</td>\n      <td>[0.9781770788959222]</td>\n      <td>[0.963066954345228]</td>\n      <td>[0.8376790142787733]</td>\n      <td>[0.9670939322696593]</td>\n      <td>[0.9758990146932517]</td>\n      <td>[0.9592157151705922]</td>\n      <td>[0.979695082216353]</td>\n    </tr>\n    <tr>\n      <th>precision_score_micro</th>\n      <td>[0.8907435508345979]</td>\n      <td>[0.790288315629742]</td>\n      <td>[0.6622154779969651]</td>\n      <td>[0.8227617602427921]</td>\n      <td>[0.7559939301972686]</td>\n      <td>[0.910773899848255]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.757814871016692]</td>\n      <td>[0.735660091047041]</td>\n      <td>[0.8955993930197269]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9071320182094081]</td>\n      <td>[0.8977238239757208]</td>\n      <td>[0.7517450682852808]</td>\n      <td>[0.8992412746585736]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.8883156297420334]</td>\n      <td>[0.9116843702579667]</td>\n    </tr>\n    <tr>\n      <th>weighted_accuracy</th>\n      <td>[0.9829629587898414]</td>\n      <td>[0.7872356595245353]</td>\n      <td>[0.6279239982077891]</td>\n      <td>[0.84175311064373]</td>\n      <td>[0.7565706639630971]</td>\n      <td>[0.9595981069341017]</td>\n      <td>[0.9843450583187134]</td>\n      <td>[0.9843450583187134]</td>\n      <td>[0.7597650948182824]</td>\n      <td>[0.7360888940294933]</td>\n      <td>[0.9830537880576069]</td>\n      <td>[0.960525485255363]</td>\n      <td>[0.9843450583187134]</td>\n      <td>[0.9537972210153172]</td>\n      <td>[0.9733551768141164]</td>\n      <td>[0.7542127821614079]</td>\n      <td>[0.9765071823530921]</td>\n      <td>[0.9766010009385309]</td>\n      <td>[0.9843874836349229]</td>\n      <td>[0.9514937218005303]</td>\n    </tr>\n    <tr>\n      <th>matthews_correlation</th>\n      <td>[0.15296310841947716]</td>\n      <td>[0.4245590509208634]</td>\n      <td>[0.38116521042998763]</td>\n      <td>[0.38112542853516945]</td>\n      <td>[0.349078638506184]</td>\n      <td>[0.4965812945659966]</td>\n      <td>[0.0]</td>\n      <td>[0.0]</td>\n      <td>[0.34538546170550394]</td>\n      <td>[0.31729065225060094]</td>\n      <td>[0.2466338262655897]</td>\n      <td>[0.5127151291642515]</td>\n      <td>[0.0]</td>\n      <td>[0.488678780261868]</td>\n      <td>[0.31858669063458983]</td>\n      <td>[0.3329431800225301]</td>\n      <td>[0.32208584960149145]</td>\n      <td>[0.3976739324324451]</td>\n      <td>[0.049063912865329386]</td>\n      <td>[0.5323740218566827]</td>\n    </tr>\n    <tr>\n      <th>AUC_weighted</th>\n      <td>[0.9005213514199394]</td>\n      <td>[0.8875301705853695]</td>\n      <td>[0.8925834542009122]</td>\n      <td>[0.8381411770371976]</td>\n      <td>[0.8643059978104907]</td>\n      <td>[0.9446000440865653]</td>\n      <td>[0.8729385362889854]</td>\n      <td>[0.8933558952814408]</td>\n      <td>[0.8650376866037969]</td>\n      <td>[0.824191391264562]</td>\n      <td>[0.9003606577419158]</td>\n      <td>[0.9419057621881755]</td>\n      <td>[0.8897122703284451]</td>\n      <td>[0.9392346349984347]</td>\n      <td>[0.8756073480078614]</td>\n      <td>[0.8392299114378704]</td>\n      <td>[0.8952647694624589]</td>\n      <td>[0.9308878256246675]</td>\n      <td>[0.8579949504211378]</td>\n      <td>[0.9450464668693167]</td>\n    </tr>\n    <tr>\n      <th>f1_score_micro</th>\n      <td>[0.8907435508345979]</td>\n      <td>[0.790288315629742]</td>\n      <td>[0.6622154779969651]</td>\n      <td>[0.8227617602427921]</td>\n      <td>[0.7559939301972686]</td>\n      <td>[0.9107738998482551]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.7578148710166921]</td>\n      <td>[0.735660091047041]</td>\n      <td>[0.8955993930197269]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9071320182094081]</td>\n      <td>[0.8977238239757208]</td>\n      <td>[0.7517450682852809]</td>\n      <td>[0.8992412746585736]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.8883156297420334]</td>\n      <td>[0.9116843702579667]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_macro</th>\n      <td>[0.743786592336894]</td>\n      <td>[0.7572002627838272]</td>\n      <td>[0.734424233215793]</td>\n      <td>[0.705522091894682]</td>\n      <td>[0.7167960153969344]</td>\n      <td>[0.8165951746335667]</td>\n      <td>[0.688261229610788]</td>\n      <td>[0.7446961685919511]</td>\n      <td>[0.7396190991649754]</td>\n      <td>[0.7122419271618234]</td>\n      <td>[0.7506021138019261]</td>\n      <td>[0.8147202692913998]</td>\n      <td>[0.717721818174303]</td>\n      <td>[0.8065229883244922]</td>\n      <td>[0.7187705891690357]</td>\n      <td>[0.7178952833685254]</td>\n      <td>[0.7352543047729461]</td>\n      <td>[0.7985126174047921]</td>\n      <td>[0.7317073954787249]</td>\n      <td>[0.8151093723721079]</td>\n    </tr>\n    <tr>\n      <th>f1_score_macro</th>\n      <td>[0.5094292803970223]</td>\n      <td>[0.6679543971328674]</td>\n      <td>[0.5796955449161503]</td>\n      <td>[0.6722250838736403]</td>\n      <td>[0.6271415735498778]</td>\n      <td>[0.7434578073061547]</td>\n      <td>[0.4703423886834914]</td>\n      <td>[0.4703423886834914]</td>\n      <td>[0.6270670510865421]</td>\n      <td>[0.6072640573082512]</td>\n      <td>[0.5526560896676928]</td>\n      <td>[0.7518365313311086]</td>\n      <td>[0.4703423886834914]</td>\n      <td>[0.7416848907681176]</td>\n      <td>[0.6249769919873013]</td>\n      <td>[0.6202346693899543]</td>\n      <td>[0.6196843276317618]</td>\n      <td>[0.6650154684677185]</td>\n      <td>[0.47312070913357085]</td>\n      <td>[0.7653697272147331]</td>\n    </tr>\n    <tr>\n      <th>recall_score_macro</th>\n      <td>[0.5192999127530578]</td>\n      <td>[0.8025838802475516]</td>\n      <td>[0.8003355580377403]</td>\n      <td>[0.7462679240599651]</td>\n      <td>[0.7536709475092016]</td>\n      <td>[0.714118537289269]</td>\n      <td>[0.5]</td>\n      <td>[0.5]</td>\n      <td>[0.7499597108069509]</td>\n      <td>[0.7339329476685061]</td>\n      <td>[0.5433483931558386]</td>\n      <td>[0.7225903820897402]</td>\n      <td>[0.5]</td>\n      <td>[0.7191727470931578]</td>\n      <td>[0.5930939692172041]</td>\n      <td>[0.7418055486091429]</td>\n      <td>[0.5880277189648178]</td>\n      <td>[0.6227454260188535]</td>\n      <td>[0.5013550135501355]</td>\n      <td>[0.7513392683482543]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_micro</th>\n      <td>[0.9684152996175677]</td>\n      <td>[0.8575855082735351]</td>\n      <td>[0.86189155701341]</td>\n      <td>[0.7765509273532566]</td>\n      <td>[0.8230085213869246]</td>\n      <td>[0.9805411439523444]</td>\n      <td>[0.9598204816316278]</td>\n      <td>[0.9646850975449291]</td>\n      <td>[0.8373579893054997]</td>\n      <td>[0.7987488934446838]</td>\n      <td>[0.9678194557998724]</td>\n      <td>[0.9799780254864652]</td>\n      <td>[0.9662155304239755]</td>\n      <td>[0.9791945367231853]</td>\n      <td>[0.963449291026005]</td>\n      <td>[0.8152336586494915]</td>\n      <td>[0.968573349610069]</td>\n      <td>[0.9766643355999638]</td>\n      <td>[0.960094413862385]</td>\n      <td>[0.9806603102489483]</td>\n    </tr>\n    <tr>\n      <th>f1_score_weighted</th>\n      <td>[0.8450645952021447]</td>\n      <td>[0.8243583119311215]</td>\n      <td>[0.7242184688117966]</td>\n      <td>[0.8446040724432422]</td>\n      <td>[0.7972373277818748]</td>\n      <td>[0.9042346763892627]</td>\n      <td>[0.8353395018439429]</td>\n      <td>[0.8353395018439429]</td>\n      <td>[0.7984264439934582]</td>\n      <td>[0.781525553205324]</td>\n      <td>[0.8566096517720779]</td>\n      <td>[0.9071286071805743]</td>\n      <td>[0.8353395018439429]</td>\n      <td>[0.9021127651963996]</td>\n      <td>[0.8731669028038096]</td>\n      <td>[0.7936601928429454]</td>\n      <td>[0.8727204740202779]</td>\n      <td>[0.885603431576398]</td>\n      <td>[0.8360791405851395]</td>\n      <td>[0.9091539479147899]</td>\n    </tr>\n    <tr>\n      <th>recall_score_micro</th>\n      <td>[0.8907435508345979]</td>\n      <td>[0.790288315629742]</td>\n      <td>[0.6622154779969651]</td>\n      <td>[0.8227617602427921]</td>\n      <td>[0.7559939301972686]</td>\n      <td>[0.910773899848255]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.757814871016692]</td>\n      <td>[0.735660091047041]</td>\n      <td>[0.8955993930197269]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9071320182094081]</td>\n      <td>[0.8977238239757208]</td>\n      <td>[0.7517450682852808]</td>\n      <td>[0.8992412746585736]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.8883156297420334]</td>\n      <td>[0.9116843702579667]</td>\n    </tr>\n    <tr>\n      <th>recall_score_weighted</th>\n      <td>[0.8907435508345979]</td>\n      <td>[0.790288315629742]</td>\n      <td>[0.6622154779969651]</td>\n      <td>[0.8227617602427921]</td>\n      <td>[0.7559939301972686]</td>\n      <td>[0.910773899848255]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.757814871016692]</td>\n      <td>[0.735660091047041]</td>\n      <td>[0.8955993930197269]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9071320182094081]</td>\n      <td>[0.8977238239757208]</td>\n      <td>[0.7517450682852808]</td>\n      <td>[0.8992412746585736]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.8883156297420334]</td>\n      <td>[0.9116843702579667]</td>\n    </tr>\n    <tr>\n      <th>log_loss</th>\n      <td>[0.25137784415065584]</td>\n      <td>[0.5083664089828028]</td>\n      <td>[0.484269722676828]</td>\n      <td>[0.5464585031912554]</td>\n      <td>[0.5095386749702528]</td>\n      <td>[0.190468543909731]</td>\n      <td>[0.26133921308850605]</td>\n      <td>[0.26926069134726394]</td>\n      <td>[0.5933679946976629]</td>\n      <td>[0.561554712792468]</td>\n      <td>[0.24073153434773964]</td>\n      <td>[0.20140669839595468]</td>\n      <td>[0.25875146426964135]</td>\n      <td>[0.1874363495858499]</td>\n      <td>[0.2575613906785161]</td>\n      <td>[0.5615865304579278]</td>\n      <td>[0.2490233975691646]</td>\n      <td>[0.33655623030329523]</td>\n      <td>[0.27374024724306756]</td>\n      <td>[0.17775706110025447]</td>\n    </tr>\n    <tr>\n      <th>accuracy</th>\n      <td>[0.8907435508345979]</td>\n      <td>[0.790288315629742]</td>\n      <td>[0.6622154779969651]</td>\n      <td>[0.8227617602427921]</td>\n      <td>[0.7559939301972686]</td>\n      <td>[0.910773899848255]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.757814871016692]</td>\n      <td>[0.735660091047041]</td>\n      <td>[0.8955993930197269]</td>\n      <td>[0.9132018209408195]</td>\n      <td>[0.8880121396054628]</td>\n      <td>[0.9071320182094081]</td>\n      <td>[0.8977238239757208]</td>\n      <td>[0.7517450682852808]</td>\n      <td>[0.8992412746585736]</td>\n      <td>[0.9062215477996965]</td>\n      <td>[0.8883156297420334]</td>\n      <td>[0.9116843702579667]</td>\n    </tr>\n    <tr>\n      <th>average_precision_score_weighted</th>\n      <td>[0.9314791958449989]</td>\n      <td>[0.9326583727065825]</td>\n      <td>[0.9290347331948864]</td>\n      <td>[0.9134452637942311]</td>\n      <td>[0.9213485712046892]</td>\n      <td>[0.9534292590474763]</td>\n      <td>[0.9137884768949512]</td>\n      <td>[0.9286681474254169]</td>\n      <td>[0.926379634336469]</td>\n      <td>[0.9117912799208243]</td>\n      <td>[0.9323316000622094]</td>\n      <td>[0.9526718948733789]</td>\n      <td>[0.9243727763723245]</td>\n      <td>[0.9505970434373063]</td>\n      <td>[0.9230853367522769]</td>\n      <td>[0.9169114563320293]</td>\n      <td>[0.929779770688238]</td>\n      <td>[0.947605275820125]</td>\n      <td>[0.9236150003775165]</td>\n      <td>[0.9531771295804466]</td>\n    </tr>\n    <tr>\n      <th>precision_score_macro</th>\n      <td>[0.8030805480408413]</td>\n      <td>[0.6489259668850806]</td>\n      <td>[0.6209371599148775]</td>\n      <td>[0.6474578884263742]</td>\n      <td>[0.6200924830551504]</td>\n      <td>[0.7879164331527493]</td>\n      <td>[0.4440060698027314]</td>\n      <td>[0.4440060698027314]</td>\n      <td>[0.6193103448275863]</td>\n      <td>[0.6075878355410955]</td>\n      <td>[0.8508102597928253]</td>\n      <td>[0.7952472622648324]</td>\n      <td>[0.4440060698027314]</td>\n      <td>[0.7723958081530135]</td>\n      <td>[0.7725672788016185]</td>\n      <td>[0.6146077517256395]</td>\n      <td>[0.7946211026863488]</td>\n      <td>[0.822098675416211]</td>\n      <td>[0.9441408621736491]</td>\n      <td>[0.7819118765348991]</td>\n    </tr>\n    <tr>\n      <th>precision_score_weighted</th>\n      <td>[0.8719874949033176]</td>\n      <td>[0.8994204407174433]</td>\n      <td>[0.9117048164875989]</td>\n      <td>[0.8822677366372598]</td>\n      <td>[0.8839497346439098]</td>\n      <td>[0.9015318191709801]</td>\n      <td>[0.788565560086672]</td>\n      <td>[0.788565560086672]</td>\n      <td>[0.8826234106012245]</td>\n      <td>[0.878009600469834]</td>\n      <td>[0.8864547707362387]</td>\n      <td>[0.9046450883564172]</td>\n      <td>[0.788565560086672]</td>\n      <td>[0.8991976076061607]</td>\n      <td>[0.8770482732548277]</td>\n      <td>[0.8800582577385105]</td>\n      <td>[0.8811111704952653]</td>\n      <td>[0.8929725418691179]</td>\n      <td>[0.9007928150046113]</td>\n      <td>[0.9072720074188747]</td>\n    </tr>\n    <tr>\n      <th>balanced_accuracy</th>\n      <td>[0.5192999127530578]</td>\n      <td>[0.8025838802475516]</td>\n      <td>[0.8003355580377403]</td>\n      <td>[0.7462679240599651]</td>\n      <td>[0.7536709475092016]</td>\n      <td>[0.714118537289269]</td>\n      <td>[0.5]</td>\n      <td>[0.5]</td>\n      <td>[0.7499597108069509]</td>\n      <td>[0.7339329476685061]</td>\n      <td>[0.5433483931558386]</td>\n      <td>[0.7225903820897402]</td>\n      <td>[0.5]</td>\n      <td>[0.7191727470931578]</td>\n      <td>[0.5930939692172041]</td>\n      <td>[0.7418055486091429]</td>\n      <td>[0.5880277189648178]</td>\n      <td>[0.6227454260188535]</td>\n      <td>[0.5013550135501355]</td>\n      <td>[0.7513392683482543]</td>\n    </tr>\n    <tr>\n      <th>norm_macro_recall</th>\n      <td>[0.038599825506115515]</td>\n      <td>[0.6051677604951031]</td>\n      <td>[0.6006711160754805]</td>\n      <td>[0.49253584811993023]</td>\n      <td>[0.5073418950184032]</td>\n      <td>[0.428237074578538]</td>\n      <td>[0.0]</td>\n      <td>[0.0]</td>\n      <td>[0.4999194216139018]</td>\n      <td>[0.4678658953370123]</td>\n      <td>[0.08669678631167721]</td>\n      <td>[0.4451807641794805]</td>\n      <td>[0.0]</td>\n      <td>[0.43834549418631563]</td>\n      <td>[0.18618793843440828]</td>\n      <td>[0.4836110972182859]</td>\n      <td>[0.17605543792963552]</td>\n      <td>[0.24549085203770704]</td>\n      <td>[0.0027100271002709064]</td>\n      <td>[0.5026785366965085]</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 17,
      "metadata": {
        "gather": {
          "logged": 1612838186869
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Retrieve the Best Model"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Retrieve best model from Pipeline Run\n",
        "best_model_output = pipeline_run.get_pipeline_output(best_model_output_name)\n",
        "num_file_downloaded = best_model_output.download('.', show_progress=True)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading azureml/d6700776-5eed-40ba-b978-2a96d733f981/model_data\n",
            "Downloaded azureml/d6700776-5eed-40ba-b978-2a96d733f981/model_data, 1 files out of an estimated total of 1\n"
          ]
        }
      ],
      "execution_count": 18,
      "metadata": {
        "gather": {
          "logged": 1612838192486
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "\n",
        "with open(best_model_output._path_on_datastore, \"rb\" ) as f:\n",
        "    best_model = pickle.load(f)\n",
        "best_model"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 19,
          "data": {
            "text/plain": "PipelineWithYTransformations(Pipeline={'memory': None,\n                                       'steps': [('datatransformer',\n                                                  DataTransformer(enable_dnn=None,\n                                                                  enable_feature_sweeping=None,\n                                                                  feature_sweeping_config=None,\n                                                                  feature_sweeping_timeout=None,\n                                                                  featurization_config=None,\n                                                                  force_text_dnn=None,\n                                                                  is_cross_validation=None,\n                                                                  is_onnx_compatible=None,\n                                                                  logger=None,\n                                                                  observer=None,\n                                                                  task=None,\n                                                                  working_dir=None))...\n                                                                     importance_type='split',\n                                                                     learning_rate=0.1,\n                                                                     max_depth=-1,\n                                                                     min_child_samples=20,\n                                                                     min_child_weight=0.001,\n                                                                     min_split_gain=0.0,\n                                                                     n_estimators=100,\n                                                                     n_jobs=1,\n                                                                     num_leaves=31,\n                                                                     objective=None,\n                                                                     random_state=None,\n                                                                     reg_alpha=0.0,\n                                                                     reg_lambda=0.0,\n                                                                     silent=True,\n                                                                     subsample=1.0,\n                                                                     subsample_for_bin=200000,\n                                                                     subsample_freq=0,\n                                                                     verbose=-10))],\n                                       'verbose': False},\n                             y_transformer={},\n                             y_transformer_name='LabelEncoder')"
          },
          "metadata": {}
        }
      ],
      "execution_count": 19,
      "metadata": {
        "gather": {
          "logged": 1612838219391
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "best_model.steps"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 20,
          "data": {
            "text/plain": "[('datatransformer',\n  DataTransformer(enable_dnn=None, enable_feature_sweeping=None,\n                  feature_sweeping_config=None, feature_sweeping_timeout=None,\n                  featurization_config=None, force_text_dnn=None,\n                  is_cross_validation=None, is_onnx_compatible=None, logger=None,\n                  observer=None, task=None, working_dir=None)),\n ('MaxAbsScaler', MaxAbsScaler(copy=True)),\n ('LightGBMClassifier',\n  LightGBMClassifier(boosting_type='gbdt', class_weight=None,\n                     colsample_bytree=1.0, importance_type='split',\n                     learning_rate=0.1, max_depth=-1, min_child_samples=20,\n                     min_child_weight=0.001, min_split_gain=0.0, n_estimators=100,\n                     n_jobs=1, num_leaves=31, objective=None, random_state=None,\n                     reg_alpha=0.0, reg_lambda=0.0, silent=True, subsample=1.0,\n                     subsample_for_bin=200000, subsample_freq=0, verbose=-10))]"
          },
          "metadata": {}
        }
      ],
      "execution_count": 20,
      "metadata": {
        "gather": {
          "logged": 1612838248526
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Test the Model\n",
        "#### Load Test Data\n",
        "For the test data, it should have the same preparation step as the train data. Otherwise it might get failed at the preprocessing step."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dataset_test = Dataset.Tabular.from_delimited_files(path='https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv')\n",
        "df_test = dataset_test.to_pandas_dataframe()\n",
        "df_test = df_test[pd.notnull(df_test['y'])]\n",
        "\n",
        "y_test = df_test['y']\n",
        "X_test = df_test.drop(['y'], axis=1)"
      ],
      "outputs": [],
      "execution_count": 21,
      "metadata": {
        "gather": {
          "logged": 1612838259230
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Testing Our Best Fitted Model\n",
        "\n",
        "We will use confusion matrix to see how our model works."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "ypred = best_model.predict(X_test)\n",
        "cm = confusion_matrix(y_test, ypred)"
      ],
      "outputs": [],
      "execution_count": 22,
      "metadata": {
        "gather": {
          "logged": 1612838265454
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualize the confusion matrix\n",
        "pd.DataFrame(cm).style.background_gradient(cmap='Blues', low=0, high=0.9)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 23,
          "data": {
            "text/plain": "<pandas.io.formats.style.Styler at 0x7fda5241c2e8>",
            "text/html": "<style  type=\"text/css\" >\n    #T_a6949d96_6a7f_11eb_8d82_43b9ae25f486row0_col0 {\n            background-color:  #63a8d3;\n            color:  #000000;\n        }    #T_a6949d96_6a7f_11eb_8d82_43b9ae25f486row0_col1 {\n            background-color:  #f7fbff;\n            color:  #000000;\n        }    #T_a6949d96_6a7f_11eb_8d82_43b9ae25f486row1_col0 {\n            background-color:  #f7fbff;\n            color:  #000000;\n        }    #T_a6949d96_6a7f_11eb_8d82_43b9ae25f486row1_col1 {\n            background-color:  #63a8d3;\n            color:  #000000;\n        }</style><table id=\"T_a6949d96_6a7f_11eb_8d82_43b9ae25f486\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >0</th>        <th class=\"col_heading level0 col1\" >1</th>    </tr></thead><tbody>\n                <tr>\n                        <th id=\"T_a6949d96_6a7f_11eb_8d82_43b9ae25f486level0_row0\" class=\"row_heading level0 row0\" >0</th>\n                        <td id=\"T_a6949d96_6a7f_11eb_8d82_43b9ae25f486row0_col0\" class=\"data row0 col0\" >28431</td>\n                        <td id=\"T_a6949d96_6a7f_11eb_8d82_43b9ae25f486row0_col1\" class=\"data row0 col1\" >827</td>\n            </tr>\n            <tr>\n                        <th id=\"T_a6949d96_6a7f_11eb_8d82_43b9ae25f486level0_row1\" class=\"row_heading level0 row1\" >1</th>\n                        <td id=\"T_a6949d96_6a7f_11eb_8d82_43b9ae25f486row1_col0\" class=\"data row1 col0\" >1246</td>\n                        <td id=\"T_a6949d96_6a7f_11eb_8d82_43b9ae25f486row1_col1\" class=\"data row1 col1\" >2446</td>\n            </tr>\n    </tbody></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 23,
      "metadata": {
        "gather": {
          "logged": 1612838267346
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Publish and run from REST endpoint\n",
        "\n",
        "Run the following code to publish the pipeline to your workspace. In your workspace in the portal, you can see metadata for the pipeline including run history and durations. You can also run the pipeline manually from the portal.\n",
        "\n",
        "Additionally, publishing the pipeline enables a REST endpoint to rerun the pipeline from any HTTP library on any platform.\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "published_pipeline = pipeline_run.publish_pipeline(\n",
        "    name=\"Bankmarketing Train\", description=\"Training bankmarketing pipeline\", version=\"1.0\")\n",
        "\n",
        "published_pipeline\n"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 24,
          "data": {
            "text/plain": "Pipeline(Name: Bankmarketing Train,\nId: b73e7285-2a4c-429c-85ed-d1a1ca0047bb,\nStatus: Active,\nEndpoint: https://uksouth.api.azureml.ms/pipelines/v1.0/subscriptions/5cf0bb76-4731-4cb5-ac3d-950c07dda6ef/resourceGroups/udacity1st/providers/Microsoft.MachineLearningServices/workspaces/ml-experiment-1/PipelineRuns/PipelineSubmit/b73e7285-2a4c-429c-85ed-d1a1ca0047bb)",
            "text/html": "<table style=\"width:100%\"><tr><th>Name</th><th>Id</th><th>Status</th><th>Endpoint</th></tr><tr><td>Bankmarketing Train</td><td><a href=\"https://ml.azure.com/pipelines/b73e7285-2a4c-429c-85ed-d1a1ca0047bb?wsid=/subscriptions/5cf0bb76-4731-4cb5-ac3d-950c07dda6ef/resourcegroups/udacity1st/workspaces/ml-experiment-1\" target=\"_blank\" rel=\"noopener\">b73e7285-2a4c-429c-85ed-d1a1ca0047bb</a></td><td>Active</td><td><a href=\"https://uksouth.api.azureml.ms/pipelines/v1.0/subscriptions/5cf0bb76-4731-4cb5-ac3d-950c07dda6ef/resourceGroups/udacity1st/providers/Microsoft.MachineLearningServices/workspaces/ml-experiment-1/PipelineRuns/PipelineSubmit/b73e7285-2a4c-429c-85ed-d1a1ca0047bb\" target=\"_blank\" rel=\"noopener\">REST Endpoint</a></td></tr></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 24,
      "metadata": {
        "gather": {
          "logged": 1612838272128
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Authenticate once again, to retrieve the `auth_header` so that the endpoint can be used"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
        "\n",
        "interactive_auth = InteractiveLoginAuthentication()\n",
        "auth_header = interactive_auth.get_authentication_header()\n",
        "\n"
      ],
      "outputs": [],
      "execution_count": 25,
      "metadata": {
        "gather": {
          "logged": 1612838274701
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Get the REST url from the endpoint property of the published pipeline object. You can also find the REST url in your workspace in the portal. Build an HTTP POST request to the endpoint, specifying your authentication header. Additionally, add a JSON payload object with the experiment name and the batch size parameter. As a reminder, the process_count_per_node is passed through to ParallelRunStep because you defined it is defined as a PipelineParameter object in the step configuration.\n",
        "\n",
        "Make the request to trigger the run. Access the Id key from the response dict to get the value of the run id.\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "rest_endpoint = published_pipeline.endpoint\n",
        "response = requests.post(rest_endpoint, \n",
        "                         headers=auth_header, \n",
        "                         json={\"ExperimentName\": \"pipeline-rest-endpoint\"}\n",
        "                        )"
      ],
      "outputs": [],
      "execution_count": 26,
      "metadata": {
        "gather": {
          "logged": 1612838278424
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "    response.raise_for_status()\n",
        "except Exception:    \n",
        "    raise Exception(\"Received bad response from the endpoint: {}\\n\"\n",
        "                    \"Response Code: {}\\n\"\n",
        "                    \"Headers: {}\\n\"\n",
        "                    \"Content: {}\".format(rest_endpoint, response.status_code, response.headers, response.content))\n",
        "\n",
        "run_id = response.json().get('Id')\n",
        "print('Submitted pipeline run: ', run_id)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Submitted pipeline run:  54733bda-7597-44cd-8370-e9d36ea4d566\n"
          ]
        }
      ],
      "execution_count": 27,
      "metadata": {
        "gather": {
          "logged": 1612838280844
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Use the run id to monitor the status of the new run. This will take another 10-15 min to run and will look similar to the previous pipeline run, so if you don't need to see another pipeline run, you can skip watching the full output."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core.run import PipelineRun\n",
        "from azureml.widgets import RunDetails\n",
        "\n",
        "published_pipeline_run = PipelineRun(ws.experiments[\"pipeline-rest-endpoint\"], run_id)\n",
        "RunDetails(published_pipeline_run).show()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …",
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "117e07bcf6664978a9e7632564de82f6"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/pipeline-rest-endpoint/runs/54733bda-7597-44cd-8370-e9d36ea4d566?wsid=/subscriptions/5cf0bb76-4731-4cb5-ac3d-950c07dda6ef/resourcegroups/udacity1st/workspaces/ml-experiment-1\", \"run_id\": \"54733bda-7597-44cd-8370-e9d36ea4d566\", \"run_properties\": {\"run_id\": \"54733bda-7597-44cd-8370-e9d36ea4d566\", \"created_utc\": \"2021-02-09T02:36:56.035842Z\", \"properties\": {\"azureml.runsource\": \"azureml.PipelineRun\", \"runSource\": \"Unavailable\", \"runType\": \"HTTP\", \"azureml.parameters\": \"{}\", \"azureml.pipelineid\": \"b73e7285-2a4c-429c-85ed-d1a1ca0047bb\"}, \"tags\": {\"azureml.pipelineid\": \"b73e7285-2a4c-429c-85ed-d1a1ca0047bb\", \"azureml.pipelineComponent\": \"pipelinerun\"}, \"end_time_utc\": \"2021-02-09T03:15:50.287267Z\", \"status\": \"Completed\", \"log_files\": {\"logs/azureml/executionlogs.txt\": \"https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.54733bda-7597-44cd-8370-e9d36ea4d566/logs/azureml/executionlogs.txt?sv=2019-02-02&sr=b&sig=q%2Bka2a%2BsX%2Fw3iOf8GvvBlu%2FoQsvo%2BdCBysiFT%2FqZWHc%3D&st=2021-02-09T05%3A28%3A02Z&se=2021-02-09T13%3A38%3A02Z&sp=r\", \"logs/azureml/stderrlogs.txt\": \"https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.54733bda-7597-44cd-8370-e9d36ea4d566/logs/azureml/stderrlogs.txt?sv=2019-02-02&sr=b&sig=UQdkQn6nn7VSnlhuybbIJi5Y4N6CDmMBzBt%2BGIuWEAQ%3D&st=2021-02-09T05%3A28%3A02Z&se=2021-02-09T13%3A38%3A02Z&sp=r\", \"logs/azureml/stdoutlogs.txt\": \"https://mlexperiment17223454014.blob.core.windows.net/azureml/ExperimentRun/dcid.54733bda-7597-44cd-8370-e9d36ea4d566/logs/azureml/stdoutlogs.txt?sv=2019-02-02&sr=b&sig=31XTeDhVLB16XDKK5WdhFtxCKacfVmQ0ABZyttrCcDA%3D&st=2021-02-09T05%3A28%3A02Z&se=2021-02-09T13%3A38%3A02Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/executionlogs.txt\", \"logs/azureml/stderrlogs.txt\", \"logs/azureml/stdoutlogs.txt\"]], \"run_duration\": \"0:38:54\"}, \"child_runs\": [{\"run_id\": \"6531af58-3319-45ae-bfdf-94cc9f4784f5\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2021-02-09T02:37:15.650879Z\", \"created_time\": \"2021-02-09T02:37:00.489862Z\", \"end_time\": \"2021-02-09T03:15:32.75773Z\", \"duration\": \"0:38:32\", \"run_number\": 2, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2021-02-09T02:37:00.489862Z\", \"is_reused\": \"\"}], \"children_metrics\": {\"categories\": null, \"series\": null, \"metricName\": null}, \"run_metrics\": [], \"run_logs\": \"[2021-02-09 02:37:00Z] Submitting 1 runs, first five are: 0e7f3c20:6531af58-3319-45ae-bfdf-94cc9f4784f5\\n[2021-02-09 03:15:50Z] Completing processing run id 6531af58-3319-45ae-bfdf-94cc9f4784f5.\\n\\nRun is completed.\", \"graph\": {\"datasource_nodes\": {\"c6a22348\": {\"node_id\": \"c6a22348\", \"name\": \"BankMarketing Dataset\"}}, \"module_nodes\": {\"0e7f3c20\": {\"node_id\": \"0e7f3c20\", \"name\": \"automl_module\", \"status\": \"Finished\", \"_is_reused\": false, \"run_id\": \"6531af58-3319-45ae-bfdf-94cc9f4784f5\"}}, \"edges\": [{\"source_node_id\": \"c6a22348\", \"source_node_name\": \"BankMarketing Dataset\", \"source_name\": \"data\", \"target_name\": \"training_data\", \"dst_node_id\": \"0e7f3c20\", \"dst_node_name\": \"automl_module\"}], \"child_runs\": [{\"run_id\": \"6531af58-3319-45ae-bfdf-94cc9f4784f5\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2021-02-09T02:37:15.650879Z\", \"created_time\": \"2021-02-09T02:37:00.489862Z\", \"end_time\": \"2021-02-09T03:15:32.75773Z\", \"duration\": \"0:38:32\", \"run_number\": 2, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2021-02-09T02:37:00.489862Z\", \"is_reused\": \"\"}]}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.20.0\"}, \"loading\": false}"
          },
          "metadata": {}
        }
      ],
      "execution_count": 28,
      "metadata": {
        "gather": {
          "logged": 1612838284518
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core import PublishedPipeline\r\n",
        "all_pub_pipelines = PublishedPipeline.list(ws)\r\n",
        "print(\"Published pipelines found in the workspace:\")\r\n",
        "for pub_pipeline in all_pub_pipelines:\r\n",
        "    print(pub_pipeline.id)\r\n",
        "    pub_pipeline_id = pub_pipeline.id\r\n",
        "print(\"Published pipeline id to be used for Schedule operations: {}\".format(pub_pipeline_id))"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Published pipelines found in the workspace:\n",
            "b73e7285-2a4c-429c-85ed-d1a1ca0047bb\n",
            "Published pipeline id to be used for Schedule operations: b73e7285-2a4c-429c-85ed-d1a1ca0047bb\n"
          ]
        }
      ],
      "execution_count": 29,
      "metadata": {
        "collapsed": true,
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1612844092817
        }
      }
    }
  ],
  "metadata": {
    "order_index": 11,
    "exclude_from_index": false,
    "task": "Demonstrates the use of AutoMLStep",
    "deployment": [
      "None"
    ],
    "authors": [
      {
        "name": "sanpil"
      }
    ],
    "star_tag": [
      "featured"
    ],
    "kernel_info": {
      "name": "python3-azureml"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.9",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "compute": [
      "AML Compute"
    ],
    "kernelspec": {
      "name": "python3-azureml",
      "language": "python",
      "display_name": "Python 3.6 - AzureML"
    },
    "tags": [
      "None"
    ],
    "datasets": [
      "Custom"
    ],
    "categories": [
      "how-to-use-azureml",
      "machine-learning-pipelines",
      "intro-to-pipelines"
    ],
    "category": "tutorial",
    "framework": [
      "Automated Machine Learning"
    ],
    "friendly_name": "How to use AutoMLStep with AML Pipelines",
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}