Skip to content

Latest commit

 

History

History
 
 

sdk

page_type languages products description
sample
python
azure-machine-learning
Top-level directory for official Azure Machine Learning Python SDK v2 sample code.

Azure Machine Learning SDK (v2) (preview) examples

code style: black license: MIT

Public Preview

We are excited to introduce the public preview of Azure Machine Learning Python SDK v2. Azure ML Python SDK v2 is an updated Python SDK package, which allows users to:

  • Submit training jobs
  • Manage data, models, environments
  • Perform managed inferencing (real time and batch)
  • Stitch together multiple tasks and production workflows using Azure ML pipelines
  • Run standalone AutoML training for various ml-tasks

Please note that this Private Preview release is subject to the Supplemental Terms of Use for Microsoft Azure Previews.

Why should I use the new SDK v2?

The SDK v2 comes with many improvements and features:

  • Use Python functions to build a single step or a complex workflow

    SDK v2 allows you to build a single command or a chain of commands like python functions - the command has a name, parameters, expects input, and returns output.

  • Move from simple to complex concepts incrementally

    SDK v2 allows you to:

    • Construct a single command.
    • Add a hyperparameter sweep on top of that command,
    • Add the command with various others into a pipeline one after the other. This construction is useful, given the iterative nature of machine learning.
  • Reusable components in pipelines

    Azure ML introduces components for managing and reusing common logic across pipelines. This functionality is available only via CLI v2 and SDK v2.

  • Managed inferencing

    Azure ML offers endpoints to streamline model deployments for both real-time and batch inference deployments. This functionality is available only via CLI v2 and SDK v2.

How do I use this feature?

Python SDK v2 can be used in various ways – in python scripts, Jupyter Notebooks to create, submit / manage jobs, pipelines, and your resources/assets. You can use the Azure ML notebooks, VS Code or other editors of your choice to manage to your code. Checkout an over view of our samples.

How can I provide feedback?

If you are facing any issues while using the new feature, please reach out to Azure ML SDK feedback. For general feedback, please submit an GitHub issue.

Prerequisites

  1. An Azure subscription. If you don't have an Azure subscription, create a free account before you begin.

Getting started

  1. Install the SDK v2
pip uninstall azure-ai-ml

pip install --pre azure-ai-ml

Clone examples repository

git clone https://github.com/Azure/azureml-examples
cd azureml-examples/sdk

Examples available

Test Status is for branch - main

Area Sub-Area Notebook Description Status
assets component component Create a component asset component
assets data data Read, write and register a data asset data
assets environment environment Create custom environments from docker and/or conda YAML environment
assets model model Create model from local files, cloud files, Runs model
endpoints batch mnist-nonmlflow Create and test batch endpoint and deployement mnist-nonmlflow
endpoints online online-endpoints-custom-container Deploy a custom container as an online endpoint. Use web servers other than the default Python Flask server used by Azure ML without losing the benefits of Azure ML's built-in monitoring, scaling, alerting, and authentication. online-endpoints-custom-container
endpoints online kubernetes-online-endpoints-safe-rollout Safely rollout a new version of a web service to production by rolling out the change to a small subset of users/requests before rolling it out completely kubernetes-online-endpoints-safe-rollout
endpoints online kubernetes-online-endpoints-simple-deployment Use an online endpoint to deploy your model, so you don't have to create and manage the underlying infrastructure kubernetes-online-endpoints-simple-deployment
endpoints online online-endpoints-safe-rollout Safely rollout a new version of a web service to production by rolling out the change to a small subset of users/requests before rolling it out completely online-endpoints-safe-rollout
endpoints online online-endpoints-simple-deployment Use an online endpoint to deploy your model, so you don't have to create and manage the underlying infrastructure online-endpoints-simple-deployment
endpoints online online-endpoints-deploy-mlflow-model Deploy an mlflow model to an online endpoint. This will be a no-code-deployment. It doesn't require scoring script and environment. online-endpoints-deploy-mlflow-model
jobs automl-standalone-jobs automl-classification-task-bankmarketing-mlflow no description - This sample is excluded from automated tests automl-classification-task-bankmarketing-mlflow
jobs automl-standalone-jobs automl-classification-task-bankmarketing no description automl-classification-task-bankmarketing
jobs automl-standalone-jobs mlflow-model-local-inference-test no description - This sample is excluded from automated tests mlflow-model-local-inference-test
jobs automl-standalone-jobs automl-forecasting-task-energy-demand-advanced-mlflow no description - This sample is excluded from automated tests automl-forecasting-task-energy-demand-advanced-mlflow
jobs automl-standalone-jobs automl-forecasting-task-energy-demand-advanced no description automl-forecasting-task-energy-demand-advanced
jobs automl-standalone-jobs mlflow-model-local-inference-test no description - This sample is excluded from automated tests mlflow-model-local-inference-test
jobs automl-standalone-jobs automl-image-classification-multiclass-task-fridge-items no description automl-image-classification-multiclass-task-fridge-items
jobs automl-standalone-jobs automl-image-classification-multilabel-task-fridge-items no description automl-image-classification-multilabel-task-fridge-items
jobs automl-standalone-jobs automl-image-instance-segmentation-task-fridge-items no description automl-image-instance-segmentation-task-fridge-items
jobs automl-standalone-jobs automl-image-object-detection-task-fridge-items-mlflow no description automl-image-object-detection-task-fridge-items-mlflow
jobs automl-standalone-jobs automl-image-object-detection-task-fridge-items no description automl-image-object-detection-task-fridge-items
jobs automl-standalone-jobs mlflow-model-local-inference-test no description - This sample is excluded from automated tests mlflow-model-local-inference-test
jobs automl-standalone-jobs automl-nlp-text-classification-multiclass-task-sentiment-mlflow no description - This sample is excluded from automated tests automl-nlp-text-classification-multiclass-task-sentiment-mlflow
jobs automl-standalone-jobs automl-nlp-text-classification-multiclass-task-sentiment no description automl-nlp-text-classification-multiclass-task-sentiment
jobs automl-standalone-jobs mlflow-model-local-inference-test no description - This sample is excluded from automated tests mlflow-model-local-inference-test
jobs automl-standalone-jobs automl-nlp-text-classification-multilabel-task-paper-cat no description automl-nlp-text-classification-multilabel-task-paper-cat
jobs automl-standalone-jobs automl-nlp-text-ner-task no description automl-nlp-text-ner-task
jobs automl-standalone-jobs automl-regression-task-hardware-performance no description automl-regression-task-hardware-performance
jobs configuration.ipynb configuration Setting up your Azure Machine Learning services workspace and configuring needed resources configuration
jobs pipelines pipeline_with_components_from_yaml Create pipeline with CommandComponents from local YAML file pipeline_with_components_from_yaml
jobs pipelines pipeline_with_python_function_components Create pipeline with command_component decorator pipeline_with_python_function_components
jobs pipelines pipeline_with_hyperparameter_sweep Use sweep (hyperdrive) in pipeline to train mnist model using tensorflow pipeline_with_hyperparameter_sweep
jobs pipelines pipeline_with_non_python_components Create a pipeline with command function pipeline_with_non_python_components
jobs pipelines pipeline_with_registered_components Register component and then use these components to build pipeline pipeline_with_registered_components
jobs pipelines train_mnist_with_tensorflow Create pipeline using components to run a distributed job with tensorflow train_mnist_with_tensorflow
jobs pipelines train_cifar_10_with_pytorch Get data, train and evaluate a model in pipeline with Components train_cifar_10_with_pytorch
jobs pipelines nyc_taxi_data_regression Build pipeline with components for 5 jobs - prep data, transform data, train model, predict results and evaluate model performance nyc_taxi_data_regression
jobs pipelines image_classification_with_densenet Create pipeline to train cnn image classification model image_classification_with_densenet
jobs pipelines image_classification_keras_minist_convnet Create pipeline to train cnn image classification model with keras image_classification_keras_minist_convnet
jobs pipelines rai_pipeline_sample Create sample RAI pipeline rai_pipeline_sample
jobs single-step lightgbm-iris-sweep Run hyperparameter sweep on a Command or CommandComponent lightgbm-iris-sweep
jobs single-step pytorch-iris Run Command to train a neural network with PyTorch on Iris dataset pytorch-iris
jobs single-step accident-prediction Run R in a Command to train a prediction model accident-prediction
jobs single-step sklearn-diabetes Run Command to train a scikit-learn LinearRegression model on the Diabetes dataset sklearn-diabetes
jobs single-step iris-scikit-learn Run Command to train a scikit-learn SVM on the Iris dataset iris-scikit-learn
jobs single-step sklearn-mnist Run a Command to train a scikit-learn SVM on the mnist dataset. sklearn-mnist
jobs single-step tensorflow-mnist-distributed-horovod Run a Distributed Command to train a basic neural network with distributed MPI on the MNIST dataset using Horovod tensorflow-mnist-distributed-horovod
jobs single-step tensorflow-mnist-distributed Run a Distributed Command to train a basic neural network with TensorFlow on the MNIST dataset tensorflow-mnist-distributed
jobs single-step tensorflow-mnist Run a Command to train a basic neural network with TensorFlow on the MNIST dataset tensorflow-mnist
resources compute compute Create compute in Azure ML workspace - This sample is only tested on demand compute
resources datastores datastore Create datastores and use in a Command - This sample is excluded from automated tests datastore
resources workspace workspace Create Azure ML workspace - This sample is only tested on demand workspace

What are the list of features?

Azure Machine Learning Python SDK v2 comes with many new features like standalone local jobs, reusable components for pipelines and managed online/batch inferencing. The SDK v2 brings consistency and ease of use across all assets of the platform. The Python SDK v2 offers the following capabilities:

  • Run Standalone Jobs - run a discrete ML activity as Job. This job can be run locally or on the cloud. We currently support the following types of jobs:
    • Command - run a command (Python, R, Windows Command, Linux Shell etc.)
    • Sweep - run a hyperparameter sweep on your Command
  • Run multiple jobs using our improved Pipelines
    • Run a series of commands stitched into a pipeline (New)
    • Components - run pipelines using reusable components (New)
  • Use your models for Managed Online inferencing (New)
  • Use your models for Managed batch inferencing
  • Manage AML resources – workspace, compute, datastores
  • Manage AML assets - Datasets, environments, models
  • AutoML - run standalone AutoML training for various ml-tasks:
    • Classification (Tabular data)
    • Regression (Tabular data)
    • Time Series Forecasting (Tabular data)
    • Image Classification (Multi*class) (New)
    • Image Classification (Multi*label) (New)
    • Image Object Detection (New)
    • Image Instance Segmentation (New)
    • NLP Text Classification (Multi*class) (New)
    • NLP Text Classification (Multi*label) (New)
    • NLP Text Named Entity Recognition (NER) (New)

Contributing

We welcome contributions and suggestions! Please see the contributing guidelines for details.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. Please see the code of conduct for details.

Reference