automl-configuration.ipynb

{
  "cells": [
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Copyright (c) Microsoft Corporation. All rights reserved.\n\nLicensed under the MIT License."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "# Configuration\n\n_**Setting up your Azure Machine Learning services workspace and configuring your notebook library**_\n\n---\n---\n\n## Table of Contents\n\n1. [Introduction](#Introduction)\n    1. What is an Azure Machine Learning workspace\n1. [Setup](#Setup)\n    1. Azure subscription\n    1. Azure ML SDK and other library installation\n    1. Azure Container Instance registration\n1. [Configure your Azure ML Workspace](#Configure%20your%20Azure%20ML%20workspace)\n    1. Workspace parameters\n    1. Access your workspace\n    1. Create a new workspace\n    1. Create compute resources\n1. [Next steps](#Next%20steps)\n\n---\n\n## Introduction\n\nThis notebook configures your library of notebooks to connect to an Azure Machine Learning (ML) workspace.  In this case, a library contains all of the notebooks in the current folder and any nested folders.  You can configure this notebook library to use an existing workspace or create a new workspace.\n\nTypically you will need to run this notebook only once per notebook library as all other notebooks will use connection information that is written here.  If you want to redirect your notebook library to work with a different workspace, then you should re-run this notebook.\n\nIn this notebook you will\n* Learn about getting an Azure subscription\n* Specify your workspace parameters\n* Access or create your workspace\n* Add a default compute cluster for your workspace\n\n### What is an Azure Machine Learning workspace\n\nAn Azure ML Workspace is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows.  In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inferencing, and the monitoring of deployed models."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Setup\n\nThis section describes activities required before you can access any Azure ML services functionality."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### 1. Azure Subscription\n\nIn order to create an Azure ML Workspace, first you need access to an Azure subscription.  An Azure subscription allows you to manage storage, compute, and other assets in the Azure cloud.  You can [create a new subscription](https://azure.microsoft.com/en-us/free/) or access existing subscription information from the [Azure portal](https://portal.azure.com).  Later in this notebook you will need information such as your subscription ID in order to create and access AML workspaces.\n\n### 2. Azure ML SDK and other library installation\n\nIf you are running in your own environment, follow [SDK installation instructions](https://docs.microsoft.com/azure/machine-learning/service/how-to-configure-environment).  If you are running in Azure Notebooks or another Microsoft managed environment, the SDK is already installed.\n\nAlso install following libraries to your environment. Many of the example notebooks depend on them\n\n```\n(myenv) $ conda install -y matplotlib tqdm scikit-learn\n```\n\nOnce installation is complete, the following cell checks the Azure ML SDK version:"
    },
    {
      "metadata": {
        "tags": [
          "install"
        ],
        "trusted": true
      },
      "cell_type": "code",
      "source": "import azureml.core\n\nprint(\"This notebook was created using version 1.0.2 of the Azure ML SDK\")\nprint(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")",
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": "This notebook was created using version 1.0.2 of the Azure ML SDK\nYou are currently using version 1.0.2 of the Azure ML SDK\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "If you are using an older version of the SDK then this notebook was created using, you should upgrade your SDK.\n\n### 3. Azure Container Instance registration\nAzure Machine Learning uses of [Azure Container Instance (ACI)](https://azure.microsoft.com/services/container-instances) to deploy dev/test web services. An Azure subscription needs to be registered to use ACI.  If you or the subscription owner have not yet registered ACI on your subscription, you will need to use the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and execute the following commands.  Note that if you ran through the AML [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) you have already registered ACI. \n\n```shell\n# check to see if ACI is already registered\n(myenv) $ az provider show -n Microsoft.ContainerInstance -o table\n\n# if ACI is not registered, run this command.\n# note you need to be the subscription owner in order to execute this command successfully.\n(myenv) $ az provider register -n Microsoft.ContainerInstance\n```\n\n---"
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "## Configure your Azure ML workspace\n\n### Workspace parameters\n\nTo use an AML Workspace, you will need to import the Azure ML SDK and supply the following information:\n* Your subscription id\n* A resource group name\n* (optional) The region that will host your workspace\n* A name for your workspace\n\nYou can get your subscription ID from the [Azure portal](https://portal.azure.com).\n\nYou will also need access to a [_resource group_](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups), which organizes Azure resources and provides a default region for the resources in a group.  You can see what resource groups to which you have access, or create a new one in the [Azure portal](https://portal.azure.com).  If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n\nThe region to host your workspace will be used if you are creating a new workspace.  You do not need to specify this if you are using an existing workspace. You can find the list of supported regions [here](https://azure.microsoft.com/en-us/global-infrastructure/services/?products=machine-learning-service).  You should pick a region that is close to your location or that contains your data.\n\nThe name for your workspace is unique within the subscription and should be descriptive enough to discern among other AML Workspaces.  The subscription may be used only by you, or it may be used by your department or your entire enterprise, so choose a name that makes sense for your situation.\n\nThe following cell allows you to specify your workspace parameters.  This cell uses the python method `os.getenv` to read values from environment variables which is useful for automation.  If no environment variable exists, the parameters will be set to the specified default values.  \n\nIf you ran the Azure Machine Learning [quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-get-started) in Azure Notebooks, you already have a configured workspace!  You can go to your Azure Machine Learning Getting Started library, view *config.json* file, and copy-paste the values for subscription ID, resource group and workspace name below.\n\nReplace the default values in the cell below with your workspace parameters"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "import os\n\nsubscription_id = os.getenv(\"SUBSCRIPTION_ID\", default=\"faa94835-ff02-41d2-a14a-bc0970e5b440\")\nresource_group = os.getenv(\"RESOURCE_GROUP\", default=\"KC-leadershiptraining-rg\")\nworkspace_name = os.getenv(\"WORKSPACE_NAME\", default=\"LeadershipTrainingMLWS\")\nworkspace_region = os.getenv(\"WORKSPACE_REGION\", default=\"southcentralus\")\ntenant_id = os.getenv(\"TENANT_ID\", default=\"fee2180b-69b6-4afe-9f14-ccd70bd4c737\")",
      "execution_count": 2,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Access your workspace\n\nThe following cell uses the Azure ML SDK to attempt to load the workspace specified by your parameters.  If this cell succeeds, your notebook library will be configured to access the workspace from all notebooks using the `Workspace.from_config()` method.  The cell can fail if the specified workspace doesn't exist or you don't have permissions to access it. "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.core import Workspace\nfrom azureml.core.authentication import InteractiveLoginAuthentication\n\ntry:\n    my_auth = InteractiveLoginAuthentication(tenant_id=tenant_id)\n    ws = Workspace(workspace_name=workspace_name, auth=my_auth, subscription_id=subscription_id, resource_group=resource_group)\n    # write the details of the workspace to a configuration file to the notebook library\n    ws.write_config()\n    print(\"Workspace configuration succeeded. Skip the workspace creation steps below\")\nexcept:\n    print(\"Workspace not accessible. Change your parameters or create a new workspace below\")",
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Performing interactive authentication. Please follow the instructions on the terminal.\n",
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": "To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code FKA8VJA3T to authenticate.\n",
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": "Interactive authentication successfully completed.\nWrote the config file config.json to: /home/nbuser/library/aml_config/config.json\nWorkspace configuration succeeded. Skip the workspace creation steps below\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Create a new workspace\n\nIf you don't have an existing workspace and are the owner of the subscription or resource group, you can create a new workspace.  If you don't have a resource group, the create workspace command will create one for you using the name you provide.\n\n**Note**: As with other Azure services, there are limits on certain resources (for example AmlCompute quota) associated with the Azure ML service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota.\n\nThis cell will create an Azure ML workspace for you in a subscription provided you have the correct permissions.\n\nThis will fail if:\n* You do not have permission to create a workspace in the resource group\n* You do not have permission to create a resource group if it's non-existing.\n* You are not a subscription owner or contributor and no Azure ML workspaces have ever been created in this subscription\n\nIf workspace creation fails, please work with your IT admin to provide you with the appropriate permissions or to provision the required resources."
    },
    {
      "metadata": {
        "tags": [
          "create workspace"
        ],
        "trusted": false
      },
      "cell_type": "code",
      "source": "from azureml.core import Workspace\n\n# Create the workspace using the specified parameters\nws = Workspace.create(name = workspace_name,\n                      subscription_id = subscription_id,\n                      resource_group = resource_group, \n                      location = workspace_region,\n                      auth = my_auth;\n                      create_resource_group = True,\n                      exist_ok = True)\nws.get_details()\n\n# write the details of the workspace to a configuration file to the notebook library\nws.write_config()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### Create compute resources for your training experiments\n\nMany of the sample notebooks use Azure ML managed compute (AmlCompute) to train models using a dynamically scalable pool of compute. In this section you will create default compute clusters for use by the other notebooks and any other operations you choose.\n\nTo create a cluster, you need to specify a compute configuration that specifies the type of machine to be used and the scalability behaviors.  Then you choose a name for the cluster that is unique within the workspace that can be used to address the cluster later.\n\nThe cluster parameters are:\n* vm_size - this describes the virtual machine type and size used in the cluster.  All machines in the cluster are the same type.  You can get the list of vm sizes available in your region by using the CLI command\n\n```shell\naz vm list-skus -o tsv\n```\n* min_nodes - this sets the minimum size of the cluster.  If you set the minimum to 0 the cluster will shut down all nodes while note in use.  Setting this number to a value higher than 0 will allow for faster start-up times, but you will also be billed when the cluster is not in use.\n* max_nodes - this sets the maximum size of the cluster.  Setting this to a larger number allows for more concurrency and a greater distributed processing of scale-out jobs.\n\n\nTo create a **CPU** cluster now, run the cell below. The autoscale settings mean that the cluster will scale down to 1 nodes when inactive and up to 6 nodes when busy."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.core.compute import ComputeTarget, AmlCompute\nfrom azureml.core.compute_target import ComputeTargetException\n\n# Choose a name for your CPU cluster\ncpu_cluster_name = \"cpucluster\"\n\n# Verify that cluster does not exist already\ntry:\n    cpu_cluster = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n    print(\"Found existing cpucluster\")\nexcept ComputeTargetException:\n    print(\"Creating new cpucluster\")\n    \n    # Specify the configuration for the new cluster\n    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_D2_V2\",\n                                                           min_nodes=1,\n                                                           max_nodes=6)\n\n    # Create the cluster with the specified name and configuration\n    cpu_cluster = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n    \n    # Wait for the cluster to complete, show the output log\n    cpu_cluster.wait_for_completion(show_output=True)",
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Creating new cpucluster\nCreating\nSucceeded....................\nAmlCompute wait for completion finished\nMinimum number of nodes requested have been provisioned\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "To create a **GPU** cluster, run the cell below. Note that your subscription must have sufficient quota for GPU VMs or the command will fail. To increase quota, see [these instructions](https://docs.microsoft.com/en-us/azure/azure-supportability/resource-manager-core-quotas-request). "
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "from azureml.core.compute import ComputeTarget, AmlCompute\nfrom azureml.core.compute_target import ComputeTargetException\n\n# Choose a name for your GPU cluster\ngpu_cluster_name = \"gpucluster\"\n\n# Verify that cluster does not exist already\ntry:\n    gpu_cluster = ComputeTarget(workspace=ws, name=gpu_cluster_name)\n    print(\"Found existing gpu cluster\")\nexcept ComputeTargetException:\n    print(\"Creating new gpucluster\")\n    \n    # Specify the configuration for the new cluster\n    compute_config = AmlCompute.provisioning_configuration(vm_size=\"STANDARD_NC6\",\n                                                           min_nodes=1,\n                                                           max_nodes=4)\n    # Create the cluster with the specified name and configuration\n    gpu_cluster = ComputeTarget.create(ws, gpu_cluster_name, compute_config)\n\n    # Wait for the cluster to complete, show the output log\n    gpu_cluster.wait_for_completion(show_output=True)",
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": "Creating new gpucluster\nCreating\nSucceeded...........\nAmlCompute wait for completion finished\nMinimum number of nodes requested have been provisioned\n",
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "---\n\n## Next steps\n\nIn this notebook you configured this notebook library to connect easily to an Azure ML workspace.  You can copy this notebook to your own libraries to connect them to you workspace, or use it to bootstrap new workspaces completely.\n\nIf you came here from another notebook, you can return there and complete that exercise, or you can try out the [Tutorials](./tutorials) or jump into \"how-to\" notebooks and start creating and deploying models.  A good place to start is the [train in notebook](./how-to-use-azureml/training/train-in-notebook) example that walks through a simplified but complete end to end machine learning process."
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "authors": [
      {
        "name": "roastala"
      }
    ],
    "kernelspec": {
      "name": "python36",
      "display_name": "Python 3.6",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}