diff --git a/.env.example b/.env.example old mode 100644 new mode 100755 diff --git a/.gitignore b/.gitignore old mode 100644 new mode 100755 diff --git a/.pipelines/abtest.yml b/.pipelines/abtest.yml old mode 100644 new mode 100755 diff --git a/.pipelines/code-quality-template.yml b/.pipelines/code-quality-template.yml old mode 100644 new mode 100755 diff --git a/.pipelines/diabetes_regression-ci-image.yml b/.pipelines/diabetes_regression-ci-image.yml old mode 100644 new mode 100755 diff --git a/.pipelines/diabetes_regression-ci.yml b/.pipelines/diabetes_regression-ci.yml old mode 100644 new mode 100755 diff --git a/.pipelines/diabetes_regression-get-model-version-template.yml b/.pipelines/diabetes_regression-get-model-version-template.yml old mode 100644 new mode 100755 diff --git a/.pipelines/diabetes_regression-variables-template.yml b/.pipelines/diabetes_regression-variables-template.yml old mode 100644 new mode 100755 diff --git a/.pipelines/helm-install-template.yml b/.pipelines/helm-install-template.yml old mode 100644 new mode 100755 diff --git a/.pipelines/helm-upgrade-template.yml b/.pipelines/helm-upgrade-template.yml old mode 100644 new mode 100755 diff --git a/.pipelines/pr.yml b/.pipelines/pr.yml old mode 100644 new mode 100755 diff --git a/LICENSE b/LICENSE old mode 100644 new mode 100755 diff --git a/README.md b/README.md old mode 100644 new mode 100755 index 0f9ab4a6..12270419 --- a/README.md +++ b/README.md @@ -1,36 +1,19 @@ --- +test page_type: sample languages: - python -products: -- azure -- azure-machine-learning-service -- azure-devops -description: "Code which demonstrates how to set up and operationalize an MLOps flow leveraging Azure Machine Learning and Azure DevOps." ---- -# MLOps with Azure ML +# MLOps [![Build Status](https://aidemos.visualstudio.com/MLOps/_apis/build/status/microsoft.MLOpsPython?branchName=master)](https://aidemos.visualstudio.com/MLOps/_build/latest?definitionId=151&branchName=master) -MLOps will help you to understand how to build a Continuous Integration and Continuous Delivery pipeline for an ML/AI project. We will be using the Azure DevOps Project for build and release/deployment pipelines along with Azure ML services for model retraining pipeline, model management and operationalization. +MLOps will help you to understand how to build a Continuous Integration and Continuous Delivery pipeline for an ML/AI project. We will be using the DevOps Project for build and release/deployment pipelines along with ML services for model retraining pipeline, model management and operationalization. ![ML lifecycle](/docs/images/ml-lifecycle.png) This template contains code and pipeline definitions for a machine learning project that demonstrates how to automate an end to end ML/AI workflow. -## Architecture and Features - -Architecture Reference: [Machine learning operationalization (MLOps) for Python models using Azure Machine Learning](https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/ai/mlops-python) - -This reference architecture shows how to implement continuous integration (CI), continuous delivery (CD), and retraining pipeline for an AI application using Azure DevOps and [Azure Machine Learning](/azure/machine-learning/service/overview-what-is-azure-ml). The solution is built on the scikit-learn diabetes dataset but can be easily adapted for any AI scenario and other popular build systems such as Jenkins and Travis. - -The build pipelines include DevOps tasks for data sanity tests, unit tests, model training on different compute targets, model version management, model evaluation/model selection, model deployment as realtime web service, staged deployment to QA/prod and integration testing. - -## Prerequisite - -- Active Azure subscription -- At least contributor access to Azure subscription ## Getting Started @@ -40,14 +23,6 @@ To deploy this solution in your subscription, follow the manual instructions in You can find the details of the code and scripts in the repository [here](/docs/code_description.md) -### References - -- [Azure Machine Learning (Azure ML) Service Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/overview-what-is-azure-ml) -- [Azure ML CLI](https://docs.microsoft.com/en-us/azure/machine-learning/service/reference-azure-machine-learning-cli) -- [Azure ML Samples](https://docs.microsoft.com/en-us/azure/machine-learning/service/samples-notebooks) -- [Azure ML Python SDK Quickstart](https://docs.microsoft.com/en-us/azure/machine-learning/service/quickstart-create-workspace-with-python) -- [Azure DevOps](https://docs.microsoft.com/en-us/azure/devops/?view=vsts) - ## Contributing This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit diff --git a/bootstrap/README.md b/bootstrap/README.md old mode 100644 new mode 100755 diff --git a/bootstrap/bootstrap.py b/bootstrap/bootstrap.py old mode 100644 new mode 100755 diff --git a/charts/abtest-istio/Chart.yaml b/charts/abtest-istio/Chart.yaml old mode 100644 new mode 100755 diff --git a/charts/abtest-istio/templates/istio-canary.yaml b/charts/abtest-istio/templates/istio-canary.yaml old mode 100644 new mode 100755 diff --git a/charts/abtest-istio/values.yaml b/charts/abtest-istio/values.yaml old mode 100644 new mode 100755 diff --git a/charts/abtest-model/Chart.yaml b/charts/abtest-model/Chart.yaml old mode 100644 new mode 100755 diff --git a/charts/abtest-model/templates/deployment.yaml b/charts/abtest-model/templates/deployment.yaml old mode 100644 new mode 100755 diff --git a/charts/abtest-model/templates/service.yaml b/charts/abtest-model/templates/service.yaml old mode 100644 new mode 100755 diff --git a/charts/abtest-model/values.yaml b/charts/abtest-model/values.yaml old mode 100644 new mode 100755 diff --git a/data/data_test.py b/data/data_test.py old mode 100644 new mode 100755 diff --git a/data/diabetes.csv b/data/diabetes.csv old mode 100644 new mode 100755 diff --git a/data/diabetes_bad_dist.csv b/data/diabetes_bad_dist.csv old mode 100644 new mode 100755 diff --git a/data/diabetes_bad_schema.csv b/data/diabetes_bad_schema.csv old mode 100644 new mode 100755 diff --git a/data/diabetes_missing_values.csv b/data/diabetes_missing_values.csv old mode 100644 new mode 100755 diff --git a/diabetes_regression/.amlignore b/diabetes_regression/.amlignore old mode 100644 new mode 100755 diff --git a/diabetes_regression/ci_dependencies.yml b/diabetes_regression/ci_dependencies.yml old mode 100644 new mode 100755 diff --git a/diabetes_regression/conda_dependencies.yml b/diabetes_regression/conda_dependencies.yml old mode 100644 new mode 100755 diff --git a/diabetes_regression/evaluate/evaluate_model.py b/diabetes_regression/evaluate/evaluate_model.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/parameters.json b/diabetes_regression/parameters.json old mode 100644 new mode 100755 diff --git a/diabetes_regression/register/register_model.py b/diabetes_regression/register/register_model.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/scoring/deployment_config_aci.yml b/diabetes_regression/scoring/deployment_config_aci.yml old mode 100644 new mode 100755 diff --git a/diabetes_regression/scoring/deployment_config_aks.yml b/diabetes_regression/scoring/deployment_config_aks.yml old mode 100644 new mode 100755 diff --git a/diabetes_regression/scoring/inference_config.yml b/diabetes_regression/scoring/inference_config.yml old mode 100644 new mode 100755 diff --git a/diabetes_regression/scoring/score.py b/diabetes_regression/scoring/score.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/scoring/scoreA.py b/diabetes_regression/scoring/scoreA.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/scoring/scoreB.py b/diabetes_regression/scoring/scoreB.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/training/R/r_train.r b/diabetes_regression/training/R/r_train.r old mode 100644 new mode 100755 diff --git a/diabetes_regression/training/R/train_with_r.py b/diabetes_regression/training/R/train_with_r.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/training/R/train_with_r_on_databricks.py b/diabetes_regression/training/R/train_with_r_on_databricks.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/training/R/weight_data.csv b/diabetes_regression/training/R/weight_data.csv old mode 100644 new mode 100755 diff --git a/diabetes_regression/training/test_train.py b/diabetes_regression/training/test_train.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/training/train.py b/diabetes_regression/training/train.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/training/train_aml.py b/diabetes_regression/training/train_aml.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/util/__init__.py b/diabetes_regression/util/__init__.py old mode 100644 new mode 100755 diff --git a/diabetes_regression/util/model_helper.py b/diabetes_regression/util/model_helper.py old mode 100644 new mode 100755 diff --git a/docs/canary_ab_deployment.md b/docs/canary_ab_deployment.md old mode 100644 new mode 100755 diff --git a/docs/code_description.md b/docs/code_description.md old mode 100644 new mode 100755 diff --git a/docs/custom_container.md b/docs/custom_container.md old mode 100644 new mode 100755 diff --git a/docs/custom_model.md b/docs/custom_model.md old mode 100644 new mode 100755 diff --git a/docs/development_setup.md b/docs/development_setup.md old mode 100644 new mode 100755 diff --git a/docs/getting_started.md b/docs/getting_started.md old mode 100644 new mode 100755 diff --git a/docs/images/build-connect.png b/docs/images/build-connect.png old mode 100644 new mode 100755 diff --git a/docs/images/ci-build-pipeline-configure.png b/docs/images/ci-build-pipeline-configure.png old mode 100644 new mode 100755 diff --git a/docs/images/create-rm-service-connection.png b/docs/images/create-rm-service-connection.png old mode 100644 new mode 100755 diff --git a/docs/images/created-resources.png b/docs/images/created-resources.png old mode 100644 new mode 100755 diff --git a/docs/images/deploy-aci.png b/docs/images/deploy-aci.png old mode 100644 new mode 100755 diff --git a/docs/images/deploy-aks.png b/docs/images/deploy-aks.png old mode 100644 new mode 100755 diff --git a/docs/images/library_variable_groups.png b/docs/images/library_variable_groups.png old mode 100644 new mode 100755 diff --git a/docs/images/ml-lifecycle.png b/docs/images/ml-lifecycle.png old mode 100644 new mode 100755 diff --git a/docs/images/ml-ws-svc-connection.png b/docs/images/ml-ws-svc-connection.png old mode 100644 new mode 100755 diff --git a/docs/images/model-artifact-cd-trigger.png b/docs/images/model-artifact-cd-trigger.png old mode 100644 new mode 100755 diff --git a/docs/images/model-artifact.png b/docs/images/model-artifact.png old mode 100644 new mode 100755 diff --git a/docs/images/multi-stage-aci-aks.png b/docs/images/multi-stage-aci-aks.png old mode 100644 new mode 100755 diff --git a/docs/images/multi-stage-aci.png b/docs/images/multi-stage-aci.png old mode 100644 new mode 100755 diff --git a/docs/images/multi-stage-webapp.png b/docs/images/multi-stage-webapp.png old mode 100644 new mode 100755 diff --git a/docs/images/release-task-createimage.PNG b/docs/images/release-task-createimage.PNG old mode 100644 new mode 100755 diff --git a/docs/images/release-task-webappdeploy.PNG b/docs/images/release-task-webappdeploy.PNG old mode 100644 new mode 100755 diff --git a/docs/images/release-webapp-pipeline.PNG b/docs/images/release-webapp-pipeline.PNG old mode 100644 new mode 100755 diff --git a/docs/images/run-iac-pipeline.png b/docs/images/run-iac-pipeline.png old mode 100644 new mode 100755 diff --git a/docs/images/scoring_image.png b/docs/images/scoring_image.png old mode 100644 new mode 100755 diff --git a/docs/images/select-iac-pipeline.png b/docs/images/select-iac-pipeline.png old mode 100644 new mode 100755 diff --git a/docs/images/trained-model.png b/docs/images/trained-model.png old mode 100644 new mode 100755 diff --git a/docs/images/training-pipeline.png b/docs/images/training-pipeline.png old mode 100644 new mode 100755 diff --git a/environment_setup/Dockerfile b/environment_setup/Dockerfile old mode 100644 new mode 100755 diff --git a/environment_setup/arm-templates/cloud-environment.json b/environment_setup/arm-templates/cloud-environment.json old mode 100644 new mode 100755 diff --git a/environment_setup/docker-image-pipeline.yml b/environment_setup/docker-image-pipeline.yml old mode 100644 new mode 100755 diff --git a/environment_setup/iac-create-environment-pipeline-arm.yml b/environment_setup/iac-create-environment-pipeline-arm.yml old mode 100644 new mode 100755 diff --git a/environment_setup/iac-create-environment-pipeline-tf.yml b/environment_setup/iac-create-environment-pipeline-tf.yml old mode 100644 new mode 100755 diff --git a/environment_setup/iac-remove-environment-pipeline.yml b/environment_setup/iac-remove-environment-pipeline.yml old mode 100644 new mode 100755 diff --git a/environment_setup/tf-templates/backend.tf b/environment_setup/tf-templates/backend.tf old mode 100644 new mode 100755 diff --git a/environment_setup/tf-templates/main.tf b/environment_setup/tf-templates/main.tf old mode 100644 new mode 100755 diff --git a/experimentation/Diabetes Ridge Regression Experimentation Pipeline.ipynb b/experimentation/Diabetes Ridge Regression Experimentation Pipeline.ipynb old mode 100644 new mode 100755 index 8b04a5c5..2d706baa --- a/experimentation/Diabetes Ridge Regression Experimentation Pipeline.ipynb +++ b/experimentation/Diabetes Ridge Regression Experimentation Pipeline.ipynb @@ -34,6 +34,15 @@ "from azureml.core import Workspace" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"Loading Data...\")" + ] + }, { "cell_type": "code", "execution_count": null, @@ -345,7 +354,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.4" + "version": "3.8.5" } }, "nbformat": 4, diff --git a/experimentation/Diabetes Ridge Regression Parameter Experimentation.ipynb b/experimentation/Diabetes Ridge Regression Parameter Experimentation.ipynb old mode 100644 new mode 100755 diff --git a/experimentation/Diabetes Ridge Regression Scoring.ipynb b/experimentation/Diabetes Ridge Regression Scoring.ipynb old mode 100644 new mode 100755 diff --git a/experimentation/Diabetes Ridge Regression Training.ipynb b/experimentation/Diabetes Ridge Regression Training.ipynb old mode 100644 new mode 100755 diff --git a/ml_service/__init__.py b/ml_service/__init__.py old mode 100644 new mode 100755 diff --git a/ml_service/pipelines/__init__.py b/ml_service/pipelines/__init__.py old mode 100644 new mode 100755 diff --git a/ml_service/pipelines/diabetes_regression_build_train_pipeline.py b/ml_service/pipelines/diabetes_regression_build_train_pipeline.py old mode 100644 new mode 100755 diff --git a/ml_service/pipelines/diabetes_regression_build_train_pipeline_with_r.py b/ml_service/pipelines/diabetes_regression_build_train_pipeline_with_r.py old mode 100644 new mode 100755 diff --git a/ml_service/pipelines/diabetes_regression_build_train_pipeline_with_r_on_dbricks.py b/ml_service/pipelines/diabetes_regression_build_train_pipeline_with_r_on_dbricks.py old mode 100644 new mode 100755 diff --git a/ml_service/pipelines/diabetes_regression_verify_train_pipeline.py b/ml_service/pipelines/diabetes_regression_verify_train_pipeline.py old mode 100644 new mode 100755 diff --git a/ml_service/pipelines/load_sample_data.py b/ml_service/pipelines/load_sample_data.py old mode 100644 new mode 100755 diff --git a/ml_service/pipelines/run_train_pipeline.py b/ml_service/pipelines/run_train_pipeline.py old mode 100644 new mode 100755 diff --git a/ml_service/util/__init__.py b/ml_service/util/__init__.py old mode 100644 new mode 100755 diff --git a/ml_service/util/attach_compute.py b/ml_service/util/attach_compute.py old mode 100644 new mode 100755 diff --git a/ml_service/util/create_scoring_image.py b/ml_service/util/create_scoring_image.py old mode 100644 new mode 100755 diff --git a/ml_service/util/create_scoring_image.sh b/ml_service/util/create_scoring_image.sh old mode 100644 new mode 100755 diff --git a/ml_service/util/env_variables.py b/ml_service/util/env_variables.py old mode 100644 new mode 100755 diff --git a/ml_service/util/manage_environment.py b/ml_service/util/manage_environment.py old mode 100644 new mode 100755 diff --git a/ml_service/util/smoke_test_scoring_service.py b/ml_service/util/smoke_test_scoring_service.py old mode 100644 new mode 100755