ML Ops Framework Setup : 4-Week Implementation

Objective: Setup MLOPs practice with tracking of 2 ML models and enable clients with a clear MLOPs practice to onboard newer ML models in terms of tracking model performance.

Key Challenges Addressed:

1. Data/model drift cause models to become ineffective over time
2. No centralized way to measure model performance
3. Poor model performance improvement statistics due to lack of explanation

Outcome:

1. A centralized model monitoring system that measures the following: a. Model Drift which includes Overall drift, drift trend, feature wise drift b. Explainable AI which provides Global feature Importance, Local explanation c. Persona based insights for business users and data engineer (Top 5 KPIs)
2. A Visual provenance graph to track model execution.

Implementation Plan The break-up of the implementation plan is as below:

1. Week 1: Spent on ‘discovery’ to understand the business and ML models, data sources and downstream applications.
2. Week 2: Integration of model pipelines and drift calculation for two models
3. Week 3-4: Model drift calculation for two models along with Explainable AI and activation of visual provenance graph. Centralized model monitoring, Documentation and a MLOps roadmap for the future.

This implementation uses the following native Azure components:

1. Azure Git: Allowing changes to the repository in a controlled way, allowing coordination between many people without accidentally overwriting or corrupting files
2. App Services: The monitoring web app and python backend code is hosted on azure Linux app services. Both the apps can be scaled automatically or manually on demand.
3. Microsoft Azure Data Factory: Used to fetch the status information of Data factory pipelines to track.
4. Databricks Workspace: MLFlow component of Databricks is used to fetch the data stored by notebook during execution.
5. Cosmos DB: With the flexibility of schema and changing nature of data, NoSQL helps accommodate requirements.