MLOps Methodology

Training a good model is only a small part of a real machine learning system. Around the model, a production system needs data collection and validation, feature pipelines, serving infrastructure, monitoring and a process to retrain and redeploy. MLOps (Machine Learning Operations) is the set of practices that brings the ideas of DevOps to machine learning, so that models can be deployed reliably and kept useful over time.

MLOps at the intersection of Machine Learning, DevOps and Data Engineering
MLOps lies at the intersection of Machine Learning, DevOps and Data Engineering. Image: Cmbreuel, Wikimedia Commons, CC BY-SA 4.0

This course follows the full life cycle of a machine learning project: a refresher on machine learning, the stages of a project, deployment strategies, monitoring and packaging, and the specific way models should be developed when they are meant for production.

This page gives an overview of the course. The slides and notebooks are provided to students during the course.

Part 1 – Machine learning refresher

Families of machine learning

The data itself can be structured (tables, databases) or unstructured (images, audio, text), which strongly influences the techniques used to label, augment and analyze it.

Three classic algorithms

Linear regression
Linear regression fits a straight line that minimizes the squared error. Sewaqu, Wikimedia Commons, public domain
Logistic curve
Logistic regression turns a linear score into a probability with the sigmoid function. Qef, Wikimedia Commons, public domain
k-nearest neighbors classification
k-nearest neighbors assigns the majority class of the k closest examples. AnAj, Wikimedia Commons, CC BY-SA 3.0

The scikit-learn workflow

With scikit-learn, every supervised model follows the same four steps, which makes it easy to compare several estimators on the same data.

from sklearn.linear_model import LinearRegression

model = LinearRegression()   # 1. choose an estimator and its hyperparameters
model.fit(X, y)              # 2. train the model
model.score(X, y)            # 3. evaluate it
model.predict(X_new)         # 4. use it

The usual working method: split the data into training and test sets, train a first simple model, evaluate it on data it has never seen, then iterate on the features and hyperparameters.

Part 2 – The machine learning project life cycle

A machine learning project goes through four stages, with frequent back-and-forth between them. The course uses a speech recognition system as a running example, and a visual inspection system (detecting defects on products in a factory) as a second case study.

StageKey questionsSpeech recognition example
ScopingWhich problem? Which metrics? Which resources and timeline? Decide to work on voice search; key metrics are accuracy, latency and throughput.
DataHow is the data defined, collected and labeled consistently? Is the transcription convention consistent (« um », punctuation)? How much silence before and after each clip?
ModelingWhich model, which baseline, how to analyze errors? Error analysis shows where the model is weak, for instance with background car noise.
DeploymentHow to serve, monitor and maintain the model? A mobile app sends audio to a prediction server; the system is monitored after release.
CRISP-DM process diagram
The CRISP-DM process, a classic model of the iterative life cycle of a data project. Kenneth Jensen, Wikimedia Commons, CC BY-SA 3.0

Data drift and concept drift

Once in production, the world keeps changing, and a model that was correct when it was trained slowly becomes wrong:

Takeaway: deployment is not the end of a project, but the beginning of its maintenance.

Part 3 – Deployment strategies

Three common deployment cases: a new product or capability, the automation or assistance of a manual task, and the replacement of a previous ML system. In all cases, the goal is to send traffic gradually to the new model, monitor it, and be able to roll back. Like modeling, deployment is iterative.

Shadow mode Input Human ML model decideslogged, compared Canary deployment Traffic Old model New model 95%5%, then ramp up Blue-green deployment Router Blue (old) Green (new) kept for rollback100% after switch Thick arrow: traffic used for decisions · dashed: kept aside or not used
Three ways to put a new model in production with limited risk.

Degrees of automation

Deployment is not all-or-nothing. A system can move step by step from human decisions to full automation, stopping at the level that makes sense for the application:

Human onlyShadow modeAI assistancePartial automationFull automation
no ML ML runs, human decides ML highlights, human decides ML decides when confident, otherwise asks a human ML decides

In the visual inspection example, AI assistance could highlight the suspicious area on a picture for a human inspector, while partial automation would let the model accept or reject products by itself and send only the uncertain cases to a human.

Part 4 – Monitoring and packaging

Monitoring makes sure the system keeps meeting expectations after deployment. A good practice is to brainstorm everything that could go wrong, define metrics that would detect each problem, start with many of them, and gradually remove those that turn out not to be useful. Each metric gets a threshold that triggers an alert.

MetricsWhat they detectExamples (speech recognition)
Softwarethe health of the servicememory, compute, latency, throughput, server load
Inputchanges in the data (data drift)average input length or volume, number of missing values, average image brightness
Outputchanges in the behaviour of the model or of the usersnumber of empty outputs, number of times a user redoes a search or switches to typing
Grafana dashboard
A Grafana dashboard: metrics collected by Prometheus are displayed as time series, with alerts on thresholds. Linux Screenshots (xmodulo), Wikimedia Commons, CC BY 2.0

In practice

Part 5 – Modeling for production

Model-centric vs. data-centric AI

The model-centric approach keeps the data fixed and improves the code of the model. The data-centric approach keeps the model fixed and improves the quality of the data: consistent labels, better coverage of rare cases, cleaning of noisy examples. In many practical projects, the second one brings more progress for less effort.

An iterative process

Model development is a loop: choose a model, hyperparameters and data → train → run an error analysis → decide what to improve. Error analysis groups mistakes by category (for instance car noise, people noise, low bandwidth) to focus the effort where it pays off most.

Beyond the average test score

A good average score on the test set is not enough to deploy a model:

Establishing a baseline

Before trying to improve a model, establish a baseline: human-level performance (especially for unstructured data such as images and audio), a literature search, open-source implementations, or an older system. The baseline tells what is possible and where the irreducible error lies.

Getting started

Takeaway: a reasonable algorithm with good data often outperforms a great algorithm with poor data.

Further reading