Machine learning systems in production
Serving, features, drift and evaluation, treated as an operational problem rather than a modelling one.
Serving a model
intermediateA model in production is a service with unusual resource requirements and entirely ordinary operational needs. It has a latency budget, a capacity limit, a deployment process and a rollback path, and treating it as a research artefact that happens to be reachable over HTTP is how most of the problems start.
The first decision is online against batch. Batch inference computes predictions on a schedule and stores them, which is simple, cheap, easy to monitor, and only works when the input is known in advance: recommendations for known users, risk scores updated nightly. Online inference computes on request, which is required when the input arrives with the request and costs you a latency budget and capacity planning for peak.
Batching and hardware dominate the cost of the online path. Model inference is far more efficient on batches than on single requests, so a server collects requests for a few milliseconds and runs them together, which trades a little latency for a large increase in throughput. Deciding that window is the main tuning knob, and it is the same accumulate-then-flush trade as everywhere else in this material.
Loading is the operational trap. Large models take a long time to load into memory, so a naive autoscaling policy scales up long after the traffic arrived and a rolling deploy briefly halves capacity. Pre-warming, keeping a spare instance and treating model load time as a first-class number in the deployment plan are what stop that being discovered during a spike.
Everything else is ordinary discipline applied to an unusual payload. Version the model, be able to roll back to the previous one without a rebuild, keep the preprocessing code alongside the weights so they cannot drift apart, and log the inputs and outputs, which is the only way anything in the next three topics is possible.
Most production failures in machine learning systems are ordinary systems failures: capacity, deployment, versioning and rollback. Treating the model as a normal service with an expensive payload gets you most of the way, and the parts that are genuinely different come later, in the data.
Serving frameworks batch incoming requests on a short window precisely because inference is far more efficient per item on a batch, which is why the request-level trade is the first thing tuned in any latency-sensitive deployment.
Why keep preprocessing code alongside the model weights?
Features, and the skew that ruins them
advancedA model consumes features, which are values computed from raw data: how many orders this customer placed in the last week, the average basket size, whether the address changed recently. Computing those consistently is most of the engineering, and getting it inconsistent is the defining failure of production machine learning.
Training and serving skew is that failure. Training features are computed in a batch job over historical data, and serving features are computed in a service on a live request, and the two are written by different people at different times in different languages. Any difference in a default, a time zone, a rounding rule or a null handling produces a model that performs well in evaluation and worse in production, with nothing failing anywhere.
Time travel is the subtler version and it flatters the model instead. If a training example for a purchase on Tuesday includes a feature computed from data that only existed on Wednesday, the model has learned from the future, and its offline accuracy is excellent and meaningless. Every feature has to be computed as of the moment the prediction would have been made, which is a discipline the data makes very easy to violate.
A feature store exists to make those two problems structural rather than a matter of care. One definition per feature, used to compute both the training set and the serving values, with point-in-time correct joins for the historical case. Whether the tool is worth its complexity depends on scale, and the property it enforces is worth having even when the implementation is a shared library and a convention.
The freshness question then splits the store in two. Features derived from slow-moving data can be computed in batch and read at serving time, while features that depend on what the user did thirty seconds ago need a streaming path. Most production systems run both, and knowing which features need which is a product decision rather than an engineering preference.
The model is rarely the problem. Features computed one way for training and another for serving produce a system that evaluates well and performs badly, and nothing in the pipeline reports an error, which is why the same definition has to feed both paths.
Feature stores exist specifically to serve one definition to both training and inference, with point-in-time correct joins for historical data. The property matters more than the product: even a shared library beats two implementations that agree by coincidence.
A model performs well in evaluation and worse in production. What is the classic cause?
Drift, monitoring and knowing it still works
advancedA deployed model degrades without anything changing in its code, because the world it was trained on moves. Data drift is the input distribution shifting: new customer segments, a new device type, a marketing campaign bringing different traffic. Concept drift is the relationship changing: behaviour that predicted fraud last year predicts nothing now because the fraudsters adapted.
That makes monitoring a model different from monitoring a service. Latency and error rate say nothing about whether the predictions are still any good, and a model that has quietly stopped working produces perfectly healthy dashboards. What has to be watched is the input distribution against training, the output distribution over time, and, wherever possible, the actual outcome.
Ground truth usually arrives late and sometimes never. Whether a transaction was fraudulent is known when a chargeback arrives weeks later; whether a recommendation was good is inferred from a click that may mean several things. So monitoring has to work in two layers: proxy signals available immediately, such as the score distribution and the rate of predictions near a decision boundary, and the real measure when it eventually lands.
Feedback loops are the failure that is specific to this domain. A model that ranks items influences which items are seen, which produces the training data for the next model, which learns from a world its predecessor shaped. Without deliberate randomisation, or holdout traffic that the model does not influence, a system can become extremely confident about a world it created rather than one it observed.
Retraining is therefore a scheduled operational activity rather than a response to a complaint, and it needs the same rigour as a deploy: a new model version, an evaluation against a held-out set, a shadow or canary period against live traffic, and a rollback path. A model updated by copying a file onto a server is a deployment with none of the controls anyone would accept for code.
A model can stop working while every service metric stays green, because the failure is in the relationship between input and outcome rather than in the software. That is why input and output distributions, and a path to real outcomes however delayed, are the monitoring that actually matters.
Holdout traffic that the model does not influence is the standard defence against feedback loops, because it is the only source of data about a world the system did not shape. It costs a slice of performance and it is what keeps the evaluation honest.
What is the difference between data drift and concept drift?