Skip to content
mlmentorship

Design ML monitoring

Separate service health, input contracts, model behavior, mature outcomes and causal comparisons; attach each alert to evidence and an owner.

Published · 7 min read ·Core ·Advanced

ML system design · active recall

Practice before you read

40 minutes. Scope first; then cover data, baseline, model, evaluation, serving, monitoring, and iteration.

Back to my workbook How practice works

ML system design · closed-book attempt

Design ML monitoring

Scope first; then cover data, baseline, model, evaluation, serving, monitoring, and iteration.

Problem

Context. An online model uses timestamped session features, while its outcome labels require a seven-day observation window. Task. Specify signals, immediate actions, and later comparisons for the incident below, including owners and evidence for recovery. Constraints. Distinguish service availability, feature freshness, distribution changes, and measured outcome quality. Account for missing label joins and important slices; unavailable mature labels cannot establish current quality. At 12:00, an ML endpoint meets its latency and error objectives. Its session-feature snapshot has as_of=11:40, despite a two-minute maximum-age contract. Model score distributions also changed. The outcome definition requires seven days of observation. Design the signals, immediate response and later comparison needed to decide what happened. Specify how missing label joins and important slices affect the conclusion.

40:00recommended time

Closing or reloading clears the scratchpad. Only score, weak rubric dimensions, attempt count, and retry date can be stored locally.
Reading depth

Design monitoring and an incident response when a model service looks healthy but its inputs and scores have changed.

Problem

Context. An online model uses timestamped session features, while its outcome labels require a seven-day observation window.

Task. Specify signals, immediate actions, and later comparisons for the incident below, including owners and evidence for recovery.

Constraints. Distinguish service availability, feature freshness, distribution changes, and measured outcome quality. Account for missing label joins and important slices; unavailable mature labels cannot establish current quality.

At 12:00, an ML endpoint meets its latency and error objectives. Its session-feature snapshot has as_of=11:40, despite a two-minute maximum-age contract. Model score distributions also changed. The outcome definition requires seven days of observation.

Design the signals, immediate response and later comparison needed to decide what happened. Specify how missing label joins and important slices affect the conclusion.

Picture: Different Evidence Arrives at Different Times

The snapshot is twenty minutes old, exceeding the two-minute limit. That is a data-contract breach now, even though requests return successfully. The score change is another observable fact, not proof of lower accuracy. Seven-day outcomes cannot establish current cohort quality before their window matures.

Use the declared fallback for affected requests while the data owner investigates. It might use approved rules or a model that does not require the stale feature. Do not replace missing values with zero unless that behavior was validated. Record affected versions, cohorts, actions and exposure so later outcomes remain interpretable.

Staleness can breach a contract before outcome labels mature

A freshness violation can be known before quality labels matureA stale snapshot breaches its current contract immediately. Quality inference still waits for mature outcomes and valid joins.Later evidence cannot cross theearlier readout boundarysnapshot stalefreshness failure knownmaturation waitquality readout
Derivation
Freshness can fail now while outcome quality is still unknownAt noon, an 11:40 snapshot is twenty minutes old and fails a two-minute freshness contract immediately. Outcome evidence matures after a seven-day window and requires request identity and join coverage. A changed score distribution does not establish changed quality or causal product impact.Same request, different evidence clocksAt 12:00: freshness can fail NOWsnapshot 11:40age 20 min > 2 min limitLater: outcome evidence must matureday 0day 7Mature label + request ID + join coverageScore drift alone does not establish a qualitychange.
The stale snapshot breaches a contract immediately. Outcome quality needs mature labels and adequate joins. Score drift alone proves neither quality nor causal impact.
Check: Which alert is justified immediately while model quality is still unknown?
Read it this way: A stale snapshot breaches its current contract immediately. Quality inference still waits for mature outcomes and valid joins.
Which failure is already knowable before quality outcomes mature?

Give Each Signal a Claim and Response

LayerSignalSupported conclusionOwner and response
ServiceLatency, errors, throughput, resourcesEndpoint operating conditionService owner: capacity/failure runbook
DataSchema, nulls, freshness, distributionsContract breach or input changeData owner: repair source; approved consumer fallback
BehaviorScore/rank distributions and slicesOutputs changedML owner: inspect versions, inputs and applicable baseline
OutcomeMature labels, join coverage, product effectsObserved quality/impact in covered cohortProduct/ML: investigate; causal claim needs valid comparison
CostCost/request, utilization, batchingResource/economic changeService/platform: evaluate throughput and task-success effects

Declare which signals are proxies, directly observed outcomes or causal estimates. An engagement proxy or confidence shift is not a substitute for correctness labels.

PredictionRecord
	prediction_id, request_id, decision_time, cohort_id
	model_version, feature_version, feature_as_of, action_id
	policy_version, exposure_assignment, score, slice_keys
LabelRecord
	prediction_id, target_definition, outcome_time
	available_time, maturity_time, join_status, censoring_reason
Alert
	signal_id, window, baseline, slice, numerator, denominator
	magnitude, uncertainty, severity, owner, runbook, deadline

Check the Denominator Before Trusting Quality

Suppose 1,000 mature label IDs are eligible, but only 900 join to predictions. Of the joined rows, 810 are correct: reported accuracy is 90%, and join coverage is 90%. Without resolving the missing rows, full-cohort accuracy could range from 81% to 91%. A critical slice might account for all 100 missing joins.

Track eligibility, maturity, missingness and coverage by slice alongside quality. Distinguish source delay, join failure and censoring from an observed negative outcome.

Change 1: A Campaign Changes Normal Traffic

New task. Input distributions shift during a seasonal campaign.

Picture and change. Compare appropriate seasonal/context baselines, sample sizes and practical effect sizes. Inspect contract breaches and mature quality rather than equating a small statistical p-value with degradation.

Check. A detectable distribution change can be harmless. Historical context helps diagnosis but does not excuse a broken freshness limit.

Use when. Traffic composition has legitimate causes.

Change 2: One Failed Job Creates Many Alerts

New task. Many feature alerts share one upstream failure.

Picture and change. Group by lineage and root cause; separate pages, tickets and dashboard signals. Retain consequential slice alerts, a named responder and the fallback state. Record acknowledgment, action and resolution evidence for later threshold review.

Check. Silencing a root alert does not repair stale features. There is no universal useful-alert percentage that validates the monitoring system.

Use when. Noise hides actionable incidents.

Failure: The Global Metric Stays Calm

A stable aggregate can hide lost label coverage or worse performance in a critical slice. Monitor counts, uncertainty and severe failures, not just an average.

Transfer Check

Can a shadow deployment establish a new model’s lift in user outcomes?

Reasoned Answer

Not generally. Shadow traffic can check integration and compare predictions without changing user exposure. User effects require actual exposure with a valid experiment or justified identification assumptions.

Select Diagnostics, Not Universal Thresholds

KS tests compare continuous distributions; chi-square tests compare categorical counts under their assumptions. PSI depends on binning and reference choice; it is not a universal quality test. Consider dependence, repeated testing, sample size, effect magnitude and historical noise before setting thresholds.

Monitor schema and freshness contracts separately from statistical drift. Track target distributions, label availability and calibration on supported mature cohorts. An input shift alone does not establish a changed conditional relationship between inputs and targets.

Predetermine important product and safety slices, then investigate newly affected groups. Sparse slices need uncertainty and sufficient observation, not automatic reassurance from a quiet test. Govern sensitive attributes and trace access under the applicable data policy.

Connect Observation to Operation

Service and data-contract breaches need explicit objectives, fallback rules and response deadlines. Behavior alerts require diagnosis; retraining is not a default repair for stale inputs or broken joins. Record incident ownership, affected versions, containment, recovery evidence and rollback scope.

Audit representative mature samples to evaluate proxies; retain selection and observation limits. Version baselines, target definitions, thresholds and runbooks with model and feature changes. Review alert volume, action usefulness, missed incidents and time to resolution without imposing an arbitrary success percentage.

Related: multi-team ML platform design, ML data lineage and versioning, delayed and selective labels, and A/B testing for ML systems.

Found an error or stale claim? Report a correction or send a private note.