Observability and Drift

The three layers of observability#

Most teams cover the first layer with off-the-shelf tools, then forget the other two. The other two are where production CV breaks silently; model monitoring is the discipline that keeps them visible.

Metrics worth emitting per frame#

import time
import logging

log = logging.getLogger("yolo.inference")

for r in model.track("rtsp://camera-1.local/stream", stream=True, persist=True):
    inf_ms = r.speed["inference"]
    n_boxes = len(r.boxes)
    mean_conf = float(r.boxes.conf.mean()) if n_boxes else None

    log.info(
        "inference",
        extra={
            "camera": "camera-1",
            "inference_ms": inf_ms,
            "detections": n_boxes,
            "mean_confidence": mean_conf,
        },
    )

Per camera, send to your metric system:

• inference_ms — inference latency. Alert on p95 above your budget.
• detections — count of returned boxes. Alert on a sudden drop or surge — a cheap anomaly detection signal.
• mean_confidence — distributional signal. A drift down often means domain change before mAP does.
• dropped_frames — capacity signal. > 0 means the system is overloaded.

The weekly holdout check#

Define a small holdout set — 200–500 production-realistic labeled images, sampled before launch — and validate against it weekly:

yolo val model=production.engine data=holdout.yaml > weekly_metrics.txt

Pipe the mAP into your metrics system. Alert when it drops by more than 1 mAP point from the previous week. The most common cause is a model swap that was supposed to be a no-op; the second most common is data drift (next section). A confusion matrix on the same holdout makes per-class regressions obvious.

Distribution drift#

The model's behavior changes when its inputs change — the textbook definition of data drift. Two cheap distributional signals:

1. Confidence histogram. Plot a histogram of detection confidence weekly. If the distribution shifts left (fewer high-confidence detections, more uncertain ones), the input domain has drifted.
2. Object size distribution. Plot box width × height. New camera mounts, new optics, weather conditions all show up here.

A Kolmogorov-Smirnov test compares two distributions and gives a p-value. A p-value < 0.01 means "these distributions are reliably different" — a useful trigger for investigation, and often the input to an active learning loop or a continual learning refresh. When drift fires, the data collection and annotation guide is the right starting point for what new data to collect, and the fine-tuning guide covers the warm-start retrain.

from scipy.stats import ks_2samp
import numpy as np

baseline = np.load("baseline_confidences.npy")     # collected at deploy
recent = np.load("recent_confidences.npy")         # last 24 hours

statistic, p_value = ks_2samp(baseline, recent)
if p_value < 0.01:
    print(f"DRIFT: KS={statistic:.3f}, p={p_value:.4f}")

What to alert on#

Be selective. Alerts that fire weekly without action become noise.

The discipline: every alert should map to a concrete next action. If you can't say what an on-call would do when the alert fires, the alert is noise.