Observability and Drift

Three Layers of Signals#

Most teams cover the first layer with standard tooling and stop. The second and third are where vision systems fail silently; the model monitoring guide covers the full maintenance loop.

If you deploy through Ultralytics Platform, the deployment page gives you the infrastructure layer (requests, error rate, P95 latency) and request logs you can filter to errors, as above; its Monitor Health and Accuracy lesson covers them. The behavior and accuracy layers are yours to build, and the rest of this lesson builds them.

Emit Per-Frame Metrics#

Every Results object carries timings and boxes. Write one JSON line per frame and ship it to whatever metrics system you use:

import json
import time

from ultralytics import YOLO

model = YOLO("yolo26n.pt")
video = "https://github.com/ultralytics/assets/releases/download/v0.0.0/solutions_ci_demo.mp4"
with open("metrics.jsonl", "w") as log:
    for r in model.predict(video, stream=True, verbose=False):
        conf = r.boxes.conf
        record = {
            "ts": time.time(),
            "camera": "lobby-1",
            "inference_ms": round(r.speed["inference"], 1),
            "detections": len(r.boxes),
            "mean_conf": round(conf.mean().item(), 3) if len(conf) else None,
            "confs": [round(c, 3) for c in conf.tolist()],  # raw values for the drift test
        }
        log.write(json.dumps(record) + "\n")
print({k: v for k, v in record.items() if k != "confs"})

{'ts': 1792003254.33, 'camera': 'lobby-1', 'inference_ms': 34.7, 'detections': 22, 'mean_conf': 0.543}

What each field is for:

• inference_ms: alert when p95 exceeds your budget.
• detections: a sudden drop to near zero usually means a blocked or broken camera; a surge means a scene change or false positives.
• mean_conf: a quick trend line, but too coarse to detect drift on its own.
• confs: the full distribution, which the drift test below needs.

Test Confidence Distributions for Drift#

Data drift shows up in confidences long before anyone labels new data. Collect confidences at launch as a baseline, collect a recent window, and compare the two distributions with a two-sample Kolmogorov-Smirnov test (pip install scipy). This script simulates dusk by darkening the same clip:

import cv2
import numpy as np
from scipy.stats import ks_2samp

from ultralytics import YOLO
from ultralytics.utils.downloads import safe_download

model = YOLO("yolo26n.pt")
video = str(safe_download("https://github.com/ultralytics/assets/releases/download/v0.0.0/solutions_ci_demo.mp4"))

def confidences(darken=1.0):
    """Collect every detection confidence from the clip, optionally darkened to simulate dusk."""
    cap, out = cv2.VideoCapture(video), []
    while True:
        ok, frame = cap.read()
        if not ok:
            break
        out += model((frame * darken).astype(np.uint8), verbose=False)[0].boxes.conf.tolist()
    return np.array(out)

baseline = confidences()  # what "normal" looked like at launch
for name, recent in (("same scene", confidences()), ("dusk (35% brightness)", confidences(darken=0.35))):
    stat, p = ks_2samp(baseline, recent)
    print(f"{name:22s} n={len(recent):4d} mean_conf={recent.mean():.3f} KS={stat:.3f} p={p:.2g}")

same scene             n=1376 mean_conf=0.519 KS=0.000 p=1
dusk (35% brightness)  n=1337 mean_conf=0.508 KS=0.064 p=0.0068

Read it line by line:

• Same scene: identical distributions give a KS statistic of 0 and p = 1, the control.
• Dusk: 39 fewer detections and a mean only 0.011 lower, which no one would notice on a chart. The KS statistic (the largest gap between the two cumulative distributions) is 0.064, and p = 0.0068 < 0.01 says the shift is real.

A drift alert is a reason to look at sample frames, not proof of failure. When it is real, collect and label frames from the affected cameras: the data collection and annotation guide covers what to collect, and the fine-tuning guide covers the retrain. Box sizes (width × height) make a second cheap drift signal: new mounts and optics show up there first.

Validate a Holdout Set Weekly#

Behavior signals say something changed; only labels say whether accuracy dropped. Keep a holdout set of 200–500 labeled production images and validate every model version, plus every week:

yolo val model=yolo26n.pt data=coco8.yaml > weekly_metrics.txt  # use your production model and holdout YAML

Record mAP50-95 and per-class AP over time, and alert when mAP drops more than 1 point week over week. The most common cause is a model swap that was supposed to be a no-op; the second is drift. A confusion matrix on the same holdout makes per-class regressions obvious.

The holdout is what answers the question in this chart. Retrain when it shows quality starting to fall, or when you have enough new labeled data from the drifted conditions, not on a fixed calendar.

Alert on Actions, Not Numbers#

Pitfalls#