Compare Runs and Pick a Winner

Compare Runs in a Project#

A project lists its models with checkboxes. Checked models overlay on the Charts dashboard; Table view lines up training arguments and final metrics, and Diff hides identical columns so you see at a glance that two runs differ only in imgsz. Comparison works within one project, so keep related experiments together (models can be cut and pasted between projects). When reading curves, compare the final value, how fast each run got there, and how smoothly.

With two or more completed models, Compare with AI asks Ask AI to summarize differences and propose the next experiment. Treat its proposal as a hypothesis for your next run, not a verdict.

Make Every Run Explainable#

Names carry the meaning: run_5 says nothing, v3_pallets_added and v4_imgsz1024 say which dataset version and what changed. For the reasoning, open More actions > Information on the model page and save a JSON note as Custom Metadata:

{
  "hypothesis": "Pallet AP is low because val holds only 7 pallets",
  "change": "Dataset v3: +80 pallet images from cam03 and cam07; fine-tuned from v2",
  "result": "Pallet AP 0.31 -> 0.58, overall mAP50-95 0.51 -> 0.59",
  "next": "Forklift recall drops at night; collect dawn and dusk footage"
}

Platform records the rest: the Run Information card shows GPU, hourly rate, total cost, ultralytics version, host, parent model, pinned dataset version, Git commit when available, and a copyable yolo train command. Comparisons are only fair between runs on the same dataset version, which is why every run should have one pinned.

Choose the Metric and Gates Before You Look#

The YOLO performance metrics guide explains each. A candidate ships only if it clears every gate:

Worked Example: A Promotion Gate Script#

Download the finalists' best.pt files and score them on identical data with the same code. This example uses two official checkpoints so it runs anywhere:

from ultralytics import YOLO

DATA = "coco8.yaml"  # identical held-out data for both runs; use your test split for the final call
BASELINE, CANDIDATE = "yolo26n.pt", "yolo26s.pt"  # e.g. best.pt from two Platform runs
NOISE, CLASS_FLOOR, LATENCY_BUDGET_MS = 0.005, 0.90, 100.0

def evaluate(weights):
    m = YOLO(weights).val(data=DATA, plots=False, verbose=False)
    per_class = {m.names[int(c)]: ap for c, ap in zip(m.box.ap_class_index, m.box.ap)}
    return m.box.map, per_class, m.speed["inference"]

(base_map, base_ap, _), (cand_map, cand_ap, cand_ms) = evaluate(BASELINE), evaluate(CANDIDATE)
weak = [n for n, ap in base_ap.items() if cand_ap.get(n, 0) < CLASS_FLOOR * ap]
gates = {
    f"mAP50-95 {base_map:.3f} -> {cand_map:.3f} beats noise": cand_map - base_map > NOISE,
    f"every class keeps {CLASS_FLOOR:.0%} of its AP (weak: {', '.join(weak) or 'none'})": not weak,
    f"inference {cand_ms:.1f} ms within {LATENCY_BUDGET_MS:.0f} ms": cand_ms <= LATENCY_BUDGET_MS,
}
for name, ok in gates.items():
    print("PASS" if ok else "FAIL", name)
print("verdict:", "ship the candidate" if all(gates.values()) else "keep the baseline, run more experiments")
raise SystemExit(0 if all(gates.values()) else 1)  # non-zero exit blocks a CI release step

PASS mAP50-95 0.669 -> 0.693 beats noise
FAIL every class keeps 90% of its AP (weak: elephant, potted plant)
PASS inference 44.1 ms within 100 ms
verdict: keep the baseline, run more experiments

What happened: the larger model wins overall by 2.4 points, but two classes lose more than 10% of their AP, so it fails. That's the regression a headline comparison hides. The per-class dictionary comes from ap_class_index and ap, which cover only classes present in the data; latency is measured on the machine running the script, so run the gate on hardware like your deployment target. COCO8 has just 8 validation images, far too few for a real decision. Use a held-out set of at least a few hundred images, and pass split="test" exactly once, for the final call.

Dig Into the Weak Images#

For detection runs trained with a saved dataset version, the Validation view adds Validation insights: the worst- and best-performing images by per-image F1 drawn with ground truth, a characteristic explorer correlating F1 with image size, aspect ratio, and instance count, and the classes most associated with failures. Find similar on the worst images adds look-alikes to the training set as unlabeled images, turning a failed gate into a concrete data task.

Pitfalls#