Know When the Dataset Is Ready
Gate the first training run on eight pass/fail checks, automate the three a script can verify, and know which fix each failure calls for.
Every project reaches the moment the team wants to train. Sometimes the dataset is ready; sometimes it still has a split leak or an under-labeled class that will waste a week of GPU time and produce a number nobody should trust. Eight checks, each scored yes, no, or documented exception, make that call objective instead of a matter of mood.
Hands-on
The eight checks#
| # | Check | Pass criteria | If it fails, go back to |
|---|---|---|---|
| 1 | Clear objective | Goal, task, classes, and an acceptance test are written down and agreed | Start With the Business Objective |
| 2 | Defined classes | Each class has a definition and a named look-alike it excludes | Objective and labeling guide |
| 3 | Representative images | Every spec scenario meets its quota, mostly from production sources | Collect High-Quality, Representative Data |
| 4 | Sufficient volume | ≥ 1,500 images and ≥ 10,000 instances per class, or a documented exception | Spec and collection |
| 5 | Consistent labels | Guide written, calibration round done, agreement ≥ 90% | Annotation Best Practices |
| 6 | QC complete | Audit script clean, suspects reviewed, QC notes written | Dataset Quality Control |
| 7 | Clean splits | Grouped split, no group in two splits, every class in every split | Split the Dataset Correctly |
| 8 | Background images | 0–10% of images are labeled-empty frames | Spec and collection |
The checklist gates the same decisions the data collection and annotation guide covers one by one; here they become a single go/no-go.
Automate the countable checks#
Checks 4, 7, and 8 are arithmetic, so a script scores them the same way every time. This one reads the practice set and the split lists written by group_split.py in the previous lesson:
from collections import Counter
from pathlib import Path
root = Path("datasets/practice") # images/, labels/, and the train/val/test.txt lists from group_split.py
splits = {s: [Path(p).stem for p in (root / f"{s}.txt").read_text().split()] for s in ("train", "val", "test")}
def classes(stem):
label = root / "labels" / f"{stem}.txt"
return [int(r.split()[0]) for r in label.read_text().splitlines() if r.strip()] if label.exists() else []
stems = [s for split in splits.values() for s in split]
instances = Counter(c for s in stems for c in classes(s))
images = Counter(c for s in stems for c in set(classes(s)))
background = sum(not classes(s) for s in stems) / len(stems)
groups = {split: {s.rsplit("_", 1)[0] for s in names} for split, names in splits.items()}
per_split = {split: {c for s in names for c in classes(s)} for split, names in splits.items()}
shared = (groups["train"] & groups["val"]) | (groups["train"] & groups["test"]) | (groups["val"] & groups["test"])
weakest = min(instances, key=lambda c: (images[c], instances[c]))
checks = {
"4 volume": (images[weakest] >= 1500 and instances[weakest] >= 10_000,
f"weakest class {weakest}: {images[weakest]} images, {instances[weakest]} instances"),
"7 no shared groups": (not shared, f"{len(shared)} groups appear in more than one split"),
"7 every class in every split": (all(per_split[s] == set(instances) for s in splits),
", ".join(f"{s} misses {len(set(instances) - per_split[s])}" for s in splits)),
"8 background 0-10%": (background <= 0.10, f"{background:.1%} background images"),
}
for name, (ok, detail) in checks.items():
print(f"{'PASS' if ok else 'FAIL'} {name:<29} {detail}")FAIL 4 volume weakest class 30: 1 images, 1 instances
PASS 7 no shared groups 0 groups appear in more than one split
FAIL 7 every class in every split train misses 5, val misses 36, test misses 41
PASS 8 background 0-10% 1.6% background imagesThe grouped split fixed the leak, but the practice set is nowhere near ready: its weakest class (30, skis) has a single labeled instance, and val can't score half the classes. That's the expected verdict for 128 demo images, and exactly the verdict this check exists to deliver before a week of training.
The script covers what can be counted. Checks 1, 2, 3, 5, and 6 need a person: an agreed objective, written definitions, spec coverage, a measured agreement score, and QC notes. Score them by hand, with one line of evidence each.
Decide: train, ship with a gap, or fix#
When a check fails, the tempting next step is almost always something other than fixing the data:
| Tempting shortcut | Why it doesn't work |
|---|---|
| "Try a bigger model." | A larger model learns label noise just as faithfully. |
| "Tune the learning rate." | No optimizer setting adds a missing class or scenario. |
| "Train for more epochs." | More epochs on narrow data overfit faster. |
| "Add more augmentation." | Augmentation can't create scenarios the data never contained. |
Three almost-ready patterns#
| Pattern | Failing check | What it looks like | Fix |
|---|---|---|---|
| Volume without variance | 3 | 5,000 images, one camera, no edge cases | Collect from more cameras, sites, and times before training |
| Variance without consistency | 5 | Annotators disagree on box extent and occlusion | Finish the guide, calibrate, and relabel the affected sample |
| Quiet leakage | 7 | Random per-image split on video frames | Re-split by clip, camera, or day; expect val mAP to drop, because that drop is reality |
Ship with a known gap#
Often the right call is a focused first model before every check is green: a pilot, a prototype, or a v1 on a subset of cameras. A model in production shows what's actually wrong faster than polishing the dataset in the dark. When you ship with a gap, write down three things:
- Which check failed.
- The expected impact, for example "no night data; expect lower recall after dark".
- The collection plan that closes it while v1 runs.
"Volume: exception" tells the next person nothing. "Volume: pallet at 700 instances, expect lower pallet recall in crowded bays; 2,000 more labeled by June" is a managed risk. Every exception needs the expected impact on the acceptance test.
Rerun before every retrain#
A dataset that was ready for v1 is not automatically ready for v3. New classes, new sources, and new annotators can each break a check that used to pass, and production data drifts while new labels pile up. Rerun readiness.py and rescore the manual checks before every retrain; it takes minutes, and skipping it can cost a week of training on a broken dataset.
Show solution
A scored checklist for the practice set:
| # | Check | Score | Evidence |
|---|---|---|---|
| 1 | Clear objective | yes | objective.md and acceptance_test.py from lesson 1 |
| 2 | Defined classes | no | COCO's 80 names have no written definitions or excluded look-alikes |
| 3 | Representative images | no | coverage.py: cam2–cam4 below quota |
| 4 | Sufficient volume | no | readiness.py: weakest class has 1 image |
| 5 | Consistent labels | no | Agreement 47% in the calibration round |
| 6 | QC complete | partial | Audit run; suspects in 000000000438.jpg still open |
| 7 | Clean splits | no | No shared groups, but val misses 36 of 71 classes |
| 8 | Background images | yes | 1.6% |
Decision: fix first. Ranked by risk to the acceptance test: (1) cut the class list to the classes the objective needs, which shrinks checks 2, 4, and 7 at once; (2) collect more groups for those classes; (3) finish the guide rule on distant people and relabel. A demo set fails this check, and it should.