Collect High-Quality, Representative Data

Variance beats volume#

Compare two datasets aimed at the same deployment:

The first dataset's validation score is high only because validation measures the one condition it covers. In production, a model is only as good as the worst-covered scenario it meets. Optimize collection for coverage of the spec first and volume second.

Choose sources in the right order#

Most of the dataset should come from the real production setup. The data collection and annotation guide covers each source in more detail:

Use staged, public, and synthetic images to fill specific, named gaps in the spec, and track their share of the dataset so you can measure whether they help. On Ultralytics Platform, Generate similar images adds generated variations of an existing image to a dataset, unlabeled and ready for annotation.

Put the metadata in the filename#

Every image needs at least a camera ID, a capture day, and a frame or clip number. Without them you can't split by group later, trace a regression to its source, or collect more of an under-represented scenario. The most durable carrier is the filename: cam2_2026-05-01_000124.jpg survives every tool, export, and upload. Ultralytics Platform dataset search matches filenames and custom metadata, so these keys stay searchable after upload.

This script builds the practice dataset the rest of the course uses. It copies COCO128's labeled images and renames them as if four cameras captured them over four days, with cam1 over-collected the way a convenient camera usually is:

import shutil
from pathlib import Path

from ultralytics.data.utils import check_det_dataset, img2label_paths

data = check_det_dataset("coco128.yaml")  # 128 COCO images stand in for your camera captures
out = Path("datasets/practice")
(out / "images").mkdir(parents=True, exist_ok=True)
(out / "labels").mkdir(parents=True, exist_ok=True)

for i, image in enumerate(sorted(Path(data["train"]).glob("*.jpg"))):
    camera = ("cam1", "cam1", "cam2", "cam3", "cam4")[i % 5]  # cam1 over-collected, like a real site
    name = f"{camera}_2026-05-0{i // 32 + 1}_{i:06d}"  # camera, capture day, frame number
    shutil.copy(image, out / "images" / f"{name}.jpg")
    label = Path(img2label_paths([str(image)])[0])
    if label.exists():
        shutil.copy(label, out / "labels" / f"{name}.txt")

print(len(list((out / "images").glob("*.jpg"))), "images written to", out)

128 images written to datasets/practice

Uploading to Ultralytics Platform keeps those filenames. The upload dialog also accepts videos and whole datasets, and its settings include face blurring for privacy.

Track coverage against the spec#

With metadata in filenames, coverage is a count. Set the quotas from your spec and let the script say what to collect next:

from collections import Counter
from pathlib import Path

QUOTA = {"cam1": 30, "cam2": 30, "cam3": 30, "cam4": 30}  # images per camera, from your spec

images = sorted(Path("datasets/practice/images").glob("*.jpg"))
cameras = Counter(p.stem.split("_")[0] for p in images)
days = Counter(p.stem.split("_")[1] for p in images)

for camera, quota in QUOTA.items():
    gap = quota - cameras[camera]
    print(f"{camera}: {cameras[camera]:3d} / {quota}  {'ok' if gap <= 0 else f'collect {gap} more'}")
print("days:", dict(sorted(days.items())))

cam1:  52 / 30  ok
cam2:  26 / 30  collect 4 more
cam3:  25 / 30  collect 5 more
cam4:  25 / 30  collect 5 more
days: {'2026-05-01': 32, '2026-05-02': 32, '2026-05-03': 32, '2026-05-04': 32}

The decision rule: stop collecting from any source that is over quota, and spend the effort on the gaps. Here, cam1 already holds 40% of the images. Another week of its footage would add volume but no new coverage.

Sample video without flooding the dataset#

Video is the cheapest source and the easiest one to misuse. At 30 frames per second, one minute yields 1,800 nearly identical frames, and labeling the same moment twice costs money and inflates metrics. Keep a frame only when the scene has changed since the last frame you kept:

from pathlib import Path

import cv2

from ultralytics.utils.downloads import safe_download

video = "solutions_ci_demo.mp4"  # a short public demo clip; swap in your own footage
safe_download(f"https://github.com/ultralytics/assets/releases/download/v0.0.0/{video}")
Path("frames").mkdir(exist_ok=True)

cap = cv2.VideoCapture(video)
fps = cap.get(cv2.CAP_PROP_FPS)
read = kept = 0
last = None
while True:
    ok, frame = cap.read()
    if not ok:
        break
    thumb = cv2.cvtColor(cv2.resize(frame, (64, 64)), cv2.COLOR_BGR2GRAY).astype(float)
    if last is None or abs(thumb - last).mean() > 10:  # keep a frame only when the scene has changed
        cv2.imwrite(f"frames/clip01_{read:06d}.jpg", frame)  # the clip ID stays in the filename
        kept, last = kept + 1, thumb
    read += 1

print(f"{read} frames at {fps:.0f} FPS -> {kept} kept")

62 frames at 30 FPS -> 6 kept

The two-second clip holds 62 frames but only 6 visually distinct moments. The script compares tiny grayscale thumbnails, so it is fast on hours of footage. Raise the threshold of 10 to keep fewer frames from static cameras; lower it for fast-moving scenes. Spread the budget across clips as well: 20 frames from each of 50 clips beat 1,000 frames from one.

Avoid the collection pitfalls#

Bias and consent#

Bias is a collection problem, not a modeling problem: no hyperparameter fixes a group the data never contained. The data collection guide lists four levers: diverse sources, balanced representation, regular re-audits of coverage, and mitigation such as over-sampling under-represented groups.

If images contain faces, license plates, or other personal data, you need consent and a retention policy before collection starts. The data privacy glossary entry is a good starting point, and Privacy, Regions, and Compliance covers keeping data in the right jurisdiction with dataRegion.