Collect High-Quality, Representative Data
Capture real production conditions, not staged-only images — and avoid the collection biases that quietly kill models.
Most dataset failures are collection failures. The team captures clean, well-lit examples on a single camera in one week, then the model fails in production on the long tail. Quality over quantity, but variance over both.
Hands-on
Quality over quantity — and variance over both#
A useful rule from the Ultralytics training-results guidance: a model's ceiling is set by the worst-represented scenario in its training data. So:
- 10,000 nearly-identical images from one camera at noon → great metrics on the val set, terrible on a different camera at dusk.
- 1,500 images that span 12 scenarios → modest val metrics, but they generalize.
Optimize for variance first, volume second.
Where to source data#
Prioritize data from your actual production setup — see the data collection and annotation guide for deeper coverage on the sources below:
| Source | When it helps | When it hurts |
|---|---|---|
| Production cameras (live) | Closest to deployment; ground truth | Slow; may need data agreements |
| Production cameras (archived) | Lots of variance for free | Old footage may not match current setup |
| Public datasets (COCO, Open Images, VOC) | Bootstrap quickly; add diversity | Domain shift from your real scenes |
| Web scraping | Cheap variance | Inconsistent quality, licensing risk |
| Synthetic data | Fill rare classes (faults, hazards) | Overfit to renderer's style |
| Staged captures | Edge cases you can't wait for | Easy to over-rely on |
The large majority of your data should come from your real production setup. Public datasets help only in specific cases (e.g. license-plate detection), and synthetic data only to fill rare classes you can't yet capture.
Common collection pitfalls#
The failure modes that show up over and over:
| Pitfall | Symptom | Fix |
|---|---|---|
| Single-day collection | High val mAP, drops at dusk in prod | Spread across ≥ 7 days, ideally a full month |
| One camera | Model fails on second camera | Capture from ≥ 3 different cameras |
| Staged-only data | Model fails on real chaos | At least 80% from genuine production |
| Duplicate frames | Inflated metrics, leaked val | Dedup at upload (Ultralytics Platform does this with content hashing) |
| Biased operator | Class imbalance you didn't plan for | Sample across shifts, sites, and people |
| Missing rare cases | Model misses safety events | Stratified sampling or synthetic top-up |
| Time correlation | Val frames ~1s after train frames | Split by day or scene, not random per-image |
Bias is a collection problem, not a modeling problem#
A biased dataset produces a biased model — and you can't fix it later with hyperparameters. The data collection guide on bias calls out four levers:
- Diverse sources — don't collect from one site, one shift, or one team.
- Balanced representation — across age, gender, ethnicity, geography (where applicable to your task).
- Continuous monitoring — re-audit the dataset every few weeks for drift in coverage.
- Mitigation techniques — oversample rare classes, augment heavily on minority groups, fairness-aware sampling.
For face / person detection in particular, ethical collection matters as much as technical correctness — see the AI ethics glossary entry.
Log everything you capture#
Every image should carry, at minimum: camera ID, timestamp, location, and any operator-relevant metadata (shift, weather, conditions). Without this you can't:
- Stratify splits later (lesson 6).
- Diagnose a regression to a specific source.
- Re-collect more of an under-represented scenario.
Ultralytics Platform preserves this metadata as you upload; for self-hosted pipelines, encode it in filenames or a sidecar JSON. Either way: don't drop it.
Ethics and consent#
If your data includes faces, license plates, or any biometric / PII data, you'll need consent and a retention policy. The data privacy glossary entry is the right place to start, and the regions / compliance lesson in the Build with Ultralytics Platform course covers dataRegion for keeping data in the right jurisdiction.