Curate, Dedup, and Split

Post-labeling dedup

Even after Discovery dedup, you'll find duplicates after labeling. Why? Because:

• Two visually similar frames with different labels are different (one missed something).
• Two visually different frames with the same labels are similar in label space.

Platform exposes both. The useful one for dataset quality is the second — it surfaces frames where the labels are essentially identical and the visual difference doesn't matter for training.

Class balance

Look at a histogram of class counts:

   forklift  ████████████████████████ 800
   person    ████████████ 400
   pallet    ██ 70

If your val/test splits are random and pallet only has 70 examples total, val might end up with 7 of them — too few for a meaningful AP. Two fixes:

1. Collect more pallets. Always the better answer.
2. Stratified split. Force class proportions to match across splits — Platform does this automatically when you ask for a stratified split. The class-imbalance discussion in the data-collection guide is worth a re-read.

For severely imbalanced classes you may also want oversampling at training time plus aggressive data augmentation — both are knobs in the YOLO trainer.

Split by the right unit

You learned this in Computer Vision Foundations: random per-image splits leak when frames are correlated. On Platform, you can split by:

• Image (default, dangerous for video data).
• Scene (the source video or capture session).
• Camera (group all images from camera-1 together).
• Day (all images captured on the same day).
• Custom tag — anything you tagged in lesson 2.

The right choice is the one that matches the deployment generalization you care about. If users will deploy a new camera that's never been seen — split by camera. If users deploy on the same cameras but on different days — split by day. For very small datasets, k-fold cross-validation — itself a flavor of cross-validation — is the right way to squeeze a meaningful signal out.

Version the dataset

Every training run should be tagged with the exact dataset version it used. Platform versions automatically — every save creates an immutable snapshot you can reference forever. This is the Platform version of the reproducibility discipline behind training data.

   v1   forklift_dataset @ 2026-04-01 — 1200 images, 3 classes
   v2   forklift_dataset @ 2026-04-15 — 1800 images, 3 classes (+ pallets)
   v3   forklift_dataset @ 2026-05-01 — 2400 images, 4 classes (+ scissor lift)

A run logs which version it trained on; the version remains accessible by ID. The full datasets catalog follows the same convention — every published dataset has a stable identifier so notebooks and CI keep working.

Sanity checks before training

The 5-minute pre-training audit:

The third one is the leak check — easy to do, easy to skip, expensive to discover later.

Promoting from raw to "production"

A useful Platform pattern: keep two dataset states.

• Working dataset. Where new annotations land. Sometimes messy.
• Production dataset. A version-tagged snapshot of the working dataset that has passed your audit. Training runs use the production dataset only.

This separation prevents "I trained on whatever was there at the time" — every run is reproducible, and pairs naturally with the active learning loop you'll run after each retrain.