Custom Keypoints and OKS Metrics

Design the keypoint schema#

Good keypoints share three properties: unambiguous (two annotators put the point in the same place), stable (the point exists in every pose the object takes), and useful (some downstream calculation reads it). Take a forklift seen from a ceiling camera, where the product question is "are the forks raised while the truck moves?":

Every point earns its place by feeding the calculation: the fork tips give direction, and the mast pair gives height. Run every candidate point through the same filter:

Resist adding points "just in case": each one costs labeling time on every instance, and points nobody reads only add noise to the metric. Public datasets show the range: Tiger-Pose uses 12 points without visibility, Dog-Pose 24, and Hand Keypoints 21:

Write the dataset YAML#

path: forklift-pose
train: images/train
val: images/val

kpt_shape: [4, 3] # 4 keypoints, each x, y, visibility
flip_idx: [1, 0, 2, 3] # fork tips swap, mast points stay
kpt_oks_sigmas: [0.05, 0.05, 0.08, 0.08] # tips are sharp, mast ends are fuzzier

names:
  0: forklift

kpt_names:
  0:
    - left_fork_tip
    - right_fork_tip
    - mast_top
    - mast_base

• kpt_names documents the order for people and tools, per class.
• flip_idx must undo itself: applying it twice returns every index to itself.
• kpt_oks_sigmas is optional and weights both the keypoint loss in training and OKS in validation; if present, it must hold one positive value per keypoint or the run stops with a ValueError.

On Ultralytics Platform, the same schema is a skeleton template. A new template can start from an existing one (Start from…), so you rename and trim points instead of placing every keypoint by hand — keep the index order from your schema table:

Check labels before training#

Pose datasets fail quietly: a row missing one keypoint shifts every later value, and a visibility of 3 from a buggy exporter is read as a number like any other. This checker loads the YAML through Ultralytics' own loader and validates every row. It runs as-is on COCO8-pose; swap in your YAML path:

from pathlib import Path

from ultralytics.data.utils import check_det_dataset

data = check_det_dataset("coco8-pose.yaml")  # resolves paths, downloads if needed
k, dims = data["kpt_shape"]
flip = data.get("flip_idx", [])
assert not flip or sorted(flip) == list(range(k)), "flip_idx must list every keypoint index once"
assert all(flip[flip[i]] == i for i in range(len(flip))), "flip_idx must undo itself: pairs swap, center points stay"

expected = 5 + k * dims  # class + 4 box values + k keypoints of `dims` values
bad = 0
for label in Path(data["path"], "labels").rglob("*.txt"):
    for n, row in enumerate(label.read_text().splitlines(), 1):
        values = row.split()
        if len(values) != expected:
            bad += 1
            print(f"{label.name}:{n} has {len(values)} values, expected {expected}")
        elif dims == 3 and not {float(v) for v in values[7::3]} <= {0, 1, 2}:
            bad += 1
            print(f"{label.name}:{n} has a visibility flag outside 0/1/2")
print(f"kpt_shape={data['kpt_shape']}, flip_idx ok, {bad} bad rows")

kpt_shape=[17, 3], flip_idx ok, 0 bad rows

values[7::3] starts at the first visibility flag (class, four box values, then x y v) and takes every third value after it.

How OKS scores a keypoint#

Pose mAP replaces box IoU with Object Keypoint Similarity (OKS). For each labeled keypoint, OKS turns the pixel distance between prediction and label into a score from 0 to 1, scaled by the object's size and by that keypoint's sigma, then averages over the labeled keypoints. Pose mAP50-95 averages precision at OKS thresholds from 0.50 to 0.95, exactly like box mAP over IoU thresholds; the performance metrics guide covers the shared machinery.

Sigmas are easiest to understand by measuring them. This script scores the same forklift prediction with two sigmas, using the kpt_iou function the pose validator calls:

import torch

from ultralytics.utils.metrics import kpt_iou

# One labeled forklift, 4 keypoints (x, y, visibility) in pixels, inside a 200 x 120 px box
label = torch.tensor([[[40.0, 30, 2], [40, 90, 2], [170, 20, 2], [170, 100, 2]]])
area = torch.tensor([200 * 120 * 0.53])  # Ultralytics scales box area by 0.53 to approximate object area

for offset in (2, 5, 10, 20):
    pred = label.clone()
    pred[..., 0] += offset  # every predicted keypoint lands `offset` pixels to the right
    sharp = kpt_iou(label, pred, area, sigma=[0.025] * 4).item()
    loose = kpt_iou(label, pred, area, sigma=[0.10] * 4).item()
    print(f"{offset:2d} px off: OKS {sharp:.2f} with sigma 0.025, {loose:.2f} with sigma 0.10")

 2 px off: OKS 0.94 with sigma 0.025, 1.00 with sigma 0.10
 5 px off: OKS 0.67 with sigma 0.025, 0.98 with sigma 0.10
10 px off: OKS 0.21 with sigma 0.025, 0.91 with sigma 0.10
20 px off: OKS 0.00 with sigma 0.025, 0.67 with sigma 0.10

On a 200 × 120-pixel forklift, a prediction 10 pixels off fails the 0.50 OKS threshold with sigma 0.025 and passes every threshold up to 0.90 with sigma 0.10. Same model, same error, very different pose mAP.

Sigmas should reflect how consistently annotators place each point. Measure it: have two people label the same 50 images and compare per-keypoint distances. Points they disagree on deserve larger sigmas; otherwise the metric punishes the model for human noise.