Pick the Task and Plan the Pipeline

One photo, four different answers#

Ask one question of one photo — how much of this frame is bus, and how far away is it? — and let four Ultralytics YOLO tasks answer it:

from ultralytics import YOLO

image = "https://ultralytics.com/images/bus.jpg"
det = YOLO("yolo26n.pt")(image, verbose=False)[0]
seg = YOLO("yolo26n-seg.pt")(image, retina_masks=True, verbose=False)[0]
sem = YOLO("yolo26n-sem.pt")(image, verbose=False)[0]
depth = YOLO("yolo26n-depth.pt")(image, verbose=False)[0]
frame = det.orig_shape[0] * det.orig_shape[1]  # pixels in the photo

i = [det.names[int(c)] for c in det.boxes.cls].index("bus")
x1, y1, x2, y2 = det.boxes.xyxy[i].tolist()
print(f"detect    bus box covers   {(x2 - x1) * (y2 - y1) / frame:.1%} of the frame")

i = [seg.names[int(c)] for c in seg.boxes.cls].index("bus")
bus_mask = seg.masks.data[i].bool()  # H x W, True on bus pixels
print(f"segment   bus mask covers  {bus_mask.sum().item() / frame:.1%} of the frame")

ratios = {row["name"]: row["pixel_ratio"] for row in sem.summary()}
print(f"semantic  'train' pixels   {ratios['train']:.1%} of the frame")

meters = depth.depth.data[bus_mask].median().item()  # depth map and mask share the original size
print(f"depth     bus is about     {meters:.1f} m from the camera")

detect    bus box covers   47.7% of the frame
segment   bus mask covers  29.2% of the frame
semantic  'train' pixels   30.9% of the frame
depth     bus is about     4.5 m from the camera

Read it line by line:

• detect answers where. The box also contains the road under the bus and the four people standing in front of it, so as an area it is 63% too large.
• segment answers what shape. 29.2% is the bus's real footprint in the image, the number to use for any area, coverage, or overlap rule.
• semantic answers which pixels belong to which class. The area is close to the mask's, but the model calls the bus train: it labels surfaces with the classes of its training set (Cityscapes, lesson 4) and never separates one vehicle from another.
• depth answers how far. The median depth inside the bus mask is 4.5 m — two tasks combined to answer a question neither answers alone.

The seven tasks at a glance#

Computer Vision Foundations covers how to choose between tasks in detail — revisit The Seven Vision Tasks if the choice is not obvious. This is the working recap for the rest of the course. Every row is a task the installed ultralytics package trains, validates, predicts, and exports with the same API; the tasks overview is the full reference.

You already ship the first row. Lessons 2 to 8 teach the five middle rows, and lessons 9 and 10 chain and ship them. The last row, monocular depth estimation, predicts a distance in meters for every pixel of a single image. Its labels come from depth sensors or 3D renderers rather than annotators, so it has its own dataset format; this course includes it wherever the workflow is shared: this plan, the validation loop, and the export check in the last lesson.

Choose one question at a time#

Walk the questions in order and stop at the first yes:

The chart covers the common cases. Four tie-breakers cover most of the rest:

The task is fixed when you create the dataset — on Ultralytics Platform it is the Task Type field of the New Dataset dialog — so this choice comes before the first label:

One API, seven result fields#

Every task uses the same call; the model reads its task from the weights file. What changes is which field of the result comes back filled, and that field is what your code will read:

from ultralytics import YOLO

WEIGHTS = ["yolo26n.pt", "yolo26n-seg.pt", "yolo26n-sem.pt", "yolo26n-pose.pt", "yolo26n-obb.pt", "yolo26n-cls.pt", "yolo26n-depth.pt"]
FIELDS = ("boxes", "masks", "semantic_mask", "keypoints", "obb", "probs", "depth")

for weights in WEIGHTS:
    model = YOLO(weights)
    result = model("https://ultralytics.com/images/bus.jpg", verbose=False)[0]
    fields = [f for f in FIELDS if getattr(result, f) is not None]
    print(f"{weights:17s} task={model.task:9s} fields={fields}")

yolo26n.pt        task=detect    fields=['boxes']
yolo26n-seg.pt    task=segment   fields=['boxes', 'masks']
yolo26n-sem.pt    task=semantic  fields=['semantic_mask']
yolo26n-pose.pt   task=pose      fields=['boxes', 'keypoints']
yolo26n-obb.pt    task=obb       fields=['obb']
yolo26n-cls.pt    task=classify  fields=['probs']
yolo26n-depth.pt  task=depth     fields=['depth']

Segment and pose results keep boxes, so detection code you already have keeps working on them. Classify, semantic, and depth results describe the whole image and have no per-object fields at all.

When one model is not enough#

Most production systems that go beyond detection take one of three shapes:

• Detect, then classify. The detector finds small objects in a large frame; an image classifier makes a fine-grained call on each full-resolution crop: helmet type, defect grade, product SKU.
• Pose plus tracking. Keypoints describe one frame; a tracker turns them into a per-person time series you can count repetitions or detect falls from.
• Segment, then measure. The model returns pixels; plain code turns them into areas, widths, coverage ratios, or (with depth) distances.

The labels behind a segment-then-measure pipeline are one polygon per object, like these parcels on warehouse robots in the Ultralytics Platform editor:

Pitfalls#