Semantic Segmentation: Label Every Pixel

Instance or semantic?#

Predict a class map#

result.semantic_mask.data is an H × W tensor at the original image size; each value is a class ID and 255 means ignore. result.summary() turns it into per-class pixel ratios:

from ultralytics import YOLO

model = YOLO("yolo26n-sem.pt")  # Cityscapes, 19 classes
result = model("https://ultralytics.com/images/bus.jpg")[0]

class_map = result.semantic_mask.data  # H x W tensor of class ids, 255 = ignore
print("class map:", tuple(class_map.shape), class_map.dtype)
for row in result.summary():
    if row["pixel_ratio"] >= 0.02:
        print(f"{row['name']:12s} {row['pixel_ratio']:.1%}")

class map: (1080, 810) torch.uint8
road         27.5%
sidewalk     2.0%
building     19.3%
vegetation   4.5%
person       14.6%
train        30.9%

Look at the last row. Cityscapes has a bus class, yet the model calls the bus a train. It learned from car-mounted footage of mostly German streets, and a bus photographed from the sidewalk at close range is outside that distribution. Semantic models inherit their training set's viewpoint and class definitions — its dataset bias — more rigidly than detectors do, so fine-tune on your own camera before trusting the map.

Turn the class map into an answer#

A class map becomes a product number with plain tensor comparisons. Here is "how much of the area a pedestrian would walk into is road or sidewalk, and how much is people?":

from ultralytics import YOLO

result = YOLO("yolo26n-sem.pt")("https://ultralytics.com/images/bus.jpg", verbose=False)[0]
class_map = result.semantic_mask.data  # H x W class ids at the original size, 255 = ignore
ids = {name: i for i, name in result.names.items()}

h = class_map.shape[0]
lower = class_map[h // 2 :]  # the half of the frame a pedestrian would walk into
walkable = (lower == ids["road"]) | (lower == ids["sidewalk"])
people = lower == ids["person"]
print(f"lower half: {walkable.float().mean():.0%} road or sidewalk, {people.float().mean():.0%} person")
result.save("semantic_overlay.jpg")  # colored class map over the photo, for a visual check

lower half: 59% road or sidewalk, 23% person

Three habits make these numbers trustworthy:

• Look up IDs by name from result.names, never by hard-coded number; a model trained on your own classes will number them differently.
• Restrict to the region that matters. Whole-frame ratios mix in sky and buildings; slice the class map (or multiply by a region mask) first.
• Open the overlay. semantic_overlay.jpg shows in one glance whether the "road" the number counts is really road.

The dataset format#

Semantic datasets pair each image with a single-channel mask image whose pixel values are class IDs. The semantic dataset guide is the reference; the layout mirrors images/ file for file:

cityscapes8/
├── images/
│   ├── train/aachen_000000_000019.png
│   └── val/aachen_000004_000019.png
└── masks/
    ├── train/aachen_000000_000019.png
    └── val/aachen_000004_000019.png

The YAML adds masks_dir and, when the IDs stored in your masks differ from the training IDs, a label_mapping block:

path: cityscapes8
train: images/train
val: images/val
masks_dir: masks # semantic mask directory

names:
  0: road
  1: sidewalk
  # ... 19 classes in total

label_mapping: # raw mask value -> training id
  0: ignore_label
  7: 0 # road
  8: 1 # sidewalk
  # ...

Cityscapes masks store raw label IDs from -1 to 33; the mapping keeps 19 of them and sends the rest to ignore_label, stored as 255. Pixels set to 255 are excluded from both the loss and the metrics — use the same value for regions your annotators could not decide on.

If you already have polygon labels and no masks_dir, the loader rasterizes the polygons for you and adds a background class for uncovered pixels.

Train and read mIoU#

Cityscapes8 has 8 urban-scene images, 4 for training and 4 for validation, enough for a CPU smoke test:

from ultralytics import YOLO

model = YOLO("yolo26n-sem.pt")
model.train(data="cityscapes8.yaml", epochs=1, imgsz=320, device="cpu")
metrics = model.val()
print(f"mIoU: {metrics.miou:.3f}  pixel accuracy: {metrics.pixel_accuracy:.3f}")

                 Class     Images     Pixels       mIoU     PixAcc
                   all          4     192315      0.415      0.906
                  road          4      78529      0.974      0.996
            vegetation          4      37602      0.844      0.936
                person          3        722          0          0
                 rider          1         21          0          0
mIoU: 0.415  pixel accuracy: 0.906

(Per-class rows trimmed.) Pixel accuracy says 91%; mIoU says 42%; the per-class rows say the model never found a single person. All three are true. Road, building, and vegetation own most of the pixels — road alone is 78,529 of 192,315 — so a model that paints them well scores high pixel accuracy while missing every small class. mIoU averages per-class IoU with equal weight, so each missed class drags it down.