Cost and Latency Tuning

Measure the Whole Pipeline First#

Every Results object splits its own time into stages:

from ultralytics import YOLO

model = YOLO("yolo26n.pt")
video = "https://github.com/ultralytics/assets/releases/download/v0.0.0/solutions_ci_demo.mp4"
totals = {"preprocess": 0.0, "inference": 0.0, "postprocess": 0.0}
for n, r in enumerate(model.predict(video, stream=True, verbose=False), 1):
    for stage in totals:
        totals[stage] += r.speed[stage]
print({stage: round(ms / n, 1) for stage, ms in totals.items()}, "ms per frame")

{'preprocess': 0.7, 'inference': 36.1, 'postprocess': 0.7} ms per frame

Here inference is 95% of the model's time, so model levers will help. Time decode and your own sinks the same way (time.perf_counter() around each), then apply the rule for what you find:

The Lever Leaderboard#

Try from the top and stop when you meet the budget:

The numbers come from the YOLO26, TensorRT, and OpenVINO benchmark tables and the measurements below; yours will differ, so measure.

Smaller Model#

The YOLO26 benchmark table on COCO:

On CPU each step down is 2.2–2.5× faster. The COCO accuracy gap is the worst case: a 3-class warehouse detector with large objects usually loses far less from m → n than an 80-class general detector, so the smaller model is always worth testing on your own val set.

Smaller Input#

Compute scales roughly with pixel count, so imgsz=480 needs about 56% of the work of 640. Measured on the test laptop's CPU with ONNX Runtime at batch 1 (median of 25 runs on bus.jpg):

Decision rule: if your smallest important object still spans about 10 pixels or more at the new size, try it; if objects are small or distant (drones, shelves far from the camera), keep the resolution and shrink the model instead.

Fewer Frames#

If inference bounds your frame rate and the scene moves slowly, run detection on every Nth frame with vid_stride=N and let the tracker bridge the gaps:

import time

from ultralytics import YOLO

model = YOLO("yolo26n.pt")
video = "https://github.com/ultralytics/assets/releases/download/v0.0.0/solutions_ci_demo.mp4"
model.predict(video, vid_stride=30, verbose=False)  # warm up so the first timing is fair

for stride in (1, 2, 3):
    start, ids, frames = time.perf_counter(), set(), 0
    for r in model.track(video, stream=True, vid_stride=stride, tracker="bytetrack.yaml", classes=[0], verbose=False):
        frames += 1
        if r.boxes.is_track:
            ids.update(r.boxes.id.int().tolist())
    print(f"vid_stride={stride}: {frames} frames in {time.perf_counter() - start:.2f} s, {len(ids)} person IDs")

vid_stride=1: 62 frames in 2.66 s, 32 person IDs
vid_stride=2: 31 frames in 1.19 s, 30 person IDs
vid_stride=3: 20 frames in 0.95 s, 28 person IDs

Half the frames cost about half the time, and the tracker kept 30 of 32 identities. At vid_stride=3 it lost 4, which is the cue to stop for this walking-pace scene.

Cut Cloud Cost#

Cost is instance hours divided by useful work. A card like this one has the two numbers you need: what an hour costs ($0.17 for 1 vCPU and 4 GiB here) and how much work it does (59 requests in 24 hours at 226 ms P95, a mostly idle endpoint). Then:

• Raise utilization. A GPU at 30% utilization wastes 70% of its bill. Batch streams (previous lessons) or serve through Triton Inference Server.
• Right-size. Benchmark on a smaller GPU, or on a CPU with OpenVINO, before assuming you need a large one.
• Use spot or preemptible capacity for batch jobs. Reprocessing archived video tolerates interruptions; live streams do not.

Pitfalls#