Resume and Iterate
Resume an interrupted run exactly, warm-start a new round from best.pt with a learning rate that actually applies, and compare runs side by side.
Real projects are a series of training runs. Some die halfway — a closed laptop, a dropped SSH session, a preempted cloud GPU — and restarting from zero throws away hours. Others finish, and the next round should build on them instead of starting from COCO again.
In this lesson you'll interrupt a run and resume it, warm-start round two from best.pt with a learning rate that is actually applied, and compare every run in one table. You'll also learn the two resume behaviors that quietly waste runs.
Hands-on
Pick the Right Kind of Run#

| Situation | Run type | Starts from |
|---|---|---|
| Power cut, disconnect, or out-of-memory partway through | Resume | last.pt, with its optimizer state and schedule |
| Finished run; more data, a labeling fix, or longer training | Warm start | best.pt as a pretrained model |
Classes added or removed, many labels changed, or imgsz changed a lot | Fresh | yolo26n.pt |
Worked Example: Resume an Interrupted Run#
resume=True restores the model, optimizer state, learning-rate schedule, and epoch counter from last.pt, then continues the original plan:
yolo train resume=True model=runs/detect/pills_resume/weights/last.ptReal output from a 5-epoch run stopped after epoch 2:
Resuming training runs/detect/pills_resume/weights/last.pt from epoch 3 to 5 total epochs
3/5 0G 1.232 2.393 0.005478 206 320: 100% 6/6 4.5s
4/5 0G 1.345 1.859 0.005833 208 320: 100% 6/6 4.5s
5/5 0G 1.31 1.625 0.005854 191 320: 100% 6/6 4.4s
3 epochs completed in 0.004 hours.from epoch 3 to 5 total epochs: it picked up exactly after the last saved epoch, with the original target.- You didn't pass
dataorepochs: every argument came from the checkpoint. A few runtime settings such asbatch,device, andworkerscan be overridden on resume; anything else is ignored with aResume ignores ...warning.
Worked Example: Warm-Start Round Two#
Most "next round" experiments aren't resumes; they're new runs that start from your last good model — plain transfer learning within your own project. Use a lower learning rate so the model refines rather than relearns:
from ultralytics import YOLO
model = YOLO("runs/detect/pills_v1/weights/best.pt") # warm start from round 1
model.train(
data="my_dataset/data.yaml",
epochs=5,
imgsz=320,
optimizer="AdamW", # set explicitly: optimizer="auto" ignores lr0
lr0=0.0005, # gentler than round 1's auto-selected 0.002
name="pills_v2_warm",
)The log confirms the setting took effect: optimizer: AdamW(lr=0.0005, momentum=0.937). The fine-tuning guide covers layer freezing and two-stage training; the hyperparameter tuning guide covers sweeps.
Worked Example: Compare Runs#
Name every run after what changed — pills_v2_warm, pills_v3_imgsz640, pills_v4_edge_frames — never train2 or final. Each run writes a results.csv, so a comparison takes a few lines:
from pathlib import Path
import pandas as pd
for csv in sorted(Path("runs/detect").glob("*/results.csv")):
df = pd.read_csv(csv)
best = df["metrics/mAP50-95(B)"].max()
print(f"{csv.parent.name:<14} epochs={len(df):<3} best mAP50-95={best:.3f}")pills_resume epochs=5 best mAP50-95=0.579
pills_v1 epochs=10 best mAP50-95=0.659
pills_v2_warm epochs=5 best mAP50-95=0.694Five warm-started epochs lifted mAP50-95 from 0.659 to 0.694, while five epochs from COCO weights (pills_resume) reached only 0.579. Warm-starting is cheaper progress — as long as the classes and labels haven't changed underneath it. Past a handful of runs, a tracker pays for itself: Ultralytics Platform (see Compare Runs and Pick a Winner), or the Comet and Weights & Biases integrations, log every run automatically.
Keep Each Round Cheap#
Most rounds answer one question, such as "did the new frames help?". These arguments buy faster answers while you explore:
| Argument | Example | Effect |
|---|---|---|
fraction | fraction=0.5 | Trains on the first half of the training images (sorted by file name) |
time | time=0.25 | Stops after 0.25 hours, overriding epochs |
patience | patience=10 | Stops when validation fitness hasn't improved for 10 epochs |
imgsz | imgsz=320 | A quarter of the pixels of 640; compare runs only at the same size |
cache | cache=True | Keeps decoded images in RAM so later epochs load faster |
Change one thing per round, and confirm the winner with a full-size run before you ship it. When you do need a fresh start, it's the same call with the pretrained weights:
from ultralytics import YOLO
model = YOLO("yolo26n.pt") # back to the COCO-pretrained weights
model.train(data="my_dataset/data.yaml", epochs=10, imgsz=320, name="pills_v3_relabeled")Avoid These Mistakes#
Symptom: WARNING ⚠️ model '.../last.pt' is not a resumable training checkpoint ... Starting new training instead, followed by a brand-new run — on the default coco8 dataset unless you passed data. Cause: resume=True only continues runs that stopped early. Fix: stop it, and start a new run from best.pt on purpose.
Symptom: optimizer: 'optimizer=auto' found, ignoring 'lr0=0.0005' and determining best 'optimizer' and 'lr0' automatically... Cause: the default optimizer=auto picks its own learning rate. Fix: set optimizer explicitly (for example "AdamW" or "SGD") whenever you set lr0, and confirm the optimizer: log line.
Training with name="pills_v1" a second time writes to pills_v1-2, which is easy to mix up later. Use a new, descriptive name for every run.
Show solution
If Ctrl+C is hard to time on a fast machine, a callback can stop the run for you after epoch 2:
from ultralytics import YOLO
def power_cut(trainer):
if trainer.epoch == 1: # 0-indexed: stop after epoch 2 is saved
raise KeyboardInterrupt
model = YOLO("yolo26n.pt")
model.add_callback("on_model_save", power_cut)
model.train(data="my_dataset/data.yaml", epochs=5, imgsz=320, name="pills_resume")yolo train resume=True model=runs/detect/pills_resume/weights/last.pt
python round2.py
python compare_runs.pypills_v2_warm has the best mAP50-95 (0.694 vs 0.659 for pills_v1), with the same
classes and labels, so round three warm-starts from pills_v2_warm/weights/best.pt.