Automate the Retraining Loop
Turn one deployment into a repeatable loop: a runbook, a scripted promotion gate, an automated release with rollback, and a weekly review.
The first deployment is a project; every one after it is a process. Teams that script the loop retrain in an afternoon whenever the holdout alert fires; teams that don't rebuild it from memory under pressure, and ship regressions. This lesson runs the loop once end to end on the pill model, fine-tune, gate, and release, and writes the runbook that makes the next time routine.
Hands-on
The Loop, With an Owner for Every Step#
Lineage comes free if you used the earlier lessons: a model's Run Information card names its parent model and pinned dataset version, the dataset's Models tab lists every model trained on each version, and the deployment page links to the model it serves. When accuracy drops, start at the deployment and walk backwards.
Agents automates the collect step. Collect Uncertain Detections predicts through your deployment and copies images with detections in a confidence range into a review dataset; Capture and Alert saves matching images and posts to Slack. Workflows run when you click Run, so make a run part of the weekly review.
Worked Example: One Full Cycle on the Pill Model#
1. Fine-tune from the deployed model. On Platform, pick the deployed model under My Models, enable Save Dataset Version, and set the optimizer explicitly. Locally, the same step is:
from ultralytics import YOLO
model = YOLO("pills-v1.pt") # the deployed model's best.pt
model.train(data="medical-pills.yaml", epochs=5, imgsz=320, optimizer="SGD", lr0=0.001, name="v2-finetune")This is the fine-tune from Train, Read, and Fine-Tune Runs: mAP50-95 0.659 → 0.687.
2. Gate it against the deployed model. Reuse gate.py from lesson 6 with the holdout and the two checkpoints:
DATA = "medical-pills.yaml" # your holdout dataset YAML or ul:// URI
BASELINE, CANDIDATE = "pills-v1.pt", "pills-v2.pt" # deployed model vs retrained candidatePASS mAP50-95 0.659 -> 0.687 beats noise
PASS every class keeps 90% of its AP (weak: none)
PASS inference 4.0 ms within 100 ms
verdict: ship the candidateThe gate exits 0, so CI may continue to the release step. (In this demo the medical-pills val split stands in for a holdout; in production, use a holdout version the model never saw.)
3. Release it. Upload or train the candidate into the same workspace, then let CI call Replace model through the ultralytics-platform SDK and confirm what's serving:
import time
from ultralytics_platform import Platform
OWNER, DEPLOYMENT = "your-username", "pill-detector-st-ghislain"
PROJECT, NEW_MODEL = "pill-inspection", "v2-finetune"
with Platform() as client: # reads ULTRALYTICS_API_KEY
body = {"action": "replace", "project": PROJECT, "model": NEW_MODEL}
reply = client.deployments.update(OWNER, DEPLOYMENT, body=body)
print(reply["status"], "-", reply["message"]) # "deploying" while the new model starts
while (deployment := client.deployments.retrieve(OWNER, DEPLOYMENT)["deployment"])["status"] == "deploying":
time.sleep(15)
print(f"status={deployment['status']} model={deployment.get('model')}")
if deployment.get("model") != NEW_MODEL: # a failed rollout leaves the previous model serving
raise SystemExit("Replacement failed; the previous model is still live")Against a stubbed API that reports deploying once and then ready, it prints:
deploying - Replacement model is deploying
status=ready model=v2-finetunedeployments.update sends PATCH /api/deployments/{owner}/{deployment} with action: "replace"; the old model serves until the new one is ready. The loop polls retrieve until the status leaves deploying, then checks the serving model, because a failed rollout also ends ready, just with the old model. Rolling back is the same script with NEW_MODEL set to the previous version. The ul cloud command-line tool that ships with ultralytics wraps the same API, and the API reference documents every call.
4. Reset the baseline. Run holdout_check.py from lesson 8 with the new checkpoint and a fresh history file.
Write the Runbook#
# Retraining runbook: pill-detector
## Trigger
- Holdout mAP50-95 more than 1 point below baseline for 2 consecutive weeks, OR
- Batch Annotation edit rate above 25% on a new batch.
## Steps
1. Collect: Save to dataset from Monitoring, plus the "Collect Uncertain Detections" agent into pill-review.
2. Label: Batch Annotation with the deployed model at the swept threshold; review every image; record the edit rate.
3. Merge into the training dataset; rerun the duplicate check, class audit, and Clustering.
4. Create dataset version v(n+1) with a description of what changed.
5. Fine-tune from the deployed model with Save Dataset Version on; optimizer SGD, lr0 0.001.
6. Run gate.py against the deployed model on the holdout. Stop here on any FAIL.
7. ML lead approves; CI runs release.py.
8. Watch endpoint_check.py for 24 hours; start a new holdout baseline.
## Rollback
- Run release.py with NEW_MODEL set to the previous version. Never delete that model while it's the rollback target.
## Owner
- ML lead approves step 7. On-call runs steps 1-6 and 8.Keep a 15-minute weekly review on the calendar with two questions: is anything red (endpoint check, holdout, edit rate), and did anything change in the deployment, and why? Most weeks both answers are no, and the meeting takes five minutes. A typical quarter looks like this:
Week 1 : Holdout alert. Collect, label, retrain, gate, release v(n+1).
Week 2-3 : Quiet. Checks green.
Week 4 : New defect class requested. Label, new version, retrain v(n+2).
Week 5-7 : Quiet.
Week 8 : Second plant in Poland. Check regions; add an endpoint in europe-central2.
Week 9-11: Quiet.
Week 12 : Quarterly holdout refresh and review.Pitfalls#
Symptom: a release goes wrong, and the previous model is gone. Fix: keep at least the last two production models in the project. Deleting a model moves it to Trash, and a permanently deleted model also removes any deployment still serving it.
Symptom: you planned a 10%/50%/100% canary, and Replace model switches all traffic at once. Fix: endpoints serve one model. For a canary, deploy the candidate to a second endpoint and split traffic in your own router, then replace and delete the second endpoint.
Symptom: every candidate passes the gate, and production still regresses. Fix: the gate is only honest on a holdout version that no run trained on. Keep the holdout in its own private dataset and never merge it into training data.
Show solution
Worked cycle for the pill line. Trigger: the night-shift holdout check dropped 0.05. Collect: 180 evening-shift Monitoring examples saved to pill-review plus 120 from the Collect Uncertain Detections agent; Batch Annotation at 0.35 with the deployed v2-finetune, edit rate 18%; reviewed and merged; audits clean; version v6: +300 night-shift frames. Fine-tune v3-night from v2-finetune, SGD, lr0 0.001, 20 epochs. Gate on the 300-image holdout, including 100 night images: mAP50-95 0.66 → 0.71, no weak classes, 4 ms: PASS. release.py with NEW_MODEL = "v3-night" reports status=ready model=v3-night; holdout_check.py writes a new baseline of 0.71. v2-finetune stays in the project as the rollback target, and the runbook, gate, and release script are committed together.