Published Nov 08, 2026 - Author: Guoxiang (Shawn) Zhou
Training a detector that reaches a good mean average precision (mAP) on a benchmark takes an afternoon. Getting the same detector to run on a robot, at a fixed rate, inside a power budget, on images from cameras and places it has never seen, and to keep it working as the fleet grows, is most of the work. Machine Learning Operations (MLOps) is the name for that work: the data, tooling and processes that take a model from training to production and keep it healthy there.
Robotics adds constraints that web MLOps does not have:
This post walks through the full workflow with one running example: deploy a 2D object detector (a You Only Look Once, or YOLO-family, model) on a Robot Operating System 2 (ROS 2) robot with an NVIDIA Jetson, and turn its 2D boxes into 3D obstacle positions for the planner.
The top row runs offline. The bottom row runs on the robot. The dashed green arrow is what makes it MLOps instead of a one-off project: failures seen in the field become training data for the next model, the so-called data flywheel.
Write the requirements down before collecting a single image. They decide the model size, the input resolution and what “good enough” means.
| Question | Example answer |
|---|---|
| What classes, and why? |
person, forklift, pallet, door; persons trigger a speed limit, the rest feed the costmap |
| Which sensors? | Front and rear RGB-D (color plus depth) cameras, 1280x720, 15 frames per second (fps) |
| Rate and latency? | 10 Hz per camera, under 100 ms photon to message |
| Compute share? | At most 30% of the Orin GPU, the rest belongs to SLAM and the elevation map |
| Operating domain? | Indoor industrial halls, outdoor yards, day and night, rain |
| Acceptance metric? | Person recall above 0.95 within 5 m, mAP@0.5 above 0.80 on the site test set |
| Failure behavior? | If the detector stops publishing, the robot falls back to geometric obstacle detection |
The last two rows matter most. A single headline mAP hides the cases that hurt; the acceptance metric should be tied to the safety function the model serves. And the rest of the stack must not trust the model blindly: a learned detector is an additional signal on top of geometric perception, not a replacement for it.
The best training data comes from the target cameras mounted on the target robot, because it has the right lens distortion, exposure behavior, mounting height and motion blur. With ROS this means recording rosbags and extracting frames:
ros2 bag record /front_camera/color/image_raw /front_camera/color/camera_info /tf /tf_static
Three rules save a lot of trouble later:
Start from a model pretrained on a large public dataset such as Common Objects in Context (COCO), which already covers person and many everyday objects. For classes that are rare or dangerous to stage, synthetic data helps: a simulator such as Isaac Sim with Replicator renders scenes and writes perfect labels, and domain randomization varies lighting, textures, object poses, camera pose and backgrounds so the model learns the object instead of the simulator’s look. NVIDIA reports an off-the-shelf detector trained on randomized synthetic indoor scenes that detects doors in a real building. Studies that compare real-only, synthetic-only and mixed training find that synthetic data works best mixed with real data, not as a replacement.
Labeling is usually the most expensive step. A typical setup:
person? Is a half-visible pallet labeled? How tight is a box? Inconsistent labels put a ceiling on mAP that no model can break.Fine-tune a pretrained detector rather than training from scratch. With Ultralytics this is a few lines:
from ultralytics import YOLO
model = YOLO("yolov8s.pt") # COCO-pretrained weights
model.train(
data="robot_dataset.yaml", # paths to train/val splits and class names
epochs=100,
imgsz=640, # must match the inference resolution
batch=32,
)
Choices that matter for a robot:
A predicted box is matched to a ground-truth box by Intersection over Union (IoU):
\[\text{IoU}(A, B) = \frac{\text{area}(A \cap B)}{\text{area}(A \cup B)}.\]A prediction is a true positive (TP) if its IoU with an unmatched ground-truth box of the same class exceeds a threshold, otherwise a false positive (FP). Ground-truth boxes that nothing matched are false negatives (FN).
Two ratios summarize these counts. Precision is $\text{TP} / (\text{TP} + \text{FP})$: of the boxes the model reported, how many were real. Recall is $\text{TP} / (\text{TP} + \text{FN})$: of the real objects, how many the model found.
Every prediction carries a confidence score, and only predictions above a confidence threshold are kept. Sweeping that threshold trades precision against recall and traces a precision-recall curve. This is an evaluation step, not part of training: the model runs once on the test set with a very low threshold, the detections are sorted by confidence, and each cut-off point in that list gives one point on the curve. The trade-off comes from what moves when the threshold changes. The denominator of recall, $\text{TP} + \text{FN}$, is the number of ground-truth objects, which is fixed. So lowering the threshold can only keep recall the same or raise it: every extra prediction either finds another object or adds nothing. The denominator of precision is the number of kept predictions, which grows as the threshold drops. A well-trained detector gives its highest scores to its most obvious, mostly correct detections. As the threshold falls, it admits less certain predictions, and a growing share of them are background clutter or duplicate boxes, so FP grows faster than TP and precision falls. At a very high threshold the model reports only a few confident boxes: precision is high, but most objects are missed. At a very low threshold it reports almost everything: recall approaches its maximum, and the false positives pull precision down. The curve is not perfectly smooth, because precision jumps up briefly each time one more true positive is admitted. So before taking the area under the curve, benchmarks first smooth out these zigzags: precision at each recall is replaced by the best precision at that recall or higher, which turns the curve into a staircase that never goes up. AP is the area under that staircase.
Average precision (AP) is the area under it, computed per class, and mAP is the mean over classes. PASCAL Visual Object Classes (VOC) uses a single IoU threshold of 0.5 (mAP@0.5). COCO averages over ten thresholds from 0.50 to 0.95 (mAP@0.5:0.95), which also rewards tight boxes. A model with high mAP@0.5 but low mAP@0.75 finds objects but localizes them loosely; for a robot that converts boxes into 3D positions, loose boxes mean wrong distances.
mAP on the whole test set is necessary but not sufficient. Also measure:
If the model is not good enough, the fix is almost always in the data (Step 2 and 3), not in the architecture. This is the red arrow in the lifecycle figure.
A PyTorch checkpoint needs Python and the PyTorch runtime, and it does not use the edge GPU efficiently. The standard path on NVIDIA hardware is PyTorch to Open Neural Network Exchange (ONNX) to TensorRT:
Export to ONNX. ONNX is a framework-neutral graph format. Fix the input shape to the deployment resolution:
from ultralytics import YOLO
YOLO("runs/detect/train/weights/best.pt").export(format="onnx", imgsz=640, opset=17, simplify=True)
Build a TensorRT engine on the target. TensorRT fuses layers, picks the fastest kernel for each layer on this exact GPU and lowers the precision. The engine is tied to the GPU model and the TensorRT version, so build it on the Jetson (or in a continuous integration (CI) job on identical hardware), never copy it from the workstation:
# on the Jetson
trtexec --onnx=model.onnx --saveEngine=model_fp16.engine --fp16
Ultralytics can also export an engine directly with format="engine"; check the export docs of your version for the precision argument.
Choose the precision. The options are 32-bit floating point (FP32), 16-bit floating point (FP16) and 8-bit integer (INT8). INT8 comes in two flavors: post-training quantization (PTQ) and quantization-aware training (QAT).
| Precision | Speed | Accuracy | What it needs |
|---|---|---|---|
| FP32 | baseline | reference | nothing |
| FP16 | large speedup on Tensor Cores | usually indistinguishable | nothing, start here |
| INT8 (PTQ) | faster again, less memory | often a small drop, model dependent | a calibration set of a few hundred representative images |
| INT8 (QAT) | same as PTQ if the quantize/dequantize (Q/DQ) nodes fuse | recovers most of the PTQ drop | retraining with fake quantization |
For INT8 post-training quantization, TensorRT runs the calibration images through the network, records the activation range of each layer and picks the scale that loses the least information. The calibration images must come from the deployment distribution: calibrating on daylight images and running at night is a classic accuracy loss. Accuracy-sensitive parts, typically the detection head, can be kept in FP16 while the backbone runs in INT8.
Verify after every conversion. Run the same images through PyTorch, ONNX and TensorRT and compare boxes and scores, then rerun the mAP evaluation with the engine. The most common bug is not numerical: it is a preprocessing mismatch (blue-green-red (BGR) versus red-green-blue (RGB) channel order, letterbox versus stretch resize, 0 to 1 versus 0 to 255 normalization) between the training code and the robot code.
On the robot, inference becomes a set of ROS 2 nodes. NVIDIA’s Isaac ROS splits it into three stages, which is a good structure even without Isaac ROS:
vision_msgs/Detection2DArray.Isaac ROS uses NVIDIA Isaac Transport for ROS (NITROS) to pass images between these nodes on the GPU without copying them to the central processing unit (CPU), which matters at high resolution. Without it, run the stages as components in one process so ROS 2 can use intra-process communication.
A minimal decoder publishing standard messages looks like this. Note the header: copying the image timestamp and frame is what lets downstream nodes look up the correct transform.
from vision_msgs.msg import Detection2D, Detection2DArray, ObjectHypothesisWithPose
def to_msg(image_msg, boxes, scores, class_ids, class_names):
out = Detection2DArray()
out.header = image_msg.header # same stamp and camera frame as the image
for (x1, y1, x2, y2), score, cid in zip(boxes, scores, class_ids):
det = Detection2D()
det.header = image_msg.header
det.bbox.center.position.x = (x1 + x2) / 2.0
det.bbox.center.position.y = (y1 + y2) / 2.0
det.bbox.size_x = float(x2 - x1)
det.bbox.size_y = float(y2 - y1)
hyp = ObjectHypothesisWithPose()
hyp.hypothesis.class_id = class_names[cid]
hyp.hypothesis.score = float(score)
det.results.append(hyp)
out.detections.append(det)
return out
A detector does not output one box per object. It scores thousands of candidate boxes (one per anchor or grid cell), and every cell near an object predicts a slightly shifted box around it with a high score. Without post-processing, one person becomes five overlapping detections. NMS keeps the best box of each cluster, that is, of each object, and drops the rest:
The assumption is that two boxes overlapping that much describe the same object, so the lower-scored one is a duplicate. Boxes on other objects barely overlap the kept box and survive step 4; the loop exists for them. Each pass keeps the best box of one object, so the result is one box per object, not one box per image. NMS runs per class (class-aware), so a person standing in front of a pallet does not suppress the pallet.
import numpy as np
def nms(boxes, scores, iou_thresh=0.5):
"""@brief Greedy non-maximum suppression for one class.
@param boxes (N, 4) array of [x1, y1, x2, y2].
@param scores (N,) array of confidences.
@param iou_thresh Boxes overlapping a kept box by more than this are removed.
@return Indices of the kept boxes, highest score first.
"""
x1, y1, x2, y2 = boxes.T
area = (x2 - x1) * (y2 - y1)
order = scores.argsort()[::-1]
keep = []
while order.size > 0:
i, rest = order[0], order[1:]
keep.append(i)
w = np.clip(np.minimum(x2[i], x2[rest]) - np.maximum(x1[i], x1[rest]), 0, None)
h = np.clip(np.minimum(y2[i], y2[rest]) - np.maximum(y1[i], y1[rest]), 0, None)
iou = w * h / (area[i] + area[rest] - w * h)
order = rest[iou <= iou_thresh]
return keep
The IoU threshold is a trade-off with the same shape as the confidence threshold. Too high, and duplicates survive: each extra box is a false positive, and a tracker may spawn two tracks for one person. Too low, and genuinely separate objects that overlap in the image, such as people walking side by side, are merged into one, which is a false negative that no later stage can recover. Crowded scenes are where greedy NMS fails; Soft-NMS decays the scores of overlapping boxes instead of deleting them, which helps there at a small cost.
Practical consequences for deployment:
EfficientNMS_TRT) that can be appended to the engine during export, so the engine outputs final boxes.A planner needs positions in the map, not pixels. With a depth image aligned to the color image and the pinhole intrinsics $(f_x, f_y, c_x, c_y)$ from camera_info, a pixel $(u, v)$ with depth $Z$ back-projects to the camera frame as
Then transform the point with TF (the ROS transform library) from the camera frame into odom or map at the image timestamp. Practical details:
PoseWithCovariance in Detection3DArray so the tracker can weight it.The network is usually only a third of the end-to-end latency. Measure the whole chain with timestamps: the image header stamp is the capture time, so now - header.stamp at the consumer is the true age of a detection. Also check that the GPU share fits next to the other GPU users, and that the Jetson power mode and thermal throttling do not change the numbers after twenty minutes of operation.
The model’s behavior depends on more than its weights. Ship them together and version them together:
Embed the model version in the output, for example in a diagnostics topic, so every detection in a recording can be traced to the exact model that produced it. In practice the bundle lives in a container image or a versioned package deployed by the fleet’s update system.
Once deployed, the model’s input distribution keeps changing: new sites, new seasons, a dirty lens, a new type of forklift. Nobody labels production data, so monitoring relies on proxies:
These signals become triggers that save a short clip of sensor data around the event, since uploading everything is not possible. The saved clips go back to Step 2: they are labeled with priority, added to the training set and, the important part, to a growing regression set of known hard cases. This is hard example mining, and selecting frames where the model was uncertain is active learning. Tesla called the same loop its data engine: cases where the network performed poorly are sourced from the fleet, labeled and added to training, and each new model then meets harder cases.
| Stage | Typical tools |
|---|---|
| Data recording |
ros2 bag, on-robot trigger logic, MCAP (the default ROS 2 bag file format) |
| Synthetic data | Isaac Sim + Replicator, domain randomization |
| Labeling | CVAT, model-assisted pre-labels, written labeling guide |
| Data versioning and browsing | DVC, FiftyOne |
| Training and tracking | Ultralytics / PyTorch, MLflow, Weights & Biases or ClearML |
| Export and optimization | ONNX, TensorRT (trtexec), FP16 or INT8 with calibration |
| On-robot inference | Isaac ROS (encoder, TensorRT or Triton node, decoder), or a custom node with TensorRT or ONNX Runtime |
| Messages |
vision_msgs/Detection2DArray, Detection3DArray, TF |
| Deployment | Containers, fleet over-the-air (OTA) update with shadow, canary and rollback |
| Monitoring | Latency and output statistics, drift proxies, hard example mining |
The model architecture is the part that changes least from one robot to the next. Most of the effort goes into the data loop around it, the conversion and integration, and the guard rails that keep a statistical component safe on a machine that moves.