I want to tell you about the three weeks I spent making YOLO and ROS 2 talk to each other inside a simulator, and why it was nothing like the tutorials made it look.

This is not a clean architecture post. This is a "here is what broke, here is what I changed, here is what I actually learned" post.

If you are building robotics perception systems, or you are just curious what it looks like to wire YOLOv11, ByteTrack, StrongSORT, CARLA, and ROS 2 into something that runs in real time, this one is for you.

What I was building and why

The goal was a real-time robotic perception system that could:

  • detect objects in a moving vehicle simulation
  • track those objects across frames with stable identities
  • feed structured perception data into a ROS 2 navigation stack
  • run fast enough to be genuinely useful for autonomous decision making

CARLA was the simulator. YOLOv11 was the detector. ByteTrack and StrongSORT handled multi-object tracking. ROS 2 was the middleware. ONNX and TensorRT handled edge optimisation.

On paper this is a well-defined problem. In practice, every one of those components has opinions about data formats, timing, and threading that you only discover by running them together.

Problem one: getting camera data out of CARLA cleanly

CARLA has its own Python API and its own event-driven callback system. ROS 2 has its own message types, publisher model, and executor. Getting them to cooperate without dropping frames or blocking the main thread is not automatic.

Here is a simplified version of how I handled the camera subscription and published frames into ROS 2:

import carla
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import numpy as np

class CARLACameraNode(Node):
    def __init__(self):
        super().__init__('carla_camera_node')
        self.publisher = self.create_publisher(Image, '/carla/camera/rgb', 10)
        self.bridge = CvBridge()

    def camera_callback(self, image):
        array = np.frombuffer(image.raw_data, dtype=np.uint8)
        array = array.reshape((image.height, image.width, 4))
        bgr = array[:, :, :3]
        msg = self.bridge.cv2_to_imgmsg(bgr, encoding='bgr8')
        msg.header.stamp = self.get_clock().now().to_msg()
        self.publisher.publish(msg)

Enter fullscreen mode Exit fullscreen mode

The key thing I got wrong initially: I was not stamping the header with the ROS 2 clock.

That one missing line caused timestamp mismatches downstream that made the tracker behave erratically. ByteTrack and StrongSORT both calculate displacement and velocity across frames using time deltas. If those deltas are wrong or inconsistent, the association algorithm produces noisy, unstable tracks that look like a model problem but are actually a data problem.

That bug cost me two days.

Add the timestamp. Always. It is not optional in robotics.

Problem two: YOLO runs fast, ROS 2 callbacks do not wait

YOLOv11 at a reasonable resolution on a GPU is quick. But "quick" in model benchmarking terms and "quick" in robotics middleware terms are genuinely different things.

ROS 2 uses a callback-based execution model. If your inference call is blocking the subscription callback thread, you are missing messages while you compute. The subscriber queue fills. You start processing frames that are already stale by the time inference finishes. Your tracker is now seeing the world as it was, not as it is.

The fix was decoupling ingestion from inference using a bounded queue and a dedicated inference thread:

import threading
import queue
from rclpy.node import Node
from sensor_msgs.msg import Image

class PerceptionNode(Node):
    def __init__(self):
        super().__init__('perception_node')
        self.frame_queue = queue.Queue(maxsize=5)
        self.subscription = self.create_subscription(
            Image,
            '/carla/camera/rgb',
            self.image_callback,
            10
        )
        self.inference_thread = threading.Thread(
            target=self.run_inference,
            daemon=True
        )
        self.inference_thread.start()

    def image_callback(self, msg):
        if not self.frame_queue.full():
            self.frame_queue.put(msg)

    def run_inference(self):
        while rclpy.ok():
            msg = self.frame_queue.get()
            # run YOLOv11 inference here

Enter fullscreen mode Exit fullscreen mode

The maxsize=5 on the queue is deliberate. When inference cannot keep up, I drop old frames rather than accumulate a growing backlog of stale images.

This is a real-time systems principle worth internalising: a stale frame processed late is usually worse than a dropped frame. Timeliness matters as much as completeness in a live perception pipeline.

Problem three: tracking across frames is not free

ByteTrack and StrongSORT both work by associating new detections to existing tracks across frames using motion models and, in StrongSORT's case, appearance embeddings. This sounds clean until you hit occlusion, rapidly moving objects, or a detector that briefly misses something due to a lighting change or partial obstruction.

What I observed specifically in CARLA: when the ego vehicle turned sharply, vehicles and pedestrians would leave the camera frustum and re-enter within a few frames. The tracker sometimes assigned fresh IDs to those returning objects, breaking the continuity that downstream navigation reasoning depended on.

The parameters that mattered most here were:

tracker_config = {
    "max_age": 30,        # frames a track survives without a detection match
    "min_hits": 3,        # detections needed before a track is confirmed
    "iou_threshold": 0.3  # minimum overlap for detection-to-track association
}

Enter fullscreen mode Exit fullscreen mode

Increase max_age too much and you get ghost tracks for objects that have genuinely left the scene. Decrease it too much and you lose real tracks during brief occlusions. Tune iou_threshold too high and the tracker drops tracks during fast motion. Too low and it starts merging distinct objects.

There is no universally correct set of values. They depend on your scene density, your detector confidence, your frame rate, and what your downstream system needs from the tracker. The only way to find the right values is to run scenarios, observe failures, and adjust deliberately.

Problem four: edge hardware changes the problem completely

Everything above ran well on my development GPU. Then I exported the model and tested closer to constrained hardware conditions.

The frame rate dropped. The tracker fell behind. The pipeline that felt smooth in development started introducing the kind of latency that makes real-time perception meaningless.

The fix was ONNX export followed by TensorRT optimisation:

import torch

model = load_yolov11("yolov11n.pt")
model.eval()

dummy_input = torch.randn(1, 3, 640, 640).cuda()

torch.onnx.export(
    model,
    dummy_input,
    "perception.onnx",
    opset_version=17,
    input_names=["images"],
    output_names=["output0"],
    dynamic_axes={"images": {0: "batch"}}
)

Enter fullscreen mode Exit fullscreen mode

After TensorRT conversion and INT8 quantisation, inference time dropped enough to bring the full pipeline back into a usable range. But the deeper lesson is not about TensorRT specifically. It is that edge deployment is a different problem to development, and discovering that at the end of the project means redesigning under pressure.

Test on target hardware early. Not day one necessarily, but not month three either.

Problem five: your confidence threshold is a system design decision

This one is subtle and I did not fully appreciate it until I had the full pipeline running.

YOLO outputs detections with confidence scores. It is tempting to pick a threshold like 0.5 and move on. But in a live robotics system, that threshold is not a model hyperparameter. It is an architectural decision about when your robot is allowed to act on what it sees.

A threshold that is too low sends noisy, unreliable detections to the tracker. The tracker generates unstable tracks. The navigation stack gets bad inputs. The robot behaves erratically.

A threshold that is too high causes the detector to miss real objects in challenging conditions. Tracks are lost. The robot navigates as if obstacles do not exist.

What helped was treating the threshold not as a single number but as part of a validation layer:

def is_valid_detection(detection, track_age, platform_stable):
    if detection.confidence < CONFIDENCE_THRESHOLD:
        return False
    if track_age < MIN_TRACK_AGE:
        return False
    if not platform_stable:
        return False
    return True

Enter fullscreen mode Exit fullscreen mode

Multiple signals need to agree before a detection influences navigation. This pattern, requiring consensus across confidence, temporal stability, and platform state, is something I now consider non-negotiable in any perception pipeline with real-world consequences.

Five things I know now that I did not know at the start

Timestamps are not optional. Misaligned timestamps produce bugs that look like model bugs and take days to diagnose.

Decouple your threads early. Blocking the callback thread is an easy mistake with expensive downstream consequences.

The tracker is part of your perception system. Not an add-on you configure at the end. Design around it from day one.

Test on target hardware before you think you are done. Edge deployment is a different engineering problem to development.

Log everything during development. You cannot replay failures you did not record, and in robotics you will need to replay failures.

Closing thought

Building this system taught me that robotics perception is not a model problem. It is a systems problem where the model is one moving part among many.

The interesting engineering is in how you connect those parts, how you handle failures at the boundaries between them, and how you make the full pipeline behave reliably when nothing goes quite according to plan.

Which is always.

If you are building something similar or want to dig into any part of this, drop a comment. I am always happy to talk robotics and perception systems.