Overview
- When and what — April to July 2026, Cognitive Automobile Lab (Kognitive Automobile Labor) at KIT’s Institute of Measurement and Control Systems (MRT), team project
- My role — mainly perception (dataset, detection model, RGB-D 3D localisation and tracking); also involved in planning, control and testing on the car
- Task — drive the right lane of a track for an unknown number of laps, overtake a parked obstacle car via the opposite lane, then enter a cone corridor and stop at its end
- Platform — MuSHR-based research car with an Intel RealSense RGB-D front camera, ROS 2 Jazzy
What I did
- Moved perception from classical vision to YOLO with RGB-D — the first version used OpenCV for cones (HSV segmentation), lane lines (Canny / Hough) and the obstacle car (RANSAC ground plane plus height segmentation). An offline replay of 6 recordings with 1,770 frames ran at 8.1–9.6 ms per frame, but exposed misaligned RGB and depth, radiators and walls detected as cars, and lane-line validity as low as 1.5% in edge cases. I replaced it with depth alignment, YOLO detection and multi-frame confirmation
- Built the dataset and iterated the detector — 400 images from the real car (334 train / 66 validation) with 1,232 boxes in 3 classes: cone, obstacle car and traffic sign (the sign class only prevents signs being taken for cones). Fine-tuned from the previous model: on the same validation set mAP50 rose from 67.7% to 97.4% and recall from 57.4% to 96.7%
- Improved YOLOv8n for distant cones — a far cone covers only about 4×8 pixels at 640 input, so I added Coordinate Attention to the mid-to-deep C2f blocks and a high-resolution P2 detection head. In a controlled comparison recall rose from 91.6% to 99.4% and mAP50-95 from 56.8% to 59.6%, at 2.0 instead of 1.5 ms per image. The deployed weights include Coordinate Attention; a small compatibility shim lets the stock Ultralytics runtime on the car load the custom modules
- RGB-D 3D localisation and tracking — reprojected raw RealSense depth (848×480) into the colour image (1280×720), clustered foreground depth inside each box for a 3D position, and passed objects to planning only after 3 consecutive confirmations; perception publishes JSON events at 10 Hz
- Tests — 18 unit tests for the perception chain (depth alignment, 3D projection, track confirmation, JSON interface and more); offline replay of recordings for debugging
- Beyond perception — contributed to planning, control and integration testing on the car
Technical details
Architecture: five ROS 2 packages
| Package | Language | Role |
|---|---|---|
kal_perception_yolo |
Python | RGB-D sync, depth alignment, YOLO inference, 3D post-processing, tracking, JSON output |
trajectory |
Python | Path selection, behaviour state machine, overtaking, cone memory and corridor centreline, path timestamps |
kal_controller_PID |
C++ | ROS-independent control law combining Pure Pursuit and PID |
kal_controller_ros_tool_PID |
C++ | ROS 2 component wrapper: parameters, TF, safety checks, cone slowdown |
autonomous_car_bringup |
Python | One launch for perception, planning and control |
Data flow: RGB + raw depth + calibration → depth alignment → YOLO boxes → 3D position and multi-frame confirmation → JSON perception events → state machine and trajectory → Pure Pursuit + PID → Ackermann steering and speed commands.
Planning and control (team work I contributed to)
- Path selection — two prerecorded closed loops; distance voting picks the one the car is on
- Behaviour state machine — follow path → overtake → cone corridor → stop
- Overtaking — a Gaussian lateral offset (up to 0.4 m) along the path’s left normal within 3.5 m before and after the obstacle; the path is frozen within 1.5 m so blind-spot re-detections cannot distort the manoeuvre, and restored 2 m after passing
- Cone corridor — a separate topic first slows the controller to cone speed; cones remembered by track ID are clustered into left and right rows, a constrained parabola gives the centreline, which is spliced into the main path; with only one row visible, it is treated as a barrier and the car stops in front of it
- Control — Pure Pursuit steers and PID corrects the cross-track error; a missing or stale trajectory or a TF failure immediately commands zero speed and zero steering
Detector iterations
| Version | Training data and method | mAP50 | Recall |
|---|---|---|---|
| v1 | Earlier dataset | 67.7% | 57.4% |
| v2 | 400 real-car images, fine-tuned from v1 | 97.4% | 96.7% |
Both rows are evaluated on the same validation set (66 images, 213 boxes).
Small-object comparison, same dataset and settings (640 input, batch 8, 100 epochs):
| Model | Recall | mAP50 | mAP50-95 | Per image |
|---|---|---|---|---|
| YOLOv8n | 91.6% | 96.7% | 56.8% | 1.5 ms |
| YOLOv8n + CA + P2 | 99.4% | 99.2% | 59.6% | 2.0 ms |
These are results from the improvement branch; the model running on the car is the v3 checkpoint with Coordinate Attention.
Limitations and lessons
- The controller has no automated tests — perception has 18 and cone memory 11 unit tests, but neither the Pure Pursuit and PID core nor its wrapper is tested; this is the clearest gap in the project
- Simple overtaking — always to the left, without checking that the left side is free or predicting the obstacle’s motion (the obstacle car in the task is parked)
- Lesson from the first version — classical vision already ran at 8.1–9.6 ms per frame, so speed was never the bottleneck; misaligned depth and false detections were. In the upgrade, depth alignment and multi-frame confirmation mattered as much as switching detectors