Overview
- When and what — September 2022 to April 2023, bachelor thesis in Robot Engineering at Guangzhou University, done on my own
- Problem — recognise hand gestures with an ordinary monocular camera and turn them into motion commands for a mobile robot car, with better recognition accuracy and tighter coupling between recognition and control
- Outcome — 81.3% mAP for static gesture detection; up to 83.3% accuracy for dynamic gestures; real-time static-gesture control of the car with 10 motion commands
What I did
- Built a static gesture dataset — recorded Chinese number gestures 0–5 with a laptop webcam across 5 people, 4,488 images in total (3,740 train / 748 validation), including face-and-hand scenes, 45° rotations and mirrored samples to match real use
- Improved YOLOv5s for static gestures — compared three modifications (CA attention, CBAM attention, CoordConv) and chose CBAM-YOLOv5s: mAP@0.5:0.95 rose from 80.2% to 81.3% while compute fell from 16.0 to 15.8 GFLOPs, and the baseline’s background false positives on digits 4 and 5 were reduced
- Improved C3D for dynamic gestures — the original C3D reached only 19.2% on five gesture classes from DSL-46, and its loss became NaN. I extended the data to 1,260 clips with binarised videos, added batch normalisation after the convolutional layers and cut the learning rate from 0.001 to 0.0001, reaching 83.3% — ahead of R3D (55%) and R(2+1)D (54%) under the same setup
- Built the gesture-controlled car — Raspberry Pi 4B car with a TB6612FNG dual H-bridge and GPIO PWM speed control; the host PC recognises gestures and sends commands over Wi-Fi via TCP/IP for 10 motions: stop, forward, backward, spin left and right, speed up, and two-hand combinations for turning left and right and reversing left and right
Technical details
Static gestures: three modifications compared
Same dataset and training setup (100 epochs, batch 8):
| Model | Change | mAP@0.5:0.95 | Compute (GFLOPs) |
|---|---|---|---|
| YOLOv5s (baseline) | — | 80.2% | 16.0 |
| CA-YOLOv5s | Coordinate Attention after the 2nd, 3rd and 4th C3 blocks of the backbone | 80.6% | 16.0 |
| CBAM-YOLOv5s (chosen) | CBAM (channel and spatial attention) inside the backbone C3 blocks | 81.3% | 15.8 |
| CoordConv-YOLOv5s | Coordinate convolution replacing 1×1 convolutions in the head, also before the detection heads | 80.8% | 21.7 |
On the same dataset, the newer YOLOv7 reached 79.7% and YOLOv8s 82.3%. Among the three YOLOv5 modifications, CBAM had both the highest accuracy and the lowest compute.
Dynamic gestures: from 19.2% to 83.3%
- Data — five everyday gestures from the public DSL-46 dataset (come, don’t want, fine, go, hello), 120 clips each, split into frames for training
- Baseline — the original C3D (8 convolutional layers with 3×3×3 kernels, 3 fully connected layers), trained for 20 epochs; the loss did not decrease
- Changes — binarised copies of the videos grew the data to 1,260 clips (804 train / 204 validation / 252 test); batch normalisation (momentum 0.9) after the convolutional layers; learning rate 0.0001; 100 epochs
| Network | Best accuracy |
|---|---|
| BN-C3D | 83.3% |
| R3D | 55% |
| R(2+1)D | 54% |
From recognition to motion
- Pipeline — host webcam → CBAM-YOLOv5s in real time → gesture-to-command mapping → Wi-Fi (TCP/IP) → Raspberry Pi 4B → TB6612FNG → DC motors
- Hardware — aluminium chassis, differential drive, metal-gear DC motors, two 7.4 V lithium batteries; host PC with an i7-12650H and an RTX 4060 Laptop GPU
- Software — Python 3.8, PyTorch 1.13, OpenCV 4.7; motion program deployed on the Pi over VNC
- Commands — one hand: 0 stop · 1 forward · 2 backward · 3 spin left · 4 spin right · 5 speed up; two hands: 1+3 turn left · 1+4 turn right · 2+3 reverse left · 2+4 reverse right
Limitations and lessons
- Dynamic gestures never reached the car — BN-C3D fluctuated heavily on validation and recognised my own recordings poorly, so real-time dynamic recognition did not meet the control requirements and the car was driven with static gestures only. The causes: only 120 clips per class, and a plain monocular camera without depth
- Modest gains — gesture detection is easier than general object detection, so the baseline already reached 80.2% and the improvement added about one percentage point; YOLOv8s reached 82.3% on the same data (its loss is different, so the numbers are not strictly comparable)
- Where it led — distant cones in the KAL project at KIT were the same small-object problem; there I used attention again (Coordinate Attention) and added a depth camera