Overview
My master’s thesis, completed at the Georgia Institute of Technology in December 2024, investigates real-time traffic sign recognition for mobile and edge devices.
The research addresses the limitations of traditional traffic-sign inventories, which are often based on expensive manual surveys or offline video processing. In particular, the U.S. Manual on Uniform Traffic Control Devices (MUTCD) defines more than 500 distinct traffic-sign types, creating a demanding recognition and inventory problem.
The objective was to develop a flexible system capable of detecting, classifying, tracking, and localizing traffic signs from road video while remaining suitable for near-real-time operation on modern mobile hardware.
Main contributions
The thesis presents two related processing pipelines:
- A flexible pipeline combining YOLOv8 for detection with CLIP-based classification. This approach uses text prompts describing traffic signs, allowing the system to scale to a large number of sign types without retraining the classifier for every new category.
- An optimized pipeline that adds a classification head to a YOLOv8-nano detector. This head produces feature embeddings compatible with the CLIP text encoder, removing the need for a separate image encoder and reducing computational cost.
The complete inventory workflow also includes object tracking and camera localization. Tracking filters inconsistent detections, while localization associates detected signs with their position along the recorded route.
Technical approach
Dataset preparation
The study combines traffic-sign data collected from road recordings with existing datasets, including GTSC and Mapillary data. The data was annotated, cleaned, and split into training, validation, and test subsets. Different split strategies were evaluated to understand the impact of dataset complexity and generalization.
Detection with YOLOv8-nano
YOLOv8-nano was selected as a compact object-detection model suitable for deployment on constrained devices. The detector was trained using both a single-class configuration, where every sign is labeled as a traffic sign, and a multi-class configuration with more detailed sign categories.
Prompt-based classification with CLIP
CLIP was used to classify detected signs by comparing image and text embeddings in a shared latent space. The text prompts combine MUTCD sign descriptions with visual attributes such as color and shape. This enables flexible classification and zero-shot evaluation across a large vocabulary of traffic-sign categories.
Optimized YOLO head
The proposed YOLO head learns to generate feature vectors aligned with CLIP’s text-embedding space. Knowledge distillation transfers information from a visual transformer into the detection model. The resulting architecture avoids the expensive image-encoding stage during inference and is designed for real-time edge deployment.
Large-scale validation
The system was evaluated on two real-world road recordings:
- Test Road A: a 24-minute recording in Foresight County, Georgia, with conditions related to the training data but no direct overlap with the training images.
- Test Road B: a 10-minute recording in Pima County, Arizona, including an unpaved road and traffic signs in a significantly different environment.
A sign was considered successfully inventoried when it was detected consistently over multiple frames. The evaluation measured detection, classification, tracking, and localization performance, as well as processing time.
Results
The prompt-based pipeline demonstrated strong flexibility and generalization across more than 556 traffic-sign types. In the real-world scenarios, it classified over 97% of the inventoried signs with at least 93% classification accuracy, but its computational cost remained too high for real-time mobile deployment.
The optimized YOLO-based pipeline substantially reduced inference cost. It reached approximately 8.7 ms per inference on an NVIDIA L40S and ran at approximately 25 FPS on a Jetson Orin Nano after TensorRT export. Its classification accuracy was lower than the original pipeline, reaching 36.9% top-1 accuracy and 44.5% top-5 accuracy in the reported training experiment.
Across the complete inventory system, the best evaluated configuration achieved 77% recall and 53% precision. These results demonstrate a functional system capable of producing traffic-sign inventories while reducing manual review effort, while also highlighting the remaining trade-off between classification flexibility, accuracy, and execution speed.
Conclusion
This work shows that combining object detection with vision-language models can provide a scalable approach to traffic-sign inventory. Prompt-based classification is particularly promising when the number of sign categories is large or changes over time. The optimized YOLO head demonstrates a path toward mobile deployment, although further work is needed to improve knowledge distillation and classification accuracy.
Potential future directions include refining the distillation strategy, generating feature descriptors per detection anchor, improving the training data, and investigating more efficient architectures for mobile hardware.
CVPR 2024
As part of my master’s work in computer vision, I attended the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) in Seattle. The conference provided an opportunity to explore current research, attend technical presentations, and engage with the broader computer-vision community.
Thesis document
The complete dissertation is available on request.