Traffic light recognition using deep learning and prior maps for autonomous cars

Abstract

At complex intersections, human drivers can easily identify which traffic lights are relevant for the route they intend to follow, and what are their states (red, yellow, or green). However, this remains a challenging task for autonomous vehicles. In the literature, an effective solution to this problem is to merge traffic light recognition with prior maps of traffic lights. Deep learning techniques have showed great performance and power of generalization including for traffic related problems. Motivated by the advances in deep learning, some recent works leveraged some state-of-the-art deep detectors to locate traffic lights and classify their state from 2D camera images. However, none of them combine the power of deep learning-based detectors with prior maps to identify the state of the relevant traffic lights. Based on that, this work proposes to integrate the power of deep learning-based detection with prior maps of traffic light into our car platform, IARA (acronym for Intelligent Autonomous Robotic Automobile), to recognize the relevant traffic lights of predefined routes. The process is divided in two phases: an offline phase for map construction and traffic lights annotation; and an online phase for traffic light recognition and identification of the relevant ones. Two different types of model for detection and classification of traffic lights are approached. One is a single model, deep learning detector, that detects and classify the state of traffic lights in a single step. The other uses a deep learning detector for locating traffic lights, and a separate model for classifying their states. The proposed system was evaluated on five test cases (routes) in the city of Vitória, each case being composed of a video sequence and a prior map of relevant traffic lights for the route. Results showed that the proposed technique is able to correctly identify the relevant traffic light along the trajectory