Traffic signal mapping and detection

• US 8,559,673 B2
• Filed: 06/21/2010
• Issued: 10/15/2013
• Est. Priority Date: 01/22/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Urmson, Christopher, Thrun, Sebastian
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US12/819,575
• Publication Number:
  US 20110182475A1
• Time in Patent Office: 1,212 Days
• Field of Search: 342/357.25, 382103-106, 340/901, 340/907
• US Class Current: 382/104
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence

Abstract

A system and method provides maps identifying the 3D location of traffic lights. The position, location, and orientation of a traffic light may be automatically extrapolated from two or more images. The maps may then be used to assist robotic vehicles or human drivers to identify the location and status of a traffic signal.

Claims

1. A method of determining three-dimensional locations of traffic signals, the method comprising:

receiving a plurality of images, each image of the plurality of images is associated with geographic location and orientation information;

selecting, by a computer, one or more images of the plurality of images which are associated with the geographic locations proximate to traffic intersections;

for each selected image, identifying by the computer brightly-colored red, yellow, and green objects within the selected image;

generating a set of traffic signal labels based on the identified brightly-colored red, yellow and green objects;

identifying associated labels within two or more of the selected images, based on the geographic location and orientation information of the two or more selected images;

filtering the set of traffic signal labels to remove labels for brightly-colored red, yellow, or green objects that correspond to objects other than a traffic signal light;

determining the three-dimensional locations of traffic signals based on (1) the identified associated labels, (2) the filtered set of traffic signal labels, and (3) the geographic location and orientation information of the two or more selected images;

storing the three-dimensional locations of the traffic signals in memory accessible by the computer; and

using the three-dimensional locations of the traffic signal to assist an autonomous vehicle.

2. The method of claim 1, further comprising generating a map including the three-dimensional locations of the traffic signals.

3. The method of claim 2, wherein each of the plurality of images is collected by one or more cameras, each camera associated with a vehicle.

4. The method of claim 3, wherein each of the one or more cameras is mounted on the vehicle.

5. The method of claim 3, wherein the geographic location and orientation information associated with each image are generated based on the geographic location and orientation information of the camera as determined by a geographic position device.

6. The method of claim 1, wherein the geographic location and orientation information associated with each image are determined by a laser positioning device.

7. The method of claim 1, wherein the geographic location and orientation information associated with each image are determined by a GPS positioning device.

8. The method of claim 1, wherein the geographic location and orientation information associated with each image are determined by an inertial positioning device.

9. The method of claim 1, wherein the geographic location information is GPS latitude and longitude coordinates.

10. A device for determining three-dimensional locations of traffic signals, the device comprising:

a processor; and

memory; and

the processor being configured to;

receive a plurality of images, each image of the plurality of images is associated with geographic location and orientation information;

select one or more images of the plurality of images which are associated with geographic locations proximate to traffic intersections;

for each selected image, identify brightly-colored red, yellow, and green objects within the selected image;

generate a set of traffic signal labels based on the identified brightly colored red, yellow, and green objects;

identify associated labels within two or more of the selected images, based on the geographic location and orientation information of the two or more selected images;

filter the set of traffic signal labels to remove labels for brightly colored red, yellow, or green objects that correspond to an object other than a traffic signal light based on direct motion compensation;

determine the three-dimensional locations of traffic signals based on (1) the identified associations between the two or more of the selected images, (2) the filtered set of traffic signal labels, and (3) the geographic location and orientation information of the two or more selected images;

store the three-dimensional locations of traffic signals in memory accessible by the device; and

use the three-dimensional locations of the traffic signal to assist an autonomous vehicle.

11. The device of claim 10, wherein each of the plurality of images is collected by one or more cameras, each camera being associated with a vehicle and having a respective gain and shutter speed which are set to avoid saturation of a traffic signal light.

12. The device of claim 10, wherein each of the plurality of images is collected by one or more cameras, each camera being associated with a vehicle and each image of the plurality of images is uploaded from each of the one or more cameras to the device via a network.

13. The device of claim 10, wherein each of the plurality of images is collected by one or more cameras, each camera being associated with a vehicle and being positioned to minimally obstruct the view of a driver of the vehicle.

14. The device of claim 10, wherein the identified brightly colored red, yellow, and green objects are the appropriate size and aspect ratios to correspond to traffic signals.

15. The device of claim 10, wherein identifying the associated labels is based on an association distance between labeled objects of the two or more selected images.

16. The device of claim 10, wherein identifying the associated labels is based on the physical dimensions of a traffic signal.

17. The device of claim 10, wherein identifying the associated labels is based on direct motion compensation between the selected images, wherein each selected image is taken by a camera mounted on a moving vehicle.

18. The device of claim 10, the processor is further configured to determine a lane associated with the particular traffic signal based on comparing the determined three-dimensional location of the particular traffic signal to a map of lanes through an intersection.

19. A method of determining the status of a traffic signal, the method comprising:

repeatedly determining a current location of a client device;

determining an estimated location of the boundaries of a traffic signal based on a comparison of the current location of the client device to a map of three-dimensional locations of traffic signals;

collecting images of the estimated location;

for each collected image, identifying by the client device brightly colored red, yellow, and green objects within the boundaries of the estimated location of the traffic signal;

generating a set of traffic signal labels based on the identified brightly colored red, yellow, and green objects;

filtering the set of traffic signal labels to remove labels for brightly colored red, yellow, or green objects within the boundaries of the estimated location of the traffic signal that correspond to an object other than a traffic signal light based on direct motion compensation; and

detecting, from the filtered set of traffic signal labels, the status of the traffic signal; and

using the estimated location of the traffic signal and the status of the traffic signal to assist an autonomous vehicle.

20. The method of claim 19, further comprising determining whether the status of the traffic signal has changed from a default status.

21. The method of claim 19, wherein when there are no identified objects within the boundaries of the estimated location of the traffic signal, determining that the status of the traffic signal is the default status.

22. The method of claim 20, wherein the default status is a yellow light.

23. The method of claim 21, wherein the default status is a red light.

24. The method of claim 19, further comprising transmitting the status of the traffic signal to a computer associated with a vehicle.

25. The method of claim 19, further comprising identifying the status of the traffic signal audibly.

26. The method of claim 19, further comprising providing driving instructions based on the status of the traffic signal.

27. A device for determining three-dimensional locations of traffic signals, the device comprising:

a processor; and

memory, including a first part for storing secondary resource files; and

the processor being configured to;

repeatedly determine the current location of a client device;

determine an estimated location of the boundaries of a traffic signal based on a comparison of the current location of the client device to a map of three-dimensional locations of traffic signals;

collect images of the estimated location;

for each collected image, identify by the client device brightly colored red, yellow, and green objects within the boundaries of the estimated location of the traffic signal;

generate a set of traffic signal labels based on the identified brightly colored red, yellow, and green objects;

filtering the set of traffic signal labels to remove labels for brightly colored red, yellow, or green objects within the boundaries of the estimated location of the traffic signal that correspond to an object other than a traffic signal light based on direct motion compensation;

detect, from the filtered set of traffic signal labels, the status of the traffic signal; and

use the estimated location of the traffic signal and the status of the traffic signal to assist an autonomous vehicle.

28. The device of claim 27, wherein the device includes a display device and the processor is further configured to identify the status of the traffic signal on the display device.

29. A method of collecting images of traffic signals, the method comprising:

setting gain and shutter speed of a camera to avoid saturation of a traffic signal light;

mounting the camera to a vehicle;

maneuvering the vehicle through a traffic intersection;

collecting a plurality of images of the traffic intersection;

associating geographic location and orientation information with each of the plurality of images based on a positioning device;

storing the plurality of images and associated geographic locations in memory;

transmitting the plurality of images and the associated geographic locations to a computer over a network;

receiving the plurality of images and the associated geographic location and orientation information;

selecting one or more images of the plurality of images which are associated with geographic locations proximate to traffic intersections;

for each selected image, identifying brightly colored red, yellow, and green objects within the selected image;

identifying associated labels within two or more of the selected images, based on the geographic location and orientation information of the two or more selected images;

filtering the set of traffic signal labels to remove labels for brightly colored red, yellow, or green objects within the selected image that correspond to an object other than a traffic signal light based on direct motion compensation;

determining the three-dimensional locations of traffic signals based on (1) the identified associated labels, (2) the filtered set of traffic signal labels, and (3) the geographic location and orientation information of the two or more selected images; and

storing the three-dimensional locations of traffic signals in memory accessible by the computer; and

using the three-dimensional locations of the traffic signal to assist an autonomous vehicle.

30. The method of claim 29, further comprising determining a lane associated with the particular traffic signal based on comparing the determined the three-dimensional location of the particular traffic signal to a map of lanes through an intersection.