TRAFFIC SIGNAL MAPPING AND DETECTION

• US 20110182475A1
• Filed: 06/21/2010
• Published: 07/28/2011
• Est. Priority Date: 01/22/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Thrun, Sebastian, Urmson, Christopher, Fairfield, Nathaniel
• Granted Patent: US 8,559,673 B2
• Time in Patent Office: Days
• US Class Current: 382/104
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence

Abstract

A system and method provides maps identifying the 3D location of traffic lights. The position, location, and orientation of a traffic light may be automatically extrapolated from two or more images. The maps may then be used to assist robotic vehicles or human drivers to identify the location and status of a traffic signal.

Claims

1. A method of determining three-dimensional locations of traffic signals, the method comprising:

receiving a plurality of images, each image of the plurality of images is associated with geographic location and orientation information;

selecting, by a computer, one or more images of the plurality of images which are associated with the geographic locations proximate to traffic intersections;

for each selected image, identifying by the computer red, yellow, and green objects within the selected image;

identifying associated ones of the red, yellow, and green objects within two or more of the selected images, based on the geographic location and orientation information of the two or more selected images;

determining the three-dimensional locations of traffic signals based on (1) the identified associations between the two or more of the selected images and (2) the geographic location and orientation information of the two or more selected images; and

storing the three-dimensional locations of the traffic signals in memory accessible by the computer.

2. The method of claim 1, further comprising generating a map including the three-dimensional locations of the traffic signals.

3. The method of claim 2, wherein each of the plurality of images is collected by one or more cameras, each camera associated with a vehicle.

4. The method of claim 3, wherein each of the one or more cameras is mounted on the vehicle.

5. The method of claim 3, wherein the geographic location and orientation information associated with each image are generated based on the geographic location and orientation information of the camera as determined by a geographic position device.

6. The method of claim 1, wherein the geographic location and orientation information associated with each image are determined by a laser positioning device.

7. The method of claim 1, wherein the geographic location and orientation information associated with each image are determined by a GPS positioning device.

8. The method of claim 1, wherein the geographic location and orientation information associated with each image are determined by an inertial positioning device.

9. The method of claim 1, wherein the geographic location information is GPS latitude and longitude coordinates.

10. A device for determining three-dimensional locations of traffic signals, the device comprising:

a processor; and

memory; and

the processor being configured to;

receive a plurality of images, each image of the plurality of images is associated with geographic location and orientation information;

select one or more images of the plurality of images which are associated with geographic locations proximate to traffic intersections;

for each selected image, identify red, yellow, and green objects within the selected image;

identify associated ones of the red, yellow, and green objects within two or more of the selected images, based on the geographic location and orientation information of the two or more selected images;

determine the three-dimensional locations of traffic signals based on (1) the identified associations between the two or more of the selected images and (2) the geographic location and orientation information of the two or more selected images; and

store the three-dimensional locations of traffic signals in memory accessible by the device.

11. The device of claim 10, wherein each of the plurality of images is collected by one or more cameras, each camera being associated with a vehicle and having a respective gain and shutter speed which are set to avoid saturation of a traffic signal light.

12. The device of claim 10, wherein each of the plurality of images is collected by one or more cameras, each camera being associated with a vehicle and each image of the plurality of images is uploaded from each of the one or more cameras to the device via a network.

13. The device of claim 10, wherein each of the plurality of images is collected by one or more cameras, each camera being associated with a vehicle and being positioned to minimally obstruct the view of a driver of the vehicle.

14. The device of claim 10, wherein the identified red, yellow, and green objects are the appropriate size and aspect ratios to correspond to traffic signals.

15. The device of claim 10, wherein the identifying associated ones of the red, yellow, and green objects of two or more selected images is based on an association distance between identified objects of the two or more selected images.

16. The device of claim 10, wherein the identifying associated ones of the red, yellow, and green objects of two or more selected images is based on the physical dimensions of a traffic signal.

17. The device of claim 10, wherein the identifying associated red, yellow, and green objects of two or more selected images is based on direct motion compensation between the selected images, wherein each selected image is taken by a camera mounted on a moving vehicle.

18. The device of claim 10, wherein the processor is further configured to identify the identified ones of red, yellow, and green objects within a selected image as an object other than a traffic signal light based on the direct motion compensation.

19. The device of claim 10, the processor is further configured to determine a lane associated with the particular traffic signal based on comparing the determined the three-dimensional location of the particular traffic signal to a map of lanes through an intersection.

20. A method of determining the status of a traffic signal, the method comprising:

repeatedly determining a current location of a client device;

determining an estimated location of the boundaries of a traffic signal based on a comparison of the current location of the client device to a map of three-dimensional locations of traffic signals;

collecting images of the estimated location;

for each collected image, identifying by the client device red, yellow, and green objects within the boundaries of the estimated location of the traffic signal; and

determining the status of the traffic signal based on the color of the identified object.

21. The method of claim 20, further comprising determining whether the status of the traffic signal has changed from a default status.

22. The method of claim 20, wherein if there are no identified objects within the boundaries of the estimated location of the traffic signal, determining that the status of the traffic signal is the default status.

23. The method of claim 21, wherein the default status is a yellow light.

24. The method of claim 22, wherein the default status is a red light.

25. The method of claim 20, further comprising transmitting the status of the traffic signal to a computer associated with a vehicle.

26. The method of claim 20, further comprising identifying the status of the traffic signal audibly.

27. The method of claim 20, further comprising providing driving instructions based on the status of the traffic signal.

28. A device for determining three-dimensional locations of traffic signals, the device comprising:

a processor; and

memory, including a first part for storing secondary resource files; and

the processor being configured to;

repeatedly determine the current location of a client device;

determine an estimated location of the boundaries of a traffic signal based on a comparison of the current location of the client device to a map of three-dimensional locations of traffic signals;

collect images of the estimated location;

for each collected image, identify by the client device red, yellow, and green objects within the boundaries of the estimated location of the traffic signal; and

determine the status of the traffic signal based on the color of the identified object.

29. The device of claim 28, wherein the device includes a display device and the processor is further configured to identify the status of the traffic signal on the display device.

30. A method of collecting images of traffic signals, the method comprising:

setting gain and shutter speed of a camera to avoid saturation of a traffic signal light;

mounting the camera to a vehicle;

maneuvering the vehicle through a traffic intersection;

collecting a plurality of images of the traffic intersection;

associating geographic location and orientation information with each of the plurality of images based on a positioning device;

storing the plurality of images and associated geographic locations in memory; and

transmitting the plurality of images and the associated geographic locations to a computer over a network.

31. The method of claim 30, further comprising:

receiving the plurality of images and the associated geographic location and orientation information;

selecting one or more images of the plurality of images which are associated with geographic locations proximate to traffic intersections;

for each selected image, identifying red, yellow, and green objects within the selected image;

identifying associated ones of the red, yellow, and green objects within two or more of the selected images, based on the geographic location and orientation information of the two or more selected images;

determining the three-dimensional locations of traffic signals based on (1) the identified associations between the two or more of the selected images and (2) the geographic location and orientation information of the two or more selected images; and

storing the three-dimensional locations of traffic signals in memory accessible by the computer.

32. The method of claim 30, further comprising determining a lane associated with the particular traffic signal based on comparing the determined the three-dimensional location of the particular traffic signal to a map of lanes through an intersection.