Traffic signal analysis system

• US 10,726,280 B2
• Filed: 03/20/2018
• Issued: 07/28/2020
• Est. Priority Date: 03/09/2016
• Status: Active Grant

• Current Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Original Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Inventors: Wellington, Carl, Green, Colin, Milstein, Adam
• Primary Examiner(s): Hilgendorf, Dale W
• Application Number:
  US15/926,211
• Publication Number:
  US 20180218226A1
• Time in Patent Office: 861 Days
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

A traffic signal analysis system for an autonomous vehicle (AV) can receive image data from one or more cameras, the image data including an upcoming traffic signaling system located at an intersection. The system can determine an action for the AV through the intersection and access a matching signal map specific to the upcoming traffic signaling system. Using the matching signal map, the system can generate a signal template for the upcoming traffic signaling system and determine a first subset and a second subset of the plurality of traffic signal faces that apply to the action. The system can dynamically analyze the first subset and the second subset to determine a state of the upcoming traffic signaling system for the action, and generate an output for the AV indicating the state of the upcoming traffic signaling system for the action.

Claims

1. A traffic signal analysis system for an autonomous vehicle (AV), the traffic signal analysis system comprising:

one or more processors; and

one or more memory resources storing instructions that, when executed by the one or more processors, cause the one or more processors to;

receive image data from one or more cameras, the image data including an upcoming traffic signaling system located at an intersection;

determine an action for the AV through the intersection;

access a matching signal map specific to the upcoming traffic signaling system;

using the matching signal map, (i) generate a signal template for the upcoming traffic signaling system, the signal template defining at least one region of interest that comprises a plurality of traffic signal faces of the upcoming traffic signaling system, and (ii) determine a first subset and a second subset of the plurality of traffic signal faces that apply to the action;

dynamically analyze the first subset and the second subset to determine a state of the upcoming traffic signaling system for the action; and

generate an output for the AV indicating the state of the upcoming traffic signaling system for the action.

2. The traffic signal analysis system of claim 1, wherein the executed instructions cause the one or more processors to dynamically analyze the first and second subsets by performing a matching operation for one or more portions of the at least one region of interest to determine a state of the first and second subsets for each traffic signal face in the first and second subsets.

3. The traffic signal analysis system of claim 2, wherein the first and second subsets comprise more than one of the traffic signal faces that apply to the action, and wherein the executed instructions further cause the one or more processors to:

when one of the traffic signal faces is at least partially occluded, prioritize a remainder of the other traffic signal faces in the first and second subsets, during the matching operation, to determine the state of the upcoming traffic signaling system for the action.

4. The traffic signal analysis system of claim 1, wherein the executed instructions further cause the one or more processors to:

determine a plurality of pass-through options for the plurality of traffic signal faces;

wherein the generated signal template comprises a signal subset for each of the plurality of pass-through options.

5. The traffic signal analysis system of claim 4, wherein the plurality of pass-through options comprises two or more of a left turn option, a right turn option, a U-turn option, or a straight-through option.

6. The traffic signal analysis system of claim 1, wherein the one or more cameras comprise a dedicated signal camera specific for identifying traffic signals.

7. The traffic signal analysis system of claim 6, wherein the dedicated signal camera comprises a set of camera settings configured to detect traffic signals, the set of camera settings being different from camera settings of a set of perception cameras utilized by a control system of the AV.

8. The traffic signal analysis system of claim 1, wherein the generated output for the AV, indicating the state of the upcoming traffic signaling system for the action, comprises either (i) a command instructing a control system of the AV to stop at the intersection, (ii) an indication to the control system that the AV has a right-of-way through the intersection, or (iii) an indication to the control system to yield to right-of-way vehicles before passing through the intersection.

9. The traffic signal analysis system of claim 1, wherein the executed instructions further cause the one or more processors to:

predict the state of the upcoming traffic signaling system for the action when the AV arrives at the intersection;

wherein the generated output comprises the predicted state of the upcoming traffic signaling system for the action.

10. An autonomous vehicle (AV) comprising:

a sensor system to dynamically generate sensor data indicating a situational environment of the AV;

acceleration, braking, and steering systems; and

a control system comprising one or more processing resources executing instructions that cause the control system to;

process the sensor data to autonomously operate the acceleration, braking, and steeling systems of the AV;

receive image data from one or more cameras, the image data including an upcoming traffic signaling system located at an intersection;

determine an action for the AV through the intersection;

access a matching signal map specific to the upcoming traffic signaling system;

using the matching signal map, (i) generate a signal template for the upcoming traffic signaling system, the signal template defining at least one region of interest that comprises a plurality of traffic signal faces of the upcoming traffic signaling system, and (ii) determine a first subset and a second subset of the plurality of traffic signal faces that apply to the action;

dynamically analyze the first subset and the second subset to determine a state of the upcoming traffic signaling system for the action; and

execute a corresponding action using the acceleration, braking, and steering systems based on the state of the upcoming traffic signaling system for the action.

11. The AV of claim 10, wherein the executed instructions cause the control system to analyze the first and second subsets by performing a matching operation for one or more portions of the at least one region of interest to determine a state of the first and second subsets for each traffic signal face in the first and second subsets.

12. The AV of claim 11, wherein the first and second subsets comprise more than one of the traffic signal faces that apply to the action, and wherein the executed instructions further cause the control system to:

when one of the traffic signal faces is at least partially occluded, prioritize a remainder of the other traffic signal faces in the first and second subsets, during the matching operation, to determine the state of the upcoming traffic signaling system for the action.

13. The AV of claim 10, wherein the at least one region of interest comprises more than one of the traffic signal faces, and wherein the executed instructions further cause the control system to:

determine a plurality of pass-through options for the traffic signal faces of the at least one region of interest;

wherein the generated signal template comprises a signal subset for each of the plurality of pass-through options.

14. The AV of claim 13, wherein the plurality of pass-through options comprises two or more of a left turn option, a right turn option, a U-turn option, or a straight-through option.

15. The AV of claim 10, wherein the one or more cameras comprise a dedicated signal camera specific for identifying traffic signals.

16. The AV of claim 15, wherein the dedicated signal camera comprises a set of camera settings configured to detect traffic signals, the set of camera settings being different from camera settings of a set of perception cameras of the sensor system.

17. A non-transitory computer readable medium storing instructions that, when executed by one or more processors of an autonomous vehicle (AV), cause the one or more processors to:

receive image data from one or more cameras, the image data including an upcoming traffic signaling system located at an intersection;

determine an action for the AV through the intersection;

access a matching signal map specific to the upcoming traffic signaling system;

using the matching signal map, (i) generate a signal template for the upcoming traffic signaling system, the signal template defining at least one region of interest that comprises a plurality of traffic signal faces of the upcoming traffic signaling system, and (ii) determine a first subset and a second subset of the plurality of traffic signal faces that apply to the action;

dynamically analyze the first subset and the second subset to determine a state of the upcoming traffic signaling system for the action; and

generate an output for the AV indicating the state of the upcoming traffic signaling system for the action.

18. The non-transitory computer readable medium of claim 17, wherein the executed instructions cause the one or more processors to dynamically analyze the first and second subsets by performing a matching operation for one or more portions of the at least one region of interest to determine a state of the first and second subsets for each traffic signal face in the first and second subsets.

19. The non-transitory computer readable medium of claim 18, wherein the first and second subsets comprise more than one of the traffic signal faces that apply to the action, and wherein the executed instructions further cause the one or more processors to:

when one of the traffic signal faces is at least partially occluded, prioritize a remainder of the other traffic signal faces in the first and second subsets, during the matching operation, to determine the state of the upcoming traffic signaling system for the action.

20. The non-transitory computer readable medium of claim 17, wherein the executed instructions further cause the one or more processors to:

determine a plurality of pass-through options for the plurality of traffic signal faces;

wherein the generated signal template comprises a signal subset for each of the plurality of pass-through options.