Fusion-based traffic light recognition for autonomous driving

• US 10,467,487 B1
• Filed: 12/11/2018
• Issued: 11/05/2019
• Est. Priority Date: 12/11/2018
• Status: Active Grant

• Current Assignee: Chongqing Jinkang New Energy Automobile Co., Ltd., SF Motors, Inc. (SERES Group Co., Ltd.)
• Original Assignee: Chongqing Jinkang New Energy Automobile Co., Ltd.
• Inventors: Wang, Fan, Chai, Huajun, Huang, Ke, Singh, Saurabh, Yu, Rui, Tang, Yifan
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US16/216,731
• Time in Patent Office: 329 Days
• Field of Search: None
• CPC Class Codes: B60R 1/005 specially adapted for viewing traffic-lights
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

Various arrangements are presented for determining a state of a traffic light. A first analysis of a scene may be performed that includes the traffic light. The first analysis may result in the output of a first classification result and a first confidence score. A second, different analysis of the scene may be performed that also outputs a classification result and a confidence score. A fusion process may be performed to fuse the classification results into a fused classification result based on the confidence scores and one or more weighting metrics. A vehicle may then be driven at least partially based on the fused classification result.

Claims

1. A method for determining a state of a traffic light, the method comprising:

performing, by an onboard vehicle processing system, a first analysis of a scene that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality of probabilities corresponds to a different traffic light state;

performing, by the onboard vehicle processing system, a second analysis of the scene that includes the traffic light, wherein;

the second analysis is a different type of analysis than the first analysis;

the second analysis outputs a second plurality of probabilities; and

each probability of the second plurality of probabilities corresponds to a different traffic light state;

performing, by the onboard vehicle processing system, a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of;

an offline weighting metric; and

an online weighting metric;

determining the online weighting metric, which comprises;

determining that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities;

in response to determining the first traffic light state is different than the second traffic light state, performing a traffic light sequence analysis to determine an expected classification result; and

comparing the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state; and

driving, by the onboard vehicle processing system, a vehicle at least partially based on the fused classification result.

2. The method for determining the state of the traffic light of claim 1, wherein the first analysis comprises using a computer vision analysis and the second analysis comprises using a deep learning analysis.

3. The method for determining the state of the traffic light of claim 1, wherein the first analysis additionally outputs a first timestamp and the second analysis additionally outputs a second timestamp.

4. The method for determining the state of the traffic light of claim 3, wherein the first timestamp and the second timestamp are compared to a defined time threshold to determine that the performed first analysis and the performed second analysis are sufficiently recent.

5. The method for determining the state of the traffic light of claim 1, wherein the offline weighting metric is based on weather, distance, time-of-day, or some combination thereof.

6. The method for determining the state of the traffic light of claim 1, wherein the online weighting metric comprises:

decreasing a first amount of weight applied to the first analysis based on the first traffic light state not matching the expected classification result from the traffic light sequence analysis; and

increasing a second amount of weight applied to the second analysis based on the second traffic light state matching the expected classification result from the traffic light sequence analysis.

7. The method for determining the state of the traffic light of claim 6, wherein the first amount of weight applied to the first analysis is decreased only if the first weighting is greater than a minimum weighting threshold.

8. The method for determining the state of the traffic light of claim 1, wherein:

the first analysis comprises analyzing a first image of the scene captured using a long-range image sensor and the second analysis comprises analyzing a second image of the scene captured using a short-range image sensor.

9. A system for determining a state of a traffic light, the system comprising:

one or more image sensors attached with a vehicle;

an onboard vehicle processing system, comprising one or more processors, that receives images from the one or more image sensors, wherein the onboard vehicle processing system is located onboard the vehicle, wherein the one or more processors are configured to;

perform a first analysis of a scene from a first image received from the one or more image sensors that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality corresponds to a different traffic light state;

perform a second analysis of the scene from a second image received from the one or more image sensors that includes the traffic light, wherein;

the second analysis is a different type of analysis than the first analysis;

the second analysis outputs a second plurality of probabilities; and

each probability of the second plurality corresponds to a different traffic light state;

perform a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of;

an offline weighting metric; and

an online weighting metric, wherein;

the online weighting metric is determined by the one or more processors of the onboard vehicle processing system being configured to;

determine that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities;

perform a traffic light sequence analysis to determine an expected classification result in response to determining the first traffic light state is different than the second traffic light state; and

compare the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state; and

drive the vehicle at least partially based on the fused classification result.

10. The system for determining the state of the traffic light of claim 9, wherein the first analysis comprises the onboard vehicle processing system using a computer vision analysis and the second analysis comprises the onboard vehicle processing system using a deep learning analysis.

11. The system for determining the state of the traffic light of claim 9, wherein the first analysis comprises additionally outputs a first timestamp and the second analysis additionally outputs a second timestamp.

12. The system for determining the state of the traffic light of claim 11, wherein the one or more processors are further configured to compare the first timestamp and the second timestamp to a defined time threshold to determine that the performed first analysis and the performed second analysis are sufficiently recent.

13. The system for determining the state of the traffic light of claim 9, wherein the offline weighting metric is based on weather, distance, time-of-day, or some combination thereof.

14. The system for determining the state of the traffic light of claim 9, wherein the online weighting metric further comprises the one or more processors of the onboard vehicle processing system being configured to:

decrease a first amount of weight applied to the first analysis based on the first traffic light state not matching the expected classification result from the traffic light sequence analysis; and

increase a second amount of weight applied to the second analysis based on the second traffic light state matching the expected classification result from the traffic light sequence analysis.

15. The system for determining the state of the traffic light of claim 14, wherein the first amount of weight applied to the first analysis is decreased only if the first weighting is greater than a minimum weighting threshold.

16. The system for determining the state of the traffic light of claim 9, wherein:

the first analysis comprises the one or more processors being configured to analyze the first image of the scene captured using a long-range image sensor of the one or more image sensors and the second analysis comprises analyzing a second image of the scene captured using a short-range image sensor of the one or more image sensors.

17. The system for determining the state of the traffic light of claim 9, wherein the first image and the second image are the same image.

18. A non-transitory processor-readable medium for determining the state of a traffic light comprising processor-readable instructions configured to cause one or more processors to:

perform a first analysis of a scene from a first image that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality corresponds to a different traffic light state;

perform a second analysis of the scene from a second image that includes the traffic light, wherein;

the second analysis is a different type of analysis than the first analysis;

the second analysis outputs a second plurality of probabilities; and

each probability of the second plurality corresponds to a different traffic light state;

perform a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of;

an offline weighting metric; and

an online weighting metric, wherein;

the online weighting metric is determined by the processor-readable instructions being configured to cause the one or more processors to;

determine that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities;

perform a traffic light sequence analysis to determine an expected classification result in response to determining the first traffic light state is different than the second traffic light state; and

compare the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state; and

cause a vehicle to be driven at least partially based on the fused classification result.