Fusion-based traffic light recognition for autonomous driving
First Claim
Patent Images
1. A method for determining a state of a traffic light, the method comprising:
- performing, by an onboard vehicle processing system, a first analysis of a scene that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality of probabilities corresponds to a different traffic light state;
performing, by the onboard vehicle processing system, a second analysis of the scene that includes the traffic light, wherein;
the second analysis is a different type of analysis than the first analysis;
the second analysis outputs a second plurality of probabilities; and
each probability of the second plurality of probabilities corresponds to a different traffic light state;
performing, by the onboard vehicle processing system, a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of;
an offline weighting metric; and
an online weighting metric;
determining the online weighting metric, which comprises;
determining that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities;
in response to determining the first traffic light state is different than the second traffic light state, performing a traffic light sequence analysis to determine an expected classification result; and
comparing the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state; and
driving, by the onboard vehicle processing system, a vehicle at least partially based on the fused classification result.
2 Assignments
0 Petitions
Accused Products
Abstract
Various arrangements are presented for determining a state of a traffic light. A first analysis of a scene may be performed that includes the traffic light. The first analysis may result in the output of a first classification result and a first confidence score. A second, different analysis of the scene may be performed that also outputs a classification result and a confidence score. A fusion process may be performed to fuse the classification results into a fused classification result based on the confidence scores and one or more weighting metrics. A vehicle may then be driven at least partially based on the fused classification result.
-
Citations
18 Claims
-
1. A method for determining a state of a traffic light, the method comprising:
-
performing, by an onboard vehicle processing system, a first analysis of a scene that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality of probabilities corresponds to a different traffic light state; performing, by the onboard vehicle processing system, a second analysis of the scene that includes the traffic light, wherein; the second analysis is a different type of analysis than the first analysis; the second analysis outputs a second plurality of probabilities; and each probability of the second plurality of probabilities corresponds to a different traffic light state; performing, by the onboard vehicle processing system, a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of;
an offline weighting metric; and
an online weighting metric;determining the online weighting metric, which comprises; determining that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities; in response to determining the first traffic light state is different than the second traffic light state, performing a traffic light sequence analysis to determine an expected classification result; and comparing the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state; and driving, by the onboard vehicle processing system, a vehicle at least partially based on the fused classification result. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A system for determining a state of a traffic light, the system comprising:
-
one or more image sensors attached with a vehicle; an onboard vehicle processing system, comprising one or more processors, that receives images from the one or more image sensors, wherein the onboard vehicle processing system is located onboard the vehicle, wherein the one or more processors are configured to; perform a first analysis of a scene from a first image received from the one or more image sensors that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality corresponds to a different traffic light state; perform a second analysis of the scene from a second image received from the one or more image sensors that includes the traffic light, wherein; the second analysis is a different type of analysis than the first analysis; the second analysis outputs a second plurality of probabilities; and each probability of the second plurality corresponds to a different traffic light state; perform a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of;
an offline weighting metric; and
an online weighting metric, wherein;
the online weighting metric is determined by the one or more processors of the onboard vehicle processing system being configured to;determine that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities; perform a traffic light sequence analysis to determine an expected classification result in response to determining the first traffic light state is different than the second traffic light state; and compare the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state; and drive the vehicle at least partially based on the fused classification result. - View Dependent Claims (10, 11, 12, 13, 14, 15, 16, 17)
-
-
18. A non-transitory processor-readable medium for determining the state of a traffic light comprising processor-readable instructions configured to cause one or more processors to:
-
perform a first analysis of a scene from a first image that includes the traffic light, wherein the first analysis outputs a first plurality of probabilities, wherein each probability of the first plurality corresponds to a different traffic light state;
perform a second analysis of the scene from a second image that includes the traffic light, wherein;the second analysis is a different type of analysis than the first analysis; the second analysis outputs a second plurality of probabilities; and each probability of the second plurality corresponds to a different traffic light state; perform a fusion process to fuse the first plurality of probabilities indicative of the state of the traffic light and the second plurality of probabilities indicative of the state of the traffic light into a fused classification result, wherein the fusion process is based on at least one weighting metric selected from a group consisting of;
an offline weighting metric; and
an online weighting metric, wherein;
the online weighting metric is determined by the processor-readable instructions being configured to cause the one or more processors to;determine that a first traffic light state having a first greatest probability of the first plurality of probabilities is different than a second traffic light state having a second greatest probability of the second plurality of probabilities; perform a traffic light sequence analysis to determine an expected classification result in response to determining the first traffic light state is different than the second traffic light state; and compare the expected classification result from the traffic light sequence analysis to the first traffic light state and the second traffic light state; and cause a vehicle to be driven at least partially based on the fused classification result.
-
Specification