Classifier hierarchies for traffic light and traffic indicator detection

• US 9,690,297 B1
• Filed: 08/16/2016
• Issued: 06/27/2017
• Est. Priority Date: 08/15/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Ferguson, David Ian Franklin, Wendel, Andreas
• Primary Examiner(s): Jabr, Fadey
• Assistant Examiner(s): Ingram, Thomas
• Application Number:
  US15/238,372
• Time in Patent Office: 315 Days
• CPC Class Codes: G01J 3/0289 Field-of-view determination; Aiming or pointing of a spectrometer; Adjusting alignment; Encoding angular position; Size of measurement area; Position tracking
• CPC Class Codes: G01J 3/46 Measurement of colour; Colour measuring devices, e.g. colorimeters measuring colour temperature G01J5/60
• CPC Class Codes: G01J 3/505 measuring the colour produced by lighting fixtures other than screens, monitors, displays or CRTs
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/0104 Measuring and analyzing of parameters relative to traffic conditions
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/09675 where a selection from the received information takes place in the vehicle
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

The present disclosure is directed to an autonomous vehicle having a vehicle control system. The vehicle control system includes an image processing system. The image processing system receives an image that includes a light indicator. The light indicator includes an illuminated component. The image processing system determines a color of the illuminated component of the light indicator and an associated confidence level of the determination of the color of the illuminated component. The image processing system also determines a shape of the illuminated component of the light indicator and an associated confidence level of the determination of the shape of the illuminated component. The determined confidence levels represent an estimated accuracy of the determinations of the shape and color. Additionally, the image processing system provides instructions executable by a computing device to control the autonomous vehicle based on at least one of the determined confidence levels exceeding a threshold value.

Claims

1. A non-transitory computer-readable medium having stored thereon instructions that, upon execution by at least one processor of a computing device, cause the computing device to perform functions comprising:

receiving, from an image-capture device coupled to an autonomous vehicle, an image that includes a light indicator, wherein the light indicator comprises an illuminated component;

determining, using the image, a color of the illuminated component of the light indicator and an associated confidence level of the determination of the color based on first training data indicative of a first plurality of illuminated components of light indicators;

determining, using the image, a shape of the illuminated component of the light indicator and an associated confidence level of the determination of the shape based on second training data indicative of a second plurality of illuminated components of light indicators;

based on at least one of the confidence level of the determination of the color and the confidence level of the determination of the shape exceeding a threshold value, determining a characteristic of a state of the light indicator; and

controlling the autonomous vehicle based on the determined characteristic of the state of the light indicator.

2. The non-transitory computer-readable medium of claim 1, wherein the first training data includes positive examples and negative examples corresponding to a color, wherein the positive examples indicate a first range of attribute values associated with the color, and wherein the negative examples indicate a second range of attribute values dissociated with the color, and

wherein determining the confidence level of the determination of the color comprises:

determining an attribute value using the image;

determining a first likelihood that the determined attribute is within the first range of attribute values;

determining a second likelihood that the determined attribute is within the second range of attribute values; and

determining the confidence level of the determination of the color based on the first likelihood and the second likelihood.

3. The non-transitory computer-readable medium of claim 1, wherein the second training data includes positive examples and negative examples corresponding to a shape, wherein the positive examples indicate a first range of attribute values associated with the shape, and wherein the negative examples indicate a second range of attribute values dissociated with the particular shape, and wherein determining the confidence level of the determination of the shape comprises:

determining an attribute value using the image;

determining a first likelihood that the determined attribute is within the first range of attribute values;

determining a second likelihood that the determined attribute is within the second range of attribute values; and

determining the confidence level of the determination of the shape based on the first likelihood and the second likelihood.

4. The non-transitory computer-readable medium of claim 1, wherein the functions further comprise:

determining, using the image, a luminance of the illuminated component of the light indicator and an associated confidence level of the determination of the luminance based on third training data indicative of a third plurality of illuminated components of light indicators.

5. The non-transitory computer-readable medium of claim 1, wherein the functions further comprise:

determining a length of time since controlling the autonomous vehicle based on the determined characteristic of the state of the light indicator; and

based on the length of time exceeding a threshold length of time, repeating the functions of (i) receiving the image, (ii) determining the color of the illuminated component of the light indicator and the associated confidence level of the determination of the color, (iii) determining the shape of the illuminated component of the light indicator and the associated confidence level of the determination of the color, (iv) determining the characteristic of the state of the traffic indicator, and (v) controlling the autonomous vehicle based on the determined characteristic of the state of the traffic indicator.

6. The non-transitory computer-readable medium of claim 1, wherein the function of determining the color of the illuminated component of the light indicator and the function of determining the shape of the illuminated component of the light indictor are performed in parallel.

7. The non-transitory computer-readable medium of claim 1, wherein the function of determining the color of the illuminated component of the light indicator and the function of determining the shape of the illuminated component of the light indictor are performed in a sequential order.

8. The non-transitory computer-readable medium of claim 1, wherein the functions further comprise:

based on the confidence level of the determination of the color being below a first threshold value and the confidence level of the determination of the shape being below a second threshold value, causing the autonomous vehicle to stop.

9. The non-transitory computer-readable medium of claim 1, wherein the functions further comprise:

determining an image patch using the image, wherein the image patch comprises a portion of the image containing at least the illuminated component of the light indicator, wherein the color of the illuminated component of the light indicator is determined from the image patch, and wherein the shape of the illuminated component of the light indicator is determined from the image patch.

10. A system comprising:

an image-capture device;

a processor; and

a memory having stored thereon instructions that, upon execution by the processor, cause the image processing system to perform functions comprising;

receiving, using an image-capture device coupled to an autonomous vehicle, an image that includes a light indicator having an illuminated component;

determining, using the image and a first set of training data, a color of the illuminated component and a confidence level of the determination of the color;

determining, using the image and a second set of training data, a shape of the illuminated component and a confidence level of the determination of the shape;

based on at least one of the confidence level of the determination of the color and the confidence level of the determination of the shape exceeding a threshold value, determining a state of the light indicator; and

controlling the autonomous vehicle based on the state of the light indicator.

11. The system of claim 10, wherein the first set of training data includes positive examples indicative of a first range of attribute values associated with a color and negative examples indicative of a second range of attribute values dissociated with the color.

12. The system of claim 11, wherein determining the confidence level of the determination of the color comprises:

determining, using the image, an attribute value;

determining a first likelihood that the attribute value is within the first range of attribute values associate with the color;

determining a second likelihood that the attribute value is within the second range of attribute values dissociated with the color; and

based on the first likelihood and the second likelihood, determining the confidence level of the determination of the color.

13. The system of claim 10, wherein the first set of training data includes positive examples indicative of a first range of attribute values associated with a shape and negative examples indicative of a second range of attribute values dissociated with the shape.

14. The system of claim 13, wherein determining the confidence level of the determination of the shape comprises:

determining, using the image, an attribute value;

determining a first likelihood that the attribute value is within the first range of attribute values associate with the shape;

determining a second likelihood that the attribute value is within the second range of attribute values dissociated with the shape; and

based on the first likelihood and the second likelihood, determining the confidence level of the determination of the shape.

15. A method comprising:

receiving, using an image-capture device coupled to an autonomous vehicle, an image that includes a light indicator having an illuminated component;

determining, using the image and a first set of training data, a color of the illuminated component and a confidence level of the determination of the color;

determining, using the image and the second set of training data, a shape of the illuminated component and a confidence level of the determination of the shape;

based on at least one of the confidence level of the determination of the color and the confidence level of the determination of the shape exceeding a threshold value, determining a state of the light indicator; and

controlling the autonomous vehicle based on the state of the light indicator.

16. The method of claim 15, wherein determining the color of the illuminated component and determining the shape of the illuminated component are performed in parallel.

17. The method of claim 15, wherein determining the color of the illuminated component and determining the shape of the illuminated component are performed in a sequential order.

18. The method of claim 15, further comprising:

based on the confidence level of the determination of the color having a value less than a first threshold value and the confidence level of the determination of the shape having a value less than a second threshold value, causing the autonomous vehicle to stop.

19. The method of claim 15, wherein determining, using the image and the first set of training data, the color of the illuminated component and the confidence level of the determination of the color comprises:

determining that the color of the illuminated component is red and that the confidence level of the determination of the color exceeds a first threshold value; and

based on determining that the color is red and that the confidence level of the determination of the color exceeds the first threshold value, causing the autonomous vehicle to stop.

20. The method of claim 15, wherein the first set of training data corresponds to a first plurality of illuminated components of light indicators, and wherein the second set of training data corresponds to a second plurality of illuminated components of light indicators.