Classifier hierarchies for traffic light and traffic indicator detection

• US 9,442,487 B1
• Filed: 08/15/2014
• Issued: 09/13/2016
• Est. Priority Date: 08/15/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ferguson, David Ian Franklin, Wendel, Andreas
• Primary Examiner(s): Beaulieu, Yonel
• Assistant Examiner(s): INGRAM, THOMAS P
• Application Number:
  US14/461,031
• Time in Patent Office: 760 Days
• US Class Current: 1/1
• CPC Class Codes: G01J 3/0289 Field-of-view determination; Aiming or pointing of a spectrometer; Adjusting alignment; Encoding angular position; Size of measurement area; Position tracking
• CPC Class Codes: G01J 3/46 Measurement of colour; Colour measuring devices, e.g. colorimeters measuring colour temperature G01J5/60
• CPC Class Codes: G01J 3/505 measuring the colour produced by lighting fixtures other than screens, monitors, displays or CRTs
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G08G 1/0104 Measuring and analyzing of parameters relative to traffic conditions
• CPC Class Codes: G08G 1/09623 Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/09675 where a selection from the received information takes place in the vehicle
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

The present disclosure is directed to an autonomous vehicle having a vehicle control system. The vehicle control system includes an image processing system. The image processing system receives an image that includes a light indicator. The light indicator includes an illuminated component. The image processing system determines a color of the illuminated component of the light indicator and an associated confidence level of the determination of the color of the illuminated component. The image processing system also determines a shape of the illuminated component of the light indicator and an associated confidence level of the determination of the shape of the illuminated component. The determined confidence levels represent an estimated accuracy of the determinations of the shape and color. Additionally, the image processing system provides instructions executable by a computing device to control the autonomous vehicle based on at least one of the determined confidence levels exceeding a threshold value.

Claims

1. A method comprising:

receiving, from an image-capture device coupled to an autonomous vehicle, an image that includes a light indicator, wherein the light indicator comprises an illuminated component;

determining, from the received image, a color of the illuminated component of the light indicator and an associated confidence level of the determination of the color of the illuminated component;

determining, from the received image, a shape of the illuminated component of the light indicator and an associated confidence level of the determination of the shape of the illuminated component;

based on at least one of the confidence level of the determination of the color and the confidence level of the determination of the shape exceeding a threshold value, determining a characteristic of a state of the light indicator; and

providing instructions executable by a computing device to control the autonomous vehicle based on the determined characteristic of the state of the light indicator.

2. The method of claim 1, wherein determining the color of the illuminated component of the light indicator and determining the shape of the illuminated component of the light indicator are performed in parallel.

3. The method of claim 1, wherein determining the color of the illuminated component of the light indicator and determining the shape of the illuminated component of the light indicator are performed sequentially.

4. The method of claim 1, wherein the instructions comprise first instructions executable by the computing device, wherein the characteristic is a first characteristic, and wherein the method further comprises:

based on the confidence level of the determination of the color exceeding a first threshold value and the confidence level of the determination of the shape being below a second threshold, determining a second characteristic of the state of the light indicator; and

providing instructions executable by the computing device corresponding to the determined color to control the autonomous vehicle based on the determined second characteristic of the state of the light indicator.

5. The method of claim 1, further comprising:

based on the confidence level of the determination of the color being below a first threshold value and the confidence level of the determination of the shape being below a second threshold value, providing instructions executable by the computing device to stop the autonomous vehicle.

6. The method of claim 1, wherein determining the color of the illuminated component is performed independently of determining the shape of the illuminated component.

7. The method of claim 1, wherein the characteristic is a first characteristic, and wherein the method further comprises:

based on the determined color of the illuminated component being red and the confidence level of the determination of the color exceeding a first threshold value, determining a second characteristic of the state of the light indicator; and

providing second instructions executable by the computing device to stop the autonomous vehicle based on the determined second characteristic of the state of the light indicator.

8. The method of claim 1, further comprising:

determining an image patch from the received image, wherein the image patch comprises a portion of the received image containing at least the illuminated component of the light indicator,

wherein the color of the illuminated component of the light indicator is determined from the image patch, and

wherein the shape of the illuminated component of the light indicator is determined from the image patch.

9. An image processing system of an autonomous vehicle comprising:

an image-capture device;

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the image processing system to perform functions comprising;

receiving, from the image-capture device coupled to the autonomous vehicle, an image that includes a particular traffic indicator;

determining, from one or more predetermined colors, (i) a color of the particular traffic indicator in the received image and (ii) an associated confidence level of the determination of the color representative of an estimated accuracy of the determined color;

determining, from one or more predetermined shapes, (i) a shape of the particular traffic indicator in the received image and (ii) an associated confidence level of the determination of the shape representative of an estimated accuracy of the determined shape; and

based on at least one of the confidence level of the determination of the color and the confidence level of the determination of the shape exceeding a threshold value, providing executable instructions to control the autonomous vehicle.

10. The image processing system of claim 9,

wherein (i) determining the color of the traffic indicator and the associated confidence level of the determination of the color and (ii) determining the shape of the traffic indicator and the associated confidence level of the determination of the shape are performed sequentially, and

wherein, based on the confidence level of the determination of the color exceeding the threshold value, the executable instructions are provided prior to determining the shape of the traffic indicator and the associated confidence level of the determination of the shape.

11. The image processing system of claim 9, wherein the executable instructions comprise first executable instructions, and wherein the method further comprises:

based on the confidence level of the determination of the color exceeding a first threshold value and the confidence level of the determination of the shape exceeding a second threshold, providing second instructions executable by the computing device to control the autonomous vehicle corresponding to the determined color and the determined shape.

12. The image processing system of claim 9, wherein the traffic indicator includes a light indicator, and wherein the received image includes a plurality of light indicators, and wherein the functions further comprise:

determining, for given light indicators of the plurality of light indicators, a distance between the autonomous vehicle and the given light indicator; and

determining the particular light indicator as a light indicator nearest to the autonomous vehicle from among the given light indicators based on the determined distances.

13. The image processing system of claim 9, wherein the functions further comprise:

determining a distance between the autonomous vehicle and the particular traffic indicator;

determining whether the distance between the autonomous vehicle and the particular traffic indicator exceeds a distance threshold value; and

based on the distance between the autonomous vehicle and the particular traffic indicator being below the distance threshold value, providing executable instructions to control the autonomous vehicle.

14. The image processing system of claim 9, wherein a predetermined color of the one or more predetermined colors corresponds to providing executable instructions to stop the autonomous vehicle.

15. The image processing system of claim 9, wherein a predetermined color of the one or more predetermined colors corresponds to providing executable instructions to maintain a speed of the autonomous vehicle or accelerate the autonomous vehicle.