Vision-based detection and classification of traffic lights

• US 9,892,332 B1
• Filed: 08/22/2017
• Issued: 02/13/2018
• Est. Priority Date: 08/22/2017
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Wendel, Andreas, Ferguson, David Ian Franklin
• Primary Examiner(s): Lee, Jonathan S
• Application Number:
  US15/682,963
• Time in Patent Office: 175 Days
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

The present disclosure is directed to an autonomous vehicle having a vehicle control system. The vehicle control system includes an image processing system. The image processing system receives an image that includes a plurality of image portions. The image processing system also calculates a score for each image portion. The score indicates a level of confidence that a given image portion represents an illuminated component of a traffic light. The image processing system further identifies one or more candidate portions from among the plurality of image portions. Additionally, the image processing system determines that a particular candidate portion represents an illuminated component of a traffic light using a classifier. Further, the image processing system provides instructions to control the autonomous vehicle based on the particular candidate portion representing an illuminated component of a traffic light.

Claims

1. A method comprising:

receiving, from an image-capture device coupled to an autonomous vehicle, an image of a field of view of the autonomous vehicle, wherein the image comprises a plurality of image portions;

calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light, wherein calculating the respective score for the given image portion comprises identifying at least one area within the given image portion and determining color values and luminance values of pixels in the at least one area;

identifying, from among the plurality of image portions, one or more candidate portions as image portions having a score exceeding a threshold level of confidence;

determining, using a classifier, that a particular candidate portion represents an illuminated component of a traffic light, wherein the classifier has been trained using training data indicative of a plurality of example images that include traffic lights and a plurality of example images that do not include traffic lights; and

based on the particular candidate portion representing an illuminated component of a traffic light, providing instructions executable by a computing device to control the autonomous vehicle.

2. The method of claim 1, wherein identifying at least one area within the given image portion comprises identifying at least a first area within the given image portion and a second area within the given image portion.

3. The method of claim 2, wherein the first area is a center area within the given image portion and the second area is an area surrounding the first area.

4. The method of claim 2, wherein calculating the respective score for the given image portion further comprises:

calculating a first sub-score based on color values and luminance values of pixels in the first area; and

calculating a second sub-score based on color values and luminance values of pixels in the second area.

5. The method of claim 4, wherein calculating the respective score for the given image portion further comprises determining a difference between the second sub-score and the first sub-score.

6. The method of claim 4, wherein calculating a first sub-score based on color values and luminance values of pixels in the first area comprises:

determining, for each pixel in the first area, a respective pixel color and assigning a respective color value based on a similarity between the respective pixel color and an expected traffic light color, so as to determine a plurality of color values;

determining, for each pixel in the first area, a respective pixel luminance and assigning a respective luminance value based on the respective pixel luminance, so as to determine a plurality of luminance values; and

calculating the first sub-score based on the determined color values and the determined luminance values.

7. The method of claim 3, wherein identifying at least one area within the given image portion further comprises identifying a third area within the given image portion, wherein the third area is located between the first area and the second area.

8. An image processing system comprising:

an image-capture device;

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the image processing system to perform functions comprising;

receiving, from the image-capture device coupled to an autonomous vehicle, an image of a field of view of the autonomous vehicle, wherein the image comprises a plurality of image portions;

calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light, wherein calculating the respective score for the given image portion comprises identifying at least one area within the given image portion and determining color values and luminance values of pixels in the at least one area;

identifying, from among the plurality of image portions, one or more candidate portions as image portions having a score exceeding a threshold level of confidence;

determining that a particular candidate portion represents an illuminated component of a traffic light based on training data indicative of a plurality of example images that include traffic lights and a plurality of example images that do not include traffic lights; and

based on the particular candidate portion representing an illuminated component of a traffic light, providing executable instructions to control the autonomous vehicle.

9. The image processing system of claim 8, wherein identifying at least one area within the given image portion comprises identifying at least a first area within the given image portion and a second area within the given image portion.

10. The image processing system of claim 9, wherein the first area is a center area within the given image portion and the second area is an area surrounding the first area.

11. The image processing system of claim 9, wherein calculating the respective score for the given image portion further comprises:

calculating a first sub-score based on color values and luminance values of pixels in the first area; and

calculating a second sub-score based on color values and luminance values of pixels in the second area.

12. The image processing system of claim 11, wherein calculating the respective score for the given image portion further comprises determining a difference between the second sub-score and the first sub-score.

13. The image processing system of claim 11, wherein calculating a first sub-score based on color values and luminance values of pixels in the first area comprises:

determining, for each pixel in the first area, a respective pixel color and assigning a respective color value based on a similarity between the respective pixel color and an expected traffic light color, so as to determine a plurality of color values;

determining, for each pixel in the first area, a respective pixel luminance and assigning a respective luminance value based on the respective pixel luminance, so as to determine a plurality of luminance values; and

calculating the first sub-score based on the determined color values and the determined luminance values.

14. The image processing system of claim 10, wherein identifying at least one area within the given image portion further comprises identifying a third area within the given image portion, wherein the third area is located between the first area and the second area.

15. A non-transitory computer-readable medium having stored thereon instructions that, upon execution by at least one processor of a computing device, cause the computing device to perform functions comprising:

receiving, from an image-capture device coupled to an autonomous vehicle, an image of a field of view of the autonomous vehicle, wherein the image comprises a plurality of image portions;

calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light, wherein calculating the respective score of the given image portion comprises identifying at least one area within the given image portion and determining color values and luminance values of pixels in the at least one area;

identifying, from among the plurality of image portions, one or more candidate portions as image portions having a score exceeding a threshold level of confidence;

determining, using a classifier, that a particular candidate portion represents an illuminated component of a traffic light, wherein the classifier has been trained using training data indicative of a plurality of example images that include traffic lights and a plurality of example images that do not include traffic lights; and

based on the particular candidate portion representing an illuminated component of a traffic light, providing instructions executable by the computing device to control the autonomous vehicle.

16. The non-transitory computer-readable medium of claim 15, wherein identifying at least one area within the given image portion comprises identifying at least a first area within the given image portion and a second area within the given image portion.

17. The non-transitory computer-readable medium of claim 16, wherein the first area is a center area within the given image portion and the second area is an area surrounding the first area.

18. The non-transitory computer-readable medium of claim 16, wherein calculating the respective score for the given image portion further comprises:

calculating a first sub-score based on color values and luminance values of pixels in the first area; and

calculating a second sub-score based on color values and luminance values of pixels in the second area.

19. The non-transitory computer-readable medium of claim 18, wherein calculating the respective score for the given image portion further comprises determining a difference between the second sub-score and the first sub-score.

20. The non-transitory computer-readable medium of claim 18, wherein calculating a first sub-score based on color values and luminance values of pixels in the first area comprises:

determining, for each pixel in the first area, a respective pixel color and assigning a respective color value based on a similarity between the respective pixel color and an expected traffic light color, so as to determine a plurality of color values;

determining, for each pixel in the first area, a respective pixel luminance and assigning a respective luminance value based on the respective pixel luminance, so as to determine a plurality of luminance values; and

calculating the first sub-score based on the determined color values and the determined luminance values.