Vision-based detection and classification of traffic lights

• US 9,779,314 B1
• Filed: 11/11/2014
• Issued: 10/03/2017
• Est. Priority Date: 08/21/2014
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Wendel, Andreas, Ferguson, David Ian Franklin
• Primary Examiner(s): Lee, Jonathan S
• Application Number:
  US14/538,669
• Time in Patent Office: 1,057 Days
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

The present disclosure is directed to an autonomous vehicle having a vehicle control system. The vehicle control system includes an image processing system. The image processing system receives an image that includes a plurality of image portions. The image processing system also calculates a score for each image portion. The score indicates a level of confidence that a given image portion represents an illuminated component of a traffic light. The image processing system further identifies one or more candidate portions from among the plurality of image portions. Additionally, the image processing system determines that a particular candidate portion represents an illuminated component of a traffic light using a classifier. Further, the image processing system provides instructions to control the autonomous vehicle based on the particular candidate portion representing an illuminated component of a traffic light.

Claims

1. A method comprising:

receiving, from an image-capture device coupled to an autonomous vehicle, an image of a field of view of the autonomous vehicle, wherein the image comprises a plurality of image portions;

calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light, wherein calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light comprises;

calculating a first sub-score for a center area of a particular image portion based on at least one characteristic of the center area of the particular image portion;

calculating a second sub-score for an outer area of the particular image portion surrounding the center area of the particular image portion based on at least one characteristic of the outer area of the particular image portion; and

calculating the score based on a difference between the second sub-score and the first sub-score;

identifying, from among the plurality of image portions, one or more candidate portions as image portions having a score exceeding a threshold level of confidence;

determining, using a classifier, that a particular candidate portion represents an illuminated component of a traffic light, wherein the classifier has been trained using training data indicative of a plurality of example images that include traffic lights and a plurality of example images that do not include traffic lights; and

based on the particular candidate portion representing an illuminated component of a traffic light, providing instructions executable by a computing device to control the autonomous vehicle.

2. The method of claim 1,

wherein the particular image portion comprises a plurality of pixels,

wherein the at least one characteristic of the center area of the particular image portion includes a respective color value of each pixel within the center area of the particular image portion, and

wherein the at least one characteristic of the outer area of the particular image portion includes a respective color value of each pixel within the outer area of the particular image portion.

3. The method of claim 2,

wherein a respective color value of each given pixel is associated with a score value,

wherein a predetermined set of color values is associated with a score value that is greater than a score value associated with a color value not within the predetermined set of color values,

wherein calculating the first sub-score comprises summing the score values associated with the respective color value of each pixel within the center area of the particular image portion, and

wherein calculating the second sub-score comprises summing the score values associated with the respective color value of each pixel within the outer area of the particular image portion.

4. The method of claim 2, further comprising:

determining that at least one of the pixels of the particular candidate portion is a particular color corresponding to an expected color of an illuminated portion of a traffic light; and

based on at least one of the pixels of the particular candidate portion being the particular color, providing instructions executable by the computing device to control the autonomous vehicle.

5. The method of claim 1, wherein the threshold level of confidence is a first threshold level of confidence, and wherein determining, using the classifier, that the particular candidate portion represents an illuminated component of a traffic light comprises:

determining, for the particular candidate portion, a confidence value indicating a level of confidence that the particular candidate portion represents an illuminated component of a traffic light;

determining that the confidence value exceeds a second threshold level of confidence; and

based on the confidence value exceeding the second threshold level of confidence, providing instructions executable by the computing device to control the autonomous vehicle.

6. An image processing system comprising:

an image-capture device;

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the image processing system to perform functions comprising;

receiving, from the image-capture device coupled to an autonomous vehicle, an image of a field of view of the autonomous vehicle, wherein the image comprises a plurality of image portions;

calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light, wherein calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light comprises;

calculating a first sub-score for a center area of a particular image portion based on at least one characteristic of the center area of the particular image portion, wherein the center area is surrounded by a middle area of the particular image portion;

calculating a second sub-score for an outer area of the particular image portion surrounding the middle area of the particular image portion based on at least one characteristic of the outer area of the particular image portion, wherein the middle area lies between the outer area and the center area; and

calculating the score based on a difference between the second sub-score and the first sub-score;

identifying, from among the plurality of image portions, one or more candidate portions as image portions having a score exceeding a threshold level of confidence;

determining that a particular candidate portion represents an illuminated component of a traffic light based on training data indicative of a plurality of example images that include traffic lights and a plurality of example images that do not include traffic lights; and

based on the particular candidate portion representing an illuminated component of a traffic light, providing executable instructions to control the autonomous vehicle.

7. The image processing system of claim 6, wherein the functions further comprise:

determining an image patch from the received image, wherein the image patch comprises a portion of the received image containing at least the traffic light.

8. The image processing system of claim 7, wherein the functions further comprise:

receiving traffic light location data indicative of pre-determined locations of traffic lights, wherein determining the image patch from the received image is based on the received traffic light location data.

9. The image processing system of claim 6, wherein the functions further comprise:

determining a color of the illuminated component of the traffic light for the particular candidate portion; and

based on the determined color of the illuminated component of the traffic light for the particular candidate portion, providing executable instructions to control the autonomous vehicle.

10. The image processing system of claim 9, wherein the functions further comprise:

determining that the color of the illuminated component of the traffic light is red; and

based on determining that the color of the illuminated component of the traffic light is red, providing executable instructions to stop the autonomous vehicle.

11. The image processing system of claim 6, wherein the functions further comprise:

determining a shape of the illuminated component of the traffic light for the particular candidate portion; and

based on the determined shape of the illuminated component of the traffic light for the particular candidate portion, providing executable instructions to control the autonomous vehicle.

12. The image processing system of claim 6, wherein the functions further comprise:

determining a luminance of the illuminated component of the traffic light for the particular candidate portion; and

based on the determined luminance of the illuminated component of the traffic light for the particular candidate portion, providing executable instructions to control the autonomous vehicle.

18. The image processing system of claim 6,

wherein the particular image portion comprises a plurality of pixels,

wherein the at least one characteristic of the center area of the particular image portion includes a respective color value of each pixel within the center area of the particular image portion, and

wherein the at least one characteristic of the outer area of the particular image portion includes a respective color value of each pixel within the outer area of the particular image portion.

19. The image processing system of claim 18,

wherein a respective color value of each given pixel is associated with a score value, wherein a predetermined set of color values is associated with a score value that is greater than a score value associated with a color value not within the predetermined set of color values,

wherein calculating the first sub-score comprises summing the score values associated with the respective color value of each pixel within the center area of the particular image portion, and

wherein calculating the second sub-score comprises summing the score values associated with the respective color value of each pixel within the outer area of the particular image portion.

20. The image processing system of claim 18, wherein the functions further comprise:

determining that at least one of the pixels of the particular candidate portion is a particular color corresponding to an expected color of an illuminated portion of a traffic light; and

based on at least one of the pixels of the particular candidate portion being the particular color, providing instructions executable by the computing device to control the autonomous vehicle.

13. A non-transitory computer-readable medium having stored thereon instructions that, upon execution by at least one processor of a computing device, cause the computing device to perform functions comprising:

receiving, from an image-capture device coupled to an autonomous vehicle, an image of a field of view of the autonomous vehicle, wherein the image comprises a plurality of image portions;

calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light, wherein calculating, for each given image portion, a respective score indicating a level of confidence that the given image portion represents an illuminated component of a traffic light comprises;

calculating a first sub-score for a center area of a particular image portion based on at least one characteristic of the center area of the particular image portion;

calculating a second sub-score for an outer area of the particular image portion surrounding the center area of the particular image portion based on at least one characteristic of the outer area of the particular image portion; and

calculating the score based on a difference between the second sub-score and the first sub-score;

identifying, from among the plurality of image portions, one or more candidate portions as image portions having a score exceeding a threshold level of confidence;

determining, using a classifier, that a particular candidate portion represents an illuminated component of a traffic light, wherein the classifier has been trained using training data indicative of a plurality of example images that include traffic lights and a plurality of example images that do not include traffic lights; and

based on the particular candidate portion representing an illuminated component of a traffic light, providing instructions executable by the computing device to control the autonomous vehicle.

14. The non-transitory computer-readable medium of claim 13,

wherein the particular image portion comprises a plurality of pixels,

wherein the at least one characteristic of the center area of the particular image portion includes a respective color value of each pixel within the center area of the particular image portion, and

wherein the at least one characteristic of the outer area of the particular image portion includes a respective color value of each pixel within the outer area of the particular image portion.

15. The non-transitory computer-readable medium of claim 14,

wherein a respective color value of each given pixel is associated with a score value, wherein a predetermined set of color values is associated with a score value that is greater than a score value associated with a color value not within the predetermined set of color values,

wherein calculating the first sub-score comprises summing the score values associated with the respective color value of each pixel within the center area of the particular image portion, and

wherein calculating the second sub-score comprises summing the score values associated with the respective color value of each pixel within the outer area of the particular image portion.

16. The non-transitory computer-readable medium of claim 14, wherein the functions further comprise:

determining that at least one of the pixels of the particular candidate portion is a particular color corresponding to an expected color of an illuminated portion of a traffic light; and

based on at least one of the pixels of the particular candidate portion being the particular color, providing instructions executable by the computing device to control the autonomous vehicle.

17. The non-transitory computer-readable medium of claim 13, wherein the threshold level of confidence is a first threshold level of confidence, and wherein determining, using the classifier, that the particular candidate portion represents an illuminated component of a traffic light comprises:

determining, for the particular candidate portion, a confidence value indicating a level of confidence that the particular candidate portion represents an illuminated component of a traffic light;

determining that the confidence value exceeds a second threshold level of confidence; and

based on the confidence value exceeding the second threshold level of confidence, providing instructions executable by the computing device to control the autonomous vehicle.