Using object appearance changes due to high reflectivity for feature detection

• US 8,995,721 B1
• Filed: 02/04/2013
• Issued: 03/31/2015
• Est. Priority Date: 02/04/2013
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fairfield, Nathaniel, Ferguson, Dave
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US13/759,033
• Time in Patent Office: 785 Days
• Field of Search: 382/100, 382103-107, 382/155, 382/162, 382/168, 382/173, 382/181, 382193-194, 382/199, 382/209, 382/224, 382/232, 382/254, 382/274, 382/276, 382286-295, 382/305, 382/312, 348/148, 348/990, 348/988, 701/36, 701/1
• US Class Current: 382/103
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

Methods and apparatus are disclosed related to classifying illuminated objects. A computing device can receive a sequence of images including first and second images. The first image can be of an environment of a vehicle taken at a first time. The second image can be of the environment of the vehicle taken at a second time. The vehicle can include a light source that illuminates a portion of the environment. The first and second times can differ. The computing device can detect an object having a first size and a first brightness in the first image. The computing device can detect the object having a second size and a second brightness in the second image. The computing device can classify the object based on a brightness difference between the first brightness and second brightness and a size difference between the first size and second size.

Claims

1. A method, comprising:

receiving, by a computing device, a sequence of images comprising a first image and a second image, wherein the first image is taken at a first time of an environment of a vehicle, wherein the second image is taken at a second time of the environment of the vehicle, wherein the vehicle comprises a light source operable to illuminate a portion of the environment within an illumination boundary, and wherein the first time is different from the second time;

detecting, by the computing device, an object in the first image, wherein the object has a first size and a first brightness in the first image, and wherein the object is not within the illumination boundary in the first image;

detecting, by the computing device, the object in the second image, wherein the object has a second size and a second brightness in the second image, and wherein the object is within the illumination boundary in the second image; and

classifying the object, by the computing device, wherein classifying the object comprises classifying the object as one of an actively illuminated object or a passively illuminated object based on at least the first brightness of the object when the object is not within the illumination boundary and the second brightness of the object when the object is within the illumination boundary.

2. The method of claim 1, further comprising determining a size difference between the second size of the object and the first size of the object, wherein classifying the object further comprises:

classifying the object as an object moving toward the vehicle when the size difference exceeds a first predetermined size threshold;

classifying the object as an object moving with the vehicle when the size difference is less than the first predetermined size threshold and greater than a second predetermined size threshold; and

classifying the object as an object moving away from the vehicle when the size difference is less than the second predetermined size threshold.

3. The method of claim 1, wherein classifying the object comprises classifying the object as a passively illuminated object when the second brightness of the object is greater than the first brightness of the object.

4. The method of claim 1, further comprising:

determining a brightness difference between the second brightness of the object and the first brightness of the object;

determining whether the brightness difference exceeds a high-reflectivity threshold,

in response to determining that the brightness difference exceeds the high-reflectivity threshold, determining that the object is a highly reflective object, and

in response to determining that the brightness difference does not exceed the high-reflectivity threshold, determining that the object is a slightly reflective object.

5. The method of claim 1, further comprising:

determining a brightness difference between the second brightness of the object and the first brightness of the object;

determining whether the brightness difference is less than a potentially-active threshold;

in response to determining that the brightness difference is less than the potentially-active threshold, determining whether the first brightness exceeds an active-brightness threshold; and

in response to determining that the first brightness exceeds the active-brightness threshold, classifying the object as an actively illuminated object.

6. The method of claim 1, wherein classifying the object further comprises determining whether the object is associated with a traffic sign.

7. The method of claim 6, wherein determining whether the object is associated with the traffic sign comprises:

detecting a first illumination pattern in the first image for the object;

detecting a second illumination pattern in the second image for the object;

determining whether a sequence of illumination patterns is indicative of a traffic sign, wherein the sequence of illumination patterns comprises the first illumination pattern and second illumination pattern; and

in response to determining that the sequence of illumination patterns is indicative of a traffic sign, determining that the object is associated with the traffic sign.

8. The method of claim 7, wherein at least one illumination pattern in the sequence of illumination patterns is a partial illumination pattern.

9. The method of claim 6, wherein determining whether the object is associated with the traffic sign comprises:

performing character recognition on at least a portion of the second image associated with the object to determine text associated with the object;

searching for the text associated with the object in a traffic-sign database storing texts of traffic signs; and

in response to finding the text associated with the object in the traffic-sign database, determining that the object is associated with the traffic sign.

10. A device including a computer-readable storage medium having stored thereon program instructions that, upon execution by a computing device, cause the computing device to perform operations comprising:

receiving a sequence of images comprising a first image and a second image, wherein the first image is taken at a first time of an environment of a vehicle, wherein the second image is taken at a second time of the environment of the vehicle, wherein the vehicle comprises a light source operable to illuminate a portion of the environment within an illumination boundary, and wherein the first time is different from the second time;

detecting an object in the first image, wherein the object has a first size and a first brightness in the first image, and wherein the object is not within the illumination boundary in the first image;

detecting the object in the second image, wherein the object has a second size and a second brightness in the second image, and wherein the object is within the illumination boundary in the second image; and

classifying the object, wherein classifying the object comprises classifying the object as one of an actively illuminated object or a passively illuminated objected based on at least the first brightness of the object when the object is not within the illumination boundary and the second brightness of the object when the object is within the illumination boundary.

11. The device of claim 10, further comprising determining a size difference between the second size of the object and the first size of the object, wherein classifying the object further comprises:

classifying the object as an object moving toward the vehicle when the size difference exceeds a first predetermined size threshold;

classifying the object as an object moving with the vehicle when the size difference is less than the first predetermined size threshold and greater than a second predetermined size threshold; and

classifying the object as an object moving away from the vehicle when the size difference is less than the second predetermined size threshold.

12. The device of claim 10, wherein classifying the object comprises classifying the object as a passively reflective object when the second brightness of the object is greater than the first brightness of the object.

13. The device of claim 10, the operations further comprising:

determining a brightness difference between the second brightness of the object and the first brightness of the object;

determining whether the brightness difference exceeds a high-reflectivity threshold,

in response to determining that the brightness difference exceeds the high-reflectivity threshold, determining that the object is a highly reflective object, and

in response to determining that the brightness difference does not exceed the high-reflectivity threshold, determining that the object is a slightly reflective object.

14. The device of claim 10, the operations further comprising:

determining a brightness difference between the second brightness of the object and the first brightness of the object;

determining whether the brightness difference is less than a potentially-active threshold;

in response to determining that the difference is less than the potentially-active threshold, determining whether the first brightness exceeds an active-brightness threshold; and

in response to determining that the first brightness exceeds the active-brightness threshold, classifying the object as an actively illuminated object.

15. The device of claim 10, wherein classifying the object further comprises determining whether the object is associated with a traffic sign, based on at least the second image.

16. The device of claim 15, wherein determining whether the object is associated with a traffic sign comprises:

detecting a first illumination pattern in the first image for the object;

detecting a second illumination pattern in the second image for the object;

determining whether a sequence of illumination patterns is indicative of a traffic sign, wherein the sequence of illumination patterns comprises the first illumination pattern and second illumination pattern; and

in response to determining that the sequence of illumination patterns is indicative of a traffic sign, determining that the object is associated with the traffic sign.

17. The device of claim 15, wherein determining whether the object is associated with a traffic sign comprises:

performing character recognition on at least a portion of the second image associated with the object to determine text associated with the object;

searching for the text associated with the object in a traffic-sign database storing texts of traffic signs; and

in response to finding the text associated with the object in the traffic-sign database, determining that the object is associated with the traffic sign.

18. A computing device, comprising:

a processor; and

a non-transitory computer-readable storage medium, storing instructions thereon that, when executed by the processor, cause the computing device to perform operations, comprising;

receiving a sequence of images comprising a first image and a second image, wherein the first image is taken at a first time of an environment of a vehicle, wherein the second image is taken at a second time of the environment of the vehicle, wherein the vehicle comprises a light source operable to illuminate a portion of the environment within an illumination boundary, and wherein the first time is different from the second time,

detecting an object in the first image, wherein the object has a first size and a first brightness in the first image, and wherein the object is not within the illumination boundary in the first image,

detecting the object in the second image, wherein the object has a second size and a second brightness in the second image, and wherein the object is within the illumination boundary in the second image, and

classifying the object, wherein the classifying the object comprises classifying the object as one of an actively illuminated object or a passively illuminated object based on at least the first brightness of the object when the object is not within the illumination boundary and the second brightness of the object when the object is within the illumination boundary.

19. The computing device of claim 18, further comprising determining a size difference between the second size of the object and the first size of the object, wherein classifying the object further comprises:

classifying the object as an object moving toward the vehicle when the size difference exceeds a first predetermined size threshold;

classifying the object as an object moving with the vehicle is the size difference is less than the first predetermined size threshold and greater than a second predetermined size threshold; and

classifying the object as an object moving away from the vehicle is the size difference is less than the second predetermined size threshold.

20. The computing device of claim 18, wherein the operation of classifying the object comprises classifying the object as a passively illuminated object when the second brightness of the object is greater than the first brightness of the object.

21. The computing device of claim 18, the operations further comprising:

determining a brightness difference between the second brightness of the object and the first brightness of the object;

determining whether the difference exceeds a high-reflectivity threshold,

in response to determining that the difference exceeds the high-reflectivity threshold, determining that the object is a highly reflective object, and

in response to determining that the difference does not exceed the high-reflectivity threshold, determining that the object is a slightly reflective object.

22. The computing device of claim 18, the operations further comprising:

determining a brightness difference between the second brightness of the object and the first brightness of the object;

determining whether the brightness difference is less than a potentially-active threshold;

in response to determining that the difference is less than the potentially-active threshold, determining whether the first brightness exceeds an active-brightness threshold; and

in response to determining that the first brightness exceeds the active-brightness threshold, classifying the object as an actively illuminated object.

23. The computing device of claim 18, wherein classifying the object further comprises determining whether the object is associated with a traffic sign, based on the first and second images.