Vision-based object detection using a polar grid

• US 9,766,628 B1
• Filed: 04/04/2014
• Issued: 09/19/2017
• Est. Priority Date: 04/04/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Lo, Wan-Yen, Ferguson, David Ian Franklin, Ogale, Abhijit
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): RIVERA-MARTINEZ, GUILLERMO M
• Application Number:
  US14/244,988
• Time in Patent Office: 1,264 Days
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06V 10/421 by analysing segments intersecting the pattern
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

A computing device of a first vehicle may receive a first image and a second image of a second vehicle having flashing light signals. The computing device may determine, in the first image and the second image, an image region that bounds the second vehicle such that the image region substantially encompasses the second vehicle. The computing device may determine a polar grid that partitions the image region in the first image and the second image into polar bins, and identify portions of image data exhibiting a change in color and a change in brightness between the first image and the second image. The computing device may determine a type of the flashing light signals and a type of the second vehicle; and accordingly provide instructions to control the first vehicle.

Claims

1. A method comprising:

receiving, at a computing device of a first vehicle, a first image of a second vehicle captured by an image-capture device coupled to the first vehicle and a second image captured by the image-capture device subsequent to capturing the first image, wherein the second vehicle has one or more flashing light signals;

determining, in the first image and the second image, an image region that bounds the second vehicle such that the image region substantially encompasses the second vehicle;

determining a polar grid that partitions the image region in the first image and the second image into a plurality of polar bins, wherein each polar bin of the plurality of polar bins is defined by two lines extending from a center portion of the image region to about a boundary of the image region;

identifying, based on a comparison of image content of polar bins in the first image to image content of corresponding polar bins in the second image, one or more portions of image data exhibiting a change in color and a change in brightness between the first image and the second image;

determining a type of the one or more flashing light signals of the second vehicle and a type of the second vehicle based on (i) a number of portions of image data exhibiting the change in color and the change in brightness, (ii) the color of the one or more portions, and (iii) the brightness of the one or more portions; and

providing, by the computing device, instructions to control the first vehicle based on the type of the second vehicle and the type of the one or more flashing light signals.

2. The method of claim 1, wherein identifying the one or more portions of image data comprises:

within each polar bin in the first image, identifying a number of image portions that exhibit characteristics of a bright light having an intensity greater than a threshold intensity;

within each corresponding polar bin in the second image, identifying corresponding image portions; and

determining the change in color and the change in brightness for the image portions from the polar bin in the first image to the corresponding polar bin in the second image.

3. The method of claim 1, wherein the polar bins are non-uniform in size.

4. The method of claim 1, further comprising:

determining that several polar bins of the plurality of polar bins are void of image pixels exhibiting the change in color or the change in brightness between the first image and the second image; and

merging the several polar bins into a single polar bin.

5. The method of claim 1, wherein the second vehicle is moving relative to the first vehicle such that the image region that bounds the second vehicle in the first image is different from a respective image region that bounds the second vehicle in the second image.

6. The method of claim 1, further comprising:

assigning each pixel in the image region to a respective polar bin of the polar bins.

7. The method of claim 1, wherein determining the type of the second vehicle comprises determining that the second vehicle is an emergency vehicle characterized by multiple flashing light signals.

8. The method of claim 1, wherein determining the type of a given flashing light signal of the one or more flashing light signals comprises determining that the given flashing light is a vehicular turn signal or a brake light.

9. The method of claim 1, wherein the first image and the second image are captured consecutively.

10. The method of claim 1, wherein the computing device is configured to control the first vehicle in an autonomous operation mode.

11. A non-transitory computer readable medium having stored thereon executable instructions that, upon execution by a computing device of a first vehicle, cause the computing device to perform functions comprising:

receiving a first image of a second vehicle captured by an image-capture device coupled to the first vehicle and a second image captured by the image-capture device subsequent to capturing the first image, wherein the second vehicle has one or more flashing light signals;

determining, in the first image and the second image, an image region that bounds the second vehicle such that the image region substantially encompasses the second vehicle;

determining a polar grid that partitions the image region in the first image and the second image into a plurality of polar bins, wherein each polar bin of the plurality of polar bins is defined by two lines extending from a center portion of the image region to about a boundary of the image region;

identifying, based on a comparison of image content of polar bins in the first image to image content of corresponding polar bins in the second image, one or more portions of image data exhibiting a change in color and a change in brightness between the first image and the second image;

determining a type of the one or more flashing light signals of the second vehicle and a type of the second vehicle based on (i) a number of portions of image data exhibiting the change in color and the change in brightness, (ii) the color of the one or more portions, and (iii) the brightness of the one or more portions; and

providing instructions to control the first vehicle based on the type of the second vehicle and the type of the one or more flashing light signals.

12. The non-transitory computer readable medium of claim 11, wherein the function of identifying the one or more portions of image data comprises:

within each polar bin in the first image, identifying a number of image portions that exhibit characteristics of a bright light having an intensity greater than a threshold intensity;

within each corresponding polar bin in the second image, identifying corresponding image portions; and

determining the change in color and the change in brightness for the image portions from the polar bin in the first image to the corresponding polar bin in the second image.

13. The non-transitory computer readable medium of claim 11, wherein the polar bins are non-uniform in size.

14. The non-transitory computer readable medium of claim 11, wherein the functions further comprise:

determining that several polar bins of the plurality of polar bins are void of image pixels exhibiting the change in color or the change in brightness between the first image and the second image; and

merging the several polar bins into a single polar bin.

15. The non-transitory computer readable medium of claim 11, wherein the functions further comprise:

assigning each pixel in the image region to a respective polar bin of the polar bins.

16. A system comprising:

an image-capture device coupled to a first vehicle;

at least one processor in communication with the image-capture device; and

a memory having stored thereon executable instructions that, upon execution by the at least one processor, cause the system to perform functions comprising;

receiving a first image of a second vehicle captured by the image-capture device and a second image captured by the image-capture device subsequent to capturing the first image, wherein the second vehicle has one or more flashing light signals;

determining, in the first image and the second image, an image region that bounds the second vehicle such that the image region substantially encompasses the second vehicle;

determining a polar grid that partitions the image region in the first image and the second image into a plurality of polar bins, wherein each polar bin of the plurality of polar bins is defined by two lines extending from a center portion of the image region to about a boundary of the image region;

identifying, based on a comparison of image content of polar bins in the first image to image content of corresponding polar bins in the second image, one or more portions of image data exhibiting a change in color and a change in brightness between the first image and the second image;

determining a type of the one or more flashing light signals of the second vehicle and a type of the second vehicle based on (i) a number of portions of image data exhibiting the change in color and the change in brightness, (ii) the color of the one or more portions, and (iii) the brightness of the one or more portions; and

providing instructions to control the first vehicle based on the type of the second vehicle and the type of the one or more flashing light signals.

17. The system of claim 16, wherein the function of identifying the one or more portions of image data comprises:

within each polar bin in the first image, identifying a number of image portions that exhibit characteristics of a bright light having an intensity greater than a threshold intensity;

within each corresponding polar bin in the second image, identifying corresponding image portions; and

determining the change in color and the change in brightness for the image portions from the polar bin in the first image to the corresponding polar bin in the second image.

18. The control system of claim 16, wherein the functions further comprise:

determining that several polar bins of the plurality of polar bins are void of image pixels exhibiting the change in color or the change in brightness between the first image and the second image; and

merging the several polar bins into a single polar bin such that the polar bins become non-uniform in size.

19. The control system of claim 16, wherein the functions further comprise:

assigning each pixel in the image region to a respective polar bin of the polar bins.