Using a scene illuminating infrared emitter array in a video monitoring camera for depth determination

• US 10,389,986 B2
• Filed: 02/20/2018
• Issued: 08/20/2019
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kirmani, Ghulam Ahmed, Colaco, Andrea, Shin, Dongeek
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US15/900,766
• Publication Number:
  US 20180176514A1
• Time in Patent Office: 546 Days
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A method generates depth maps at a camera having illuminators, a lens assembly, an image sensing element, a processor, and memory. The illuminators operate in a first mode to provide illumination, the lens assembly focuses incident light on the image sensing element, the memory stores image data from the image sensing element, and the processor executes programs to control operation of the camera. The method reconfigures the illuminators to operate in a second mode, where each of a plurality of subsets of the illuminators provides illumination of a scene separately. For each subset, the process activates the illuminators in the subset without activating illuminators not in the subset and receives reflected illumination from the scene incident on the lens assembly and focused onto the image sensing element. The measured light intensity of the received reflected illumination at the image sensing element is stored in association with activation of the subset.

Claims

1. A method for generating depth maps, comprising:

in a camera having a plurality of illuminators, a lens assembly, an image sensing element, a processor, and memory, wherein the illuminators are configured to operate in a first mode to provide illumination using all of the illuminators, the lens assembly is configured to focus incident light on the image sensing element, the memory is configured to store image data from the image sensing element, and the processor is configured to execute programs to control operation of the camera, the method comprising;

reconfiguring the plurality of illuminators to operate in a second mode wherein each of a plurality of subsets of the plurality of illuminators provides illumination of a scene separately, including, for each subset;

activating the illuminators in the subset without activating the illuminators not in the subset;

receiving reflected illumination from the illuminated scene incident on the lens assembly and focused onto the image sensing element;

measuring light intensity values of the received reflected illumination at the image sensing element; and

storing to the memory the measured light intensity values associated with activation of the subset.

2. The method of claim 1, wherein differences in the stored light intensity values associated with activation of each of the subsets for a respective image sensor correlate with spatial depth of an object in the scene from which reflected light was received at the respective image sensor.

3. The method of claim 1, further comprising:

capturing a baseline image while none of the illuminators are emitting light, wherein the captured baseline image measures ambient light intensity of the scene at each of the image sensors;

storing the captured baseline image to the memory; and

for each image sensor, subtracting the baseline intensity value from the stored intensity values for the respective image sensor to correct the stored intensity values for ambient light at the scene.

4. The method of claim 1, wherein the image sensors are partitioned into a plurality of pixels;

and

for each pixel of the plurality of pixels;

using the captured IR images to form a respective vector of light intensity at the respective pixel; and

estimating a depth in the first scene at the respective pixel by looking up the respective vector in a respective lookup table.

5. The method of claim 4, wherein each respective lookup table includes a plurality of normalized IR light intensity vectors, each normalized light intensity vector corresponding to a respective depth in the first scene.

6. The method of claim 4, wherein the respective lookup table is downloaded to the camera from a remote server during an initialization process.

7. A camera system, comprising:

a plurality of illuminators;

a lens assembly;

an image sensing element;

a processor;

memory; and

one or more programs stored in the memory configured for execution by the one or more processors;

wherein the illuminators are configured to operate in a first mode to provide illumination using all of the illuminators, the lens assembly is configured to focus incident light on the image sensing element, the memory is configured to store image data from the image sensing element, and the processor is configured to execute programs to control operation of the camera, and the one or more programs are configured for;

reconfiguring the plurality of illuminators to operate in a second mode wherein each of a plurality of subsets of the plurality of illuminators provides illumination of a scene separately, including, for each subset;

activating the illuminators in the subset without activating the illuminators not in the subset;

receiving reflected illumination from the illuminated scene incident on the lens assembly and focused onto the image sensing element;

measuring light intensity values of the received reflected illumination at the image sensing element; and

storing to the memory the measured light intensity values associated with activation of the subset.

8. The camera system of claim 7, further comprising:

capturing a baseline image while none of the illuminators are emitting light, wherein the captured baseline image measures ambient light intensity of the scene at each of the image sensors;

storing the captured baseline image to the memory; and

for each image sensor, subtracting the baseline intensity value from the stored intensity values for the respective image sensor to correct the stored intensity values for ambient light at the scene.

9. The camera system of claim 7, wherein the image sensors are partitioned into a plurality of pixels;

and

for each pixel of the plurality of pixels the one or more programs are configured for;

using the captured IR images to form a respective vector of light intensity at the respective pixel; and

estimating a depth in the first scene at the respective pixel by looking up the respective vector in a respective lookup table.

10. The camera system of claim 9, wherein looking up the respective vector in the respective lookup table comprises:

computing an inner product of the respective vector with records in the lookup table; and

determining the depth in the first scene at the pixel as a depth corresponding to a record in the lookup table whose inner product with the respective vector is greatest among the computed inner products for the respective vector.

11. The camera system of claim 10, wherein computing an inner product of the respective vector with records in the lookup table comprises:

for each record in the respective lookup table, computing an inner product of the respective vector and the respective record.

12. The camera system of claim 10, wherein:

the respective vector for a respective pixel comprises a plurality of components, each of the components corresponding to a respective IR light intensity for the respective pixel for a respective captured IR image; and

computing an inner product comprises computing a dot product.

13. A non-transitory computer readable storage medium storing one or more programs configured for execution by a camera system having a plurality of illuminators, a lens assembly, an image sensing element, a processor, and memory, wherein the illuminators are configured to operate in a first mode to provide illumination using all of the illuminators, the lens assembly is configured to focus incident light on the image sensing element, the memory is configured to store image data from the image sensing element, the processor is configured to execute programs to control operation of the camera, and the one or more programs comprise instructions for:

reconfiguring the plurality of illuminators to operate in a second mode wherein each of a plurality of subsets of the plurality of illuminators provides illumination of a scene separately, including, for each subset;

activating the illuminators in the subset without activating the illuminators not in the subset;

receiving reflected illumination from the illuminated scene incident on the lens assembly and focused onto the image sensing element;

measuring light intensity values of the received reflected illumination at the image sensing element; and

storing to the memory the measured light intensity values associated with activation of the subset.

14. The computer readable storage medium of claim 13, wherein each of the subsets of illuminators is configured at a different angle relative to the image sensing element.

15. The computer readable storage medium of claim 13, wherein each of the subsets of illuminators highlights a different portion of the scene.

16. The computer readable storage medium of claim 13, further comprising:

transmitting the stored light intensity values to a depth mapping module configured to estimate spatial depths of objects in the scene based on the stored light intensity values, predetermined illumination specifications of the illuminators, and response specifications of the image sensors.

17. The computer readable storage medium of claim 13, wherein the illuminators are IR illuminators.

18. The computer readable storage medium of claim 13, wherein the illuminators comprise 8 IR illuminators and each of the subsets of the illuminators comprises 2 adjacent IR illuminators.

19. The computer readable storage medium of claim 13, wherein the image sensing element is a 2-dimensional array of image sensors.

20. The computer readable storage medium of claim 19, wherein the image sensors are partitioned into a plurality of pixels;

and

for each pixel of the plurality of pixels;

using the captured IR images to form a respective vector of light intensity at the respective pixel; and

estimating a depth in the first scene at the respective pixel by looking up the respective vector in a respective lookup table.