Using a scene illuminating infrared emitter array in a video monitoring camera for depth determination

• US 9,454,820 B1
• Filed: 06/12/2015
• Issued: 09/27/2016
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kirmani, Ghulam Ahmed, Colaco, Andrea, Shin, Dongeek
• Primary Examiner(s): Holder, Anner
• Application Number:
  US14/738,818
• Time in Patent Office: 473 Days
• Field of Search: 348/135
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A process creates a depth map of a scene. The process is performed at a computing device having one or more processors, and memory storing one or more programs configured for execution by the one or more processors. For each of a plurality of distinct subsets of illuminators of a camera system, the process receives a captured image of a first scene taken by a 2-dimensional array of image sensors of the camera system while the respective subset of illuminators are emitting light and the illuminators not in the respective subset are not emitting light. The image sensors are partitioned into a plurality of pixels. For each pixel, the process uses the captured images to form a respective vector of light intensity at the pixel and estimates a depth in the first scene at the respective pixel by looking up the respective vector in a respective lookup table.

Claims

1. A method of creating a depth map of a scene, comprising:

at a computing device having one or more processors, and memory storing one or more programs configured for execution by the one or more processors;

for each of a plurality of distinct subsets of IR illuminators of a camera system, receiving a captured IR image of a first scene taken by a 2-dimensional array of image sensors of the camera system while the respective subset of IR illuminators are emitting IR light and the IR illuminators not in the respective subset are not emitting IR light, wherein the image sensors are partitioned into a plurality of pixels; and

for each pixel of the plurality of pixels;

using the captured IR images to form a respective vector of light intensity at the respective pixel; and

estimating a depth in the first scene at the respective pixel by looking up the respective vector in a respective pre-generated lookup table.

2. The method of claim 1, wherein looking up the respective vector in the respective lookup table comprises:

computing an inner product of the respective vector with records in the lookup table; and

determining the depth in the first scene at the pixel as a depth corresponding to a record in the lookup table whose inner product with the respective vector is greatest among the computed inner products for the respective vector.

3. The method of claim 2, wherein computing an inner product of the respective vector with records in the lookup table comprises:

for each record in the respective lookup table, computing an inner product of the respective vector and the respective record.

4. The method of claim 2, wherein:

the respective vector for a respective pixel comprises a plurality of components, each of the components corresponding to a respective IR light intensity for the respective pixel for a respective captured IR image; and

computing an inner product comprises computing a dot product.

5. The method of claim 1, wherein one or more of the subsets of the IR illuminators consists of a single IR illuminator.

6. The method of claim 1, wherein the plurality of IR illuminators are orientated at a plurality of distinct angles relative to the array of image sensors.

7. A computing device, comprising:

one or more processors;

memory; and

one or more programs stored in the memory configured for execution by the one or more processors, the one or more programs comprising instructions for;

for each of a plurality of distinct subsets of IR illuminators of a camera system, receiving a captured IR image of a first scene taken by a 2-dimensional array of image sensors of the camera system while the respective subset of IR illuminators are emitting IR light and the IR illuminators not in the respective subset are not emitting IR light, wherein the image sensors are partitioned into a plurality of pixels; and

for each pixel of the plurality of pixels;

using the captured IR images to form a respective vector of light intensity at the respective pixel; and

estimating a depth in the first scene at the respective pixel by looking up the respective vector in a respective pre-generated lookup table.

9. The computing device of claim 7, wherein the one or more programs further comprise instructions for detecting a trigger event, wherein creating the depth map of the first scene is in response to detecting the trigger event.

10. The computing device of claim 9, wherein:

the first scene includes a first object positioned at a first location within the first scene; and

detecting the trigger event comprises detecting the first object positioned at a second location within the first scene, wherein the second location is distinct from the first location.

11. The computing device of claim 9, wherein the trigger event comprises a power interruption event.

12. The computing device of claim 7, wherein the lookup tables are defined for and stored in the camera system during a calibration process.

13. The computing device of claim 12, wherein the calibration process comprises instructions for:

simulating a virtual planar surface at a plurality of respective depths in the first scene; and

determining, for each pixel and each respective depth, an expected IR light intensity.

14. The computing device of claim 7, wherein each of the distinct subsets of IR illuminators comprises two adjacent IR illuminators, and the distinct subsets of IR illuminators are non-overlapping.

8. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computing device having one or more processors and memory, the one or more programs comprising instructions for:

for each of a plurality of distinct subsets of IR illuminators of a camera system, receiving a captured IR image of a first scene taken by a 2-dimensional array of image sensors of the camera system while the respective subset of IR illuminators are emitting IR light and the IR illuminators not in the respective subset are not emitting IR light, wherein the image sensors are partitioned into a plurality of pixels; and

for each pixel of the plurality of pixels;

using the captured IR images to form a respective vector of light intensity at the respective pixel; and

estimating a depth in the first scene at the respective pixel by looking up the respective vector in a respective pre-generated lookup table.

15. The computer readable storage medium of claim 8, wherein each respective lookup table includes a plurality of normalized IR light intensity vectors, each normalized light intensity vector corresponding to a respective depth in the first scene.

16. The computer readable storage medium of claim 8, wherein the computing device is a server distinct from the camera system.

17. The computer readable storage medium of claim 8, wherein the computing device is included in the camera system.

18. The computer readable storage medium of claim 17, wherein the respective lookup table is downloaded to the camera system from a remote server during an initialization process prior to creating the depth map.

19. The computer readable storage medium of claim 8, the one or more programs further comprise instructions for:

prior to capturing the IR images, switching from a first mode of the camera system to a second mode of the camera system, including deactivating the first mode and activating the second mode;

wherein the array of image sensors has an associated first pixel gain curve when the first mode is activated, and the array of image sensors has an associated second pixel gain curve when the second mode is activated.

20. The computer readable storage medium of claim 8, the one or more programs further comprise instructions for:

receiving a baseline IR image of the scene captured by the array of sensors while none of the IR illuminators are emitting IR light;

wherein forming each respective vector of light intensity at a respective pixel comprises subtracting a light intensity at the pixel of the baseline IR image from the light intensity at the pixel of each of the captured IR images.