Simulating an infrared emitter array in a video monitoring camera to construct a lookup table for depth determination

• US 9,235,899 B1
• Filed: 06/12/2015
• Issued: 01/12/2016
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kirmani, Ghulam Ahmed, Colaco, Andrea, Shin, Dongeek
• Primary Examiner(s): Lee, Young
• Application Number:
  US14/738,803
• Time in Patent Office: 214 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A process generates lookup tables for estimating spatial depth in a scene. The process identifies subsets of illuminators of a camera system that has a 2-dimensional array of image sensors and illuminators in fixed locations relative to the array, and partitions the image sensors into a plurality of pixels. For each pixel, and for each of m distinct depths from the respective pixel, the process simulates a virtual surface at the respective depth. For each of the subsets of illuminators, the process determines an expected light intensity at the pixel based on the respective depth. The process forms an intensity vector using the expected light intensities for each of the distinct subsets and normalizes the intensity vector. For each pixel, the process constructs a lookup table comprising the normalized vectors corresponding to the pixel. The lookup table associates each normalized vector with the depth of the corresponding simulated surface.

Claims

1. A method of generating a lookup table for use in estimating spatial depth in a visual scene, comprising:

at a server having one or more processors, and memory storing one or more programs configured for execution by the one or more processors;

identifying a plurality of distinct subsets of IR illuminators of a camera system that has a 2-dimensional array of image sensors and a plurality of IR illuminators in fixed locations relative to the array of image sensors;

partitioning the image sensors into a plurality of pixels;

for each pixel, and for each of m distinct depths from the respective pixel;

simulating a virtual surface at the respective depth;

for each of the plurality of distinct subsets of IR illuminators, determining an expected IR light intensity at the respective pixel based on the respective depth and based on only the respective subset of IR illuminators emitting IR light;

forming an intensity vector using the expected IR light intensities for each of the distinct subsets; and

normalizing the intensity vector; and

for each pixel, constructing a lookup table comprising the normalized vectors corresponding to the pixel, wherein the lookup table associates each respective normalized vector with the respective depth of the respective simulated surface.

2. The method of claim 1, wherein the expected IR light intensity at the respective pixel is based on characteristics of the IR illuminators of the camera system.

3. The method of claim 2, wherein the characteristics include lux.

4. The method of claim 2, wherein the characteristics include orientation of the IR illuminators relative to the array of image sensors.

5. The method of claim 2, wherein the characteristics include location of the IR illuminators relative to the array of image sensors.

6. The method of claim 1, wherein the virtual surface at the respective depth is a planar surface.

7. The method of claim 1, wherein the virtual surface at the respective depth is a spherical surface.

8. The method of claim 1, wherein the virtual surface at the respective depth is a parabolic surface.

9. The method of claim 1, wherein the virtual surface at the respective depth is a cubic surface.

10. The method of claim 1, wherein normalizing each intensity vector comprises determining a respective magnitude of the intensity vector and dividing each component of the intensity vector by the respective magnitude.

11. The method of claim 1, wherein each pixel comprises a respective single image sensor.

12. The method of claim 1, wherein each pixel comprises a respective plurality of image sensors.

13. The method of claim 1, wherein each pixel comprises more than fifty respective image sensors.

14. The method of claim 1, wherein the array of image sensors comprises more than one million image sensors.

15. A server system, comprising:

one or more processors;

memory; and

one or more programs stored in the memory configured for execution by the one or more processors, the one or more programs comprising instructions for;

identifying a plurality of distinct subsets of IR illuminators of a camera system that has a 2-dimensional array of image sensors and a plurality of IR illuminators in fixed locations relative to the array of image sensors;

partitioning the image sensors into a plurality of pixels;

for each pixel, and for each of m distinct depths from the respective pixel;

simulating a virtual surface at the respective depth;

for each of the plurality of distinct subsets of IR illuminators, determining an expected IR light intensity at the respective pixel based on the respective depth and based on only the respective subset of IR illuminators emitting IR light;

forming an intensity vector using the expected IR light intensities for each of the distinct subsets; and

normalizing the intensity vector; and

for each pixel, constructing a lookup table comprising the normalized vectors corresponding to the pixel, wherein the lookup table associates each respective normalized vector with the respective depth of the respective simulated surface.

16. The server system of claim 15, wherein the expected IR light intensity at the respective pixel is based on characteristics of the IR illuminators of the camera system, and wherein the characteristics include orientation of the IR illuminators relative to the array of image sensors.

17. The server system of claim 15, wherein the virtual surface at the respective depth is a planar surface.

18. The server system of claim 15, wherein normalizing each intensity vector comprises determining a respective magnitude of the intensity vector and dividing each component of the intensity vector by the respective magnitude.

19. The server system of claim 15, wherein each pixel comprises a respective plurality of image sensors.

20. A non-transitory computer readable storage medium storing one or more programs configured for execution by a server system having one or more processors and memory, the one or more programs comprising instructions for:

identifying a plurality of distinct subsets of IR illuminators of a camera system that has a 2-dimensional array of image sensors and a plurality of IR illuminators in fixed locations relative to the array of image sensors;

partitioning the image sensors into a plurality of pixels;

for each pixel, and for each of m distinct depths from the respective pixel;

simulating a virtual surface at the respective depth;

for each of the plurality of distinct subsets of IR illuminators, determining an expected IR light intensity at the respective pixel based on the respective depth and based on only the respective subset of IR illuminators emitting IR light;

forming an intensity vector using the expected IR light intensities for each of the distinct subsets; and

normalizing the intensity vector; and

for each pixel, constructing a lookup table comprising the normalized vectors corresponding to the pixel, wherein the lookup table associates each respective normalized vector with the respective depth of the respective simulated surface.