SYSTEM AND METHOD FOR HIGH DYNAMIC RANGE DEPTH CAPTURE USING MULTIPLE CAMERAS

• US 20170142312A1
• Filed: 11/02/2016
• Published: 05/18/2017
• Est. Priority Date: 11/12/2015
• Status: Active Grant

• Current Assignee: Packsize LLC (Packsize International, LLC)
• Original Assignee: Aquifi, Inc.
• Inventors: Rafii, Abbas, Dal Mutto, Carlo, Salvagnini, Pietro, Peruch, Francesco, Marin, Giulio, Stoppa, Michele, Hazeghi, Aryan
• Granted Patent: US 10,453,185 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths

Abstract

A method for capturing a depth map includes: controlling a plurality of cameras to capture, concurrently, a plurality of first images during a first exposure interval, each of the cameras concurrently capturing a corresponding one of the first images, the cameras having overlapping fields of view; controlling a projection source to emit light at a first illumination level during the first exposure interval; controlling the cameras to capture, concurrently, a plurality of second images during a second exposure interval, each of the cameras concurrently capturing a corresponding one of the second images; controlling the projection source to emit light at a second illumination level during the second exposure interval, the second illumination level being different from the first illumination level; combining the first images with the second images to generate a depth map; and outputting the depth map.

Claims

1. A depth camera system comprising:

a processor;

a plurality of cameras coupled to the processor and configured to have overlapping fields of view;

a projection source coupled to the processor and configured to emit light toward the fields of view of the cameras; and

memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to;

control the cameras to capture, concurrently, a plurality of first images during a first exposure interval, each of the cameras concurrently capturing a corresponding one of the first images;

control the projection source to emit light at a first illumination level during the first exposure interval;

control the cameras to capture, concurrently, a plurality of second images during a second exposure interval, each of the cameras concurrently capturing a corresponding one of the second images;

control the projection source to emit light at a second illumination level during the second exposure interval, the second illumination level being different from the first illumination level;

combine the first images with the second images to generate a depth map; and

output the depth map.

2. The depth camera system of claim 1, wherein the first illumination level is a lower illumination level than the second illumination level.

3. The depth camera system of claim 2, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

control the projection source to emit light at the first illumination level during the first exposure interval by emitting light for a first emission interval having a length shorter than the first exposure interval; and

control the projection source to emit light at the second illumination level during the second exposure interval by emitting light for a second emission interval having a length longer than the first emission interval and shorter than or equal to the second exposure interval.

4. The depth camera system of claim 2, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

control the projection source to emit light at the first illumination level during the first exposure interval by emitting light for at a first emission power; and

control the projection source to emit light at the second illumination level during the second exposure interval by emitting light at a second emission power greater than the first emission power.

5. The depth camera system of claim 1, wherein the cameras are infrared cameras, and

wherein the projection source is configured to emit infrared light.

6. The depth camera system of claim 5, wherein the projection source is configured to emit an infrared pattern.

7. The depth camera system of claim 1, wherein the projection source is configured to emit light having a pattern.

8. The depth camera system of claim 7, wherein the pattern is configured to be detectable at different illumination levels.

9. The depth camera system of claim 1, wherein the cameras are configured to capture images using a global shutter.

10. The depth camera system of claim 1, wherein the memory further stores instructions that, when executed by the processor, cause the processor to control the projection source to emit light at a third illumination level, the third illumination level being different from the first illumination level and different from the second illumination level.

11. The depth camera system of claim 1, further comprising a second projection source, and

wherein the memory further stores instructions that, when executed by the processor, cause the processor to control the second projection source to emit light at a fourth illumination level different from the first illumination level and different from the second illumination level.

12. The depth camera system of claim 1, wherein the memory further stores one or more previously captured images, and

wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

analyze a brightness of at least one of the one or more previously captured images; and

set the second illumination level in accordance with a comparison of the brightness with a model brightness.

13. The depth camera system of claim 12, wherein the projection source is configured to emit, selectively, one of a plurality of patterns at a time, each of the patterns corresponding to one of a plurality of illumination levels, and

wherein the second illumination level is set by selecting a pattern from the plurality of patterns.

14. The depth camera system of claim 1, wherein the memory further stores one or more previously captured images, and

wherein the memory further stores instructions that, when executed by the processor, cause the processor to:

identify first portions of the first images having features matching second portions of the second images;

detect motion portions of the second images in accordance with locations of the first portions and the second portions; and

compute portions of the depth map corresponding to the motion portions in accordance with only the second images.

15. The depth camera system of claim 1, wherein the memory further stores one or more previously captured images, and

wherein the memory further stores instructions that, when executed by the processor, cause the processor to compensate for motion using optical flow estimation by:

detecting portions of the images of the cameras subject to motion;

warping a previously captured image of the previously captured images; and

computing a depth estimate based on the second images and the warped previously captured images.

16. The depth camera system of claim 1, wherein the projection source is configured to emit structured light.

17. A method for capturing a depth map in a scene having high dynamic range, comprising:

controlling, by a processor, a plurality of cameras to capture, concurrently, a plurality of first images during a first exposure interval, each of the cameras concurrently capturing a corresponding one of the first images, the cameras having overlapping fields of view;

controlling, by the processor, a projection source to emit light at a first illumination level during the first exposure interval;

controlling, by the processor, the cameras to capture, concurrently, a plurality of second images during a second exposure interval, each of the cameras concurrently capturing a corresponding one of the second images;

controlling, by the processor, the projection source to emit light at a second illumination level during the second exposure interval, the second illumination level being different from the first illumination level;

combining, by the processor, the first images with the second images to generate a depth map; and

outputting, by the processor, the depth map.

18. The method of claim 17, wherein the first illumination level is a lower illumination level than the second illumination level.

19. The method of claim 18, further comprising:

controlling, by the processor, the projection source to emit light at the first illumination level during the first exposure interval by emitting light for a first emission interval having a length shorter than the first exposure interval; and

controlling, by the processor, the projection source to emit light at the second illumination level during the second exposure interval by emitting light for a second emission interval having a length longer than the first emission interval and shorter than or equal to the second exposure interval.

20. The method of claim 18, further comprising:

controlling, by the processor, the projection source to emit light at the first illumination level during the first exposure interval by emitting light for at a first emission power; and

controlling, by the processor, the projection source to emit light at the second illumination level during the second exposure interval by emitting light at a second emission power greater than the first emission power.

21. The method of claim 17, wherein the cameras are infrared cameras, and

wherein the projection source is configured to emit infrared light.

22. The method of claim 21, wherein the projection source is configured to emit an infrared pattern.

23. The method of claim 17, wherein the projection source is configured to emit light having a pattern.

24. The method of claim 23, wherein the pattern is configured to be detectable at different illumination levels.

25. The method of claim 17, wherein the cameras are configured capture images using a global shutter.

26. The method of claim 17, further comprising controlling, by the processor, the projection source to emit light at a third illumination level, third illumination level being different from the first illumination level and different from the second illumination level.

27. The method of claim 17, further comprising controlling a second projection source to emit light at a fourth illumination level different from the first illumination level and the second illumination level.

28. The method of claim 17, further comprising:

storing, by the processor, the first images in a memory coupled to the processor;

analyzing, by the processor, a brightness of at least one of the first images; and

setting, by the processor, the second illumination second illumination level in accordance with a comparison of the brightness with a model brightness.

29. The method of claim 28, wherein the projection source is configured to emit, selectively, one of a plurality of patterns at a time, each of the patterns corresponding to one of a plurality of illumination levels, and

wherein the second illumination level is set by selecting a pattern from the plurality of patterns.

30. The method of claim 17, further comprising:

storing, by the processor, the first images and the second images in a memory coupled to the processor;

identifying, by the processor, first portions of the first images having features matching second portions of the second images;

detecting, by the processor, motion portions of the second images in accordance with locations of the first portions and the second portions; and

compute portions of the depth map corresponding to motion portions in accordance with only the second images.

31. The method of claim 17, further comprising:

storing, by the processor, the first images and the second images in a memory coupled to the processor;

detecting, by the processor, portions of the images of the cameras subject to motion;

warping, by the processor, a previously captured image of the previously captured images; and

computing, by the processor, a depth estimate based on the second images and the warped previously captured images.

32. The method of claim 17, wherein the projection source is configured to emit structured light.