METHOD AND APPARATUS FOR IMAGE CAPTURING AND SIMULTANEOUS DEPTH EXTRACTION

• US 20160004925A1
• Filed: 07/06/2015
• Published: 01/07/2016
• Est. Priority Date: 07/04/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: BUCHA, Victor Valentinovich, PARAMONOV, Vladimir Petrovich, PANCHENKO, Ivan Andreevich
• Granted Patent: US 9,872,012 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths

Abstract

A system for image capturing and depth extraction includes a camera and a data processor. The camera includes: a spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; and a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis. The data processor is configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information.

Claims

1. A system for image capturing and depth extraction, the system comprising:

a lens system;

a spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; and

a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis; and

a data processor configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information.

2. The system of claim 1, wherein the different spectrum channels form a basis of the spectrum coded aperture.

3. The system of claim 2, wherein the processing basis is different from the sensor basis and the basis of the spectrum coded aperture.

4. The system of claim 1, wherein the spectrum coded aperture has three regions including a transparent region in a central portion, and two regions having spectrum bandwidths respectively corresponding to yellow and cyan.

5. The system of claim 1, wherein the at least two regions of the spectrum coded aperture have spectrum bandwidths respectively corresponding to yellow and cyan.

6. The system of claim 1, wherein the spectrum coded aperture includes three congruent regions having spectrum bandwidths respectively corresponding to yellow, cyan, and magenta.

7. The system of claim 1, wherein the spectrum coded aperture includes three non-congruent regions having spectrum bandwidths respectively corresponding to yellow, cyan, and magenta.

8. The system of claim 1, wherein the spectrum coded aperture has a smooth bandwidth change over an aperture region.

9. The system of claim 1, wherein the spectrum coded aperture is fixed to the lens system.

10. The system of claim 1, wherein the spectrum coded aperture is attachable to and detachable from the lens system.

11. The system of claim 1, wherein the spectrum coded aperture has a combination of an opaque region and a congruent region, and

the congruent region is transparent or transmits ultraviolet light, infrared light, or visible light.

12. The system of claim 1, wherein the spectrum coded aperture has a combination of an opaque region and a non-congruent region, and

the non-congruent region is transparent or transmits ultraviolet light, infrared light, or visible light.

13. The system of claim 1, wherein the data processor comprises a preprocessing unit configured to perform the converting the captured image, a disparity estimation unit configured to perform the extracting the disparity, and a conversion unit configured to perform the converting the disparity to the depth information.

14. The system of claim 13, wherein the data processor further comprises an image restoration unit configured to restore the captured image based on the extracted disparity.

15. A method of image capturing and depth extraction, the method comprising:

recording at least two shifted spectrum channels of a light field to form an image captured from a video;

converting the captured image into an image of a processing basis;

estimating a disparity based on a correlation between pixels of the spectrum channels in the processing basis to extract a disparity map;

restoring the captured image based on the extracted disparity map; and

converting the disparity map into a depth map.

16. The method of claim 15, wherein the estimating the disparity comprises:

generating candidate images having respective shifts in the spectrum channels;

computing a matching cost involved in the candidate images in the spectrum channels;

propagating a matching cost involved in a low textured region of the candidate images; and

estimating a matching cost having a sub-pixel accuracy based on the propagated matching cost.

17. The method of claim 15, wherein the correlation between the pixels of the spectrum channel for requesting the disparity estimation includes a correlation metric computed in a sparse moving window.

18. The method of claim 15, wherein the correlation between the pixels of the spectrum channel for requesting the disparity estimation is computed by using at least one stereo matching algorithm.

19. The method of claim 15, wherein the restoring the captured image comprises performing a spectrum channel alignment in the processing basis.

20. A mobile device for image capturing and depth extraction in ultraviolet light, infrared light, or visible light, the mobile device comprising:

a lens system;

at least one spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other;

a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis; and

a coded aperture fixture configured to move at least one spectrum coded aperture relatively with respect to the lens system; and

a data processor configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information.

21. The mobile device of claim 20, wherein the spectrum coded aperture has a combination of an opaque region and a congruent region, and

the congruent region is transparent or transmits ultraviolet light, infrared light, or visible light.

22. The mobile device of claim 20, wherein the spectrum coded aperture has a combination of an opaque region and a non-congruent region, and

the non-congruent region is transparent or transmits ultraviolet light, infrared light, or visible light.

23. An apparatus for image capturing, the apparatus comprising:

a lens system;

at least two spectrum coded apertures including a first aperture and a second aperture which have different characteristics of optical efficiency and depth discrimination from each other;

a coded aperture fixture adapted to dispose the first aperture in front of the lens system;

a data processor configured to obtain depth information of an image captured through the first spectrum coded aperture, and control the coded aperture fixture to determine whether to switch the first aperture to the second aperture based on the depth information.

24. The apparatus of claim 23, wherein the first aperture includes a transparent region placed in the center of the first aperture and two regions separated by the transparent region,

wherein the two regions pass different color spectrums, respectively.

25. The apparatus of claim 24, the two regions pass a yellow spectrum and a cyan spectrum, respectively.

26. The apparatus of claim 24, wherein the second aperture includes equally divided two regions which pass yellow and cyan spectrums, respectively.