APPARATUS AND METHOD FOR PROCESSING LIGHT FIELD DATA USING A MASK WITH AN ATTENUATION PATTERN

• US 20120057040A1
• Filed: 05/06/2011
• Published: 03/08/2012
• Est. Priority Date: 05/11/2010
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology, Samsung Electronics Co. Ltd.
• Original Assignee: Massachusetts Institute of Technology, Samsung Electronics Co. Ltd.
• Inventors: Raskar, Ramesh, Park, Byung Kwan, Kirmani, Ghulam Ahmed, Pandharkar, Rohit
• Granted Patent: US 9,143,678 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20048 Transform domain processing
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

Provided are an apparatus and method for processing a light field image that is acquired and processed using a mask to spatially modulate a light field. The apparatus includes a lens, a mask to spatially modulate 4D light field data of a scene passing through the lens to include wideband information on the scene, a sensor to detect a 2D image corresponding to the spatially modulated 4D light field data, and a data processing unit to recover the 4D light field data from the 2D image to generate an all-in-focus image.

Claims

1. An apparatus for processing a light field image, the apparatus comprising:

a lens;

a mask configured to spatially modulate four-dimensional (4D) light field data of a scene passing through the lens such that the spatially modulated 4D light field data includes wideband information of the scene;

a sensor configured to detect a two-dimensional (2D) image corresponding to the spatially modulated 4D light field data; and

a data processing unit configured to recover the 4D light field data from the 2D image, generate a high-resolution depth map based on the 4D light field data, and generate an all-in-focus image based on the 2D image and the high-resolution depth map.

2. The apparatus of claim 1, wherein the data processing unit comprises:

a recovering portion configured to recover the 4D light field data from the 2D image;

a depth map generating portion configured to generate a plurality of viewpoint images from the 4D light field data and generate a depth map from a plurality of the viewpoint images;

a high-resolution depth map generating portion configured to generate a high-resolution depth map with respect to the depth map; and

an all-in-focus image generating portion configured to generate an all-in-focus image which is focused in all positions based on the 2D image and the generated high-resolution depth map.

3. The apparatus of claim 2, wherein the all-in-focus image generating portion determines a spatially varying modulation function based on the high-resolution depth map, and generates an all-in-focus image based on an inverted value of the spatially varying modulation function and the 2D image.

4. The apparatus of claim 2, wherein the data processing unit further comprises a traditional image generating portion configured to generate a traditional image that is focused only at a certain depth through a reblurring process, based on the high-resolution depth map and the all-in-focus image.

5. The apparatus of claim 2, wherein the data processing unit further comprises a high-resolution light field image generating portion configured to generate a high-resolution light field image based on the high-resolution depth map and the all-in-focus image.

6. The apparatus of claim 5, wherein the high-resolution light field image generating portion generates a high-resolution light field image by shifting pixels of the all-in-focus image based on the depths of the pixels included in the high-resolution depth map.

7. The apparatus of claim 1, wherein the mask has an attenuation pattern comprising unit patterns that are repeatedly formed to attenuate light.

8. The apparatus of claim 7, wherein the spatially varying modulation functions of the unit patterns forming the attenuation pattern have non-singular characteristics, and the unit patterns are formed such that at least half of the area of each unit pattern transmits light.

9. The apparatus of claim 7, wherein each unit pattern has a 7×

7 pixel structure, and regions corresponding to second and fourth rows and second and fourth columns of each unit pattern intercept light and the other regions do not intercept light.

10. The apparatus of claim 7, wherein the mask comprises:

a transparent layer; and

a metal layer disposed on the transparent layer, the metal layer comprising an attenuation pattern formed therein to attenuate the light passing through the transparent layer.

11. A method of processing a light field image, the method comprising:

detecting a two-dimensional (2D) image corresponding to four-dimensional (4D) light field data that is spatially modulated to include wideband information of a scene, using a mask configured to spatially modulate the 4D light field data of the scene and a sensor;

recovering the 4D light field data from the 2D image;

generating a high-resolution depth map based on 4D light field data; and

generating an all-in-focus image using the 2D image and the high-resolution depth map.

12. The method of claim 11, wherein the generating of the all-in-focus image comprises:

determining a spatially varying modulation function based on the high-resolution depth map; and

generating an all-in-focus image based on an inverted value of the spatially varying modulation function and the 2D image.

13. The method of claim 11, further comprising generating a traditional image that is only focused at a certain depth through a reblurring process, based on the high-resolution depth map and the all-in-focus image.

14. The method of claim 11, further comprising generating a high-resolution light field image based on the high-resolution depth map and the all-in-focus image.

15. The method of claim 14, wherein the generating of the high-resolution light field image further comprises generating a high-resolution light field image by shifting pixels of the all-in-focus image based on the depths of the pixels included in the high-resolution depth map.

16. An attenuation mask for an imaging apparatus, the attenuation mask comprising:

a transparent layer; and

a metal layer comprising an attenuation pattern that is disposed on the transparent layer,

wherein the attenuation pattern spatially modulates four-dimensional (4D) acquired light field data such that the spatially modulated data includes wideband image information of a scene.

17. The attenuation mask of claim 16, wherein the attenuation pattern comprises a plurality of unit patterns, and each unit pattern comprises a pixel structure in which a first group of pixels block light, and a second group of pixels allow light to pass,

and the number of pixels that block light and that allow light to pass are adjustable.

18. The attenuation mask of claim 16, wherein the attenuation pattern comprises a plurality of unit patterns, each unit pattern comprises a 7×

7 pixel structure, and the second and fourth rows of pixels and second and fourth columns of pixels of each unit pattern block light and the remaining pixels of each unit pattern do not block light.