Apparatus and method for processing light field data using a mask with an attenuation pattern

• US 9,143,678 B2
• Filed: 05/06/2011
• Issued: 09/22/2015
• Est. Priority Date: 05/11/2010
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology, Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Park, Byung-Kwan, Kirmani, Ghulam Ahmed, Raskar, Ramesh, Pandharkar, Rohit
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US13/102,812
• Publication Number:
  US 20120057040A1
• Time in Patent Office: 1,600 Days
• Field of Search: 348/340, 348/342
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20048 Transform domain processing
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

Provided are an apparatus and method for processing a light field image that is acquired and processed using a mask to spatially modulate a light field. The apparatus includes a lens, a mask to spatially modulate 4D light field data of a scene passing through the lens to include wideband information on the scene, a sensor to detect a 2D image corresponding to the spatially modulated 4D light field data, and a data processing unit to recover the 4D light field data from the 2D image to generate an all-in-focus image.

Claims

1. An apparatus for processing a light field image, the apparatus comprising:

a lens;

a mask configured to spatially modulate four-dimensional (4D) light field data of a scene passing through the lens such that the spatially modulated 4D light field data includes wideband information of the scene, the mask comprising

a transparent layer, and

a metal layer disposed on the transparent layer, the metal layer comprising

an attenuation pattern configured to attenuate light passing through the transparent layer;

a sensor configured to detect a two-dimensional (2D) image corresponding to the spatially modulated 4D light field data; and

a data processing unit configured to recover the 4D light field data from the 2D image, generate a high-resolution depth map based on the 4D light field data, and generate an all-in-focus image based on the 2D image and the high-resolution depth map,

wherein the attenuation pattern comprises unit patterns, each unit pattern comprises a 7×

7 pixel structure, and each region in the second and fourth rows and second and fourth columns of the unit patterns are configured to intercept light.

2. The apparatus of claim 1, wherein the data processing unit comprises:

a recovering portion configured to recover the 4D light field data from the 2D image;

a depth map generating portion configured to generate a plurality of viewpoint images from the 4D light field data and generate a depth map from a plurality of the viewpoint images;

a high-resolution depth map generating portion configured to generate a high-resolution depth map with respect to the depth map; and

an all-in-focus image generating portion configured to generate an all-in-focus image which is focused in all positions based on the 2D image and the generated high-resolution depth map.

3. The apparatus of claim 2, wherein the all-in-focus image generating portion determines a spatially varying modulation function based on the high-resolution depth map, and generates an all-in-focus image based on an inverted value of the spatially varying modulation function and the 2D image.

4. The apparatus of claim 2, wherein the data processing unit further comprises a traditional image generating portion configured to generate a traditional image that is focused only at a certain depth through a reblurring process, based on the high-resolution depth map and the all-in-focus image.

5. The apparatus of claim 2, wherein the data processing unit further comprises a high-resolution light field image generating portion configured to generate a high-resolution light field image based on the high-resolution depth map and the all-in-focus image.

6. The apparatus of claim 5, wherein the high-resolution light field image generating portion generates a high-resolution light field image by shifting pixels of the all-in-focus image based on the depths of the pixels included in the high-resolution depth map.

7. The apparatus of claim 1, wherein the spatially varying modulation functions of the unit patterns forming the attenuation pattern have non-singular characteristics, and the unit patterns are formed such that at least half of the area of each unit pattern transmits light.

8. A method of processing a light field image, the method comprising:

detecting a two-dimensional (2D) image corresponding to four-dimensional (4D) light field data that is spatially modulated to include wideband information of a scene, using a mask configured to spatially modulate the 4D light field data of the scene and a sensor;

recovering the 4D light field data from the 2D image;

generating a high-resolution depth map based on 4D light field data; and

generating an all-in-focus image using the 2D image and the high-resolution depth map,

wherein the mask comprises a transparent layer and a metal layer disposed on the transparent layer, the metal layer comprising an attenuation pattern configured to attenuate the light passing through the transparent layer, and

the attenuation pattern comprises unit patterns, each unit pattern comprises a 7×

7 pixel structure, and each region in the second and fourth rows and second and fourth columns of the unit patterns are configured to intercept light.

9. The method of claim 8, wherein the generating of the all-in-focus image comprises:

determining a spatially varying modulation function based on the high-resolution depth map; and

generating an all-in-focus image based on an inverted value of the spatially varying modulation function and the 2D image.

10. The method of claim 8, further comprising generating a traditional image that is only focused at a certain depth through a reblurring process, based on the high-resolution depth map and the all-in-focus image.

11. The method of claim 8, further comprising generating a high-resolution light field image based on the high-resolution depth map and the all-in-focus image.

12. The method of claim 11, wherein the generating of the high-resolution light field image further comprises generating a high-resolution light field image by shifting pixels of the all-in-focus image based on the depths of the pixels included in the high-resolution depth map.

13. An attenuation mask of an imaging apparatus, the attenuation mask comprising:

a transparent layer; and

a metal layer disposed on the transparent layer,

wherein the attenuation mask is configured to spatially modulates four-dimensional (4D) acquired light field data such that the spatially modulated data includes wideband image information of a scene,

the metal layer comprising an attenuation pattern configured to attenuate light passing through the transparent layer, and

the attenuation pattern comprises unit patterns, each unit pattern comprises a 7×

7 pixel structure, and each region in the second and fourth rows and second and fourth columns of the unit patterns are configured to intercept light.