Radiance processing by demultiplexing in the frequency domain

• US 8,559,756 B2
• Filed: 08/05/2008
• Issued: 10/15/2013
• Est. Priority Date: 08/06/2007
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Georgiev, Todor G., Intwala, Chintan, Babacan, Sevket Derin
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): TORRES, JOSE
• Application Number:
  US12/186,392
• Publication Number:
  US 20090041381A1
• Time in Patent Office: 1,897 Days
• Field of Search: 382/280, 382/283, 382/306, 382/210, 382/277, 396/113, 396/268
• US Class Current: 382/280
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

Method and apparatus for radiance processing by demultiplexing in the frequency domain. A frequency domain demultiplexing module obtains a radiance image captured with a lens-based radiance camera. The image includes optically mixed spatial and angular frequency components of light from a scene. The module performs frequency domain demultiplexing on the radiance image to generate multiple parallax views of the scene. The method may extract multiple slices at different angular frequencies from a Fourier transform of the radiance image, apply a Fourier transform to each of the multiple slices to generate intermediate images, stack the intermediate images to form a 3- or 4-dimensional image, apply an inverse Fourier transform along angular dimension(s) of the 3- or 4-dimensional image, and unstack the transformed 3- or 4-dimensional image to obtain the multiple parallax views. During the method, phase correction may be performed to determine the centers of the intermediate images.

Claims

1. A system, comprising:

at least one processor; and

a memory comprising program instructions that are executable by the at least one processor to;

obtain a radiance image of a scene captured with a lens-based radiance camera comprising an array of refracting microlenses, the radiance image including optically mixed different spatial and angular frequency components;

demultiplex the radiance image in the frequency domain to generate multiple parallax views of the scene by the program instructions further executable to;

apply a Fourier transform to the radiance image to generate a transformed radiance image;

extract multiple slices of the transformed radiance image each at a different angular frequency;

apply the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;

stack the intermediate images to form a 3-dimensional image;

apply the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image; and

unstack the transformed 3-dimensional image to obtain the multiple parallax views of the scene.

2. The system as recited in claim 1, wherein the program instructions are executable by the at least one processor to:

stack the intermediate images to form a 4-dimensional image;

apply the Fourier transform along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; and

unstack the transformed 4-dimensional image to obtain the multiple parallax views of the scene.

3. The system as recited in claim 2, wherein, to apply the Fourier transform along the angular dimension of the 3-dimensional image, the program instructions are executable by the at least one processor to apply an inverse Fourier transform (IFFT) along the angular dimension of the 3-dimensional image or along the two angular dimensions of the 4-dimensional image.

4. The system as recited in claim 1, wherein the parallax views include vertical parallax views.

5. The system as recited in claim 1, wherein the parallax views include horizontal parallax views.

6. The system as recited in claim 1, wherein, to apply the Fourier transform to each of the multiple slices of the transformed radiance image, the program instructions are executable by the at least one processor to apply a 2-dimensional inverse Fourier transform (IFFT) to each of the multiple slices.

7. The system as recited in claim 1, wherein the program instructions are executable by the at least one processor to perform phase correction on the intermediate images to determine centers of the intermediate images prior to the stack of the intermediate images.

8. A computer-implemented method, comprising:

obtaining a radiance image of a scene captured with a lens-based radiance camera comprising an array of refracting microlenses, the radiance image including optically mixed different spatial and angular frequency components; and

demultiplexing the radiance image in the frequency domain to generate parallax views of the scene, said demultiplexing the radiance image comprises;

applying a Fourier transform to the radiance image to generate a transformed radiance image;

extracting multiple slices of the transformed radiance image each at a different angular frequency;

applying the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;

stacking the intermediate images to form a 4-dimensional image;

applying the Fourier transform along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; and

unstacking the transformed 4-dimensional image to obtain the multiple parallax views of the scene.

9. The computer-implemented method as recited in claim 8, wherein said demultiplexing the radiance image in the frequency domain comprises:

stacking the intermediate images to form a 3-dimensional image;

applying the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image; and

unstacking the transformed 3-dimensional image to obtain the multiple parallax views of the scene.

10. The computer-implemented method as recited in claim 9, wherein said applying a Fourier transform along the angular dimension of the 3-dimensional image comprises applying an inverse Fourier transform (IFFT) along the angular dimension of the 3-dimensional image or the along the two angular dimensions of the 4-dimensional image.

11. The computer-implemented method as recited in claim 8, wherein the parallax views include vertical parallax views.

12. The computer-implemented method as recited in claim 8, wherein the parallax views include horizontal parallax views.

13. The computer-implemented method as recited in claim 8, wherein said applying the Fourier transform to each of the multiple slices of the transformed radiance image comprises applying a 2-dimensional inverse Fourier transform (IFFT) to each of the multiple slices.

14. The computer-implemented method as recited in claim 8, further comprising performing phase correction on the intermediate images to determine centers of the intermediate images prior to said stacking the intermediate images to form the 3-dimensional image.

15. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

obtaining a radiance image of a scene captured with a lens-based radiance camera comprising an array of refracting microlenses, the radiance image including optically mixed different spatial and angular frequency components; and

demultiplexing the radiance image in the frequency domain to generate parallax views of the scene, said demultiplexing the radiance image comprises;

applying a Fourier transform to the radiance image to generate a transformed radiance image;

extracting multiple slices of the transformed radiance image each at a different angular frequency;

applying the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;

stacking the intermediate images to form a 3-dimensional or a 4-dimensional image;

applying the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image, or along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; and

unstacking the transformed 3-dimensional image or the transformed 4-dimensional image to obtain the multiple parallax views of the scene.

16. The non-transitory computer-readable storage medium as recited in claim 15, wherein the parallax views include vertical parallax views.

17. The non-transitory computer-readable storage medium as recited in claim 15, wherein the parallax views include horizontal parallax views.

18. The non-transitory computer-readable storage medium as recited in claim 15, wherein, in said applying the Fourier transform to each of the multiple slices of the transformed radiance image, the program instructions are computer-executable to implement applying a 2-dimensional inverse Fourier transform (IFFT) to each of the multiple slices.

19. The non-transitory computer-readable storage medium as recited in claim 15, wherein, in said applying the Fourier transform along the angular dimension of the 3-dimensional image comprises the program instructions are computer-executable to implement applying an inverse Fourier transform (IFFT) along the angular dimension of the 3-dimensional image or along the two angular dimensions of the 4-dimensional image.

20. The non-transitory computer-readable storage medium as recited in claim 15, wherein the program instructions are computer-executable to implement performing phase correction on the intermediate images to determine centers of the intermediate images prior to said stacking the intermediate images to form the 3-dimensional image.

21. A system, comprising:

a mask-based radiance camera configured to capture a scene as a radiance image that includes optically different spatial and angular frequency components, the mask-based radiance camera comprising a non-sinusoidal periodic mask positioned between an objective lens and a photosensor of the mask-based radiance camera; and

an image processing application configured to demultiplex the radiance image in the frequency domain to generate parallax views of the scene by the image processing application configured to;

apply a Fourier transform to the radiance image to generate a transformed radiance image;

extract multiple slices of the transformed radiance image each at a different angular frequency;

apply the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;

stack the intermediate images to form a 3-dimensional or a 4-dimensional image;

apply the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image, or along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; and

unstack the transformed 3-dimensional image or the transformed 4-dimensional image to obtain the parallax views of the scene.

22. The system as recited in claim 21, wherein the non-sinusoidal periodic mask comprises a grid of multiple horizontally and vertically arranged opaque linear elements that define horizontal rows and vertical columns of periodically spaced transparent openings between the opaque linear elements.

23. A system, comprising:

an external mask-based radiance camera configured to capture a scene as a radiance image that includes optically different spatial and angular frequency components, the external mask-based radiance camera comprising a mask positioned in front of an objective lens of the camera; and

an image processing application configured to demultiplex the radiance image in the frequency domain to generate parallax views of the scene by the image processing application configured to;

apply a Fourier transform to the radiance image to generate a transformed radiance image;

extract multiple slices of the transformed radiance image each at a different angular frequency;

apply the Fourier transform to each of the multiple slices of the transformed radiance image to generate an intermediate image from each of the multiple slices;

stack the intermediate images to form a 3-dimensional or a 4-dimensional image;

apply the Fourier transform along an angular dimension of the 3-dimensional image to generate a transformed 3-dimensional image, or along two angular dimensions of the 4-dimensional image to generate a transformed 4-dimensional image; and

unstack the transformed 3-dimensional image or the transformed 4-dimensional image to obtain the parallax views of the scene.

24. The system as recited in claim 23, wherein the mask comprises a grid of multiple horizontally and vertically arranged opaque linear elements that define horizontal rows and vertical columns of periodically spaced transparent openings between the opaque linear elements.