Generating Images from Light Fields Utilizing Virtual Viewpoints

• US 20140092281A1
• Filed: 09/30/2013
• Published: 04/03/2014
• Est. Priority Date: 09/28/2012
• Status: Abandoned Application

• Current Assignee: Fotonation Cayman Limited (Adeia Inc.)
• Original Assignee: Pelican Imaging Corporation
• Inventors: Nisenzon, Semyon, Jain, Ankit K.
• Application Number:
  US14/042,275
• Publication Number:
  US 20140092281A1
• Time in Patent Office: Days
• US Class Current: 348/262
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/95 Computational photography systems, e.g. light-field imaging systems

Abstract

Systems and methods for the synthesis of light field images from virtual viewpoints in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system includes a processor and a memory configured to store captured light field image data and an image manipulation application, wherein the captured light field image data includes image data, pixel position data, and a depth map, and wherein the image manipulation application configures the processor to obtain captured light field image data, determine a virtual viewpoint for the captured light field image data, where the virtual viewpoint includes a virtual location and virtual depth information, compute a virtual depth map based on the captured light field image data and the virtual viewpoint, and generate an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map.

Claims

1. A system configured to synthesize images using captured light field image data, comprising:

a processor; and

a memory connected to the processor and configured to store captured light field image data and an image manipulation application;

wherein the captured light field image data comprises image data, pixel position data, and a depth map;

wherein the depth map comprises depth information for one or more pixels in the image data; and

wherein the image manipulation application configures the processor to;

obtain captured light field image data;

determine a virtual viewpoint for the captured light field image data based on the pixel position data and the depth map for the captured light field image data, where the virtual viewpoint comprises a virtual location and virtual depth information;

compute a virtual depth map based on the captured light field image data and the virtual viewpoint; and

generate an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map, where the generated image comprises a plurality of pixels selected from the image data based on the pixel position data and the virtual depth map.

2. The system of claim 1, wherein the virtual viewpoint corresponds to a focal plane in an array camera utilized to create the captured light field image data.

3. The system of claim 1, wherein:

the captured light field image data further comprises a reference viewpoint within the captured light field image data; and

the virtual viewpoint is a separate viewpoint within the captured light field image data from the reference viewpoint.

4. The system of claim 3, wherein:

the captured light field image data was captured by an array camera having an imager array comprising a plurality of imagers; and

the reference viewpoint corresponds to the viewpoint of a first imager within the imager array in the array camera.

5. The system of claim 4, wherein the virtual viewpoint corresponds to the viewpoint of a second imager within the imager array, where the second imager is separate from the first imager.

6. The system of claim 4, wherein the virtual viewpoint is a viewpoint that does not correspond to the viewpoint of any of the imagers within the imager array.

7. The system of claim 4, wherein the virtual viewpoint is selected from a position selected from the group consisting of in front of the imager array and behind the imager array.

8. The system of claim 1, wherein the image manipulation application further configures the processor to generate an image from the perspective of the virtual viewpoint by projecting pixels from the captured light field image data based on the pixel position data and the depth map, where the projected pixels are described in the image data and the depth map.

9. The system of claim 8, wherein:

the captured light field image data further comprises occluded pixel information describing pixels not visible from a reference viewpoint of the captured light field image data; and

the projected pixels include at least one occluded pixel in the occluded pixel information that is visible from the perspective of the virtual viewpoint.

10. The system of claim 8, wherein:

at least one projected pixel in the generated image is not described in the image data, the pixel position data, and the depth map; and

the image manipulation application further configures the processor to generate the at least one projected pixel by resampling the image data, the pixel position data, and the depth map.

11. The system of claim 8, wherein a pinhole camera model is utilized to project pixels within the generated image based on light rays projecting from the virtual viewpoint, where each projected pixels is associated with at least one of the projected light rays.

12. The system of claim 11, wherein projected pixel depth information is determined for at least one pixel in the generated image based on the depth map, the virtual viewpoint, and the light rays associated with the projected pixel.

13. The system of claim 12, wherein the depth information for a projected pixel is based on minimizing the variance for the projected pixel across the image data within the captured light field image data.

14. The system of claim 11, wherein the image manipulation application further configures the processor to combine projected pixels having the same location within the generated image.

15. The system of claim 14, wherein the pixels are combined based on the weighted average of the pixels, where the weighted average of the pixels is the inverse of the distance from the imager from which the projected pixel originated to the virtual viewpoint.

16. The system of claim 1, further comprising an input device configured to obtain input data indicative of a position within the captured light field image data.

17. The system of claim 16, wherein the input device is a touchscreen interface.

18. The system of claim 16, wherein the input device is a sensor configured to obtain spatial location information.

19. The system of claim 16, wherein the input device is a camera configured to obtain input data selected from the group consisting of head tracking data and gaze tracking data.

20. The system of claim 16, wherein the virtual viewpoint is selected based on the input data.

21. The system of claim 20, wherein the image manipulation application further configures the processor to:

obtain a first input data indicative of a first position within the captured light field image data;

determine a first virtual viewpoint based on the first input data;

generate a first image from the perspective of the first virtual viewpoint;

obtain a second input indicative of a second position within the captured light field image data, where the second position is separate from the first position;

determine a second virtual viewpoint based on the second input data;

generate at least one intermediate virtual viewpoint by interpolating between the first virtual viewpoint and the second virtual viewpoint;

generate at least one intermediate image based on the generated at least one intermediate virtual viewpoints, where each intermediate image is from the perspective of an intermediate virtual viewpoint; and

generate a second image from the perspective of the second virtual viewpoint.

22. The system of claim 1, wherein the image is generated utilizing a super-resolution process.

23. A process for generating an image from a virtual viewpoint, comprising:

obtaining captured light field image data using an image manipulation device, where the captured light field image data comprises image data, pixel position data, and a depth map and where the depth map comprises depth information for one or more pixels in the image data;

determining a virtual viewpoint for the captured light field image data based on the pixel position data and the depth map for the captured light field image data using the image manipulation device, where the virtual viewpoint comprises a virtual location and virtual depth information;

computing a virtual depth map based on the captured light field image data and the virtual viewpoint using the image manipulation device; and

generating an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map using the image manipulation device, where the generated image comprises a plurality of pixels selected from the image data based on the pixel position data and the virtual depth map.