Generating images from light fields utilizing virtual viewpoints

• US 10,390,005 B2
• Filed: 10/06/2015
• Issued: 08/20/2019
• Est. Priority Date: 09/28/2012
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: Nisenzon, Semyon, Jain, Ankit K.
• Primary Examiner(s): Rush, Eric
• Application Number:
  US14/876,024
• Publication Number:
  US 20160255333A1
• Time in Patent Office: 1,414 Days
• Field of Search: 382100, 382154, 382276, 382293, 382312, 382284, 382294, 3482181, 348266, 348332, 396322, 396324, 396325, 396326, 396327, 396332, 396333, 396334, 345419, 345421, 345422, 345427, 345629, 345634, 345635, 345637
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/95 Computational photography systems, e.g. light-field imaging systems

Abstract

Systems and methods for the synthesis of light field images from virtual viewpoints in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system includes a processor and a memory configured to store captured light field image data and an image manipulation application, wherein the captured light field image data includes image data, pixel position data, and a depth map, and wherein the image manipulation application configures the processor to obtain captured light field image data, determine a virtual viewpoint for the captured light field image data, where the virtual viewpoint includes a virtual location and virtual depth information, compute a virtual depth map based on the captured light field image data and the virtual viewpoint, and generate an image from the perspective of the virtual viewpoint based on the captured light field image data and the virtual depth map.

Claims

1. A system configured to synthesize images using a plurality of images captured from multiple viewpoints, comprising:

a processor; and

a memory connected to the processor and configured to store the plurality of images captured from the multiple viewpoints and an image manipulation application;

wherein the plurality of images comprises;

image data and pixel position data, the plurality of images being captured using an array camera comprising a plurality of cameras that capture the plurality of images from viewpoints of the plurality of cameras, wherein multiple cameras in the array camera simultaneously capture the plurality of images, the viewpoints of the plurality of cameras including a viewpoint of a first camera within the array camera and additional viewpoints, the additional viewpoints including a viewpoint of a second camera within the array camera, and wherein the plurality of images includes occluded pixel information captured from at least one of the additional viewpoints describing pixels not visible from the viewpoint of the first camera; and

wherein the image manipulation application configures the processor to;

obtain the plurality of images;

generate a depth map from the viewpoint of the first camera for the plurality of images using the plurality of images, where the depth map comprises depth information for one or more pixels in the image data, wherein generating the depth map comprises;

(1) for each of a plurality of depth levels, shifting the plurality of images into a stack of images for a particular depth level and computing a variance for a particular pixel in the image stack; and

(2) determining a depth level for the particular pixel by minimizing the variance for the particular pixel across the image stack;

determine a virtual viewpoint for the plurality of images based on the pixel position data and the depth map for the plurality of images, where the virtual viewpoint comprises a virtual location and virtual depth information, wherein the virtual viewpoint is a separate viewpoint from the viewpoint of the first camera, and is a viewpoint from an interpolated position between the viewpoint of the first camera and the viewpoint of the second camera;

compute a virtual depth map based on the plurality of images by projecting pixel depth information from the depth map from the viewpoint of the first camera to the virtual viewpoint; and

generate an image from the perspective of the virtual viewpoint based on the plurality of images and the virtual depth map by projecting pixels from the plurality of images based on the pixel position data and the depth map, where;

the generated image comprises a plurality of pixels selected from the image data based on the pixel position data and the virtual depth map, and

the pixels that are projected include at least one occluded pixel from the occluded pixel information that is visible from the perspective of the virtual viewpoint.

2. The system of claim 1, wherein:

at least one projected pixel in the generated image is not described in the image data, the pixel position data, and the depth map; and

the image manipulation application further configures the processor to generate the at least one projected pixel by resampling the image data, the pixel position data, and the depth map.

3. The system of claim 1, wherein a pinhole camera model is utilized to project pixels within the generated image based on light rays projecting from the virtual viewpoint, where each projected pixel is associated with at least one of the projected light rays.

4. The system of claim 3, wherein projected pixel depth information is determined for at least one pixel in the generated image based on the depth map, the virtual viewpoint, and light rays associated with the at least one projected pixel.

5. The system of claim 4, wherein protected pixel depth information for a projected pixel is based on minimizing the variance for the projected pixel across the image data within the plurality of images captured from the multiple viewpoints.

6. The system of claim 3, wherein the image manipulation application further configures the processor to combine projected pixels having the same location within the generated image.

7. The system of claim 6, wherein the projected pixels having the same location are combined based on a weighted average of the pixels.

8. The system of claim 1, further comprising a sensor configured to obtain input data indicative of a position within the plurality of images captured from the multiple viewpoints.

9. The system of claim 8, wherein the sensor is a touchscreen interface.

10. The system of claim 8, wherein the sensor is configured to obtain spatial location information.

11. The system of claim 8, wherein the sensor is a camera configured to obtain input data selected from the group consisting of head tracking data and gaze tracking data.

12. The system of claim 8, wherein the virtual viewpoint is selected based on the input data.