Generating dolly zoom effect using light field image data

• US 10,594,945 B2
• Filed: 04/03/2018
• Issued: 03/17/2020
• Est. Priority Date: 04/03/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Wang, Zejing, Merritt, Joel
• Primary Examiner(s): Haskins, Twyler L
• Assistant Examiner(s): Bhuiyan, Fayez
• Application Number:
  US15/944,551
• Publication Number:
  US 20180288335A1
• Time in Patent Office: 714 Days
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A sensor is configured to acquire a light field by imaging a scene. A processor is configured to determine four-dimensional (4D) coordinates of points in a light field and generate dollied coordinates from the 4D coordinates based on a dolly transform and a dolly parameter. The processor is also configured to project rays associated with the dollied coordinates from the light field onto corresponding points in an output raster. In some cases, the processor applies an aperture function to filter the rays in the coordinate system of the dollied coordinates. The aperture function has a first value in a first region of an aperture plane and the aperture value has a second value in a second region of the aperture plane. Rays passing through the first region are accepted by the aperture function and rays passing through the second region are rejected.

Claims

1. A method comprising:

determining four-dimensional (4D) coordinates of points in a light field;

generating dollied coordinates from the 4D coordinates based on a dolly transform and a dolly parameter;

applying an aperture function to filter rays in the coordinate system of the dollied coordinates, wherein the aperture function has a first value in a first region of an aperture plane and the aperture value has a second value in a second region of the aperture plane such that rays passing through the first region are accepted by the aperture function and rays passing through the second region are rejected; and

projecting rays associated with the dollied coordinates from the light field onto corresponding points in an output raster.

2. The method of claim 1, wherein applying the aperture function to filter the rays in the coordinate system of the dollied coordinates comprises dollying a virtual aperture by a distance relative to the aperture plane.

3. The method of claim 2, wherein generating the dollied coordinates comprises applying a depth dependent dolly transformation to the 4D coordinates.

4. The method of claim 3, wherein dollying the virtual aperture produces a depth-dependent scaling of a two dimensional (2D) reconstructed image that is reconstructed from the light field.

5. The method of claim 1, further comprising:

refocusing portions of the light field using a linear refocusing transform and a refocusing parameter.

6. The method of claim 1, wherein generating the dollied coordinates comprises generating the dollied coordinates by applying a quadratic dolly transform.

7. The method of claim 1, wherein generating the dollied coordinates comprises producing tilt in a lens plane and an aperture plane.

8. An apparatus comprising:

a sensor configured to acquire a light field by imaging a scene; and

a processor configured to;

determine four-dimensional (4D) coordinates of points in a light field;

generate dollied coordinates from the 4D coordinates based on a dolly transform and a dolly parameter;

apply an aperture function to filter rays in the coordinate system of the dollied coordinates, wherein the aperture function has a first value in a first region of an aperture plane and the aperture value has a second value in a second region of the aperture plane such that rays passing through the first region are accepted by the aperture function and rays passing through the second region are rejected; and

project rays associated with the dollied coordinates from the light field onto corresponding points in an output raster.

9. The apparatus of claim 8, wherein the processor is configured to dolly a virtual aperture by a distance relative to the aperture plane.

10. The apparatus of claim 9, wherein the processor is configured to apply a depth dependent dolly transformation to the 4D coordinates.

11. The apparatus of claim 10, wherein dollying the virtual aperture produces a depth-dependent scaling of a two dimensional (2D) reconstructed image that is reconstructed from the light field.

12. The apparatus of claim 8, wherein the processor is configured to refocus portions of the light field using a linear refocusing transform and a refocusing parameter.

13. The apparatus of claim 8, wherein the processor is configured to generate the dollied coordinates by applying a quadratic dolly transform.

14. The apparatus of claim 8, wherein the processor is configured to tilt an image in a lens plane and an aperture plane applying the dolly transform.