Artificially rendering images using viewpoint interpolation and extrapolation

• US 10,726,593 B2
• Filed: 09/22/2015
• Issued: 07/28/2020
• Est. Priority Date: 09/22/2015
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Miller, Stephen David, Saelzle, Martin, Rusu, Radu Bogdan
• Primary Examiner(s): Nguyen, Anh-Tuan V
• Application Number:
  US14/860,983
• Publication Number:
  US 20170084001A1
• Time in Patent Office: 1,771 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation

Abstract

Various embodiments of the present invention relate generally to systems and methods for artificially rendering images using viewpoint interpolation and/or extrapolation. According to particular embodiments, a transformation between a first frame and a second frame is estimated, where the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location. An artificially rendered image corresponding to a third location positioned on a trajectory between the first location and the second location is then generated by interpolating a transformation from the first location to the third location and from the third location to the second location and gathering image information from the first frame and the second frame by transferring first image information from the first frame to the third frame based on the interpolated transformation and second image information from the second frame to the third frame based on the interpolated transformation. The first image information and the second image information are then combined. If an occlusion is created by a change in layer placement between the first frame and second frame, this occlusion is detected and missing data is replaced to fill the occlusion.

Claims

1. A method comprising:

moving a mobile device with a camera through space in a locally convex or locally concave motion;

obtaining, from the camera throughout the movement of the mobile device, a plurality of frames having location information, wherein the plurality of frames include a first frame and a second frame;

estimating a transformation between the first frame and the second frame, wherein the first frame includes a first image captured by the camera from a first location and the second frame includes a second image captured by the camera from a second location, the first and second frames including foregrounds and backgrounds that include one or more layers, each layer corresponding to a region of image pixels that share a similar motion; and

generating an artificially rendered image corresponding to a third location, wherein the third location is positioned on a trajectory outside of the trajectory between the first location and the second location, the artificially rendered image generated by;

interpolating the transformation from the first location to the third location and from the third location to the second location by moving control points in the one or more layers of the first frame and the second frame in a direction perpendicular to the trajectory between the first location and the second location, wherein control points in layers at a greater depth are moved less than control points in layers at a lesser depth;

gathering image information from the first frame and the second frame by transferring first image information from the first frame to the third frame based on the interpolated transformation and second image information from the second frame to the third frame based on the interpolated transformation;

combining the first image information and the second image information;

detecting a first occlusion in the artificially rendered image, wherein the occlusion is caused by movement of layers; and

replacing missing data to fill the first occlusion, wherein replacing the missing data includes obtaining and using data from the first frame, the second frame, or a third frame separate from the first and second frames.

2. The method of claim 1, wherein the transformation is estimated using homography, affine, similarity, translation, rotation, or scale.

3. The method of claim 1, wherein interpolating the transformation includes parameterizing the transformation from the first location to the third location and from the third location to the second location and linearly interpolating the parameterized transformation.

4. The method of claim 1, wherein combining the first image information and the second image information includes weighting the first image information from the first frame and the second image information from the second frame.

5. The method of claim 1, wherein the missing data is obtained from the first frame or the second frame.

6. The method of claim 1, wherein the missing data is obtained from a fourth frame, wherein the fourth frame includes fourth image information corresponding to the missing data.

7. The method of claim 1, wherein the missing data is obtained by extrapolating from surrounding image data, wherein surrounding image data includes information from frames located near the first location and second location.