Artificially rendering images using viewpoint interpolation and extrapolation

• US 10,147,211 B2
• Filed: 07/15/2015
• Issued: 12/04/2018
• Est. Priority Date: 07/15/2015
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Miller, Stephen David, Saelzle, Martin, Rusu, Radu Bogdan
• Primary Examiner(s): NGUYEN, ANH TUAN V
• Application Number:
  US14/800,642
• Publication Number:
  US 20170018055A1
• Time in Patent Office: 1,238 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation

Abstract

Various embodiments of the present invention relate generally to systems and methods for artificially rendering images using viewpoint interpolation and extrapolation. According to particular embodiments, a method includes moving a set of control points perpendicular to a trajectory between a first frame and a second frame, where the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location. The set of control points is associated with a layer and each control point is moved based on an associated depth of the control point. The method also includes generating an artificially rendered image corresponding to a third location outside of the trajectory by extrapolating individual control points using the set of control points for the third location and extrapolating pixel locations using the individual control points.

Claims

1. A method comprising:

moving a mobile device with a camera through space in a locally convex or locally concave motion;

obtaining, from the camera throughout the movement of the mobile device, a plurality of frames having location information obtained from Inertial Measuring Unit (IMU) information and depth information, wherein the plurality of frames include a first frame and a second frame;

moving a set of control points perpendicular to a trajectory between the first frame and the second frame, wherein the first frame includes a first image captured using the camera from a first location and the second frame includes a second image captured using the camera from a second location, wherein each control point is moved based on an associated depth of the control point, the depth of each of the control points being determined from the amount of frame-to-frame motion of the control points between the first frame and the second frame, wherein control points located at a further depth are moved less than control points located at a closer depth, wherein the set of control points is associated with a single layer among multiple layers;

generating an artificially rendered image corresponding to a third location outside of the trajectory by extrapolating individual control points using the set of control points for the third location and extrapolating pixel locations using the individual control points, wherein generating the artificially rendered image occurs on the fly based on content-weighted keypoint tracks between the first image and the second image, the IMU information, and the depth information; and

generating a surround view by fusing the plurality of frames along with the artificially rendered image, wherein the surround view is a three-dimensional multi-view interactive digital media representation (MIDMR) created from two-dimensional images in the plurality of frames and the artificially rendered image.

2. The method of claim 1, wherein extrapolating the pixel locations includes extrapolation using barycentric coordinates based on three or more control points.

3. The method of claim 1, wherein extrapolating the pixel locations includes extrapolation using splines.

4. The method of claim 1, wherein extrapolating the pixel locations includes extrapolation using finite elements.

5. The method of claim 1, wherein extrapolating the pixel locations includes using motion of a nearest control point.

6. The method of claim 1, wherein extrapolating pixel locations using the individual control points includes using dense optical flow.

7. The method of claim 1, wherein extrapolating pixel locations using the individual control points includes using depth data.

8. The method of claim 1, wherein each pixel in the first image corresponds to a control point in the set of control points.

9. A non-transitory computer readable medium comprising:

computer code for obtaining, from the camera throughout the movement of the mobile device, a plurality of frames having location information obtained from Inertial Measuring Unit (IMU) information and depth information, wherein the plurality of frames include a first frame and a second frame;

computer code for moving a set of control points perpendicular to a trajectory between the first frame and the second frame, wherein the first frame includes a first image captured using the camera from a first location and the second frame includes a second image captured using the camera from a second location, wherein each control point is moved based on an associated depth of the control point, the depth of each of the control points being determined from the amount of frame-to-frame motion of the control points between the first frame and the second frame, wherein control points located at a further depth are moved less than control points located at a closer depth, wherein the set of control points is associated with a single layer among multiple layers;

computer code for generating an artificially rendered image corresponding to a third location outside of the trajectory by extrapolating individual control points using the set of control points for the third location and extrapolating pixel locations using the individual control points, wherein generating the artificially rendered image occurs on the fly based on content-weighted keypoint tracks between the first image and the second image, the IMU information, and the depth information; and

computer code for generating a surround view by fusing the plurality of frames along with the artificially rendered image, wherein the surround view is a three-dimensional multi-view interactive digital media representation (MIDMR) created from two-dimensional images in the plurality of frames and the artificially rendered image.

10. The computer readable medium of claim 9, wherein extrapolating the pixel locations includes extrapolation using barycentric coordinates based on three or more control points, extrapolation using splines, or extrapolation using finite elements.

11. The computer readable medium of claim 9, wherein extrapolating the pixel locations includes using motion of a nearest control point.

12. The computer readable medium of claim 9, wherein each pixel in the first image corresponds to a control point in the set of control points.

13. The computer readable medium of claim 12, wherein extrapolating pixel locations using the individual control points includes using dense optical flow.

14. The computer readable medium of claim 12, wherein extrapolating pixel locations using the individual control points includes using depth data.

15. A system comprising:

a camera;

a processor; and

memory, wherein the memory stores instructions for;

obtaining, from the camera throughout the movement of the mobile device, a plurality of frames having location information obtained from Inertial Measuring Unit (IMU) information and depth information, wherein the plurality of frames include a first frame and a second frame;

moving a set of control points perpendicular to a trajectory between the first frame and the second frame, wherein the first frame includes a first image captured using the camera from a first location and the second frame includes a second image captured using the camera from a second location wherein each control point is moved based on an associated depth of the control point, the depth of each of the control points being determined from the amount of frame-to-frame motion of the control points between the first frame and the second frame, wherein control points located at a further depth are moved less than control points located at a closer depth, wherein the set of control points is associated with a single layer among multiple layers;

generating an artificially rendered image corresponding to a third location outside of the trajectory by extrapolating individual control points using the set of control points for the third location and extrapolating pixel locations using the individual control points, wherein generating the artificially rendered image occurs on the fly based on content-weighted keypoint tracks between the first image and the second image, the IMU information, and the depth information; and

generating a surround view by fusing the plurality of frames along with the artificially rendered image, wherein the surround view is a three-dimensional multi-view interactive digital media representation (MIDMR) created from two-dimensional images in the plurality of frames and the artificially rendered image.

16. The apparatus of claim 15, wherein extrapolating the pixel locations includes extrapolation using barycentric coordinates based on three or more control points, extrapolation using splines, or extrapolation using finite elements.

17. The apparatus of claim 15, wherein extrapolating the pixel locations includes using motion of a nearest control point.

18. The apparatus of claim 15, wherein each pixel in the first image corresponds to a control point in the set of control points.