Artificially rendering images using viewpoint interpolation and extrapolation

• US 10,242,474 B2
• Filed: 07/15/2015
• Issued: 03/26/2019
• Est. Priority Date: 07/15/2015
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Miller, Stephen David, Saelzle, Martin, Rusu, Radu Bogdan
• Primary Examiner(s): Nguyen, Anh-Tuan V
• Application Number:
  US14/800,640
• Publication Number:
  US 20170018054A1
• Time in Patent Office: 1,350 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2210/44 Morphing
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

Various embodiments of the present invention relate generally to mechanisms and processes relating to artificially rendering images using viewpoint interpolation and extrapolation. According to particular embodiments, a method includes applying a transform to estimate a path outside the trajectory between a first frame and a second frame, where the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location. The process also includes generating an artificially rendered image corresponding to a third location positioned on the path. The artificially rendered image is generated by interpolating a transformation from the first location to the third location and from the third location to the second location, gathering image information from the first frame and the second frame by transferring first image information from the first frame to the third frame and second image information from the second frame to the third frame, and combining the first image information and the second image information.

Claims

1. A method comprising:

moving a mobile device with a camera through space in a locally convex or locally concave motion;

obtaining, from the camera throughout the movement of the mobile device, a plurality of frames having location information, wherein the plurality of frames include a first frame and a second frame;

applying a transform to estimate a path outside the trajectory between the first frame and the second frame, wherein the first frame includes a first image captured by the camera from a first location and the second frame includes a second image captured by the camera from a second location;

generating an artificially rendered image corresponding to a third location, wherein the third location is positioned on the path, the artificially rendered image generated by;

interpolating a transformation from the first location to the third location and from the third location to the second location;

gathering image information from the first frame and the second frame by transferring first image information from the first frame to the third frame based on the interpolated transformation and second image information from the second frame to the third frame based on the interpolated transformation; and

combining the first image information and the second image information;

determining whether an occlusion is present due to movement of multiple layers; and

if an occlusion is detected, determining whether the layer closer to the camera is non-see-through or partially see-through, wherein if the layer closer to the camera is non-see-through, only image information from the layer closer to the camera is taken, wherein if the layer closer to the camera is partially see-through, then image information is taken from both layers,

wherein generating the artificially rendered image includes filling in missing information using viewpoint extrapolation by moving different layers of the multiple layers in a motion perpendicular to the trajectory.

2. The method of claim 1, wherein the transform warps the first image and second image according to curvature and direction of the estimated path.

3. The method of claim 1, wherein the transform implements homography.

4. The method of claim 1, wherein the transform implements spherical projection.

5. The method of claim 1, wherein the transform implements cylindrical projection.

6. The method of claim 1, wherein the first image is associated with a first layer and wherein the second image is associated with the first layer.

7. A non-transitory computer readable medium comprising:

computer code moving a mobile device with a camera through space in a locally convex or locally concave motion;

computer code for obtaining, from the camera throughout the movement of the mobile device, a plurality of frames having location information, wherein the plurality of frames include a first frame and a second frame;

computer code for applying a transform to estimate a path outside the trajectory between the first frame and the second frame, wherein the first frame includes a first image captured by the camera from a first location and the second frame includes a second image captured by the camera from a second location computer code for applying a transform to estimate a path outside the trajectory between a first frame and a second frame, wherein the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location;

computer code for generating an artificially rendered image corresponding to a third location, wherein the third location is positioned on the path, the artificially rendered image generated by;

interpolating a transformation from the first location to the third location and from the third location to the second location;

gathering image information from the first frame and the second frame by transferring first image information from the first frame to the third frame based on the interpolated transformation and second image information from the second frame to the third frame based on the interpolated transformation; and

combining the first image information and the second image information;

computer code for determining whether an occlusion is present due to movement of multiple layers; and

computer code for determining, if an occlusion is detected, whether the layer closer to the camera is non-see-through or partially see-through, wherein if the layer closer to the camera is non-see-through, only image information from the layer closer to the camera is taken, wherein if the layer closer to the camera is partially see-through, then image information is taken from both layers,

wherein generating the artificially rendered image includes filling in missing information using viewpoint extrapolation by moving different layers of the multiple layers in a motion perpendicular to the trajectory.

8. The non-transitory computer readable medium of claim 7, wherein the transform warps the first image and second image according to curvature and direction of the estimated path.

9. The non-transitory computer readable medium of claim 7, wherein the transform implements homography.

10. The non-transitory computer readable medium of claim 7, wherein the transform implements spherical projection.

11. The non-transitory computer readable medium of claim 7, wherein the transform implements cylindrical projection.

12. The non-transitory computer readable medium of claim 7, wherein the first image is associated with a first layer and wherein the second image is associated with the first layer.