Artificially rendering images using interpolation of tracked control points

• US 10,719,732 B2
• Filed: 03/26/2018
• Issued: 07/21/2020
• Est. Priority Date: 07/15/2015
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Miller, Stephen David, Saelzle, Martin, Rusu, Radu Bogdan
• Primary Examiner(s): Chang, Jon
• Application Number:
  US15/936,234
• Publication Number:
  US 20180218235A1
• Time in Patent Office: 848 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

Various embodiments of the present invention relate generally to systems and processes for artificially rendering images using interpolation of tracked control points. According to particular embodiments, a set of control points is tracked between a first frame and a second frame, where the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location. An artificially rendered image corresponding to a third location is then generated by interpolating individual control points for the third location using the set of control points and interpolating pixel locations using the individual control points. The individual control points are used to transform image data.

Claims

1. A method comprising:

tracking a set of control points between a first frame and a second frame, wherein the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location, the first and second locations corresponding to real world location positions; and

generating an artificially rendered image as a third frame corresponding to a third location, the third location being a real world location position on a trajectory between the first location and the second location, wherein generating the artificially rendered image includes;

interpolating a transformation, including interpolating individual control points for the third location using IMU data and the set of control points;

gathering weighted image information by transferring first image information from the first frame to the third frame based on the interpolated transformation and transferring second image information from the second frame to the third, wherein the image information is weighted by 1-x for the first image information and x for the second image information; and

combining the first image information and the second image information to form the artificially rendered image.

2. The method of claim 1, wherein interpolating the transformation includes interpolation using barycentric coordinates based on three or more control points.

3. The method of claim 1, wherein interpolating the transformation includes interpolation using splines.

4. The method of claim 1, wherein interpolating the transformation includes interpolation using finite elements.

5. The method of claim 1, wherein interpolating the transformation includes using motion of a nearest control point.

6. The method of claim 1, wherein the set of control points are located within a first layer, the first layer associated with particular objects viewable in the first frame and second frame.

7. The method of claim 1, wherein each pixel in the first image corresponds to a control point in the set of control points.

8. The method of claim 7, wherein interpolating the transformation includes using dense optical flow.

9. A non-transitory computer readable medium having instructions stored thereon, that when executed by a computer cause the computer to perform:

tracking a set of control points between a first frame and a second frame, wherein the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location, the first and second locations corresponding to real world location positions; and

generating an artificially rendered image as a third frame corresponding to a third location, the third location being a real world location position on a trajectory between the first location and the second location, wherein generating the artificially rendered image includes;

interpolating a transformation, including interpolating individual control points for the third location using IMU data and the set of control points;

gathering weighted image information by transferring first image information from the first frame to the third frame based on the interpolated transformation and transferring second image information from the second frame to the third, wherein the image information is weighted by 1-x for the first image information and x for the second image information; and

combining the first image information and the second image information to form the artificially rendered image.

10. The computer readable medium of claim 9, wherein interpolating the transformation includes interpolation using barycentric coordinates based on three or more control points, interpolation using splines, or interpolation using finite elements.

11. The computer readable medium of claim 9, wherein the set of control points are located within a first layer, the first layer associated with particular objects viewable in the first frame and second frame.

12. A system comprising:

a processor; and

a memory, the memory storing instructions for;

tracking a set of control points between a first frame and a second frame, wherein the first frame includes a first image captured from a first location and the second frame includes a second image captured from a second location, the first and second locations corresponding to real world location positions; and

generating an artificially rendered image as a third frame corresponding to a third location, the third location being a real world location position on a trajectory between the first location and the second location, wherein generating the artificially rendered image includes;

interpolating a transformation, including interpolating individual control points for the third location using IMU data and the set of control points;

gathering weighted image information by transferring first image information from the first frame to the third frame based on the interpolated transformation and transferring second image information from the second frame to the third, wherein the image information is weighted by 1-x for the first image information and x for the second image information; and

combining the first image information and the second image information to form the artificially rendered image.

13. The system of claim 12, wherein interpolating the transformation includes interpolation using barycentric coordinates based on three or more control points.

14. The system of claim 12, wherein interpolating the transformation includes interpolation using splines.

15. The system of claim 12, wherein interpolating the transformation includes interpolation using finite elements.

16. The system of claim 12, wherein interpolating the transformation includes using motion of a nearest control point.

17. The system of claim 12, wherein each pixel in the first image corresponds to a control point in the set of control points.

18. The system of claim 17, wherein interpolating the transformation includes using dense optical flow.

19. The system of claim 12, wherein the set of control points are located within a first layer, the first layer associated with particular objects viewable in the first frame and second frame.