System and method for infinite smoothing of image sequences

• US 10,540,773 B2
• Filed: 04/15/2019
• Issued: 01/21/2020
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Miller, Stephen David, Rusu, Radu Bogdan, Ren, Yuheng
• Primary Examiner(s): Bader, Robert
• Application Number:
  US16/384,578
• Publication Number:
  US 20190244372A1
• Time in Patent Office: 281 Days
• Field of Search: None
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Various embodiments of the present invention relate generally to systems and processes for interpolating images of an object. According to particular embodiments, a sequence of images is obtained using a camera which captures the sequence of images along a camera translation. Each image contains at least a portion of overlapping subject matter. A plurality of keypoints is identified on a first image of the sequence of images. Each keypoint from the first image are kept track of to a second image. Using a predetermined algorithm, a plurality of transformations are computed using two randomly sampled keypoint correspondences, each of which includes a keypoint on the first image and a corresponding keypoint on the second image. An optimal subset of transformations is determined from the plurality of transformations based on predetermined criteria, and transformation parameters corresponding to the optimal subset of transformations is calculated and stored for on-the-fly interpolation.

Claims

1. A method for interpolating images of an object, the method including:

obtaining a sequence of images, the sequence of images including a first image and a second image;

computing, using a predetermined algorithm, a plurality of transformations between the first and second images;

determining an optimal subset of transformations from the plurality of transformations based on predetermined criteria, wherein determining the optimal subset of transformations includes selecting an optimal transformation candidate for each pixel of the first image, wherein selecting an optimal transformation candidate for each pixel is processed as a separate thread such that the processing time for each pixel is bounded by O(1) rather than O(n), wherein the plurality of transformations includes a subset of transformation candidates, wherein determining the optimal subset of transformations includes applying each transformation candidate in the subset of transformation candidates to each pixel in the first image, thereby generating a plurality of first interpolated pixel candidates, and to a corresponding pixel in the second image, thereby generating a plurality of second interpolated pixel candidates, and then comparing the plurality of first interpolated pixel candidates to the corresponding plurality of second interpolated pixel candidates to select an optimal transformation candidate having the smallest corresponding pixel candidate difference; and

calculating transformation parameters corresponding to the optimal subset of transformations.

2. The method of claim 1, further comprising storing the transformation parameters for on-the-fly interpolation.

3. The method of claim 1, wherein computing the plurality of transformations includes identifying a plurality of correspondences between the first and second images.

4. The method of claim 1, wherein transformations are computed for a predetermined number of keypoint correspondences.

5. The method of claim 1, wherein computing a transformation includes using one or more of the following transformations:

similarity, homography, image warping, dense optical flow, and sparse optical flow.

6. The method of claim 1, wherein determining the optimal subset of transformations includes, for a particular transformation:

determining a number of inlier keypoints; and

calculating an image intensity difference value between a transformed image and the second image.

7. The method of claim 1, wherein a random sample consensus (RANSAC) algorithm is implemented to compute the plurality of transformations and determine the optimal subset of transformations.

8. The method of claim 1, wherein a transformation in the subset of transformations is selected for each pixel during on-the-fly interpolation.

9. The method of claim 1, further comprising applying the transformation parameters during runtime based on a user selection.

10. The method of claim 9, wherein the user selection corresponds to a projected physical location of a camera along a calculated camera translation.

11. A system for interpolating images of an object, the system comprising:

a processor, and

memory storing one or more programs configured for execution by the processor, the one or more programs comprising instructions for;

obtaining a sequence of images, the sequence of images including a first image and a second image;

computing, using a predetermined algorithm, a plurality of transformations between the first and second images;

determining an optimal subset of transformations from the plurality of transformations based on predetermined criteria, wherein determining the optimal subset of transformations includes selecting an optimal transformation candidate for each pixel of the first image, wherein selecting an optimal transformation candidate for each pixel is processed as a separate thread such that the processing time for each pixel is bounded by O(1) rather than O(n), wherein the plurality of transformations includes a subset of transformation candidates, wherein determining the optimal subset of transformations includes applying each transformation candidate in the subset of transformation candidates to each pixel in the first image, thereby generating a plurality of first interpolated pixel candidates, and to a corresponding pixel in the second image, thereby generating a plurality of second interpolated pixel candidates, and then comparing the plurality of first interpolated pixel candidates to the corresponding plurality of second interpolated pixel candidates to select an optimal transformation candidate having the smallest corresponding pixel candidate difference; and

calculating transformation parameters corresponding to the optimal subset of transformations.

12. The system of claim 11, wherein the one or more programs further comprises instructions for storing the transformation parameters for on-the-fly interpolation.

13. The system of claim 11, wherein computing the plurality of transformations includes identifying a plurality of correspondences between the first and second images.

14. The system of claim 11, wherein transformations are computed for a predetermined number of keypoint correspondences.

15. The system of claim 11, wherein computing a transformation includes using one or more of the following transformations:

similarity, homography, image warping, dense optical flow, and sparse optical flow.

16. The system of claim 11, wherein determining the optimal subset of transformations includes, for a particular transformation:

determining a number of inlier keypoints; and

calculating an image intensity difference value between a transformed image and the second image.

17. The system of claim 11, wherein a random sample consensus (RANSAC) algorithm is implemented to compute the plurality of transformations and determine the optimal subset of transformations.

18. The system of claim 11, wherein a transformation in the subset of transformations is selected for each pixel during on-the-fly interpolation.

19. The system of claim 11, wherein the one or more programs further comprises instructions for applying the transformation parameters during runtime based on a user selection.

20. A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

obtaining a sequence of images, the sequence of images including a first image and a second image;

computing, using a predetermined algorithm, a plurality of transformations between the first and second images;

determining an optimal subset of transformations from the plurality of transformations based on predetermined criteria, wherein determining the optimal subset of transformations includes selecting an optimal transformation candidate for each pixel of the first image, wherein selecting an optimal transformation candidate for each pixel is processed as a separate thread such that the processing time for each pixel is bounded by O(1) rather than O(n), wherein the plurality of transformations includes a subset of transformation candidates, wherein determining the optimal subset of transformations includes applying each transformation candidate in the subset of transformation candidates to each pixel in the first image, thereby generating a plurality of first interpolated pixel candidates, and to a corresponding pixel in the second image, thereby generating a plurality of second interpolated pixel candidates, and then comparing the plurality of first interpolated pixel candidates to the corresponding plurality of second interpolated pixel candidates to select an optimal transformation candidate having the smallest corresponding pixel candidate difference; and

calculating transformation parameters corresponding to the optimal subset of transformations.