Stabilizing image sequences based on camera rotation and focal length parameters

• US 10,586,378 B2
• Filed: 01/17/2017
• Issued: 03/10/2020
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Rusu, Radu Bogdan, Ren, Yuheng
• Primary Examiner(s): Bader, Robert
• Application Number:
  US15/408,270
• Publication Number:
  US 20170124680A1
• Time in Patent Office: 1,148 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 5/265 Mixing

Abstract

The present disclosure describes systems and processes for image sequence stabilization. According to particular embodiments, a sequence of images is obtained using a camera which captures the sequence of images along a camera translation. Each image contains at least a portion of overlapping subject matter. A plurality of keypoints is identified on a first image of the sequence of images. Each keypoint from the first image are kept track of to a second image. Using a predetermined algorithm, a camera rotation value and a focal length value are calculated from two randomly sampled keypoints on the first image and two corresponding keypoints on the second image. An optimal camera rotation and focal length pair corresponding to an optimal transformation for producing an image warp for image sequence stabilization is determined. The image warp for image sequence stabilization is constructed using the optimal camera and focal length pair.

Claims

1. A method for image sequence stabilization, the method including:

moving a mobile device through space in a locally convex or locally concave motion;

obtaining a sequence of images using a camera on the mobile device, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object; and

stabilizing, via a processor, each image in the sequence of images using an optimal image warp based on an optimal camera rotation value and an optimal focal length value, wherein the stabilizing includes;

identifying a plurality of keypoints on a first image of the sequence of images;

keeping track of each keypoint from the first image to a second image;

calculating, using a predetermined algorithm, a camera rotation value and a focal length value from two randomly sampled keypoints on the first image and two corresponding keypoints on the second image;

calculating a plurality of transformations between the first image and the second image;

determining an optimal transformation from the plurality of transformations for producing the optimal image warp, wherein determining the optimal transformation involves sequentially testing transformations of the plurality of transformations by

     1) determining if a current transformation results in the largest inlier group of keypoints compared to a previously selected transformation and

     2) next, determining if the current transformation resulting in the largest inlier group of keypoints also results in the lowest image intensity difference between the first image and the second image compared to the previously selected transformation,

wherein if the current transformation does not result in a larger inlier group of keypoints than the previously selected transformation, the current transformation is not tested for image intensity difference,

wherein if the current transformation results in a larger inlier group of keypoints than the previously selected transformation and results in a higher image intensity difference, the previously selected transformation is discarded and the current transformation replaces the selected transformation for the next sequential test, and otherwise the current transformation is discarded,

wherein inlier keypoints are determined by calculating a pixel distance between a transformed keypoint of the first image and a corresponding keypoint on the second image, wherein a transformed keypoint is labeled an inlier if the pixel distance calculated is less than a pixel distance threshold, wherein the pixel distance threshold is a function of image resolution of the first and second images,

wherein image intensity difference is determined by;

     1) applying a transformation to every pixel in the first image to warp the first image onto the second image,

     2) calculating an error or difference for each color channel between each corresponding pixel in the first and second image,

     3) deriving an average error of the RGB value of all color channels for each pixel, and

     4) calculating the average error for all pixels within the first and second image, wherein image intensity difference between the first and second image is a value between 0 and 255;

determining the optimal camera rotation and focal length pair corresponding to the optimal transformation for producing the optimal image warp for image sequence stabilization;

constructing the optimal image warp for image sequence stabilization using the optimal camera and focal length pair; and

applying the optimal image warp to the sequence of images by composing optimal transformations into a trajectory, wherein the trajectory is smoothed into a polynomial or linear model.

2. The method of claim 1, wherein identifying a plurality of keypoints includes using Harris-style keypoint detection.

3. The method of claim 1, wherein keeping track of each keypoint includes using Kanade-Lucas-Tomasi-style (KLT-style) keypoint tracking.

4. The method of claim 1, wherein the calculated transformations are offset to fit the smooth trajectory.

5. The method of claim 1, wherein determining the optimal camera rotation and focal length pair includes using random sample consensus on a plurality of transformations to determine the optimal transformation for image sequence stabilization.

6. The method of claim 1, wherein a plurality of camera rotation values and focal length values are calculated from the plurality of keypoints in order to determine the optimal camera rotation and focal length pair.

7. The method of claim 1, wherein the image warp is constructed as KRK⁻

1, where K is an intrinsic parameter given the optimal focal length, and R is a 3×

3 rotation matrix.

8. A system for image sequence stabilization, the system comprising:

a camera,

a processor, and

memory storing one or more programs configured for execution by the processor, the one or more programs comprising instructions for;

obtaining a sequence of images using a camera, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object, the sequence of images being obtained via moving a mobile device through space in a locally convex or locally concave motion; and

stabilizing each image in the sequence of images using an optimal image warp based on an optimal camera rotation value and an optimal focal length value, wherein the stabilizing includes;

identifying a plurality of keypoints on a first image of the sequence of images;

keeping track of each keypoint from the first image to a second image;

calculating, using a predetermined algorithm, a camera rotation value and a focal length value from two randomly sampled keypoints on the first image and two corresponding keypoints on the second image;

calculating a plurality of transformations between the first image and the second image;

determining an optimal transformation from the plurality of transformations for producing the optimal image warp, wherein determining the optimal transformation involves sequentially testing transformations of the plurality of transformations by

     1) determining if a current transformation results in the largest inlier group of keypoints compared to a previously selected transformation and

     2) next, determining if the current transformation resulting in the largest inlier group of keypoints also results in the lowest image intensity difference between the first image and the second image compared to the previously selected transformation,

wherein if the current transformation does not result in a larger inlier group of keypoints than the previously selected transformation, the current transformation is not tested for image intensity difference,

 

wherein if the current transformation results in a larger inlier group of keypoints than the previously selected transformation and results in a higher image intensity difference, the previously selected transformation is discarded and the current transformation replaces the selected transformation for the next sequential test, and otherwise the current transformation is discarded,

wherein inlier keypoints are determined by calculating a pixel distance between a transformed keypoint of the first image and a corresponding keypoint on the second image, wherein a transformed keypoint is labeled an inlier if the pixel distance calculated is less than a pixel distance threshold, wherein the pixel distance threshold is a function of image resolution of the first and second images,

wherein image intensity difference is determined by;

     1) applying a transformation to every pixel in the first image to warp the first image onto the second image,

     2) calculating an error or difference for each color channel between each corresponding pixel in the first and second image,

     3) deriving an average error of the RGB value of all color channels for each pixel, and

     4) calculating the average error for all pixels within the first and second image, wherein image intensity difference between the first and second image is a value between 0 and 255;

determining the optimal camera rotation and focal length pair corresponding to the optimal transformation for producing the optimal image warp for image sequence stabilization;

constructing the optimal image warp for image sequence stabilization using the optimal camera and focal length pair; and

applying the optimal image warp to the sequence of images by composing optimal transformations into a trajectory, wherein the trajectory is smoothed into a polynomial or linear model.

9. The system of claim 8, wherein identifying a plurality of keypoints includes using Harris-style keypoint detection.

10. The system of claim 8, wherein keeping track of each keypoint includes using Kanade-Lucas-Tomasi-style (KLT-style) keypoint tracking.

11. The system of claim 8, wherein the calculated transformations are offset to fit the smooth trajectory.

12. The system of claim 8, wherein determining the optimal camera rotation and focal length pair includes using random sample consensus on a plurality of transformations to determine the optimal transformation for image sequence stabilization.

13. The system of claim 8, wherein a plurality of camera rotation values and focal length values are calculated from the plurality of keypoints in order to determine the optimal camera rotation and focal length pair.

14. The system of claim 8, wherein the image warp is constructed as KRK⁻

1, where K is an intrinsic parameter given the optimal focal length, and R is a 3×

3 rotation matrix.

15. A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

obtaining a sequence of images using a camera, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including an object, the sequence of images being obtained via moving a mobile device through space in a locally convex or locally concave motion; and

stabilizing each image in the sequence of images using an optimal image warp based on an optimal camera rotation value and an optimal focal length value, wherein the stabilizing includes;

identifying a plurality of keypoints on a first image of the sequence of images;

keeping track of each keypoint from the first image to a second image;

calculating, using a predetermined algorithm, a camera rotation value and a focal length value from two randomly sampled keypoints on the first image and two corresponding keypoints on the second image;

calculating a plurality of transformations between the first image and the second image;

determining an optimal transformation from the plurality of transformations for producing the optimal image warp, wherein determining the optimal transformation involves sequentially testing transformations of the plurality of transformations by

     1) determining if a current transformation results in the largest inlier group of keypoints compared to a previously selected transformation and

     2) next, determining if the current transformation resulting in the largest inlier group of keypoints also results in the lowest image intensity difference between the first image and the second image compared to the previously selected transformation,

wherein if the current transformation does not result in a larger inlier group of keypoints than the previously selected transformation, the current transformation is not tested for image intensity difference,

wherein if the current transformation results in a larger inlier group of keypoints than the previously selected transformation and results in a higher image intensity difference, the previously selected transformation is discarded and the current transformation replaces the selected transformation for the next sequential test, and otherwise the current transformation is discarded,

wherein inlier keypoints are determined by calculating a pixel distance between a transformed keypoint of the first image and a corresponding keypoint on the second image, wherein a transformed keypoint is labeled an inlier if the pixel distance calculated is less than a pixel distance threshold, wherein the pixel distance threshold is a function of image resolution of the first and second images,

wherein image intensity difference is determined by;

     1) applying a transformation to every pixel in the first image to warp the first image onto the second image,

     2) calculating an error or difference for each color channel between each corresponding pixel in the first and second image,

     3) deriving an average error of the RGB value of all color channels for each pixel, and

     4) calculating the average error for all pixels within the first and second image, wherein image intensity difference between the first and second image is a value between 0 and 255;

determining the optimal camera rotation and focal length pair corresponding to the optimal transformation for producing the optimal image warp for image sequence stabilization;

constructing the optimal image warp for image sequence stabilization using the optimal camera and focal length pair; and

applying the optimal image warp to the sequence of images by composing optimal transformations into a trajectory, wherein the trajectory is smoothed into a polynomial or linear model.

16. The non-transitory computer readable medium of claim 15, wherein identifying a plurality of keypoints includes using Harris-style keypoint detection.

17. The non-transitory computer readable medium of claim 15, wherein keeping track of each keypoint includes using Kanade-Lucas-Tomasi-style (KLT-style) keypoint tracking.

18. The non-transitory computer readable medium of claim 15, wherein the calculated transformations are offset to fit the smooth trajectory.

19. The non-transitory computer readable medium of claim 15, wherein determining the optimal camera rotation and focal length pair includes using random sample consensus on a plurality of transformations to determine the optimal transformation for image sequence stabilization.

20. The non-transitory computer readable medium of claim 15, wherein the image warp is constructed as KRK⁻

1, where K is an intrinsic parameter given the optimal focal length, and R is a 3×

3 rotation matrix.