Generic platform video image stabilization

• US 8,896,715 B2
• Filed: 02/11/2010
• Issued: 11/25/2014
• Est. Priority Date: 02/11/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wu, Yongjun, Borisov, Nikola, Zhao, Weidong, Sadhwani, Shyam, Thumpudi, Naveen
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US12/704,047
• Publication Number:
  US 20110193978A1
• Time in Patent Office: 1,748 Days
• Field of Search: 348/208.99, 348/208.1, 3482084-2086, 348/208.13, 3482261-2281, 348/169, 348154-155, 348/352
• US Class Current: 348/208.6
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Video image stabilization provides better performance on a generic platform for computing devices by evaluating available multimedia digital signal processing components, and selecting the available components to utilize according to a hierarchy structure for video stabilization performance for processing parts of the video stabilization. The video stabilization has improved motion vector estimation that employs refinement motion vector searching according to a pyramid block structure relationship starting from a downsampled resolution version of the video frames. The video stabilization also improves global motion transform estimation by performing a random sample consensus approach for processing the local motion vectors, and selection criteria for motion vector reliability. The video stabilization achieves the removal of hand shakiness smoothly by real-time one-pass or off-line two-pass temporal smoothing with error detection and correction.

Claims

1. One or more computer-readable media storing computer-executable instructions for causing a computing device programmed thereby to perform a method of digital video stabilization, the one or more computer-readable media including non-volatile memory or a storage device, the method comprising:

evaluating the computing device to determine availability of any of various multimedia digital signal processing (“

DSP”

) components on the computing device;

determining from the results of said evaluating which available multimedia DSP component to utilize for at least one part of digital video stabilization according to a hierarchical structure for ordering multimedia DSP components by criteria comprising at least performance of the digital video stabilization, the digital video stabilization comprising at least local motion vector estimation to produce estimated local motion vectors, processing the estimated local motion vectors for estimation of a global motion transform representing jittery video motion, and image warping based on the global motion transform to compensate the jittery video motion; and

performing digital video stabilization of a video segment on the computing device utilizing the determined available multimedia DSP component for said at least one part of the digital video stabilization, wherein the performing the digital video stabilization includes warping at least one frame of the video segment based on the global motion transform that applies for the at least one frame of the video segment.

2. The one or more computer-readable media of claim 1 wherein the hierarchical structure comprises in order multimedia DSP ASIC units, GPU shader units, multicore CPU and single core CPU.

3. The one or more computer-readable media of claim 1 wherein, when said evaluating determines a GPU shader is available on the computing device, then the digital video stabilization utilizes the GPU shader for the local motion vector estimation.

4. The one or more computer-readable media of claim 3 wherein, when said evaluating determines the GPU shader is available on the computing device, then the digital video stabilization further utilizes the GPU shader for the image warping.

5. The one or more computer-readable media of claim 1 wherein a library of an operating system includes a set of library functions to control the digital video stabilization, and wherein the library provides a programmatic interface for applications to make use of the digital video stabilization, the library including at least one call to an operating system service or a DSP interface as part of the evaluating and including at least one call to a DSP interface to perform the at least one part of the digital video stabilization.

6. The one or more computer-readable media of claim 1 wherein an application executable controls the digital video stabilization, the application executable including at least one call to an operating system service or a DSP interface as part of the evaluating and including at least one call to a DSP interface to perform the at least one part of the digital video stabilization.

7. The one or more computer-readable media of claim 1 wherein a library of an operating system controls the digital video stabilization, the library including at least one call to a DSP interface as part of the evaluating and including at least one call to a DSP interface to perform the at least one part of the digital video stabilization.

8. A method of digital video stabilization, the method comprising performing steps on a computing device of:

performing local motion vector estimation for a frame of a plurality of frames of a scene of a video segment, wherein the local motion vector estimation comprises for said frame;

downsampling said frame and a reference frame associated with said frame by one or more times to form a plurality of versions of said frame at a plurality of resolutions, wherein a lowest resolution version of said frame has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of said frame;

for the lowest resolution version of said frame, estimating motion vectors of the blocks of said lowest resolution version frame using an initial motion vector estimation processing of the associated lowest resolution version reference frame;

for each successively higher resolution version of said frame, estimating motion vectors of the blocks of said successively higher resolution version frame using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version frame according to the pyramid structure; and

producing estimated local motion vectors for the blocks of the original resolution version of said frame;

processing the estimated local motion vectors of said frame for estimation of a global motion transform representing jittery video motion, including selecting a set of motion vectors to use for estimation of the global motion transform out of the produced estimated local motion vectors of said frame, wherein said selecting the set of motion vectors comprises one or more of;

excluding motion vectors on picture boundaries,

excluding motion vectors having a high motion compensation residual, and

excluding motion vectors for blocks with low image content variation; and

performing image warping on said frame based on the global motion transform estimated for said frame to compensate for the jittery video motion.

9. The method of claim 8 wherein said processing the estimated local motion vectors of said frame to estimate the global motion transform processes the local motion vectors of said frame using a random sample consensus for removal of outlier motion vectors.

10. The method of claim 8 wherein said global motion transform is based on a similarity model of video motion.

11. The method of claim 8 wherein said processing the estimated local motion vectors of said frame comprises:

comparing the global motion transform estimated from processing the local motion vectors of said frame to lower and upper limits;

if the estimated global motion transform exceeds the upper limit, resetting the global motion transform;

if the estimated global motion transform exceeds the lower limit but not the upper limit, limiting the global motion transform to the lower limit; and

otherwise, using the estimated global motion transform to be representative of jittery video motion.

12. The method of claim 11 further comprising:

as a first pass over the frames in the scene of the video segment, processing the frames to estimate global motion transforms for the frames in the scene of the video segment;

determining probability distributions of parameters of the global motion transforms for the frames in the scene of the video segment;

determining said lower limit and said upper limit based on the probability distributions; and

applying said lower limit and upper limit to estimates of the global motion transform in a second pass processing the frames in the scene of the video segment.

13. The method of claim 8 wherein said processing the estimated local motion vectors of said frame comprises:

processing the motion vectors of said frame using random sample consensus and least mean square error to estimate parameters for a global motion transform based on a similarity motion model;

comparing the estimated parameters of the global motion transform to lower and upper limits on said parameters;

if any of the estimated parameters of the global motion transform exceed their upper limit, resetting the global motion transform;

if any of the estimated parameters of the global motion transform exceed the lower limit but not the upper limit, limiting the estimated parameters of the global motion transform to their lower limit; and

otherwise, using the estimated global motion transform to be representative of jittery video motion.

14. The method of claim 8 wherein said processing the estimated local motion vectors of said frame comprises applying temporal smoothing to the estimated global motion transform using a Gaussian filter.

15. The method of claim 8 wherein said processing the estimated local motion vectors of said frame comprises applying temporal smoothing to the estimated global motion transform using a constrained global optimization over the frames in the scene of the video segment.

16. The method of claim 8 wherein a library of an operating system includes a set of library functions to control the digital video stabilization, and wherein the library provides a programmatic interface for applications to make use of the digital video stabilization, the library including at least one call to a DSP interface to perform one or more of the local motion vector estimation and the image warping.

17. The method of claim 8 wherein an application executable controls the digital video stabilization, the application executable including at least one call to a DSP interface to perform one or more of the local motion vector estimation and the image warping.

18. The method of claim 8 wherein a library of an operating system controls the digital video stabilization, the library including at least one call to a DSP interface to perform one or more of the local motion vector estimation and the image warping.

19. A digital video processing device for processing video to apply digital video stabilization to the video, the digital video processing device comprising:

a memory storing a generic platform video stabilization library program;

at least one digital signal processing component;

a processing unit operating to execute the video stabilization library program from the memory, wherein said execution of the video stabilization library program comprises;

evaluating the at least one digital signal processing component of the digital video processing device;

determining which of the at least one digital signal processing component to utilize for at least one part of digital video stabilization according to a hierarchical structure for ordering digital signal processing components by criteria comprising at least performance of the digital video stabilization; and

performing digital video stabilization on the video in part by processing a frame of the video to estimate local motion vectors of the frame, processing the estimated local motion vectors for estimation of a global motion transform representing jittery video motion, and warping the frame based on the global motion transform to compensate for the jittery video motion, wherein said performing the digital video stabilization executes programming functions in the generic platform video stabilization library program to utilize the determined digital signal processing component for the at least one part of the digital video stabilization.

20. The digital video processing device of claim 19 wherein said processing the frame to estimate local motion vectors of the frame comprises causing the determined at least one digital signal processing component to perform acts of:

downsampling the frame and a reference frame associated with the frame by one or more times to form a plurality of versions of the frame at a plurality of resolutions, wherein a lowest resolution version of the frame has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of the frame;

for the lowest resolution version of the frame, estimating motion vectors of the blocks of said lowest resolution version frame using an initial motion vector estimation processing of the associated lowest resolution version reference frame;

for each successively higher resolution version of the frame, estimating motion vectors of the blocks of said successively higher resolution version frame using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version frame according to the pyramid structure; and

producing the estimated local motion vectors for the blocks of the original resolution version of the frame.

21. The digital video processing device of claim 19 wherein said processing the estimated local motion vectors for estimation of the global motion transform representing jittery video motion comprises performing acts of:

selecting a set of motion vectors to use for estimation of the global motion transform out of the estimated local motion vectors of the frame, wherein said selecting the set of motion vectors comprises;

excluding motion vectors on picture boundaries;

excluding motion vectors having a high motion compensation residual; and

excluding motion vectors for blocks with low image content variation.

22. The digital video processing device of claim 19 wherein said processing the estimated local motion vectors for estimation of the global motion transform representing jittery video motion comprises processing the estimated local motion vectors of the frame using random sample consensus and least mean square error to estimate parameters for the global motion transform based on a similarity motion model.

23. The digital video processing device of claim 19 wherein said processing the estimated local motion vectors for estimation of the global motion transform representing jittery video motion comprises:

comparing estimated parameters of the global motion transform to lower and upper limits;

if any of the estimated parameters of the global motion transform exceed their upper limit, resetting the global motion transform;

if any of the estimated parameters of the global motion transform exceed the lower limit but not the upper limit, limiting the estimated parameters of the global motion transform to their lower limit; and

otherwise, using the estimated global motion transform to be representative of jittery video motion.

24. The digital video processing device of claim 23 wherein said upper limit and said lower limit are set in a first pass of the digital video stabilization, and wherein said upper limit and said lower limit are compared to the estimated parameters of the global motion transform in a second pass of the digital video stabilization.

25. The digital video processing device of claim 19 wherein said processing the estimated local motion vectors for estimation of the global motion transform representing jittery video motion further comprises applying temporal smoothing to the estimated global motion transform using a Gaussian filter.