GENERIC PLATFORM VIDEO IMAGE STABILIZATION

• US 20110193978A1
• Filed: 02/11/2010
• Published: 08/11/2011
• Est. Priority Date: 02/11/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhao, Weidong, Borisov, Nikola, Sadhwani, Shyam, Thumpudi, Naveen, Wu, Yongjun
• Granted Patent: US 8,896,715 B2
• Time in Patent Office: Days
• US Class Current: 348/208.6
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Video image stabilization provides better performance on a generic platform for computing devices by evaluating available multimedia digital signal processing components, and selecting the available components to utilize according to a hierarchy structure for video stabilization performance for processing parts of the video stabilization. The video stabilization has improved motion vector estimation that employs refinement motion vector searching according to a pyramid block structure relationship starting from a downsampled resolution version of the video frames. The video stabilization also improves global motion transform estimation by performing a random sample consensus approach for processing the local motion vectors, and selection criteria for motion vector reliability. The video stabilization achieves the removal of hand shakiness smoothly by real-time one-pass or off-line two-pass temporal smoothing with error detection and correction.

Claims

1. A method of digital video stabilization, the method comprising performing steps on a computing device of:

evaluating the computing device to determine availability of any of various multimedia digital signal processing components on the computing device;

determining from the results of said evaluating which available multimedia DSP component to utilize for processing at least one part of processing a digital video stabilization according to a hierarchical structure for ordering multimedia DSP components by criteria comprising at least performance of the digital video stabilization, the digital video stabilization comprising at least processing of a local motion vector estimation, processing motion vectors for estimation of a global motion transform representing jittery video motion, and processing an image warping based on the global motion transform to compensate the jittery video motion; and

processing digital video stabilization of a video segment on the computing device utilizing the determined available multimedia DSP component for said at least one part of processing the digital video stabilization.

2. The method of claim 1 wherein the hierarchical structure comprises in order multimedia DSP ASIC units, GPU shader units, multicore CPU and single core CPU.

3. The method of claim 1 further comprising when said evaluating determines a GPU shader is available on the computing device, then processing the digital video stabilization to utilize the GPU shader for processing of the local motion vector estimation.

4. The method of claim 3 further comprising when said evaluating determines the GPU shader is available on the computing device, then processing the digital video stabilization to further utilize the GPU shader for processing the image warping.

5. A method of digital video stabilization, the method comprising performing steps on a computing device of:

processing of a local motion vector estimation for each of a plurality of frames of each scene of a video segment, wherein the processing of the local motion vector estimation comprises for each said frame;

downsampling said each frame and a reference frame associated with said each frame by one or more times to form a plurality of versions of said each frame at a plurality of resolutions, wherein a lowest resolution version of said each frame has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of said each frame;

for the lowest resolution version of said each frame, estimating motion vectors of the blocks of said lowest resolution version frame using a full search motion vector estimation processing of the associated lowest resolution version reference frame;

for each successively higher resolution version of said each frame, estimating motion vectors of the blocks of said successively higher resolution version frame using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version frame according to the pyramid structure; and

producing estimated local motion vectors for the blocks of the original resolution version of said each frame;

processing the estimated local motion vectors of said each frame for estimation of a global motion transform representing jittery video motion; and

processing an image warping on said each based on the global motion transform to compensate the jittery video motion.

6. The method of claim 5 wherein said processing the estimated local motion vectors of said each frame comprises:

selecting a set of motion vectors to use for estimation of the global motion transform out of the produced estimated local motion vectors of said each frame, wherein said selecting the set of motion vectors comprises;

excluding motion vectors on picture boundaries;

excluding motion vectors having a high motion compensation residual; and

excluding motion vectors for blocks with low image content variation.

7. The method of claim 5 wherein said processing the estimated local motion vectors of said each frame to estimate the global motion transform processes the local motion vectors of said each frame using a random sample consensus for removal of outlier motion vectors.

8. The method of claim 5 wherein said global motion transform is based on a similarity model of video motion.

9. The method of claim 5 wherein said processing the estimated local motion vectors of said each frame comprises:

comparing the global motion transform estimated from processing the local motion vectors of said each frame to lower and upper limits;

if the estimated global motion transform exceeds the upper limit, resetting the global motion transform;

if the estimated global motion transform exceeds the lower limit but not the upper limit, limiting the global motion transform to the lower limit; and

otherwise, using the estimated global motion transform to be representative of jittery video motion.

10. The method of claim 9 further comprising:

as a first pass over the frames in a scene of a video segment, processing the frames to estimate global motion transforms for the frames;

determining probability distributions of parameters of the global motion transforms for the frames;

determining said lower limit and said upper limit based on the probability distributions; and

applying said lower limit and upper limit to estimates of the global motion transform in a second pass processing the frames of the video segment.

11. The method of claim 5 wherein said processing the estimated local motion vectors of said each frame comprises:

processing the motion vectors of said each frame using random sample consensus and least mean square error to estimate parameters for a global motion transform based on a similarity motion model; and

comparing the estimated parameters of the global motion transform to lower and upper limits on said parameters;

if any of the estimated parameters of the global motion transform exceed their upper limit, resetting the global motion transform;

if any of the estimated parameters of the global motion transform exceed the lower limit but not the upper limit, limiting the estimated parameters of the global motion transform to their lower limit; and

otherwise, using the estimated global motion transform to be representative of jittery video motion.

12. The method of claim 5 wherein said processing the estimated local motion vectors of said each frame comprises applying temporal smoothing to the estimated global motion transform using a Gaussian filter.

13. The method of claim 5 wherein said processing the estimated local motion vectors of said each frame comprises applying temporal smoothing to the estimated global motion transform using a constrained global optimization over the frames in a scene of a video segment.

14. A digital video processing device for processing a video to apply video image stabilization to the video, the digital video processing device comprising:

a memory storing a generic platform video stabilization library program;

at least one digital signal processing component;

a processing unit operating to execute the video stabilization library program from the memory, wherein said execution of the video stabilization library program comprises;

evaluating the at least one digital signal processing component of the digital video processing device;

determining which of the at least one digital signal processing component to utilize for at least one part of processing a video for video image stabilization according to a hierarchical structure for ordering digital signal processing components by criteria comprising at least performance of the digital video stabilization;

processing the video for video image stabilization in part by processing frames of the video to estimate local motion vectors of the frames, processing the motion vectors for estimation of a global motion transform representing jittery video motion of the frames, and processing an image warping of the frames based on the global motion transform to compensate the jittery video motion, wherein said processing the video executes programming functions in the generic platform video stabilization library program to utilize the determined digital signal processing component for the at least one part of processing the video for image stabilization.

15. The digital video processing device of claim 14 wherein said part of processing frames of the video to estimate local motion vectors of the frames by the digital signal components comprises causing the determined at least one digital signal processing component for each of a plurality of frames in the scene of the video segment, to perform acts of:

downsampling said each frame and a reference frame associated with said each frame by one or more times to form a plurality of versions of said each frame at a plurality of resolutions, wherein a lowest resolution version of said each frame has a plurality of blocks associated as a pyramid structure to a successively larger number of blocks at corresponding locations in each higher resolution version of said each frame;

for the lowest resolution version of said each frame, estimating motion vectors of the blocks of said lowest resolution version frame using a full search motion vector estimation processing of the associated lowest resolution version reference frame;

for each successively higher resolution version of said each frame, estimating motion vectors of the blocks of said successively higher resolution version frame using a refinement search starting from the motion vector estimated for the associated block of the preceding lower resolution version frame according to the pyramid structure; and

producing estimated local motion vectors for the blocks of the original resolution version of said each frame.

16. The digital video processing device of claim 14 wherein said part of processing motion vectors for estimation of a global motion transform representing jittery video motion of the frames comprises performing acts of:

selecting a set of motion vectors to use for estimation of the global motion transform out of the produced estimated local motion vectors of said each frame, wherein said selecting the set of motion vectors comprises;

excluding motion vectors on picture boundaries;

excluding motion vectors having a high motion compensation residual; and

excluding motion vectors for blocks with low image content variation.

17. The digital video processing device of claim 14 wherein said part of processing motion vectors for estimation of a global motion transform representing jittery video motion of the frames comprises processing the motion vectors of each frame using random sample consensus and least mean square error to estimate parameters for a global motion transform based on a similarity motion model.

18. The digital video processing device of claim 17 wherein said part of processing motion vectors for estimation of a global motion transform representing jittery video motion of the frames further comprises:

comparing the estimated parameters of the global motion transform to lower and upper limits on said parameters;

if any of the estimated parameters of the global motion transform exceed their upper limit, resetting the global motion transform;

if any of the estimated parameters of the global motion transform exceed the lower limit but not the upper limit, limiting the estimated parameters of the global motion transform to their lower limit; and

otherwise, using the estimated global motion transform to be representative of jittery video motion.

19. The digital video processing device of claim 18 wherein said part of processing motion vectors for estimation of a global motion transform representing jittery video motion of the frames further comprises:

performing two passes of said processing motion vectors for estimation of a global motion transform for frames in the scene of the video segment;

using the global motion transforms estimated in the first pass for the frames, determining probability distributions of parameters of the global motion transforms;

determining said lower limit and said upper limit based on the probability distributions; and

applying said lower limit and upper limit determined based on probability distributions from the first pass in said second pass of processing the motion vectors for estimation of the global motion transform.

20. The digital video processing device of claim 17 wherein said part of processing motion vectors for estimation of a global motion transform representing jittery video motion of the frames further comprises applying temporal smoothing to the estimated global motion transform using a Gaussian filter.