Generic platform video image stabilization

• US 9,578,240 B2
• Filed: 10/20/2014
• Issued: 02/21/2017
• Est. Priority Date: 02/11/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wu, Yongjun, Borisov, Nikola, Zhao, Weidong, Sadhwani, Shyam, Thumpudi, Naveen
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US14/518,840
• Publication Number:
  US 20150036010A1
• Time in Patent Office: 855 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Video image stabilization provides better performance on a generic platform for computing devices by evaluating available multimedia digital signal processing components, and selecting the available components to utilize according to a hierarchy structure for video stabilization performance for processing parts of the video stabilization. The video stabilization has improved motion vector estimation that employs refinement motion vector searching according to a pyramid block structure relationship starting from a downsampled resolution version of the video frames. The video stabilization also improves global motion transform estimation by performing a random sample consensus approach for processing the local motion vectors, and selection criteria for motion vector reliability. The video stabilization achieves the removal of hand shakiness smoothly by real-time one-pass or off-line two-pass temporal smoothing with error detection and correction.

Claims

1. A handheld computing device comprising:

a central processing unit (CPU);

one or more memory units;

a camera;

a graphics processing unit (GPU) including a shader unit; and

the one or more memory units storing computer-executable instructions for causing the handheld computing device, when programmed thereby, to perform real-time digital video stabilization that includes;

for each of multiple frames of a video sequence captured by the camera;

estimating, using the CPU, a motion transform that represents jittery motion associated with shakiness of the handheld computing device; and

automatically warping, using the shader unit of the GPU, at least a portion of the frame based on the motion transform to compensate for the jittery motion; and

initiating uploading of the stabilized video, from the handheld computing device to a video sharing site or social networking site, over a connection to a network.

2. The handheld computing device of claim 1, wherein the estimating the motion transform includes calculating a warping matrix that compensates for one or more of rotation, zooming and translation of the handheld computing device during video capture.

3. The handheld computing device of claim 1, wherein the real-time digital video stabilization further includes:

forming stabilized video around a display port, including, for each of at least some of the multiple frames of the video sequence, cropping boundaries of the frame, wherein the display port depends on the warping for the multiple frames, respectively.

4. The handheld computing device of claim 3, further comprising a display, wherein the storage further stores computer-executable instructions for causing the handheld computing device, when programmed thereby, to perform:

outputting the stabilized video to the display for playback.

5. The handheld computing device of claim 1, wherein the shader unit of the GPU uses a vertex shader.

6. The handheld computing device of claim 1, wherein the handheld computing device is a mobile phone.

7. The handheld computing device of claim 1, wherein the real-time digital video stabilization uses single-pass processing consistent with time constraints for playback or streaming of the stabilized video during the real-time digital video stabilization.

8. The handheld computing device of claim 1, wherein the real-time digital video stabilization further includes:

applying temporal smoothing during the estimating the motion transform.

9. A computer-readable memory or storage device storing computer-executable instructions for causing a computing device that includes a graphics processing unit (GPU), when programmed thereby, to perform video sharing with real-time digital video stabilization, the video sharing with real-time digital video stabilization including:

for each of multiple frames of a video sequence captured by a camera of the computing device;

estimating a motion transform that represents jittery motion associated with shakiness of the computing device; and

automatically warping, using a vertex shader on the GPU, at least a portion of the frame based on the motion transform to compensate for the jittery motion;

forming stabilized video around a display port, including, for each of at least some of the multiple frames of the video sequence, cropping boundaries of the frame, wherein the display port depends on the warping for the multiple frames, respectively; and

initiating uploading of the stabilized video to a video sharing site or social networking site over a connection to a network.

10. The computer-readable memory or storage device of claim 9, wherein the estimating the motion transform includes calculating a warping matrix that compensates for one or more of rotation, zooming and translation of the computing device during video capture.

11. The computer-readable memory or storage device of claim 9, wherein the real-time digital video stabilization uses single-pass processing consistent with time constraints for playback or streaming of stabilized video during the real-time digital video stabilization.

12. The computer-readable memory or storage device of claim 9, wherein the video sharing with real-time digital video stabilization further includes:

applying temporal smoothing during the estimating the motion transform.

13. The computer-readable memory or storage device of claim 12, wherein the temporal smoothing includes selecting between different filter sizes and/or filter types having different delays.

14. The computer-readable memory or storage device of claim 9, wherein the video sharing with real-time digital video stabilization further includes:

comparing the motion transform to one or more thresholds, wherein performance of the warping depends on the motion transform satisfying the one or more thresholds.

15. The computer-readable memory or storage device of claim 9, wherein a library implements the real-time digital video stabilization and provides a programmatic interface for an application program to use of the real-time digital video stabilization.

16. The computer-readable memory or storage device of claim 15, wherein the real-time digital video stabilization further includes:

evaluating digital signal processing (DSP) components of the computing device; and

choosing, from among the DSP components of the computing device, which of the DSP components to use for different stages of the real-time digital video stabilization.

17. In a computing device that includes a graphics processing unit (GPU), a method of video sharing with real-time digital video stabilization, the method comprising:

for each of multiple frames of a video sequence captured by a camera of the computing device;

estimating a motion transform that represents jittery motion associated with shakiness of the computing device; and

automatically warping, using a vertex shader on the GPU, at least a portion of the frame based on the motion transform to compensate for the jittery motion;

forming stabilized video around a display port, including, for each of at least some of the multiple frames of the video sequence, cropping boundaries of the frame, wherein the display port depends on the warping for the multiple frames, respectively; and

initiating uploading of the stabilized video, from the computing device to a video sharing site or social networking site over a connection to a network.

18. The method of claim 17, wherein the estimating the motion transform includes calculating a warping matrix that compensates for one or more of rotation, zooming and translation of the computing device during video capture.

19. The method of claim 17, wherein the real-time digital video stabilization uses single-pass processing consistent with time constraints for playback or streaming of the stabilized video during the real-time digital video stabilization.

20. The method of claim 17, further comprising, during the estimating the motion transform, applying temporal smoothing.