Generic platform video image stabilization

• US 10,257,421 B2
• Filed: 12/14/2016
• Issued: 04/09/2019
• Est. Priority Date: 02/11/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wu, Yongjun, Borisov, Nikola, Zhao, Weidong, Sadhwani, Shyam, Thumpudi, Naveen
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Quiett, Carramah J
• Application Number:
  US15/378,942
• Publication Number:
  US 20170094172A1
• Time in Patent Office: 846 Days
• Field of Search: None
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

Video image stabilization provides better performance on a generic platform for computing devices by evaluating available multimedia digital signal processing components, and selecting the available components to utilize according to a hierarchy structure for video stabilization performance for processing parts of the video stabilization. The video stabilization has improved motion vector estimation that employs refinement motion vector searching according to a pyramid block structure relationship starting from a downsampled resolution version of the video frames. The video stabilization also improves global motion transform estimation by performing a random sample consensus approach for processing the local motion vectors, and selection criteria for motion vector reliability. The video stabilization achieves the removal of hand shakiness smoothly by real-time one-pass or off-line two-pass temporal smoothing with error detection and correction.

Claims

1. A method of real-time sharing of stabilized digital video for multiple frames of a captured video sequence, comprising:

warping at least a portion of a frame, among the multiple frames of the captured video sequence, based on a motion transform that represents jittery motion of a video capture device to compensate for the jittery motion of the video capture device; and

initiating uploading of stabilized video from the video capture device to a server device associated with a service for video sharing or social networking;

wherein the motion transform has been estimated prior to warping.

2. The method of claim 1, wherein said warping comprises use of a vertex shader of a processing unit.

5. The method of claim 1, further comprising cropping two or more of the multiple frames of the captured video sequence.

6. The method of claim 5, wherein the cropping comprises adaptive cropping depending on characteristics of the two or more frames.

7. The method of claim 1, wherein the warping further comprises moving a display port within the content of one of the multiple frames of a captured video sequence.

8. The method of claim 7, wherein the stabilized video reflects the moved display port.

9. The method of claim 1, wherein said warping and initiating uploading is carried out, at least in part, by a mobile device.

10. The method of claim 1, wherein the estimation of the motion transform occurs in real time during playback.

11. The method of claim 1, wherein the estimation of the motion transform relies on statistical information from a first past estimation of the motion of the video capture device.

12. The method of claim 1, wherein the estimation of the motion transform comprises adaptive filtering.

13. The method of claim 1, wherein the warping comprises warping in real time during playback.

3. A device comprising:

one or more processing units;

one or more memory units;

a camera; and

the one or more memory units storing computer-executable instructions for causing the device, when programmed thereby, to perform real-time digital video stabilization that includes;

for each of multiple frames of a video sequence captured by the camera;

estimating, using at least one of the processing units, a motion transform that represents jittery motion of the device; and

warping at least a portion of the frame based on the motion transform to compensate for the jittery motion; and

initiating transfer of stabilized video to a social media storage location in network communication with the device.

4. The device of claim 3, wherein at least one of said processing units includes a shader unit and wherein said warping comprises using said shader unit.

14. The device of claim 3, wherein the real-time digital video stabilization further includes cropping two or more of the multiple frames of the captured video sequence.

15. The device of claim 14, wherein the cropping comprises adaptive cropping depending on characteristics of the two or more frames.

16. The device of claim 3, wherein the warping comprises moving a display port within the content of one of the multiple frames of a captured video sequence.

17. The method of claim 16, wherein the stabilized video reflects the moved display port.

18. Computer-readable hardware based storage device storing computer-executable instructions that, when executed, cause a computing device to perform video sharing with real-time digital video stabilization comprising:

capturing multiple frames of a video sequence into a captured video sequence;

warping at least a portion of a frame, among the multiple frames of the captured video sequence, to compensate for the jittery motion of a video capture device; and

initiating uploading of stabilized video from the video capture device to a social networking or video sharing website;

wherein the warping is based on an estimated motion transform that represents jittery motion.

19. The computer-readable hardware based storage device of claim 18, wherein the video sharing with real-time digital video stabilization further comprises adaptively cropping two or more of the multiple frames of the captured video sequence.

20. The computer-readable hardware based storage device of claim 18, wherein the video sharing with real-time digital video stabilization further comprises moving a display port within the content of one of the multiple frames of a captured video sequence.