SYSTEMS AND METHODS FOR DIGITAL VIDEO STABILIZATION VIA CONSTRAINT-BASED ROTATION SMOOTHING

• US 20140160309A1
• Filed: 12/09/2013
• Published: 06/12/2014
• Est. Priority Date: 12/11/2012
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Karpenko, Alexandre
• Granted Patent: US 9,071,756 B2
• Time in Patent Office: Days
• US Class Current: 348/208.6
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/91 Television signal processing therefor

Abstract

Systems and methods for digital video stabilization via constraint-based rotation smoothing are provided. Digital video data including a set of image frames having associated time stamps and a set of camera orientation data having associated time stamps may be provided. A smoothed set of camera orientation data may be generated by minimizing a rate of rotation between successive image frames while minimizing an amount of empty regions in a resulting set of smoothed image frames reoriented based on the smoothed set of camera orientation data.

Claims

1. A computer implemented method comprising:

providing, by a computer system, digital video data comprising a set of image frames having associated time stamps and a set of camera orientation data having associated time stamps; and

generating, by the computer system, a smoothed set of camera orientation data by minimizing a rate of rotation between successive image frames while minimizing an amount of empty regions in a resulting set of smoothed image frames reoriented based on the smoothed set of camera orientation data.

2. The method of claim 1, wherein the amount of empty regions in the resulting set of smoothed image frames is minimized to zero.

3. The method of claim 1, wherein the amount of empty regions in the resulting set of smoothed image frames is minimized below a threshold value.

4. The method of claim 1, further comprising warping the set of image frames based on the associated time stamps for the set of image frames and the smoothed set of camera orientation data to form a set of corrected image frames.

5. The method of claim 4, wherein the warping the set of image frames based on the associated time stamps for the set of image frames and the smoothed set of camera orientation data to form a set of corrected image frames comprises:

dividing an individual image frame into a plurality of subsections, each subsection having an associated time stamp and camera orientation; and

realigning each subsection based on the associated time stamp and camera orientation to form an individual corrected image frame.

6. The method of claim 4, further comprising displaying the set of corrected image frames as a video.

7. The method of claim 6, wherein the amount of empty regions in the resulting set of smoothed image frames is minimized below a threshold value.

8. The method of claim 7, further comprising inpainting the amount of empty regions below the threshold value.

9. The method of claim 1, wherein the set of camera orientation data having associated time stamps is provided from a gyroscope of a handheld device.

10. The method of claim 1, wherein the set of image frames is provided from a camera of a handheld device.

11. The method of claim 1, wherein the set of camera orientation data having associated time stamps is provided from a gyroscope of a handheld device comprising a mobile phone and a digital camera, and wherein the set of image frames is provided from the digital camera.

12. The method of claim 1, wherein the generating the smoothed set of camera orientation data comprises an iterative optimization based on gradient descent.

13. The method of claim 1, wherein the generating the smoothed set of camera orientation data comprises filtering based on a Gaussian filter.

14. The method of claim 1, wherein the generating the smoothed set of camera orientation data comprises filtering based on a temporal derivative.

15. The method of claim 1, wherein the set of camera orientation data includes rotations without any translations.

16. The method of claim 1, wherein the set of camera orientation data includes vectors having both rotations and translations.

17. The method of claim 1, wherein the generating the smoothed set of camera orientation data is performed by a social networking system.

18. The method of claim 17, wherein the set of image frames having associated time stamps and the set of camera orientation data having associated time stamps are uploaded to the social networking system by a user of the social networking system.

19. A system comprising:

at least one processor, and

a memory storing instructions configured to instruct the at least one processor to perform;

providing digital video data comprising a set of image frames having associated time stamps and a set of camera orientation data having associated time stamps; and

generating a smoothed set of camera orientation data by minimizing a rate of rotation between successive image frames while minimizing an amount of empty regions in a resulting set of smoothed image frames reoriented based on the smoothed set of camera orientation data.

20. A computer storage medium storing computer-executable instructions that, when executed, cause a computer system to perform computer-implemented method comprising:

providing digital video data comprising a set of image frames having associated time stamps and a set of camera orientation data having associated time stamps; and

generating a smoothed set of camera orientation data by minimizing a rate of rotation between successive image frames while minimizing an amount of empty regions in a resulting set of smoothed image frames reoriented based on the smoothed set of camera orientation data.