CASCADED CAMERA MOTION ESTIMATION, ROLLING SHUTTER DETECTION, AND CAMERA SHAKE DETECTION FOR VIDEO STABILIZATION

• US 20170195575A1
• Filed: 03/20/2017
• Published: 07/06/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Grundmann, Matthias, Kwatra, Vivek, Essa, Irfan
• Granted Patent: US 9,888,180 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

An easy-to-use online video stabilization system and methods for its use are described. Videos are stabilized after capture, and therefore the stabilization works on all forms of video footage including both legacy video and freshly captured video. In one implementation, the video stabilization system is fully automatic, requiring no input or parameter settings by the user other than the video itself. The video stabilization system uses a cascaded motion model to choose the correction that is applied to different frames of a video. In various implementations, the video stabilization system is capable of detecting and correcting high frequency jitter artifacts, low frequency shake artifacts, rolling shutter artifacts, significant foreground motion, poor lighting, scene cuts, and both long and short videos.

Claims

1. A method, comprising:

accessing a video;

estimating, for a plurality of frames of the video, values of a plurality of degrees of freedom (DOF) of a similarity motion model, each degree of freedom representing a different camera motion of an original camera used to capture the video, the values of the DOFs representing magnitudes of the different camera motions;

generating a spectrogram for each of the DOFs, each spectrogram based on the values of the DOFs over a time window comprising a plurality of adjacent frames of the video;

generating a plurality of shake features based on the spectrograms;

classifying the video based on the shake features; and

stabilizing the video based on the classification.

2. The method of claim 1, wherein the similarity motion model comprises a lateral translation DOF, a longitudinal translation DOF, a scale change DOF, and a rotation change DOF.

3. The method of claim 1, wherein the shake features are invariant with respect to a length of the video.

4. The method of claim 1, wherein the time windows of the spectrograms have overlapping coverage of at least a portion of the frames of a previous and a subsequent spectrogram.

5. The method of claim 1, wherein one of the spectrograms comprises an energy representation of contributions of a plurality of energy ranges to the values of one of the DOFs across the frames of one of the time windows, the spectrogram further comprising a contribution value for each of the energy ranges.

6. The method of claim 5, wherein the shake features comprise a mean contribution of one of the energy ranges across the windows of the video.

7. The method of claim 5, wherein generating a plurality of shake features based on the spectrograms comprises:

generating a domain histogram for one of the energy ranges, the domain histogram comprising a plurality of domain bins,

each domain bin indicating a percentile range of contributions values bounded by a maximum contribution value of the spectrogram,

a height of each domain bin indicating a number windows of the energy range having a contribution value within the percentile range; and

generating a shake feature based on the height of each domain bin.

8. A non-transitory computer readable storage medium storing executable computer program instructions for caching streaming media, the computer program instructions comprising instructions that when executed cause a computer processor to:

access a video;

estimate, for a plurality of frames of the video, values of a plurality of degrees of freedom (DOF) of a similarity motion model, each degree of freedom representing a different camera motion of an original camera used to capture the video, the values of the DOFs representing magnitudes of the different camera motions;

generate a spectrogram for each of the DOFs, each spectrogram based on the values of the DOFs over a time window comprising a plurality of adjacent frames of the video;

generate a plurality of shake features based on the spectrograms;

classify the video based on the shake features; and

stabilize the video based on the classification.

9. The computer readable medium of claim 8, wherein the similarity motion model comprises a lateral translation DOF, a longitudinal translation DOF, a scale change DOF, and a rotation change DOF.

10. The computer readable medium of claim 8, wherein the shake features are invariant with respect to a length of the video.

11. The computer readable medium of claim 8, wherein the time windows of the spectrograms have overlapping coverage of at least a portion of the frames of a previous and a subsequent spectrogram.

12. The computer readable medium of claim 8, wherein one of the spectrograms comprises an energy representation of contributions of a plurality of energy ranges to the values of one of the DOFs across the frames of one of the time windows, the spectrogram further comprising a contribution value for each of the energy ranges.

13. The computer readable medium of claim 12, wherein the shake features comprise a mean contribution of one of the energy ranges across the windows of the video.

14. The computer readable medium of claim 12, wherein the computer program instructions for generating a plurality of shake features based on the spectrograms comprise instructions that when executed cause the computer processor to:

generate a domain histogram for one of the energy ranges, the domain histogram comprising a plurality of domain bins,

each domain bin indicating a percentile range of contributions values bounded by a maximum contribution value of the spectrogram,

a height of each domain bin indicating a number windows of the energy range having a contribution value within the percentile range; and

generate a shake feature based on the height of each domain bin.

15. A computer system for caching streaming media, the system comprising:

a non-transitory computer-readable storage medium storing executable computer program instructions, the computer program instructions comprising instructions that when executed cause a computer processor to perform steps, comprising;

accessing a video;

estimating, for a plurality of frames of the video, values of a plurality of degrees of freedom (DOF) of a similarity motion model, each degree of freedom representing a different camera motion of an original camera used to capture the video, the values of the DOFs representing magnitudes of the different camera motions;

generating a spectrogram for each of the DOFs, each spectrogram based on the values of the DOFs over a time window comprising a plurality of adjacent frames of the video;

generating a plurality of shake features based on the spectrograms;

classifying the video based on the shake features; and

stabilizing the video based on the classification.

16. The system of claim 15, wherein the similarity motion model comprises a lateral translation DOF, a longitudinal translation DOF, a scale change DOF, and a rotation change DOF.

17. The system of claim 15, wherein the shake features are invariant with respect to a length of the video.

18. The system of claim 15, wherein the time windows of the spectrograms have overlapping coverage of at least a portion of the frames of a previous and a subsequent spectrogram.

19. The system of claim 15, wherein one of the spectrograms comprises an energy representation of contributions of a plurality of energy ranges to the values of one of the DOFs across the frames of one of the time windows, the spectrogram further comprising a contribution value for each of the energy ranges.

20. The system of claim 19, wherein generating a plurality of shake features based on the spectrograms comprises:

generating a domain histogram for one of the energy ranges, the domain histogram comprising a plurality of domain bins,

each domain bin indicating a percentile range of contributions values bounded by a maximum contribution value of the spectrogram,

a height of each domain bin indicating a number windows of the energy range having a contribution value within the percentile range; and

generating a shake feature based on the height of each domain bin.