Cascaded camera motion estimation, rolling shutter detection, and camera shake detection for video stabilization

• US 9,888,180 B2
• Filed: 03/20/2017
• Issued: 02/06/2018
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Grundmann, Matthias, Kwatra, Vivek, Essa, Irfan
• Primary Examiner(s): Velez, Roberto
• Assistant Examiner(s): Segura, Cynthia
• Application Number:
  US15/463,127
• Publication Number:
  US 20170195575A1
• Time in Patent Office: 323 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

An easy-to-use online video stabilization system and methods for its use are described. Videos are stabilized after capture, and therefore the stabilization works on all forms of video footage including both legacy video and freshly captured video. In one implementation, the video stabilization system is fully automatic, requiring no input or parameter settings by the user other than the video itself. The video stabilization system uses a cascaded motion model to choose the correction that is applied to different frames of a video. In various implementations, the video stabilization system is capable of detecting and correcting high frequency jitter artifacts, low frequency shake artifacts, rolling shutter artifacts, significant foreground motion, poor lighting, scene cuts, and both long and short videos.

Claims

1. A method, comprising:

accessing a video;

estimating, for a plurality of frames of the video, values of a plurality of degrees of freedom (DOF) of a similarity motion model, a value of each of the DOFs of the similarity motion model representing a magnitude of a different camera motion of an original camera used to capture the video, the DOFs of the similarity motion model comprising a scale change DOF and a rotation change DOF;

generating spectrograms for the DOFs based on the values of the DOFs over time windows comprising a plurality of adjacent frames of the video;

generating a plurality of shake features based on the spectrograms;

classifying the video based on the shake features; and

stabilizing the video based on the classification.

2. The method of claim 1, wherein the DOFs of the similarity motion model further comprise a lateral translation DOF and a longitudinal translation DOF.

3. The method of claim 1, wherein the shake features are invariant with respect to a length of the video.

4. The method of claim 1, wherein the time windows of the spectrograms have overlapping coverage of at least a portion of the adjacent frames of a previous and a subsequent spectrogram among the spectrograms.

5. The method of claim 1, wherein a spectrogram among the spectrograms comprises an energy representation of contributions of a plurality of energy ranges to the values of one of the DOFs across the adjacent frames of one of the time windows, the spectrogram further comprising a contribution value for each of the energy ranges.

6. The method of claim 5, wherein the shake features comprise a mean contribution of one of the energy ranges across the time windows of the video.

7. The method of claim 5, wherein generating the shake features based on the spectrograms comprises:

generating a domain histogram for one of the energy ranges, the domain histogram comprising a plurality of domain bins,

each domain bin among the domain bins indicating a percentile range of contribution values bounded by a maximum contribution value of the spectrogram,

a height of each domain bin among the domain bins indicating a number of the time windows of the energy range having a contribution value within the percentile range; and

generating a shake feature among the shake features for the spectrogram based on the height of each domain bin among the domain bins.

8. A non-transitory computer readable storage medium storing executable computer program instructions for caching streaming media, the computer program instructions comprising instructions that when executed cause a computer processor to:

access a video;

estimate, for a plurality of frames of the video, values of a plurality of degrees of freedom (DOF) of a similarity motion model, a value of each of the DOFs of the similarity motion model representing a magnitude of a different camera motion of an original camera used to capture the video, the DOFs of the similarity motion model comprising a scale change DOF and a rotation change DOF;

generate spectrograms for the DOFs based on the values of the DOFs over time windows comprising a plurality of adjacent frames of the video;

generate a plurality of shake features based on the spectrograms;

classify the video based on the shake features; and

stabilize the video based on the classification.

9. The computer readable medium of claim 8, wherein the DOFs of the similarity motion model further comprise a lateral translation DOF and a longitudinal translation DOF.

10. The computer readable medium of claim 8, wherein the shake features are invariant with respect to a length of the video.

11. The computer readable medium of claim 8, wherein the time windows of the spectrograms have overlapping coverage of at least a portion of the adjacent frames of a previous and a subsequent spectrogram among the spectrograms.

12. The computer readable medium of claim 8, wherein a spectrogram among the spectrograms comprises an energy representation of contributions of a plurality of energy ranges to the values of one of the DOFs across the adjacent frames of one of the time windows, the spectrogram further comprising a contribution value for each of the energy ranges.

13. The computer readable medium of claim 12, wherein the shake features comprise a mean contribution of one of the energy ranges across the time windows of the video.

14. The computer readable medium of claim 12, wherein, to generate the shake features based on the spectrograms, the computer program instructions are further to cause the computer processor to:

generate a domain histogram for one of the energy ranges, the domain histogram comprising a plurality of domain bins,

each domain bin among the domain bins indicating a percentile range of contribution values bounded by a maximum contribution value of the spectrogram,

a height of each domain bin among the domain bins indicating a number of the time windows of the energy range having a contribution value within the percentile range; and

generate a shake feature among the shake features for the spectrogram based on the height of each domain bin among the domain bins.

15. A computer system for caching streaming media, the system comprising:

a memory; and

a computer processor, coupled to the memory, to;

access a video;

estimate, for a plurality of frames of the video, values of a plurality of degrees of freedom (DOF) of a similarity motion model, a value of each of the DOFs of the similarity motion model representing a magnitude of a different camera motion of an original camera used to capture the video, the DOFs of the similarity motion model comprising a scale change DOF and a rotation change DOF;

generate spectrograms for the DOFs based on the values of the DOFs over time windows comprising a plurality of adjacent frames of the video;

generate a plurality of shake features based on the spectrograms;

classify the video based on the shake features; and

stabilize the video based on the classification.

16. The system of claim 15, wherein the DOFs of the similarity motion model further comprise a lateral translation DOF and a longitudinal translation DOF.

17. The system of claim 15, wherein the shake features are invariant with respect to a length of the video.

18. The system of claim 15, wherein the time windows of the spectrograms have overlapping coverage of at least a portion of the adjacent frames of a previous and a subsequent spectrogram among the spectrograms.

19. The system of claim 15, wherein a spectrogram among the spectrograms comprises an energy representation of contributions of a plurality of energy ranges to the values of one of the DOFs across the adjacent frames of one of the time windows, the spectrogram further comprising a contribution value for each of the energy ranges.

20. The system of claim 19, wherein, to generate the shake features based on the spectrograms, the computer processor is further to:

generate a domain histogram for one of the energy ranges, the domain histogram comprising a plurality of domain bins,

each domain bin among the domain bins indicating a percentile range of contribution values bounded by a maximum contribution value of the spectrogram,

a height of each domain bin among the domain bins indicating a number of the time windows of the energy range having a contribution value within the percentile range; and

generate a shake feature among the shake features for the spectrogram based on the height of each domain bin among the domain bins.