Video Stabilization and Reduction of Rolling Shutter Distortion

• US 20110176014A1
• Filed: 03/03/2010
• Published: 07/21/2011
• Est. Priority Date: 01/18/2010
• Status: Active Grant

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: Wei, Dennis, Hong, Wei, Batur, Aziz Umit
• Granted Patent: US 8,179,446 B2
• Time in Patent Office: Days
• US Class Current: 348/208.4
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A method of processing a digital video sequence is provided that includes estimating compensated motion parameters and compensated distortion parameters (compensated M/D parameters) of a compensated motion/distortion (M/D) affine transformation for a block of pixels in the digital video sequence, and applying the compensated M/D affine transformation to the block of pixels using the estimated compensated M/D parameters to generate an output block of pixels, wherein translational and rotational jitter in the block of pixels is stabilized in the output block of pixels and distortion due to skew, horizontal scaling, vertical scaling, and wobble in the block of pixels is reduced in the output block of pixels.

Claims

1. A method of processing a digital video sequence, the method comprising:

estimating compensated motion parameters and compensated distortion parameters (compensated M/D parameters) of a compensated motion/distortion (M/D) affine transformation for a block of pixels in the digital video sequence; and

applying the compensated M/D affine transformation to the block of pixels using the estimated compensated M/D parameters to generate an output block of pixels, wherein translational and rotational jitter in the block of pixels is stabilized in the output block of pixels and distortion due to skew, horizontal scaling, vertical scaling, and wobble in the block of pixels is reduced in the output block of pixels.

2. The method of claim 1, wherein the compensated M/D affine transformation is defined as

3. The method of claim 1, wherein estimating compensated motion parameters and compensated distortion parameters comprises:

estimating block-to-block motion parameters and block-to-block distortion parameters (block-to-block M/D parameters) between the block of pixels and a reference block of pixels;

estimating first intentional block-to-block motion parameters and first intentional block-to-block distortion parameters (first intentional block-to-block M/D parameters) between the block of pixels and the reference block of pixels using the block-to-block M/D parameters and second intentional block-to-block M/D parameters estimated for the reference block of pixels; and

estimating the compensated M/D parameters using the block-to-block M/D parameters and the first intentional block-to-block M/D parameters.

4. The method of claim 3, wherein estimating block-to-block M/D parameters comprises computing a solution for

5. The method of claim 3, wherein estimating intentional block-to-block M/D parameters comprises:

estimating first low-pass block-to-block motion parameters and first low-pass block-to-block distortion parameters (first low-pass block-to-block M/D parameters) between the block of pixels and the reference block of pixels using a low-pass filter, the block-to-block M/D parameters, and second low-pass block-to-block M/D parameters estimated for the reference block of pixels; and

using the first low-pass block-to-block M/D parameters to estimate the first intentional block-to-block M/D parameters.

6. The method of claim 5, wherein using a low-pass filter comprises applying a vector of damping factors to the block-to-block M/D parameters and the second low-pass block-to-block M/D parameters.

7. The method of claim 5, wherein using a low-pass filter comprises applying an adaptive damping vector {right arrow over (β

)}[n] to the block-to-block M/D parameters and the second intentional block-to-block M/D parameters, wherein the adaptive damping vector {right arrow over (β

)}[n] is computed as

{right arrow over (β

)}[n]=max({right arrow over (β

)}₀−

{right arrow over (ρ

)}·

|{right arrow over (p)}^L[n]|,0)

wherein {right arrow over (β

)}₀ is a vector of maximum damping factor values, {right arrow over (ρ

)} is a vector of parameters that control strength of adaptation, n is a number of the block of pixels, and {right arrow over (p)}^L[n] is a vector of the first low-pass block-to-block M/D parameters.

8. The method of claim 3, wherein estimating the compensated M/D parameters comprises:

estimating first wobble compensation parameters between the block of pixels and the reference block of pixels using the block-to-block M/D parameters, the first intentional block-to-block M/D parameters, and second wobble compensation parameters estimated for the reference block of pixels;

estimating skew and vertical scaling distortion parameters between the block of pixels and the reference block of pixels; and

computing the compensated M/D parameters as the sum of the first wobble compensation parameters and the skew and vertical scaling distortion parameters.

9. The method of claim 1, wherein the output block of pixels is cropped to removed invalid regions.

10. The method of claim 1, wherein the block of pixels is augmented with a border of pixels prior to applying the compensated M/D affine transformation.

11. The method of claim 1, wherein the output block of pixels is a same size as the block of pixels.

12. A digital system for processing a digital video sequence, the digital system comprising:

a memory configured to store a block of pixels of the video sequence and a reference block of pixels of the video sequence; and

a video stabilization component configured to

estimate compensated motion parameters and compensated distortion parameters (M/D parameters) of a motion/distortion (M/D) affine transformation for the block of pixels; and

apply the compensated M/D affine transformation to the block of pixels using the estimated compensated M/D parameters to generate an output block of pixels, wherein translational and rotational jitter in the block of pixels is stabilized in the output block of pixels and distortion due to skew, horizontal scaling, vertical scaling, and wobble in the block of pixels is reduced in the output block of pixels.

13. The digital system of claim 12, wherein the memory is further configured to store a border block of pixels and wherein the video stabilization component is further configured to augment the block of pixels with the border block of pixels prior to applying the compensated M/D affine transformation.

14. The digital system of claim 12, wherein the video stabilization component comprises an M/D compensation affine transform component configured to apply the compensated M/D affine transformation, wherein the compensated M/D affine transformation is defined as

15. The digital system of claim 14, wherein the video stabilization component comprises:

a block-to-block M/D estimation component configured to estimate block-to-block motion parameters and block-to-block distortion parameters (block-to-block M/D parameters) between the block of pixels and a reference block of pixels;

an intentional M/D estimation component configured to estimate first intentional block-to-block motion parameters and first intentional block-to-block distortion parameters (first intentional block-to-block M/D parameters) between the block of pixels and the reference block of pixels using the block-to-block M/D parameters and second intentional block-to-block M/D parameters estimated for the reference block of pixels; and

a compensated M/D estimation component configured to estimate the compensated M/D parameters using the block-to-block M/D parameters and the first intentional block-to-block M/D parameters.

16. The digital system of claim 15, wherein the intentional M/D estimation component is configured to estimate first intentional block-to-block M/D parameters by

estimating first low-pass block-to-block motion parameters and first low-pass block-to-block distortion parameters (first low-pass block-to-block M/D parameters) between the block of pixels and the reference block of pixels using a low-pass filter, the block-to-block M/D parameters, and second low-pass block-to-block M/D parameters estimated for the reference block of pixels;

and

using the first low-pass block-to-block M/D parameters to estimate the first intentional block-to-block M/D parameters.

17. The digital system of claim 16, wherein using a low-pass filter comprises applying a vector of damping factors to the block-to-block M/D parameters and the second low-pass block-to-block M/D parameters.

18. The digital system of claim 15, wherein using a low-pass filter comprises applying an adaptive damping vector {right arrow over (β

)}[n] to the block-to-block M/D parameters and the second intentional block-to-block M/D parameters, wherein the adaptive damping vector {right arrow over (β

)}[n] is computed as

{right arrow over (β

)}[n]=max({right arrow over (β

)}₀−

{right arrow over (ρ

)}·

|{right arrow over (p)}^L[n]|,0)

wherein {right arrow over (β

)}₀ is a vector of maximum damping factor values, {right arrow over (ρ

)} is a vector of parameters that control strength of adaptation, n is a number of the block of pixels, and {right arrow over (p)}^L[n] is a vector of the first low-pass block-to-block M/D parameters.

19. The digital system of claim 15, wherein the compensated M/D estimation component is configured to estimate the compensated M/D parameters by

estimating first wobble compensation parameters between the block of pixels and the reference block of pixels using the block-to-block M/D parameters, the first intentional block-to-block M/D parameters, and second wobble compensation parameters estimated for the reference block of pixels;

estimating skew and vertical scaling distortion parameters between the block of pixels and the reference block of pixels; and

computing the compensated M/D parameters as the sum of the first wobble compensation parameters and the skew and vertical scaling distortion parameters.

20. A computer readable medium storing software instructions that, when executed by a processor of a digital system, cause the digital system to perform a method of processing a digital video sequence, the method comprising:

estimating compensated motion parameters and compensated distortion parameters (M/D parameters) of a motion/distortion (M/D) affine transformation for a block of pixels in the digital video sequence; and

applying the M/D affine transformation to the block of pixels using the estimated compensated M/D parameters to generate an output block of pixels, wherein translational and rotational jitter in the block of pixels is stabilized in the output block of pixels and distortion due to skew, horizontal scaling, vertical scaling, and wobble in the block of pixels is reduced in the output block of pixels