Method to stabilize digital video motion

• US 7,768,551 B2
• Filed: 03/24/2008
• Issued: 08/03/2010
• Est. Priority Date: 01/18/2005
• Status: Active Grant

• Current Assignee: Primax Electronics Limited
• Original Assignee: Primax Electronics Limited
• Inventors: Yang, Shih-Hsuan, Jheng, Fu-Ming
• Primary Examiner(s): Tran, Sinh
• Assistant Examiner(s): BERARDESCA, PAUL M
• Application Number:
  US12/053,630
• Publication Number:
  US 20080170125A1
• Time in Patent Office: 862 Days
• Field of Search: 348/208.3, 348/208.6
• US Class Current: 348/208.99
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation

Abstract

A method for calculating an estimated hand motion vector of a plurality of images includes selecting a current video frame and a previous video frame from a video stream, dividing the current video frame into a plurality of macroblocks, selecting a subset of macroblocks from the current video frame, comparing the subset of macroblocks to the previous video frame within a two dimensional comparison range to generate a set of motion vectors, calculating a frame motion vector for the current video frame, estimating a steady motion vector for the current video frame by a first-order autoregression, and calculating a hand motion vector for the current video frame by taking a difference between the steady motion vector and the frame motion vector of the current video frame.

Claims

1. A method for calculating an estimated hand motion vector of a plurality of images, comprising:

selecting a current video frame and a previous video frame from a video stream generated by a video camera;

dividing the current video frame into a plurality of macroblocks;

selecting a subset of macroblocks from the current video frame;

comparing the subset of macroblocks to the previous video frame within a two dimensional comparison range to generate a set of motion vectors;

calculating a frame motion vector for the current video frame;

estimating a steady motion vector for the current video frame by a first-order autoregression;

calculating a hand motion vector for the current video frame by taking a difference between the steady motion vector and the frame motion vector of the current video frame;

estimating the steady motion vector for the current video frame by summing a product of a first variable and the steady motion vector for the previous video frame with a product of a second variable and the frame motion vector of the current video frame, wherein the sum of the first variable and the second variable equals a first constant, wherein the second variable is set to be a minimum of a smoothing value or a correlation value; and

calculating an estimated hand motion vector by subtracting the steady motion vector of the previous video frame from the frame motion vector of the current video frame;

wherein the correlation value is calculated by;

taking a first square root of an inner product of the estimated hand motion vector of the current video frame and the estimated hand motion vector of the current video frame;

taking a second square root of an inner product of the steady motion vector of the previous video frame and the steady motion vector of the previous video frame;

calculating a dividend value by multiplying the first square root by the second square root;

calculating a divisor value of an inner product of the estimated hand motion vector of the current video frame and the steady motion vector of the previous video frame; and

calculating the correlation value by taking an absolute value of the divisor value divided by the dividend value.

2. The method of claim 1 wherein the frame motion vector of the current video frame is calculated by:

selecting a median horizontal component from the set of motion vectors; and

selecting a median vertical component from the set of motion vectors; and

forming the frame motion vector from the median horizontal component and the median vertical component.

3. The method of claim 1 further comprising assigning the frame motion vector for an initial frame of the video stream to be a first default vector, and assigning the steady motion vector of the initial frame of the video stream to be a second default vector.

4. The method of claim 3 where the first default vector is the null vector <

0,0>

.

5. The method of claim 3 where the second default vector is the null vector <

0,0>

.

6. The method of claim 3 where the second default vector is the first default vector.

7. The method of claim 1 where the first constant is 1.

8. The method of claim 1 where the smoothing value is about 0.4.

9. The method of claim 1 further comprising:

setting the smoothing value to a first smoothing constant when a first absolute value of a horizontal component of the frame motion vector of the current video frame minus a horizontal component of the steady motion vector of the previous frame is greater than a first threshold value, and a second absolute value of a horizontal component of the frame motion vector of the previous video frame minus a horizontal component of the steady motion vector of the previous video frame of the previous video frame is greater than a second threshold value, and a third absolute value of a horizontal component of the frame motion vector of the previous video frame of the previous video frame minus a horizontal component of the steady motion vector of the previous video frame of the previous video frame of the previous video frame is greater than a third threshold value;

otherwise setting the smoothing value to a second smoothing constant.

10. The method of claim 9 where the first smoothing constant is 0.8 and the second smoothing constant is 0.1.

11. The method of claim 1 further comprising:

setting the smoothing value to a first smoothing constant when a first absolute value of a vertical component of the frame motion vector of the current video frame minus a vertical component of the steady motion vector of the previous frame is greater than a first threshold value, and a second absolute value of a vertical component of the frame motion vector of the previous video frame minus a vertical component of the steady motion vector of the previous video frame of the previous video frame is greater than a second threshold value, and a third absolute value of a vertical component of the frame motion vector of the previous video frame of the previous video frame minus a vertical component of the steady motion vector of the previous video frame of the previous video frame of the previous video frame is greater than a third threshold value;

otherwise setting the smoothing value to a second smoothing constant.

12. The method of claim 11 where the first smoothing constant is 0.8 and the second smoothing constant is 0.1.

13. The method of claim 1 further comprising:

dividing a horizontal magnitude of the hand motion vector by 2 when the horizontal magnitude of an accumulated motion vector is larger than a quantity generated by multiplying a horizontal range limit by a scaling factor; and

dividing the vertical magnitude of the hand motion vector by 2 when the vertical magnitude of the accumulated motion vector is larger than a quantity generated by multiplying a vertical range limit by a scaling factor.

14. A digital video recording device comprising a lens, a means of capturing a sequence of images, a storage device, and a processor that performs the method of claim 1.

15. A method for calculating an estimated hand motion vector of a plurality of images, comprising:

selecting a current video frame and a previous video frame from a video stream generated by a video camera;

dividing the current video frame into a plurality of macroblocks;

selecting a subset of macroblocks from the current video frame;

comparing the subset of macroblocks to the previous video frame within a two dimensional comparison range to generate a set of motion vectors;

calculating a frame motion vector for the current video frame;

estimating a steady motion vector for the current video frame by a first-order autoregression, wherein the steady motion vector for the current video frame is estimated by;

setting the horizontal component of the steady motion vector by summing a product of a first variable and a horizontal component of the steady motion vector for the previous video frame with a product of a second variable and a horizontal component of the frame motion vector of the current video frame, wherein the sum of the first variable and the second variable equals a first constant, wherein the second variable is set to a minimum of a horizontal smoothing value or a horizontal correlation value; and

setting the vertical component of the steady motion vector by summing a product of a third variable and a vertical component of the steady motion vector for the previous video frame with a product of a fourth variable and a vertical component of the frame motion vector of the current video frame, wherein the sum of the third variable and the fourth variable equals a second constant;

calculating a hand motion vector for the current video frame by taking a difference between the steady motion vector and the frame motion vector of the current video frame; and

calculating an estimated hand motion vector by subtracting the steady motion vector of the previous video frame from the frame motion vector of the current video frame;

wherein the horizontal correlation is calculated by;

taking a first square root of an inner product of the estimated hand motion vector of the current video frame and the estimated hand motion vector of the current video frame;

taking a second square root of an inner product of the steady motion vector of the previous video frame and the steady motion vector of the previous video frame;

calculating a dividend value by multiplying the first square root by the second square root;

calculating a divisor value of an inner product of the estimated hand motion vector of the current video frame and the steady motion vector of the previous video frame; and

calculating the horizontal correlation value by taking an absolute value of the divisor value divided by the dividend value.

16. The method of claim 15 where the first constant is 1.

17. The method of claim 15 where the second constant is 1.

18. The method of claim 15 where the horizontal smoothing value is about 0.4.

19. A method for calculating an estimated hand motion vector of a plurality of images, comprising:

selecting a current video frame and a previous video frame from a video stream generated by a video camera;

dividing the current video frame into a plurality of macroblocks;

selecting a subset of macroblocks from the current video frame;

comparing the subset of macroblocks to the previous video frame within a two dimensional comparison range to generate a set of motion vectors;

calculating a frame motion vector for the current video frame;

estimating a steady motion vector for the current video frame by a first-order autoregression, wherein the steady motion vector for the current video frame is estimated by;

setting the horizontal component of the steady motion vector by summing a product of a first variable and a horizontal component of the steady motion vector for the previous video frame with a product of a second variable and a horizontal component of the frame motion vector of the current video frame, wherein the sum of the first variable and the second variable equals a first constant; and

setting the vertical component of the steady motion vector by summing a product of a third variable and a vertical component of the steady motion vector for the previous video frame with a product of a fourth variable and a vertical component of the frame motion vector of the current video frame, wherein the sum of the third variable and the fourth variable equals a second constant, wherein the fourth variable is set to a minimum of a vertical smoothing value or a vertical correlation value;

calculating a hand motion vector for the current video frame by taking a difference between the steady motion vector and the frame motion vector of the current video frame; and

calculating an estimated hand motion vector by subtracting the steady motion vector of the previous video frame from the frame motion vector of the current video frame;

wherein the vertical correlation value is calculated by;

taking a first square root of an inner product of the estimated hand motion vector of the current video frame and the estimated hand motion vector of the current video frame;

taking a second square root of an inner product of the steady motion vector of the previous video frame and the steady motion vector of the previous video frame;

calculating a dividend value by multiplying the first square root by the second square root;

calculating a divisor value of an inner product of the estimated hand motion vector of the current video frame and the steady motion vector of the previous video frame; and

calculating the vertical correlation value by taking an absolute value of the divisor value divided by the dividend value.

20. The method of claim 19 where the vertical smoothing value is about 0.4.

21. The method of claim 19 further comprising:

setting the smoothing value to a first smoothing constant when a first absolute value of a horizontal component of the frame motion vector of the current video frame minus a horizontal component of the steady motion vector of the previous frame is greater than a first threshold value, and a second absolute value of a horizontal component of the frame motion vector of the previous video frame minus a horizontal component of the steady motion vector of the previous video frame of the previous video frame is greater than a second threshold value, and a third absolute value of a horizontal component of the frame motion vector of the previous video frame of the previous video frame minus a horizontal component of the steady motion vector of the previous video frame of the previous video frame of the previous video frame is greater than a third threshold value;

otherwise setting the smoothing value to a second smoothing constant.

22. The method of claim 21 where the first smoothing constant is 0.8 and the second smoothing constant is 0.1.

23. The method of claim 21 where the first smoothing constant is 0.8 and the second smoothing constant is 0.1.

24. The method of claim 19 further comprising:

setting the smoothing value to a first smoothing constant when a first absolute value of a vertical component of the frame motion vector of the current video frame minus a vertical component of the steady motion vector of the previous frame is greater than a first threshold value, and a second absolute value of a vertical component of the frame motion vector of the previous video frame minus a vertical component of the steady motion vector of the previous video frame of the previous video frame is greater than a second threshold value, and a third absolute value of a vertical component of the frame motion vector of the previous video frame of the previous video frame minus a vertical component of the steady motion vector of the previous video frame of the previous video frame of the previous video frame is greater than a third threshold value;

otherwise setting the smoothing value to a second smoothing constant.