METHOD TO STABILIZE DIGITAL VIDEO MOTION

• US 20060158524A1
• Filed: 01/18/2005
• Published: 07/20/2006
• Est. Priority Date: 01/18/2005
• Status: Active Grant

• Current Assignee: Primax Electronics Limited
• Original Assignee: Primax Electronics Limited
• Inventors: Jheng, Fu-Ming, Yang, Shih-Hsuan
• Granted Patent: US 7,489,341 B2
• Time in Patent Office: Days
• US Class Current: 348/208.600
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation

Abstract

A digital video image stabilization method features scene change detection, adaptive stabilization in two dimensions, prevention of accumulated-compensation saturation, multi-resolution motion estimation, and compatibility with MPEG video encoding standards. The method uses block-based motion estimation on a selected set of macroblocks, which generates and validates a set of motion vectors, to determine how a digital camcorder or other video recording device is moving; accumulates motion data to estimate and preserve overall motion due to panning while eliminating small variations due to shaking; and resets its accumulated motion data when it detects a scene change. The method uses this information to select a sub-image from a larger image for the selected video frame. When one or more edges of the sub-image approach the edges of the larger image, the method reduces its correction factor to transition smoothly to a faster pan and prevent saturation.

Claims

1. A method for stabilizing digital video motion comprising:

selecting a current video frame and a previous video frame from a video stream;

dividing the current video frame into a plurality of macroblocks;

selecting a subset of macroblocks from the current video frame;

comparing the subset of macroblocks to the previous video frame within a two-dimensional comparison range to generate a set of motion vectors;

calculating a frame motion vector for the current video frame;

estimating a steady motion vector for the current video frame by a first-order autoregression;

calculating a hand motion vector for the current video frame by taking the difference between the steady motion vector and the frame motion vector of the current video frame;

cropping a selected video frame from the current video frame, wherein a center of the selected video frame is offset from a center of the current video frame by the hand motion vector; and

recording the selected video frame to a storage device.

2. The method of claim 1 further comprising deselecting one or more macroblocks from the subset of macroblocks based upon a mean average difference within a search region within a first number of pixels around the macroblock being below a threshold value, a lack of features within the macroblock, or repeated patterns within the previous video frame.

3. The method of claim 1 wherein calculating the frame motion vector comprises selecting a median vertical component and a median horizontal component from the set of motion vectors and the frame motion vector for the previous video frame.

4. The method of claim 1 wherein the first-order autoregression comprises:

multiplying the frame motion vector of the current frame by a first constant; and

multiplying the steady motion vector of the previous video frame by a second constant;

wherein the first constant is a floating-point value in the range of 0-1 inclusive and the second constant is calculated by subtracting the first constant from 1.

5. The method of claim 4 wherein the first constant is chosen to be the lesser of a third constant and a correlation value.

6. The method of claim 5 further comprising:

computing an estimated hand motion vector of the current video frame by subtracting the steady motion vector of the previous video frame from the frame motion vector of the current video frame; and

calculating the correlation value between the estimated hand motion vector and the steady motion vector by taking an absolute value of an inner product of the estimated hand motion vector of the current video frame and steady motion vector of the previous video frame, and dividing the absolute value by a product of a square root of an inner product of the estimated hand motion vector of the current video frame and the estimated hand motion vector of the current video frame and a square root of an inner product of the steady motion vector of the previous video frame and the steady motion vector of the previous video frame.

7. The method of claim 5 where the third constant is substantially 0.40.

8. The method of claim 1 further comprising:

dividing a horizontal magnitude of the hand motion vector by 2 when the horizontal magnitude of the accumulated motion vector is larger than a quantity generated by multiplying a horizontal range limit by a scaling factor; and

dividing the vertical magnitude of the hand motion vector by 2 when the vertical magnitude of the accumulated motion vector is larger than a quantity generated by multiplying a vertical range limit by a scaling factor.

9. The method of claim 1 wherein a size of the macroblock is chosen to be compatible with a MPEG video encoding standard.

10. The method of claim 1 wherein the previous frame is selected to be the current frame when analyzing the first frame or when a count of the macroblocks in the subset of macroblocks is below a threshold value.

11. A digital video recording device comprising a lens, a means of capturing a sequence of images, a storage device, and a processor that performs the method of claim 1.

12. A digital video recording device comprising a lens, a means of capturing a sequence of images, a storage device, and a processor that performs the method of claim 1.

13. A method for stabilizing digital video motion comprising:

selecting a current video frame and a previous video frame from a video stream;

dividing the current video frame into a plurality of macroblocks, such that the macroblock size is suitable for either autonomously encoded or predictively encoded pictures;

selecting a subset of macroblocks from the current video frame;

comparing the subset of macroblocks to the previous video frame within a two-dimensional comparison range to generate a set of motion vectors;

calculating a frame motion vector for the current video frame by selecting a median vertical component and a median horizontal component from the set of motion vectors and the frame motion vector for the previous video frame;

estimating a steady motion vector for the current video frame by a first-order autoregression;

calculating a hand motion vector for the current video frame by taking the difference between the steady motion vector and the frame motion vector of the current video frame;

cropping a selected video frame from the current video frame, wherein a center of the selected video frame is offset from a center of the current video frame by the hand motion vector; and

recording the selected video frame to a storage device.

14. The method of claim 13 further comprising deselecting one or more macroblocks from the subset of macroblocks based upon a mean average difference within a search region within a first number of pixels around the macroblock being below a threshold value, a lack of features within the macroblock, or repeated patterns within the previous video frame.

15. The method of claim 13 wherein the first-order autoregression comprises:

multiplying the frame motion vector of the current frame by a first constant; and

multiplying the steady motion vector of the previous video frame by a second constant;

wherein the first constant is a floating-point value in the range of 0-1 inclusive and the second constant is calculated by subtracting the first constant from 1.

16. The method of claim 15 wherein the first constant is chosen to be the lesser of a third constant and a correlation value.

17. The method of claim 16 further comprising:

computing an estimated hand motion vector of the current video frame by subtracting the steady motion vector of the previous video frame from the frame motion vector of the current video frame; and

calculating the correlation value between the estimated hand motion vector and the steady motion vector by taking an absolute value of an inner product of the estimated hand motion vector of the current video frame and steady motion vector of the previous video frame, and dividing the absolute value by a product of a square root of an inner product of the estimated hand motion vector of the current video frame and the estimated hand motion vector of the current video frame and a square root of an inner product of the steady motion vector of the previous video frame and the steady motion vector of the previous video frame.

18. The method of claim 16 where the third constant is substantially 0.40.

19. The method of claim 13 further comprising:

dividing a horizontal magnitude of the hand motion vector by 2 when the horizontal magnitude of the average motion vector is larger than a quantity generated by multiplying a horizontal range limit by a scaling factor; and

dividing the vertical magnitude of the hand motion vector by 2 when the vertical magnitude of the average motion vector is larger than a quantity generated by multiplying a vertical range limit by a scaling factor.

20. The method of claim 13 wherein the previous frame is selected to be the current frame when analyzing the first frame or when a count of the macroblocks in the subset of macroblocks is below a threshold value.

21. A digital video recording device comprising a lens, a means of capturing a sequence of images, a storage device, and a processor that performs the method of claim 13.

22. A method for calculating an estimated hand motion vector of a plurality of images, comprising:

selecting a current video frame and a previous video frame from a video stream;

dividing the current video frame into a plurality of macroblocks;

selecting a subset of macroblocks from the current video frame;

comparing the subset of macroblocks to the previous video frame within a two dimensional comparison range to generate a set of motion vectors;

calculating a frame motion vector for the current video frame;

estimating a steady motion vector for the current video frame by a first-order autoregression; and

calculating a hand motion vector for the current video frame by taking a difference between the steady motion vector and the frame motion vector of the current video frame.

23. The method of claim 22 further comprising assigning the frame motion vector for an initial frame of the video stream to be a first default vector, and assigning the steady motion vector of the initial frame of the video stream to be a second default vector.

24. The method of claim 23 where the first default vector is the null vector <

0,0>

.

25. The method of claim 23 where the second default vector is the null vector <

0,0>

.

26. The method of claim 23 where the second default vector is the first default vector.

27. The method of claim 22 further comprising estimating the steady motion vector for the current video frame by summing a product of a first variable and the steady motion vector for the previous video frame with a product of a second variable and the frame motion vector of the current video frame, wherein the sum of the first variable and the second variable equals a first constant.

28. The method of claim 27 where the first constant is 1.

29. The method of claim 22 wherein the frame motion vector of the current video frame is calculated by:

selecting a median horizontal component from the set of motion vectors; and

selecting a median vertical component from the set of motion vectors; and

forming the frame motion vector from the median horizontal component and the median vertical component.

30. The method of claim 27 further comprising calculating an estimated hand motion vector by subtracting the steady motion vector of the previous video frame from the frame motion vector of the current video frame.

31. The method of claim 30 further comprising setting the second variable to a minimum of a smoothing value or a correlation value.

32. The method of claim 31 further comprising calculating the correlation value by:

taking a first square root of an inner product of the estimated hand motion vector of the current video frame and the estimated hand motion vector of the current video frame;

taking a second square root of an inner product of the steady motion vector of the previous video frame and the steady motion vector of the previous video frame;

calculating a dividend value by multiplying the first square root by the second square root;

calculating a divisor value of an inner product of the estimated hand motion vector of the current video frame and the steady motion vector of the previous video frame; and

calculating the correlation value by taking an absolute value of the divisor value divided by the dividend value.

33. The method of claim 31 where the smoothing value is about 0.4.

34. The method of claim 32 further comprising:

setting the smoothing value to a first smoothing constant when a first absolute value of a horizontal component of the frame motion vector of the current video frame minus a horizontal component of the steady motion vector of the previous frame is greater than a first threshold value, and a second absolute value of a horizontal component of the frame motion vector of the previous video frame minus a horizontal component of the steady motion vector of the previous video frame of the previous video frame is greater than a second threshold value, and a third absolute value of a horizontal component of the frame motion vector of the previous video frame of the previous video frame minus a horizontal component of the steady motion vector of the previous video frame of the previous video frame of the previous video frame is greater than a third threshold value;

otherwise setting the smoothing value to a second smoothing constant.

35. The method of claim 34 where the first smoothing constant is 0.8 and the second smoothing constant is 0.1.

36. The method of claim 32 further comprising:

setting the smoothing value to a first smoothing constant when a first absolute value of a vertical component of the frame motion vector of the current video frame minus a vertical component of the steady motion vector of the previous frame is greater than a first threshold value, and a second absolute value of a vertical component of the frame motion vector of the previous video frame minus a vertical component of the steady motion vector of the previous video frame of the previous video frame is greater than a second threshold value, and a third absolute value of a vertical component of the frame motion vector of the previous video frame of the previous video frame minus a vertical component of the steady motion vector of the previous video frame of the previous video frame of the previous video frame is greater than a third threshold value;

otherwise setting the smoothing value to a second smoothing constant.

37. The method of claim 36 where the first smoothing constant is 0.8 and the second smoothing constant is 0.1.

38. The method of claim 22 further comprising:

dividing a horizontal magnitude of the hand motion vector by 2 when the horizontal magnitude of the accumulated motion vector is larger than a quantity generated by multiplying a horizontal range limit by a scaling factor; and

dividing the vertical magnitude of the hand motion vector by 2 when the vertical magnitude of the accumulated motion vector is larger than a quantity generated by multiplying a vertical range limit by a scaling factor.

39. The method of claim 22 further comprising estimating the steady motion vector for the current video frame by:

setting the horizontal component of the steady motion vector by summing a product of a first variable and a horizontal component of the steady motion vector for the previous video frame with a product of a second variable and a horizontal component of the frame motion vector of the current video frame, wherein the sum of the first variable and the second variable equals a first constant; and

setting the vertical component of the steady motion vector by summing a product of a third variable and a vertical component of the steady motion vector for the previous video frame with a product of a fourth variable and a vertical component of the frame motion vector of the current video frame, wherein the sum of the third variable and the fourth variable equals a second constant.

40. The method of claim 39 where the first constant is 1.

41. The method of claim 39 where the second constant is 1.

42. The method of claim 39 further comprising calculating an estimated hand motion vector by subtracting the steady motion vector of the previous video frame from the frame motion vector of the current video frame.

43. The method of claim 42 further comprising setting the second variable to a minimum of a horizontal smoothing value or a horizontal correlation value.

44. The method of claim 43 further comprising calculating the horizontal correlation value by:

taking a first square root of an inner product of the estimated hand motion vector of the current video frame and the estimated hand motion vector of the current video frame;

taking a second square root of an inner product of the steady motion vector of the previous video frame and the steady motion vector of the previous video frame;

calculating a dividend value by multiplying the first square root by the second square root;

calculating a divisor value of an inner product of the estimated hand motion vector of the current video frame and the steady motion vector of the previous video frame; and

calculating the horizontal correlation value by taking an absolute value of the divisor value divided by the dividend value.

45. The method of claim 43 where the horizontal smoothing value is about 0.4.

46. The method of claim 42 further comprising setting the fourth variable to a minimum of a horizontal smoothing value or a horizontal correlation value.

47. The method of claim 46 further comprising calculating the vertical correlation value by:

taking a first square root of an inner product of the estimated hand motion vector of the current video frame and the estimated hand motion vector of the current video frame;

taking a second square root of an inner product of the steady motion vector of the previous video frame and the steady motion vector of the previous video frame;

calculating a dividend value by multiplying the first square root by the second square root;

calculating a divisor value of an inner product of the estimated hand motion vector of the current video frame and the steady motion vector of the previous video frame; and

calculating the vertical correlation value by taking an absolute value of the divisor value divided by the dividend value.

48. The method of claim 47 where the vertical smoothing value is about 0.4.

49. The method of claim 47 further comprising:

setting the smoothing value to a first smoothing constant when a first absolute value of a horizontal component of the frame motion vector of the current video frame minus a horizontal component of the steady motion vector of the previous frame is greater than a first threshold value, and a second absolute value of a horizontal component of the frame motion vector of the previous video frame minus a horizontal component of the steady motion vector of the previous video frame of the previous video frame is greater than a second threshold value, and a third absolute value of a horizontal component of the frame motion vector of the previous video frame of the previous video frame minus a horizontal component of the steady motion vector of the previous video frame of the previous video frame of the previous video frame is greater than a third threshold value;

otherwise setting the smoothing value to a second smoothing constant.

50. The method of claim 49 where the first smoothing constant is 0.8 and the second smoothing constant is 0.1.

51. The method of claim 47 further comprising:

setting the smoothing value to a first smoothing constant when a first absolute value of a vertical component of the frame motion vector of the current video frame minus a vertical component of the steady motion vector of the previous frame is greater than a first threshold value, and a second absolute value of a vertical component of the frame motion vector of the previous video frame minus a vertical component of the steady motion vector of the previous video frame of the previous video frame is greater than a second threshold value, and a third absolute value of a vertical component of the frame motion vector of the previous video frame of the previous video frame minus a vertical component of the steady motion vector of the previous video frame of the previous video frame of the previous video frame is greater than a third threshold value;

otherwise setting the smoothing value to a second smoothing constant.

52. The method of claim 49 where the first smoothing constant is 0.8 and the second smoothing constant is 0.1.

53. A digital video recording device comprising a lens, a means of capturing a sequence of images, a storage device, and a processor that performs the method of claim 22.