Video stabilization

• US 8,711,233 B2
• Filed: 11/15/2011
• Issued: 04/29/2014
• Est. Priority Date: 05/31/2011
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Skype S.Ã R.L. (Microsoft Corporation)
• Inventors: Jefremov, Andrei, Zhao, David, Carlsson, Pontus
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): CHIU, WESLEY JASON
• Application Number:
  US13/296,941
• Publication Number:
  US 20120307086A1
• Time in Patent Office: 896 Days
• Field of Search: None
• US Class Current: 348/208.4
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming

Abstract

Method, device and computer program product for stabilizing a video signal. A plurality of frames of the video signal are captured using a camera. A motion sensor associated with the camera is used to generate a plurality of samples representing motion of the camera. The samples are used to determine a displacement of the camera between a first time and a second time, wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal. The determined displacement is used to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, to thereby stabilize the video signal.

Claims

1. A method of stabilizing a video signal, the method comprising:

capturing a plurality of frames of the video signal using a camera;

using a motion sensor associated with the camera to generate a plurality of samples representing motion of the camera;

using the samples to determine a displacement of the camera between a first time and a second time, wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal; and

using the determined displacement to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, thereby stabilizing the video signal by;

determining a pixel displacement representing motion in the video signal between the first and second frames caused by the determined displacement of the camera;

filtering the pixel displacement by determining an accumulated pixel displacement based on said determined pixel displacement for the second frame and determining a filtered accumulated pixel displacement for the second frame based on a weighted sum of the determined accumulated pixel displacement for the second frame and a filtered accumulated pixel displacement for the first frame; and

shifting the image of at least one of the first and second frames in accordance with the filtered pixel displacement to thereby stabilize the video signal.

2. The method of claim 1 wherein the motion of the camera is rotational motion, the motion sensor is a rotational motion sensor and the displacement of the camera is an angular displacement of the camera.

3. The method of claim 2 wherein said using the samples to determine an angular displacement of the camera between a first time and a second time comprises:

determining an angular velocity of the camera using the samples; and

integrating the determined angular velocity over time from the first time to the second time to thereby determine the angular displacement of the camera between the first time and the second time.

4. The method of claim 3 wherein the angular displacement of the camera between the first time and the second time is given by the equation:

5. The method of claim 3 wherein said integrating the determined angular velocity comprises interpolating the angular velocity between the times at which the samples are generated.

6. The method of claim 5 further comprising delaying the video signal thereby allowing the angular velocity of the camera at the second time to be determined by said interpolating the angular velocity between the times at which the samples are generated.

7. The method of claim 3 further comprising extrapolating the angular velocity determined using the samples to thereby determine the angular velocity of the camera at the second time.

8. The method of claim 1 wherein the sample rate of the samples generated using the motion sensor is higher than the frame rate of the video signal.

9. The method of claim 1 wherein the camera and the motion sensor are situated within a mobile device.

10. The method of claim 1 further comprising:

adding a time offset to at least one of (i) the captured plurality of frames, and (ii) the generated plurality of samples, such that the timing of the captured plurality of frames matches the timing of the generated plurality of samples.

11. A device configured to stabilize a video signal, the device comprising:

a camera configured to capture a plurality of frames of the video signal;

a motion sensor, associated with the camera, configured to generate a plurality of samples representing motion of the camera, wherein the motion sensor is a rotational motion sensor and the motion of the camera is rotational motion;

a displacement determining block configured to use the samples to determine a displacement of the camera between a first time and a second time, wherein the displacement of the camera is an angular displacement of the camera, and wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal; and

a motion compensation block configured to use the determined displacement to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, to thereby stabilize the video signal, the motion compensation block further configured to;

determine an angular velocity of the camera using the samples; and

integrate the determined angular velocity over time from the first time to the second time to thereby determine the angular displacement of the camera between the first time and the second time, wherein to integrate the determined angular velocity includes interpolating the angular velocity between the times at which the samples are generated;

the device further configured to delay the video signal thereby allowing the angular velocity of the camera at the second time to be determined by said interpolating the angular velocity between the times at which the samples are generated.

12. The device of claim 11 wherein the motion sensor is a gyroscopic motion sensor.

13. The device of claim 11 wherein the device is a mobile device.

14. The device of claim 11 wherein the angular displacement of the camera between the first time and the second time is given by the equation:

15. The device of claim 11 further configured to extrapolate the angular velocity determined using the samples to thereby determine the angular velocity of the camera at the second time.

16. The device of claim 11 wherein the sample rate of the samples generated using the motion sensor is higher than the frame rate of the video signal.

17. The device of claim 11 further configured to:

add a time offset to at least one of (i) the captured plurality of frames, and (ii) the generated plurality of samples, such that the timing of the captured plurality of frames matches the timing of the generated plurality of samples.

18. A computing device comprising:

one or more processors;

one or more computer-readable storage memories embodying computer-readable instructions which, when executed by the one or more processors, cause the computing device to perform operations for stabilizing a video signal comprising;

capturing a plurality of frames of the video signal using a camera;

using a motion sensor associated with the camera to generate a plurality of samples representing motion of the camera, wherein the motion sensor is a rotational motion sensor and wherein the motion of the camera is rotational motion;

using the samples to determine a displacement of the camera between a first time and a second time, wherein the displacement of the camera is an angular displacement of the camera, and wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal, the determining comprising;

determining an angular velocity of the camera using the samples; and

integrating the determined angular velocity over time from the first time to the second time to thereby determine the angular displacement of the camera between the first time and the second time, wherein the angular displacement of the camera between the first time and the second time is given by the equation

19. The computing device of claim 18 wherein the sample rate of the samples generated using the motion sensor is higher than the frame rate of the video signal.

20. The computing device of claim 18 wherein the camera and the motion sensor are situated within a mobile device.

21. The computing device of claim 18 further comprising:

adding a time offset to at least one of (i) the captured plurality of frames, and (ii) the generated plurality of samples, such that the timing of the captured plurality of frames matches the timing of the generated plurality of samples.