Video Stabilization

• US 20120307086A1
• Filed: 11/15/2011
• Published: 12/06/2012
• Est. Priority Date: 05/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Skype S.Ã R.L. (Microsoft Corporation)
• Inventors: Jefremov, Andrei, Zhao, David, Carlsson, Pontus
• Granted Patent: US 8,711,233 B2
• Time in Patent Office: Days
• US Class Current: 348/208.4
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming

Abstract

Method, device and computer program product for stabilizing a video signal. A plurality of frames of the video signal are captured using a camera. A motion sensor associated with the camera is used to generate a plurality of samples representing motion of the camera. The samples are used to determine a displacement of the camera between a first time and a second time, wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal. The determined displacement is used to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, to thereby stabilize the video signal.

Claims

1. A method of stabilizing a video signal, the method comprising:

capturing a plurality of frames of the video signal using a camera;

using a motion sensor associated with the camera to generate a plurality of samples representing motion of the camera;

using the samples to determine a displacement of the camera between a first time and a second time, wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal; and

using the determined displacement to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, to thereby stabilize the video signal.

2. The method of claim 1 wherein the motion of the camera is rotational motion, the motion sensor is a rotational motion sensor and the displacement of the camera is an angular displacement of the camera.

3. The method of claim 2 wherein said using the samples to determine an angular displacement of the camera between a first time and a second time comprises:

determining an angular velocity of the camera using the samples; and

integrating the determined angular velocity over time from the first time to the second time to thereby determine the angular displacement of the camera between the first time and the second time.

4. The method of claim 3 wherein the angular displacement of the camera between the first time and the second time is given by the equation:

5. The method of claim 3 wherein said integrating the determined angular velocity comprises interpolating the angular velocity between the times at which the samples are generated.

6. The method of claim 5 further comprising delaying the video signal thereby allowing the angular velocity of the camera at the second time to be determined by said interpolating the angular velocity between the times at which the samples are generated.

7. The method of claim 3 further comprising extrapolating the angular velocity determined using the samples to thereby determine the angular velocity of the camera at the second time.

8. The method of claim 1 wherein the sample rate of the samples generated using the motion sensor is higher than the frame rate of the video signal.

9. The method of claim 1 wherein the camera and the motion sensor are situated within a mobile device.

10. The method of claim 1 wherein said using the determined displacement to compensate for motion in the video signal between the first and second frames caused by the motion of the camera comprises:

determining a pixel displacement representing motion in the video signal between the first and second frames caused by the determined displacement of the camera;

filtering the pixel displacement; and

shifting the image of at least one of the first and second frames in accordance with the filtered pixel displacement to thereby stabilize the video signal.

11. The method of claim 10 wherein said filtering the pixel displacement comprises:

determining an accumulated pixel displacement based on said determined pixel displacement for the second frame; and

determining a filtered accumulated pixel displacement for the second frame based on a weighted sum of the determined accumulated pixel displacement for the second frame and a filtered accumulated pixel displacement for the first frame.

12. The method of claim 1 further comprising:

adding a time offset to at least one of (i) the captured plurality of frames, and (ii) the generated plurality of samples, such that the timing of the captured plurality of frames matches the timing of the generated plurality of samples.

13. A device configured to stabilize a video signal, the device comprising:

a camera configured to capture a plurality of frames of the video signal;

a motion sensor, associated with the camera, configured to generate a plurality of samples representing motion of the camera;

a displacement determining block configured to use the samples to determine a displacement of the camera between a first time and a second time, wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal; and

a motion compensation block configured to use the determined displacement to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, to thereby stabilize the video signal.

14. The device of claim 13 wherein the motion sensor is a gyroscopic motion sensor.

15. The device of claim 13 wherein the device is a mobile device.

16. A method of stabilizing a video signal, the method comprising:

capturing a plurality of frames of the video signal using a camera;

determining a pixel displacement representing motion in the video signal between first and second frames of the video signal caused by motion of the camera; and

shifting an image of at least one of the first and second frames in accordance with the pixel displacement to thereby stabilize the video signal, wherein the image comprises a stride value indicating memory space of the image, a plurality of pixel values, a pointer indicating the position of a first pixel of the image, and a width value indicating the width of the image, wherein said shifting of the image comprises adjusting the pointer and the width value without adjusting the stride value.

17. The method of claim 16 wherein the image comprises a plurality of image planes which are each represented by a respective plurality of pixel values, a respective pointer indicating the position of a first pixel of the image plane, and a respective width value indicating the width of the image plane, wherein the image planes are shifted together by adjusting their respective pointers and width values without adjusting the stride value.

18. The method of claim 16 further comprising rounding the determined pixel displacement to an integer-pixel displacement.

19. The method of claim 16 further comprising:

using a motion sensor associated with the camera to generate a plurality of samples representing motion of the camera; and

using the samples to determine a displacement of the camera between a first frame of the video signal and a second frame of the video signal, wherein the determined displacement is used to determine said pixel displacement.

20. The method of claim 16 further comprising filtering the pixel displacement.

21. The method of claim 16 wherein the motion of the camera is rotational motion.

22. A device configured to stabilize a video signal, the device comprising:

a camera configured to capture a plurality of frames of the video signal;

a pixel displacement determining block configured to determine a pixel displacement representing motion in the video signal between first and second frames of the video signal caused by motion of the camera; and

an image shifting block configured to shift an image of at least one of the first and second frames in accordance with the pixel displacement to thereby stabilize the video signal, wherein the image comprises a stride value indicating memory space of the image, a plurality of pixel values, a pointer indicating the position of a first pixel of the image, and a width value indicating the width of the image, wherein the image shifting block is configured to shift the image by adjusting the pointer and the width value without adjusting the stride value.

23. A method of stabilizing a video signal, the method comprising:

capturing a plurality of frames of the video signal using a camera;

determining a portion of motion of the camera occurring whilst a shutter of the camera is closed and not whilst the shutter of the camera is open;

determining a pixel displacement representing motion in the video signal between first and second frames of the video signal caused by, and corresponding to, the portion of motion of the camera; and

shifting an image of at least one of the first and second frames in accordance with the pixel displacement to thereby stabilize the video signal.

24. The method of claim 23 further comprising:

using a motion sensor associated with the camera to generate a plurality of samples representing the portion of motion of the camera; and

using the samples to determine a displacement of the camera between the first frame of the video signal and the second frame of the video signal, wherein the determined displacement is used to determine said pixel displacement.

25. The method of claim 24 wherein the motion of the camera is rotational motion, the motion sensor is a rotational motion sensor and the displacement of the camera is an angular displacement of the camera.

26. The method of claim 23 further comprising filtering the determined pixel displacement.

27. A device configured to stabilize a video signal, the device comprising:

a camera configured to capture a plurality of frames of the video signal;

a motion determining block configured to determine a portion of motion of the camera occurring whilst a shutter of the camera is closed and not whilst the shutter of the camera is open;

a pixel displacement determining block configured to determine a pixel displacement representing motion in the video signal between first and second frames of the video signal caused by, and corresponding to, the portion of motion of the camera; and

an image shifting block configured to shift an image of at least one of the first and second frames in accordance with the pixel displacement to thereby stabilize the video signal.

28. A computer program product for stabilizing a video signal, the computer program product being embodied on a non-transient computer-readable medium and configured so as when executed on a processor of a device to perform the operations of:

capturing a plurality of frames of the video signal using a camera;

using a motion sensor associated with the camera to generate a plurality of samples representing motion of the camera;

using the samples to determine a displacement of the camera between a first time and a second time, wherein the first time corresponds to an exposure time midpoint of a first frame of the video signal and the second time corresponds to an exposure time midpoint of a second frame of the video signal; and

using the determined displacement to compensate for motion in the video signal between the first and second frames caused by the motion of the camera, to thereby stabilize the video signal.

29. A computer program product for stabilizing a video signal, the computer program product being embodied on a non-transient computer-readable medium and configured so as when executed on a processor of a device to perform the operations of:

capturing a plurality of frames of the video signal using a camera;

determining a pixel displacement representing motion in the video signal between first and second frames of the video signal caused by motion of the camera; and

shifting an image of at least one of the first and second frames in accordance with the pixel displacement to thereby stabilize the video signal, wherein the image comprises a stride value indicating memory space of the image, a plurality of pixel values, a pointer indicating the position of a first pixel of the image, and a width value indicating the width of the image, wherein said shifting of the image comprises adjusting the pointer and the width value without adjusting the stride value.

30. A computer program product for stabilizing a video signal, the computer program product being embodied on a non-transient computer-readable medium and configured so as when executed on a processor of a device to perform the operations of:

capturing a plurality of frames of the video signal using a camera;

determining a portion of motion of the camera occurring whilst a shutter of the camera is closed and not whilst the shutter of the camera is open;

determining a pixel displacement representing motion in the video signal between first and second frames of the video signal caused by, and corresponding to, the portion of motion of the camera; and

shifting an image of at least one of the first and second frames in accordance with the pixel displacement to thereby stabilize the video signal.