Generating a stabilized video sequence based on motion sensor data

• US 8,488,010 B2
• Filed: 09/21/2010
• Issued: 07/16/2013
• Est. Priority Date: 09/21/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Voss, Shane D., Brake, Wilfred F., Goris, Andrew C.
• Primary Examiner(s): Daniels, Anthony J
• Application Number:
  US12/887,034
• Publication Number:
  US 20120069203A1
• Time in Patent Office: 1,029 Days
• Field of Search: 34820899-2083
• US Class Current: 348/208.99
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20

Abstract

A machine-implemented method of generating a stabilized video sequence includes receiving an input video sequence captured by an image capture device. The input video sequence includes a plurality of pairs of successive frames. Motion sensor data indicative of motion of the image capture device while the input video sequence was being captured is received. A set of matching features for each pair of successive frames is identified. Global motion features are identified in each set of matching features and qualified based on the motion sensor data. The global motion features are indicative of movement of the image capture device. A stabilized video sequence is generated based on the input video sequence and the identified global motion features.

Claims

1. A machine-implemented method of generating a stabilized video sequence, comprising:

receiving an input video sequence captured by an image capture device, the input video sequence including a plurality of pairs of successive frames;

receiving motion sensor data indicative of motion of the image capture device while the input video sequence was being captured;

identifying a set of matching features for each pair of successive frames;

identifying a first feature in a first frame of a first pair of the successive frames;

identifying a location for a search window in a second frame of the first pair of the successive frames based on the motion sensor data;

identifying global motion features in each set of matching features based on the motion sensor data, the global motion features indicative of movement of the image capture device; and

generating a stabilized video sequence based on the input video sequence and the identified global motion features.

2. The machine-implemented method of claim 1, wherein the motion sensor data is determined independently from image data in the input video sequence.

3. The machine-implemented method of claim 1, and further comprising:

generating the motion sensor data with at least one of a linear motion sensor and a rotational motion sensor.

4. The machine-implemented method of claim 1, and further comprising:

generating the motion sensor data with a linear motion sensor and a rotational motion sensor.

5. The machine-implemented method of claim 1, and further comprising:

generating the motion sensor data with an accelerometer.

6. The machine-implemented method of claim 1, and further comprising:

generating the motion sensor data with a gyroscope.

7. The machine-implemented method of claim 1, and further comprising:

generating the motion sensor data with a magnetometer.

8. The machine-implemented method of claim 1, and further comprising:

generating the motion sensor data with at least two of an accelerometer, a gyroscope, and a magnetometer.

9. The machine-implemented method of claim 1, and further comprising:

identifying a location of the first feature in the second frame of the first pair of the successive frames based on the motion sensor data.

10. The machine-implemented method of claim 1, and further comprising:

searching for the first feature in the search window of the second frame.

11. The machine-implemented method of claim 1, and further comprising:

identifying a potentially matching feature corresponding to the first feature in the second frame of the first pair of the successive frames; and

verifying whether the potentially matching feature actually matches the first feature based on the motion sensor data.

12. The machine-implemented method of claim 1, and further comprising:

identifying non-global motion features in each set of matching features based on the motion sensor data, the non-global motion features indicative of subject motion; and

discarding the identified non-global motion features so that they are not used in generating the stabilized video sequence.

13. The machine-implemented method of claim 1, and further comprising:

computing local motion vectors based on the identified global motion features;

generating a stabilization transformation matrix based on the local motion vectors; and

generating the stabilized video sequence using the stabilization transformation matrix.

14. A system for generating a stabilized video sequence, comprising:

a feature identification and matching unit configured to receive an input video sequence captured by an image capture device, identify a set of matching features for pairs of successive frames in the input video sequence, identify a first feature in a first frame of a first pair of the successive frames, identify a location for a search window in a second frame of the first pair of the successive frames based on the motion sensor data, and identify global motion features in each set of matching features based on motion sensor data indicative of motion of the image capture device while the input video sequence was being captured, wherein the global motion features are indicative of movement of the image capture device; and

a motion stabilization unit configured to generate a stabilized video sequence based on the input video sequence and the identified global motion features.

15. The system of claim 14, wherein the motion sensor data comprises data from at least one of an accelerometer, a gyroscope, and a magnetometer.

16. The system of claim 14, wherein the motion sensor data comprises data from at least two of an accelerometer, a gyroscope, and a magnetometer.

17. The system of claim 14, wherein the feature identification and matching unit is configured to search for the first feature in the search window of the second frame.

18. An image capture device, comprising:

an image sensor configured to capture an input video sequence;

a motion sensor configured to generate motion sensor data indicative of motion of the image capture device while the input video sequence is being captured; and

an image processing system configured to receive the input video sequence and the motion sensor data, identify a set of matching features for pairs of successive frames in the input video sequence, identify a first feature in a first frame of a first pair of the successive frames, identify a location for a search window in a second frame of the first pair of the successive frames based on the motion sensor data, and search for the first feature in the search window of the second frame, identify global motion features in each set of matching features based on the motion sensor data, and generate a stabilized video sequence based on the input video sequence and the identified global motion features, wherein the global motion features are indicative of global motion.

19. The image capture device of claim 18, wherein the motion sensor comprises at least one of an accelerometer, a gyroscope, and a magnetometer.

20. The image capture device of claim 18, wherein the motion sensor comprises at least two of an accelerometer, a gyroscope, and a magnetometer.