Accelerometer / gyro-facilitated video stabilization

• US 8,558,903 B2
• Filed: 04/07/2010
• Issued: 10/15/2013
• Est. Priority Date: 03/25/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Liu, Yuxin, Shi, Xiaojin, Normile, James Oliver, Wu, Hsi-Jung
• Primary Examiner(s): Misleh, Justin P
• Application Number:
  US12/755,620
• Publication Number:
  US 20110234825A1
• Time in Patent Office: 1,287 Days
• Field of Search: None
• US Class Current: 348/208.99
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Embodiments of the present invention provide a control system for video processes that selectively control the operation of motion stabilization processes. According to the present invention, motion sensor data indicative of motion of a mobile device may be received and processed. A determination may be made by comparing processed motion sensor data to a threshold. Based on the determination, motion stabilization may be suspended on select portions of a captured video sequence.

Claims

1. A video processing method, comprising:

capturing video via a camera to generate a captured video sequence,

aggregating motion sample data from a motion detection device over a period of each video frame in the captured video sequence, the motion sample data being generated by the motion detection device at a rate in excess of one motion sample per video frame and each motion sample generated during the period of a single video frame represents motion of the camera at a different time during the period,

comparing the aggregated motion data corresponding to an amount of camera motion for each frame to a threshold,

identifying, based on the comparison, portions of the captured video sequence that represent a scene change, and

performing video stabilization on portions of the captured video sequence that are outside the scene change.

2. The method of claim 1, wherein the motion detection device is a gyroscope.

3. The method of claim 1, wherein the motion detection device is an accelerometer.

4. The method of claim 1, further comprising normalizing aggregated motion data over each frame.

5. The method of claim 1, further comprising interpolating aggregated motion data for a frame for which motion sample data is unavailable.

6. The method of claim 1, further comprising filtering the aggregated motion data prior to the comparing.

7. The method of claim 1, further comprising:

performing global motion estimation on the captured video to derive motion data therefrom for each frame,

comparing the derived motion data to another threshold, wherein the scene change identification is based at least in part on the derived motion data comparison.

8. The method of claim 7, wherein the global motion estimation includes estimating a motion vector representing motion of each frame from a preceding frame based on content motion in the frames.

9. The method of claim 1, further comprising:

estimating a motion vector representing motion of each frame from a preceding frame based on content motion between the frames; and

comparing the motion vector to a second threshold.

10. The method of claim 9, wherein the portions of the captured video sequence that represent the scene change includes portions of the video sequence that have the aggregated motion exceeding the first threshold and the motion vector that exceeds the second threshold.

11. The method of claim 1, wherein each motion sample from the motion detection device corresponds to the motion of the whole frame.

12. The method of claim 1, wherein the motion detection device is a motion sensor sensing the motion of the camera.

13. The method of claim 1, wherein aggregating the motion sample data includes summing the motion sample data associated with each frame to provide a single aggregated motion value per frame, the single aggregated motion value representing the motion of the camera of the respective frame.

14. The method of claim 13, wherein comparing the aggregated motion data includes comparing the single aggregated motion value of each frame to the threshold.

15. A video processing method, comprising:

estimating motion of a camera device for each video frame in a video sequence based on motion sample data from a camera motion detector, the motion sample data being generated by the camera motion detector at a rate in excess of one motion sample per video frame to provide a plurality of motion samples for each video frame and each motion sample of the plurality of motion samples for each video frame represents motion of the camera at a different point in time,

comparing the estimated motion of each video frame to a threshold, and

for each frame in a captured video sequence, performing video stabilization on the video sequence if the motion comparisons are below the threshold both for the respective frame and for a predetermined number of frames prior to the respective frame.

16. The method of claim 15, wherein the estimating comprises aggregating the motion sample data over a period of each frame in the captured video sequence.

17. The method of claim 15, wherein the estimating comprises filtering motion sample data over a period of each frame in the captured video sequence.

18. The method of claim 15, wherein the estimating comprises normalizing motion sample data over a period of each frame in the captured video sequence.

19. The method of claim 15, wherein the motion detection device is a gyroscope.

20. The method of claim 15, wherein the motion detection device is an accelerometer.

21. The method of claim 15, further comprising:

estimating a motion vector representing motion of each frame from a preceding frame based on content motion between the frames; and

comparing the motion vector to a second threshold.

22. The method of claim 21, further comprising disabling the video stabilization if it is determined that the motion vector is above the second threshold.

23. The method of claim 22, further comprising:

disabling the video stabilization for at least a predetermined number of frames if the video stabilization is disabled.

24. A video processing method, comprising:

estimating motion of a camera device for each video frame in a video sequence based on motion sample data from a camera motion detector, the motion sample data being generated by the camera motion detector at a rate in excess of one motion sample per video frame to provide a plurality of motion samples for each video frame and each motion sample of the plurality of motion samples for each video frame represents motion of the camera at a different point in time,

comparing the estimated motion of each video frame to a threshold,

when the estimated motion crosses the threshold, determining if the estimated motion remains on a crossed side of the threshold for a predetermined number of frames, and

if so, changing operational state of a video stabilization process respectively between an enabled state and a disabled state.

25. The method of claim 24, further comprising:

estimating a motion vector representing motion of each frame from a preceding frame based on content motion in the frames; and

comparing the motion vector to a second threshold.

26. The method of claim 24, wherein the camera motion detector is one of an accelerometer and a gyroscope.

27. Control apparatus for a video stabilization processor comprising:

a motion detector processor to generate estimated motion of a camera for each video frame in a video sequence based on motion sample data received from a motion sensor, the motion sample data being generated by the motion sensor at a rate in excess of one motion sample per video frame to provide a plurality of motion samples for each video frame and each motion sample of the plurality of motion samples for each video frame represents motion of the camera at a different point in time,

a scene change detector to compare the estimated camera motion to a threshold and to identify scene changes within portions of a captured video sequence based on the comparison, and

a video stabilization unit selectively enabled or disabled based on the identification from the scene change detector.

28. The control apparatus of claim 27, wherein the motion detector processor includes an accumulator to sum motion sample data from the motion sensor.

29. The control apparatus of claim 27, wherein the motion detector processor includes a filter to filter motion sample data from the motion sensor.

30. The control apparatus of claim 27, wherein the motion detector processor includes an interpolator to estimate motion of a frame for which motion sample data is unavailable based on estimated motion of neighboring frames.

31. The control apparatus of claim 27, wherein the scene change detector performs a second comparison of motion estimates derived from the captured video sequence to a second threshold and the scene change identification is based on the second comparison.

32. The control apparatus of claim 27, further comprising:

global motion unit estimating a motion vector representing motion of each frame from a preceding frame based on content motion within the frames and comparing the motion vector to a second threshold.

33. The control apparatus of claim 27, wherein the motion sensor is one of an accelerometer and a gyroscope.

34. Computer readable medium storing program instructions that, when executed by a processor, cause the processor to:

estimate motion of a camera device for each video frame in a video sequence based on motion sample data from a camera motion detector, the motion sample data being generated by the camera motion detector at a rate in excess of one motion sample per video frame to provide a plurality of motion samples for each video frame and each motion sample of the plurality of motion samples for each video frame represents motion of the camera at a different point in time,

compare the estimated motion of each video frame to a threshold, and

for each frame in a captured video sequence, perform a video stabilization process on the video sequence if the motion comparisons are below the threshold both for the respective frame and for a predetermined number of frames prior to the respective frame.