Motion filtering for video stabilization

• US 7,558,405 B2
• Filed: 06/30/2005
• Issued: 07/07/2009
• Est. Priority Date: 06/30/2005
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Tico, Marius, Vehvilainen, Markku
• Primary Examiner(s): Tabatabai; Abolfazl
• Application Number:
  US11/174,297
• Publication Number:
  US 20070002146A1
• Time in Patent Office: 1,468 Days
• Field of Search: 382/103, 382/107, 382/210, 382/211, 382/236, 382/260, 382/261, 382/262, 382/263, 382/264, 382/265, 348/14.1, 348/69, 348/154, 348/155, 348/208.1, 348/208.2, 348/208.16, 348/352, 348/407.1, 348/431.1, 348/97, 348/620, 348/665, 348/667, 348/835, 396/153, 396/492
• US Class Current: 382/107
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure

Abstract

For applying a motion filter of a video stabilization system to a sequence of video frames, an estimate of a motion in the current video frame compared to a first video frame of the sequence of video frames is received. Based on the received motion estimate and on at least one state variable of the motion filter, a correction value for the motion in the current video frame is computed. The at least one state variable is updated in the computation. In case the computed correction value exceeds a system constraint of the video stabilization system, the at least one state variable is adjusted in accordance with an extent by which the system constraint is exceeded. The correction value is then recomputed based on the motion estimate and on the adjusted state variable.

Claims

1. A method for applying a motion filter of a video stabilization system to a sequence of video frames, said method comprising for a current video frame:

a processor or processors to perform the following;

receiving an estimate of a motion in said current video frame compared to a first video frame of said sequence of video frames;

computing a correction value for said motion in said current video frame based on said motion estimate and on at least one state variable of said motion filter, wherein said at least one state variable is updated in said computation;

in case said computed correction value exceeds a system constraint of said video stabilization system, adjusting said at least one state variable in accordance with an extent by which said system constraint is exceeded and recomputing said correction value based on said motion estimate and on said adjusted state variable.

2. The method according to claim 1, wherein said system constraint is related to a difference between a size of video frames input to said video stabilization system and a size of video frames output by said video stabilization system.

3. The method according to claim 1, wherein said computing of a correction value for said motion in said current video frame comprises using an extended Kalman filter and wherein said at least one state variable is a state variable of said extended Kalman filter, said extended Kalman filter being extended in that it implements the adjusting of said at least one state variable and the recomputing of said correction value.

5. The method according to claim 1, wherein said motion is at least one of a horizontal translation, a vertical translation and a rotation.

6. The method according to claim 5, wherein said correction value is computed and, if required, recomputed in sequence for a rotation and for a translation in horizontal and vertical direction, and wherein a system constraint for at least one of said types of motion is determined as a function of a correction value resulting for at least one other of said types of motion.

4. The method according to claim l, wherein said at least one state variable comprises a variable representing an intended motion in said current video frame and a variable representing an assumed velocity of said intended motion.

7. An apparatus comprising a video stabilization system including a motion filter, said motion filter comprising:

a processor configured to receive an estimate of a motion in a current video frame compared to a first video frame of a sequence of video frames and to compute a correction value for said motion in said current video frame based on said motion estimate and on at least one state variable of said motion filter, wherein said at least one state variable is updated in said computation; and

configured to adjust said at least one state variable, in case said computed correction value exceeds a system constraint of said video stabilization system, in accordance with an extent by which said system constraint is exceeded and to recompute a computed correction value based on said motion estimate and on said adjusted state variable.

8. The apparatus according to claim 7, wherein said processor configured to adjust said at least one state variable is configured to adjust a variable representing an intended motion in said current video frame and a variable representing an assumed velocity of said intended motion.

9. The apparatus according to claim 7, wherein said motion is at least one of a horizontal translation, a vertical translation and a rotation, wherein said processor is configured to compute and, if required, recompute a correction value in sequence for a rotation and for a translation in horizontal and vertical direction, and wherein said processor is configured to determine a system constraint for at least one of said types of motion as a function of a correction value resulting for at least one other of said types of motion.

10. The apparatus according to claim 7, further including a motion estimator configured to determine said motion estimate and a motion corrector configured to correct a motion in a respective video frame based on a correction value computed or recomputed by said motion filter.

11. The apparatus according to claim 7, wherein said electronic device comprises a video processing chipset including at least one chip, said video stabilization system being integrated in said chipset.

12. The apparatus according to claim 7, wherein said apparatus is one of a camera module and a mobile electronic device.

13. The apparatus according to claim 7, further comprising a video recording component configured to capture a sequence of video frames.

17. The apparatus according to claim 7, wherein said system constraint is related to a difference between a size of video frames input to said video stabilization system and a size of video frames output by said video stabilization system.

18. The apparatus according to claim 7, wherein said processor is configured to compute a correction value for said motion in said current video frame by using an extended Kalman filter and wherein said at least one state variable is a state variable of said extended Kalman filter, said extended Kalman filter being extended in that it implements the adjustment of said at least one state variable and the recomputation of said correction value.

19. The apparatus according to claim 7, wherein said motion is at least one of a horizontal translation, a vertical translation and a rotation.

14. A data processor readable medium embodied with software code for applying a motion filter of a video stabilization system to a sequence of video frames is stored, said software code realizing the following steps-when being executed by a data processor of an electronic device:

receiving an estimate of a motion in said current video frame compared to a first video frame of said sequence of video frames;

computing a correction value for said motion in said current video frame based on said motion estimate and on at least one state variable of said motion filtering, wherein said at least one state variable is updated in said computation; and

in case said computed correction value exceeds a system constraint of said video stabilization system, adjusting said at least one state variable in accordance with an extent by which said system constraint is exceeded and recomputing said correction value based on said motion estimate and on said adjusted state variable.

15. The data processor readable medium according to claim 14 wherein said at least one state variable comprises a variable representing an intended motion in said current video frame and a variable representing an assumed velocity of said intended motion.

16. The data processor readable medium according to claim 14 wherein said motion is at least one of a horizontal translation, a vertical translation and a rotation, and wherein said correction value is computed and, if required, recomputed in sequence for a rotation and for a translation in horizontal and vertical direction, and wherein a system constraint for at least one of said types of motion is determined as a function of a correction value resulting for at least one other of said types of motion.

20. An apparatus comprising a video stabilization system, said video stabilization system including:

means for receiving an estimate of a motion in a current video frame compared to a first video frame of a sequence of video frames;

means for computing a correction value for said motion in said current video frame based on said motion estimate and on at least one state variable, wherein said at least one state variable is updated in said computation;

means for adjusting said at least one state variable, in case said computed correction value exceeds a system constraint of said video stabilization system, in accordance with an extent by which said system constraint is exceeded and for recomputing said correction value based on said motion estimate and on said adjusted state variable.