Image stabilizing apparatus, image-pickup apparatus and image stabilizing method

• US 7,929,043 B2
• Filed: 04/03/2008
• Issued: 04/19/2011
• Est. Priority Date: 04/06/2007
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Saito, Mitsuhiro, Oshino, Takahiro, Tsubaki, Hidetoshi
• Primary Examiner(s): Hannett; James M
• Application Number:
  US12/062,007
• Publication Number:
  US 20080246848A1
• Time in Patent Office: 1,111 Days
• Field of Search: 348/348, 348/208.1, 348/208.6
• US Class Current: 348/348
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

An image stabilizing apparatus includes a motion vector calculating part that calculates a motion vector between a plurality of images including a displacement caused by a motion of an image-pickup apparatus, a shake-correction parameter calculating part that receives the motion vector as input to calculate a shake correction amount, and an image transforming part that performs geometric transformation of the image in accordance with the shake correction amount. The shake-correction parameter calculating part performs variation amount calculation, variation amount correction and correction amount calculation based on the motion information between the plurality of images. The image stabilizing apparatus preserves a motion in video from an intended camera work and allows image stabilization for an unintended shake.

Claims

1. An image stabilizing apparatus comprising:

a motion vector calculating part that calculates a motion vector between a plurality of images including a displacement caused by a motion of an image-pickup apparatus;

a shake-correction parameter calculating part that receives the motion vector as input to calculate a shake correction amount; and

an image transforming part that performs geometric transformation of the image in accordance with the shake correction amount,

wherein the shake-correction parameter calculating part includes;

a variation amount calculating part that calculates an image variation amount between the plurality of images based on the motion vector;

a variation amount correcting part that calculates, based on the image variation amount, motion information in which a component distorting the image is excluded; and

a correction amount calculating part that calculates the shake correction amount based on the motion information between the plurality of images.

2. The image stabilizing apparatus according to claim 1, wherein the image variation amount is a homography, and

wherein the shake-correction parameter calculating part changes a direction of a normal to a reference plane of the homography.

3. The image stabilizing apparatus according to claim 2, wherein the shake-correction parameter calculating part changes the direction of the normal to the reference plane such that the normal is directed in parallel with a normal to an image plane before the displacement.

4. The image stabilizing apparatus according to claim 1, wherein the image variation amount is associated with rotation and translation, and

wherein the shake-correction parameter calculating part calculates the motion information based on information on the rotation and translation.

5. The image stabilizing apparatus according to claim 4, wherein the shake-correction parameter calculating part changes a reference plane such that the reference plane is directed in parallel with an image plane before the displacement.

6. The image stabilizing apparatus according to claim 1, wherein the correction amount calculating part performs filtering for outputting the shake correction amount based on the motion information.

7. The image stabilizing apparatus according to claim 1, wherein the variation amount calculating part selects, as the image variation amount to be calculated, one of a homography and an image variation amount associated with rotation and translation by investigating planarity of a spatial distribution of detection points of a plurality of the motion vectors.

8. The image stabilizing apparatus according to claim 1, wherein the shake-correction amount calculating part transforms the motion information into components of horizontal translation, vertical translation, scaling, in-plane rotation, shear, horizontal foreshortening, and vertical foreshortening, and

wherein the shake-correction amount calculating part performs logarithmic transformation of the scaling component and then filtering thereon, and performs exponential transformation of the filtering result to restore the image variation amount.

9. The image stabilizing apparatus according to claim 1, wherein the shake-correction amount calculating part transforms the motion information into components of horizontal translation, vertical translation, scaling, in-plane rotation, shear, horizontal foreshortening, and vertical foreshortening, and

wherein the shake-correction amount calculating part multiplies the foreshortening component by the scaling component and then performs filtering on the scaling component as the product thereof, and uses the filtering result to restore the image variation amount.

10. An image-pickup apparatus comprising:

an image-pickup system that photoelectrically converts an object image to produce a plurality of frame images constituting video; and

the image stabilizing apparatus according to claim 1.

11. An image stabilizing apparatus comprising:

a motion vector calculating part that calculates a motion vector that represents displacement of corresponding feature portions between a plurality of images, the displacement being caused by a motion of an image-pickup apparatus;

a shake-correction parameter calculating part that receives the motion vector as input to calculate a shake correction amount; and

an image transforming part that performs geometric transformation of the image in accordance with the shake correction amount,

wherein the shake-correction parameter calculating part includes;

a variation amount calculating part that calculates an image variation amount between the plurality of images based on the motion vector,

the image variation amount is associated with rotation and translation;

a variation amount correcting part that calculates motion information based on the image variation amount between the plurality of images; and

a correction amount calculating part that calculates the shake correction amount based on the motion information,

wherein the shake-correction parameter calculating part calculates a homography based on information on the rotation and translation.

12. An image-pickup apparatus comprising:

an image-pickup system that photoelectrically converts an object image to produce a plurality of frame images constituting video; and

the image stabilizing apparatus according to claim 11.

13. An image stabilizing apparatus comprising:

a motion vector calculating part that calculates a motion vector that represents displacement of corresponding feature portions between a plurality of images, the displacement being caused by a motion of an image-pickup apparatus;

a shake-correction parameter calculating part that receives the motion vector as input to calculate a shake correction amount; and

an image transforming part that performs geometric transformation of the image in accordance with the shake correction amount,

wherein the shake-correction parameter calculating part includes;

a variation amount calculating part that calculates an image variation amount between the plurality of images based on the motion vector, the image variation amount is a fundamental matrix;

a variation amount correcting part that calculates motion information based on the image variation amount between the plurality of images; and

a correction amount calculating part that calculates the shake correction amount based on the motion information,

wherein the shake-correction parameter calculating part calculates the motion information based on the fundamental matrix.

14. An image-pickup apparatus comprising:

an image-pickup system that photoelectrically converts an object image to produce a plurality of frame images constituting video; and

the image stabilizing apparatus according to claim 13.

15. An image-stabilizing method comprising:

a motion vector calculating step of calculating a motion vector between a plurality of images including a displacement caused by a motion of an image-pickup apparatus;

a shake-correction parameter calculating step of receiving the motion vector as input to calculate a shake correction amount; and

an image transforming step of performing geometric transformation of the image in accordance with the shake correction amount,

wherein the shake-correction parameter calculating step includes;

a variation amount calculating step of calculating an image variation amount between the plurality of images based on the motion vector;

a variation amount correcting step of calculating, based on the image variation amount, motion information in which a component distorting the image is excluded; and

a correction amount calculating step of calculating the shake correction amount based on the motion information between the plurality of images.

16. A storage media storing a computer program that causes a computer to perform processing corresponding to the image-stabilizing method according to claim 15.

17. An image-stabilizing method comprising:

a motion vector calculating step of calculating a motion vector that represents displacement of corresponding feature portions between a plurality of images, the displacement being caused by a motion of an image-pickup apparatus;

a shake-correction parameter calculating step of receiving the motion vector as input to calculate a shake correction amount; and

an image transforming step of performing geometric transformation of the image in accordance with the shake correction amount,

wherein the shake-correction parameter calculating step includes;

a variation amount calculating step of calculating an image variation amount between the plurality of images based on the motion vector,

the image variation amount is associated with rotation and translation;

a variation amount correcting step of calculating motion information based on the image variation amount between the plurality of images; and

a correction amount calculating step of calculating the shake correction amount based on the motion information,

wherein the shake-correction parameter calculating part calculates a homography based on information on the rotation and translation.

18. A storage media storing a computer program that causes a computer to perform processing corresponding to the image-stabilizing method according to claim 17.

19. An image-stabilizing method comprising:

a motion vector calculating step of calculating a motion vector that represents displacement of corresponding feature portions between a plurality of images, the displacement being caused by a motion of an image-pickup apparatus;

a shake-correction parameter calculating step of receiving the motion vector as input to calculate a shake correction amount; and

an image transforming step of performing geometric transformation of the image in accordance with the shake correction amount,

wherein the shake-correction parameter calculating step includes;

a variation amount calculating step of calculating an image variation amount between the plurality of images based on the motion vector, the image variation amount is a fundamental matrix;

a variation amount correcting step of calculating motion information based on the image variation amount between the plurality of images; and

a correction amount calculating step of calculating the shake correction amount based on the motion information;

wherein the shake-correction parameter calculating part calculates the motion information based on the fundamental matrix.

20. A storage media storing a computer program that causes a computer to perform processing corresponding to the image-stabilizing method according to claim 19.