Image processing device, focal plane distortion component calculation method, image processing program, and recording medium

• US 20100214423A1
• Filed: 02/12/2010
• Published: 08/26/2010
• Est. Priority Date: 02/19/2009
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ogawa, Nobuhiro
• Granted Patent: US 8,363,116 B2
• Time in Patent Office: Days
• US Class Current: 348/208.400
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations

Abstract

An image processing device includes: a motion vector receiving unit configured to receive, from frame image data made up of frame images, motion vectors representing motion of the frame images; a modeling unit configured to model the motion vector, received from the motion vector receiving unit, to a component separation expression in which a camera motion component and a focal plane distortion component are separated, using component parameters respectively representing camera motion which is motion of a camera, and the amount in change in focal plane distortion; and a component calculation unit configured to calculate the component parameters used in the component separation expression, thereby calculating the focal plane distortion component in the motion vector.

Claims

1. An image processing device, comprising:

a motion vector receiving unit configured to receive, from frame image data made up of frame images, motion vectors representing motion of said frame images;

a modeling unit configured to model said motion vector, received from said motion vector receiving unit, to a component separation expression in which a camera motion component and a focal plane distortion component are separated, using component parameters respectively representing camera motion which is motion of a camera, and the amount in change in focal plane distortion; and

a component calculation unit configured to calculate said component parameters used in said component separation expression, thereby calculating said focal plane distortion component in said motion vector.

2. The image processing device according to claim 1, wherein said modeling unit expresses said component separation expression as a matrix expression.

3. The image processing device according to claim 2, wherein said modeling unit expresses said camera motion component as

a translational element representing translational speed in the vertical direction and horizontal direction as to said frame image,

an enlargement/reduction element representing enlargement/reduction speed in an enlargement/reduction direction, and

a rotational element representing angular change in one direction.

4. The image processing device according to claim 3, wherein said modeling unit

multiplies said component separation expression by

an origin correction matrix for moving the origin to the center, and

an aspect ratio correction matrix for changing the aspect ratio of pixels to 1:

1,

before said rotational element, and

multiplies said component separation expression by

an origin correction inverse matrix for moving the origin from the center back to the original position, and

an aspect ratio correction inverse matrix for changing the aspect ratio of pixels back to the original aspect ratio,

after said rotational element.

5. The image processing device according to claim 2, wherein said modeling unit models said focal plane distortion component as in the following Expression

6. The image processing device according to claim 5, wherein said modeling unit models said camera motion component as in the following Expression

7. The image processing device according to claim 6, wherein said modeling unit models said motion vector as in the following Expression

8. The image processing device according to claim 1, further comprising:

a correction unit configured to correct focal plane distortion in said frame image, based on said focal plane distortion component calculated by said component calculation unit.

9. The image processing device according to claim 8, wherein said modeling unit expresses said focal plane distortion component as an FP distortion parallelogram element and an FP distortion vertical direction enlargement/reduction element.

10. The image processing device according to claim 9, said correction unit including:

a correction amount calculation unit configured to calculate a focal plane distortion correction amount representing the amount of correction regarding said focal plane distortion performed as to said frame image, based on said focal plane distortion component calculated by said component calculation unit;

a correction vector generating unit configured to generate a correction vector for correcting said focal plane distortion in said motion vector, based on said focal plane distortion correction amount calculated by said correction amount calculation unit; and

a motion compensation unit configured to apply said correction vector generated by said correction vector generating unit to a processing frame image to be processed out of said frame images.

11. The image processing device according to claim 10, wherein said correction amount calculation unit selects, from said processing frame image and frame images immediately prior to and following said processing frame image of said frame images, a frame image where said focal plane distortion is the smallest, and calculates said focal plane distortion such that said focal plane distortion is at the same level of that of said selected frame image.

12. The image processing device according to claim 11,

wherein said modeling unit expresses said camera motion component as at least

a translational element and

an enlargement/reduction element;

and wherein said correction amount calculation unit selects a frame image with the smallest focal plane distortion, based on said translational element.

13. The image processing device according to claim 12,

wherein said modeling unit expresses said translational element as translational speed in the vertical direction and the horizontal direction;

and wherein said correction amount calculation unit

calculates said focal plane distortion correction amount corresponding to said FP distortion vertical direction enlargement/reduction element, based on said vertical direction translational speed, and

calculates said focal plane distortion correction amount corresponding to said FP distortion parallelogram element, based on said horizontal direction translational speed.

14. The image processing device according to claim 13, wherein, in the event that said frame image has been imaged with the global shutter method, said correction amount calculation unit replaces said focal plane distortion with a unit matrix.

15. The image processing device according to claim 14,

wherein said modeling unit expresses said camera motion component as at least

a translational element and

an enlargement/reduction element;

and wherein said correction amount calculation unit takes an accumulation value of said focal plane component as said focal plane distortion correction amount;

and wherein, in the event that the absolute value of said translational element is below a reset threshold, said correction amount calculation unit resets the accumulation value of said focal plane component.

16. The image processing device according to claim 15,

wherein said modeling unit expresses said camera motion component as at least

a translational element and

an enlargement/reduction element;

and wherein said correction amount calculation unit takes said translational element and

an accumulation value of said focal plane component as said focal plane distortion correction amount;

and wherein said correction amount calculation unit reduces or eliminates the effects of said focal plane component regarding which correlation with said translational element is small.

17. The image processing device according to claim 8, further comprising:

a motion vector reliability generating unit configured to generate reliability information representing the reliability of said motion vector;

and wherein said compensation amount calculation unit generates said focal plane distortion correction amount using only focal plane components regarding which the reliability of said vector is high, based on said reliability information generated by said motion vector reliability generating unit.

18. A focal plane distortion component calculation method, comprising the steps of:

receiving, from frame image data made up of frame images, motion vectors representing motion of said frame images;

modeling said motion vector, received in said receiving, to a component separation expression in which a camera motion component and a focal plane distortion component are separated, using component parameters respectively representing camera motion which is motion of a camera, and the amount in change in focal plane distortion; and

component separation expression, thereby calculating said focal plane distortion component in said motion vector.

19. An image processing program causing a computer to execute the steps of:

receiving, from frame image data made up of frame images, motion vectors representing motion of said frame images;

modeling said motion vector, received in said receiving, to a component separation expression in which a camera motion component and a focal plane distortion component are separated, using component parameters respectively representing camera motion which is motion of a camera, and the amount in change in focal plane distortion; and

calculating said component parameters used in said component separation expression, thereby calculating said focal plane distortion component in said motion vector.

20. A recording medium storing an image processing program causing a computer to execute the steps of:

receiving, from frame image data made up of frame images, motion vectors representing motion of said frame images;

modeling said motion vector, received in said receiving, to a component separation expression in which a camera motion component and a focal plane distortion component are separated, using component parameters respectively representing camera motion which is motion of a camera, and the amount in change in focal plane distortion; and

calculating said component parameters used in said component separation expression, thereby calculating said focal plane distortion component in said motion vector.