METHOD FOR CORRECTING DISTORTION OF CAPTURED IMAGE, DEVICE FOR CORRECTING DISTORTION OF CAPTURED IMAGE, AND IMAGING DEVICE

• US 20070188619A1
• Filed: 02/09/2007
• Published: 08/16/2007
• Est. Priority Date: 02/13/2006
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kurata, Tohru
• Granted Patent: US 7,692,688 B2
• Time in Patent Office: Days
• US Class Current: 348/208.990
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A method for correcting distortion of a captured image is provided. The method includes receiving a detection output from a sensor; optically correcting distortion of a captured image in image data from the element due to the positional change of the imaging element by controlling a control mechanism depending on the detection output in the receiving step, the mechanism being configured to control a position at which incident light from a target object enters the imaging element; receiving image data resulting from the optical correction of a captured image in the correcting step from the element, and detecting a motion vector per one screen of a captured image from the image data; and further correcting distortion of a captured image due to the positional change of the imaging element for the image data from the imaging element based on the motion vector detected in the motion vector detecting step.

Claims

1. A method for correcting distortion of a captured image, the method comprising:

receiving a detection output from a sensor that detects a change corresponding to a positional change of an imaging element in a horizontal direction and/or a vertical direction of a captured image at the time of imaging;

optically correcting distortion of a captured image in image data from the imaging element due to the positional change of the imaging element by controlling a control mechanism depending on the detection output received from the sensor in the detection output receiving step, the control mechanism being configured to control a position at which incident light from a target object enters the imaging element;

receiving image data resulting from the optical correction of a captured image in the correcting step from the imaging element, and detecting a motion vector per one screen of a captured image from the image data; and

further correcting distortion of a captured image due to the positional change of the imaging element for the image data from the imaging element based on the motion vector detected in the motion vector detecting step.

2. The method for correcting distortion of a captured image according to claim 1, wherein

in the motion vector detecting step, the motion vector is detected from the image data from the imaging element with reference to a detection output from the sensor.

3. The method for correcting distortion of a captured image according to claim 1, wherein

when detecting the motion vector,

at least one target block that is composed of a plurality of pixels and has a predetermined size is defined at a predetermined position in each of divided image regions in an original screen,

a plurality of reference blocks each having the same size as the size of the target block are defined in a search range defined in a reference screen that is a screen of interest,

a reference block having a strong correlation with the target block is detected from the plurality of reference blocks, and

the motion vector is detected based on an amount of a position shift on a screen between the detected reference block and the target block.

4. The method for correcting distortion of a captured image according to claim 3, wherein

detecting the motion vector includes:

obtaining a sum of an absolute difference between a pixel value of each pixel in the reference block and a pixel value of a pixel at a corresponding position in the target block;

obtaining a reference reduced vector arising from reduction in a size of a reference vector by a predetermined reduction ratio, the reference vector indicating an amount and a direction of a position shift between a position of the reference block on the reference screen and a position of the target block on a screen;

creating a reduced sum-of-absolute-difference table in which values obtained based on a sum of an absolute difference about each of the reference blocks are stored as a plurality of table elements, the number of the table elements being obtained by reducing the number of the reference blocks depending on the predetermined reduction ratio, the table elements each having an address indicated by a reference vector with a size near or equivalent to a size of a reference reduced vector; and

calculating a motion vector that corresponds to a difference between the reference screen and the original screen for each of the divided image regions by using at least a reference vector corresponding to a minimum value among the values stored in the reduced sum of absolute difference table, and

creating the table includes;

detecting a plurality of reference vectors near to the reference reduced vector obtained in the reference reduced vector obtaining step, as a plurality of near reference vectors;

calculating distribution values, each corresponding to a respective one of the near reference vectors detected in the detecting step from the sum of an absolute difference calculated in the sum of absolute difference obtaining step; and

adding the distribution values calculated in the distribution value calculating step to previous values in the reduced SAD table each corresponding to a respective one of the near reference vectors.

5. The method for correcting distortion of a captured image according to claim 3, wherein

in the motion vector detecting step, the motion vector is detected after the search range is defined with reference to a detection output from the sensor.

6. The method for correcting distortion of a captured image according to claim 4, wherein

in the motion vector detecting step, the motion vector is detected after the search range is defined with reference to a detection output from the sensor.

7. A method for correcting distortion of a captured image, the method comprising:

receiving a detection output from a sensor that detects a positional change of an imaging element in a horizontal direction and/or a vertical direction of a captured image at the time of imaging;

optically correcting distortion of a captured image in image data from the imaging element due to the positional change of the imaging element by controlling a control mechanism depending on the detection output received from the sensor in the detection output receiving step, the control mechanism being configured to control a position at which incident light from a target object enters the imaging element; and

correcting distortion of a captured image due to the positional change of the imaging element with reference to a detection output from the sensor, for image data from the imaging element resulting from the optical correction of a captured image in the optical correcting step.

8. The method for correcting distortion of a captured image according to claim 7, wherein

when correcting the distortion, if it is determined based on the reference to the detection output from the sensor that the positional change of the imaging element has occurred, a blur is detected about the image data from the imaging element and correction is carried out so that the blur is removed.

9. A device for correcting distortion of a captured image, comprising:

a sensor configured to detect a positional change of an imaging element in a horizontal direction and/or a vertical direction of a captured image at the time of imaging;

an optical corrector configured to optically correct distortion of a captured image in image data from the imaging element due to the positional change of the imaging element by controlling a control mechanism depending on a detection output from the sensor, the control mechanism being configured to control a position at which incident light from a target object enters the imaging element;

a motion vector detector configured to receive image data resulting from the optical correction of a captured image from the imaging element and detect a motion vector per one screen of a captured image from the image data; and

an image distortion correction processor configured to further correct distortion of a captured image due to the positional change of the imaging element for the image data from the imaging element based on the motion vector detected by the motion vector detector.

10. An imaging device comprising:

a sensor configured to detect a positional change of an imaging element in a horizontal direction and/or a vertical direction of a captured image at the time of imaging;

an optical corrector configured to optically correct distortion of a captured image in image data from the imaging element due to the positional change of the imaging element by controlling a control mechanism depending on a detection output from the sensor, the control mechanism being configured to control a position at which incident light from a target object enters the imaging element;

a motion vector detector configured to receive image data resulting from the optical correction of a captured image from the imaging element and detect a motion vector per one screen of a captured image from the image data;

an image distortion correction processor configured to further correct distortion of a captured image due to the positional change of the imaging element for the image data from the imaging element based on the motion vector detected by the motion vector detector; and

a recorder configured to record image information of the captured image corrected by the image distortion correction processor in a recording medium.

11. The imaging device according to claim 10, wherein

the motion vector detector detects the motion vector from the image data from the imaging element with reference to a detection output from the sensor.

12. The imaging device according to claim 10, wherein

the motion vector detector calculates a motion vector that corresponds to a difference between a reference screen that is a screen of interest and an original screen previous to the reference screen,

at least one target block that is composed of a plurality of pixels and has a predetermined size is defined at a predetermined position in each of the divided image regions in the original screen, and a plurality of reference blocks each having the same size as the size of the target block are defined in a search range defined in the reference screen,

a reference block having a strong correlation with the target block is detected from the plurality of reference blocks, and

the motion vector is detected based on an amount of a position shift on a screen between the detected reference block and the target block.

13. The imaging device according to claim 12, wherein

the motion vector detector includes:

a sum-of-absolute-difference calculator that obtains a sum of an absolute difference between a pixel value of each pixel in the reference block and a pixel value of a pixel at a corresponding position in the target block;

a reference reduced vector obtainer that obtains a reference reduced vector arising from reduction in a size of a reference vector by a predetermined reduction ratio, the reference vector indicating an amount and a direction of a position shift between a position of the reference block on the reference screen and a position of the target block on a screen;

a table creator that creates a reduced sum of absolute difference table in which values obtained based on a sum of an absolute difference about each of the reference blocks are stored as a plurality of table elements, the number of the table elements being obtained by reducing the number of the reference blocks depending on the predetermined reduction ratio, the table elements each having an address indicated by a reference vector with a size near or equivalent to a size of a reference reduced vector; and

a motion vector calculator that calculates a motion vector corresponding to a difference between the reference screen and the original screen for each of the divided image regions by using at least a reference vector corresponding to a minimum value among the values stored in the reduced sum of absolute difference table, and wherein

the table creator includes;

a near reference vector detector that detects a plurality of reference vectors near to the reference reduced vector obtained by the reference reduced vector obtainer, as a plurality of near reference vectors;

a distribution value calculator that calculates distribution values each corresponding to a respective one of the near reference vectors from the sum of an absolute difference of the reference block, calculated by the sum of absolute difference calculator; and

a distribution adder that adds the distribution values each corresponding to a respective one of the near reference vectors, calculated by the distribution value calculator, to previous values in the reduced sum of absolute difference table each corresponding to a respective one of the near reference vectors.

14. The imaging device according to claim 12, wherein

the motion vector detector detects the motion vector after defining the search range with reference to a detection output from the sensor.

15. The imaging device according to claim 13, wherein

the motion vector detector detects the motion vector after defining the search range with reference to a detection output from the sensor.

16. An imaging device comprising:

a sensor configured to detect a positional change of an imaging element in a horizontal direction and/or a vertical direction of a captured image at the time of imaging;

an optical corrector configured to optically correct distortion of a captured image in image data from the imaging element due to the positional change of the imaging element by controlling a control mechanism depending on a detection output from the sensor, the control mechanism being configured to control a position at which incident light from a target object enters the imaging element; and

an image distortion correction processor configured to correct distortion of a captured image due to the positional change of the imaging element with reference to a detection output from the sensor, for image data from the imaging element resulting from the optical correction of a captured image by the optical corrector.

17. The imaging device according to claim 16, wherein

the image distortion correction processor detects a blur about the image data from the imaging element and carries out correction so that the blur is removed, if the image distortion correction processor has determined based on the reference to the detection output from the sensor that the positional change of the imaging element has occurred.