Image processing device, image processing method, and program for image processing

• US 8,798,387 B2
• Filed: 11/08/2011
• Issued: 08/05/2014
• Est. Priority Date: 11/11/2010
• Status: Expired due to Fees

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Yamada, Hitoshi, Ohmiya, Jun
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Dunphy, David F
• Application Number:
  US13/521,169
• Publication Number:
  US 20120281922A1
• Time in Patent Office: 1,001 Days
• Field of Search: None
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

An image processing device including: a motion amount estimating unit estimating a first motion amount indicating a displacement amount of a second picture with respect to a first picture; an area setting unit setting an area to be used for post-processing for each of the first picture and a first corrected picture, using feature points extracted from the first picture and the second picture; a post-processing unit generating a first cutout picture from the first picture and a second cutout picture from the first corrected picture and estimating a second motion amount indicating a displacement amount of the second cutout picture with respect to the first cutout picture; and a motion amount obtaining unit obtaining a motion amount indicating a displacement amount between the first picture and the second picture, from the estimated first motion amount and the estimated second motion amount.

Claims

1. An image processing device which corrects an amount of shake between pictures obtained by temporally-continuous capturing, the image processing device comprising:

a motion amount estimating unit configured to estimate a first shake amount indicating a rotational displacement amount of a second picture with respect to a first picture, the rotational displacement amount being obtained based on a combination of axis rotational directions of mutually perpendicular three axes, and the second picture being captured temporally after the first picture;

an area setting unit configured to set an area to be used for post-processing for each of the first picture and a first corrected picture, using feature points extracted from the first picture and the second picture, the first corrected picture being generated from the second picture whose displacement is corrected using the estimated first shake amount;

a post-processing unit configured to perform the post-processing, by generating a first cutout picture and a second cutout picture and estimating a second shake amount indicating a displacement amount of the second cutout picture with respect to the first cutout picture, the first cutout picture being generated from the area set in the first picture and the second cutout picture being generated from the area set in the first corrected picture;

a motion amount obtaining unit configured to obtain a shake amount indicating the amount of shake between the first picture and the second picture, from the estimated first shake amount and the estimated second shake amount; and

a picture correcting unit configured to correct the amount of shake between the first picture and the second picture using the shake amount obtained by the motion amount obtaining unit,

wherein the motion amount estimating unit includes;

a feature point extracting unit configured to extract the feature points from each of the first picture and the second picture;

a distortion correcting unit configured to correct distortion (i) developed due to optical strain of an optical system used for capturing and (ii) found between the feature points of the first picture and the feature points of the second picture, the feature points being extracted by the feature point extracting unit; and

a feature point matching unit configured to match the feature points of the first picture and the feature points of the second picture with each other to estimate the first shake amount, the first picture and the second picture having the distortion corrected by the distortion correcting unit.

2. The image processing device according to claim 1, wherein the post-processing unit includes:

a cutout picture generating unit configured to generate (i) the first cutout picture by cutting out the area set in the first picture and (ii) the second cutout picture by cutting out the area set in the first corrected picture; and

a cutout picture positioning unit configured to match the area of the first cutout picture and the area of the second cutout picture with each other to estimate the second shake amount of the second cutout picture with respect to the first cutout picture.

3. The image processing device according to claim 2,

wherein the pictures are captured by a fisheye optical system,

the motion amount estimating unit further includes a feature point coordinate transforming unit configured to transform coordinates of the feature points of the first picture and coordinates of the feature points of the second picture into coordinates to be used in a projection technique of the fisheye optical system, the feature points of the first picture and of the second picture being extracted by the feature point extracting unit, and

the feature point matching unit is configured to match the feature points of the first picture and the feature points of the second picture with each other to estimate the first shake amount indicating the displacement amount of the second picture with respect to the first picture, the first picture and the second picture having the coordinates transformed by the coordinate transforming unit.

4. The image processing device according to claim 3, further comprising a post-processing area coordinate transforming unit configured to transform coordinates of the area to be used for the post-processing into coordinates to be used in the projection technique of the fisheye optical system, the area to be used for the post-processing being set in each of the first picture and the first corrected picture by the area setting unit,

wherein the cutout picture generating unit is configured to generate (i) the first cutout picture by cutting out the area set in the first picture and (ii) the second cutout picture by cutting out the area set in the first corrected picture, the areas each of which being used for the post-processing and having the coordinates transformed by the post-processing area coordinate transforming unit.

5. The image processing device according to claim 1, wherein the area setting unit is configured to set the area to be used for the post-processing for each of the first picture and the first corrected picture, the area being set using feature points (i) found among the feature points extracted from the first picture and the second picture and (ii) used for estimating the first shake amount by the motion amount estimating unit.

6. The image processing device according to claim 5, wherein the area setting unit includes:

a corrected picture generating unit configured to generate the first corrected picture generated from the second picture whose displacement is corrected using the first shake amount estimated by the motion amount estimating unit;

a post-processing area setting unit configured to set an area for each of the first picture and the first corrected picture, using first feature points (i) found among the feature points extracted from the first picture and the second picture and (ii) used for estimating the first shake amount by the motion amount estimating unit; and

a determining unit configured to (i) determine whether or not a number of the first feature points, included in the area for the first corrected picture, is greater than or equal to a threshold value and (ii) set the area for the first corrected picture as the area to be used for the post-processing in a case where a result of the determination shows that the number is greater than or equal to the threshold value.

7. The image processing device according to claim 5, wherein the feature points used for estimating the first shake amount are an inlier.

8. The image processing device according to claim 1, further comprising a sensor estimating unit configured to measure either an angle of rotation about an axis of the optical system used for the capturing or an angle of rotation about at least one of two axes perpendicular to each other with respect to the axis of the optical system, and estimate the measured angle of rotation as the first shake amount of the second picture with respect to the first picture, the first shake amount being obtained based on the combination of the axis rotational directions of the mutually perpendicular three axes.

9. The image processing device according to claim 8, wherein the sensor estimating unit includes at least one of an angular acceleration sensor, an angular velocity sensor, an acceleration sensor, and a direction sensor.

10. The image processing device according to claim 8, wherein the motion amount estimating unit further includes a correcting technique determining unit configured to (i) determine the first shake amount, estimated by either the feature point matching unit or the sensor estimating unit, to be used for correcting the amount of shake between the pictures and (ii) provide the determined first shake amount to the area setting unit and the motion amount obtaining unit.

11. The image processing device according to claim 1, further comprising a motion amount determining unit configured to determine, before the setting by the area setting unit, whether or not the first shake amount is successfully calculated,

wherein, in a case where a result of the determination by the motion amount determining unit shows that the first shake amount is successfully calculated, the motion amount determining unit is configured to provide the first shake amount to the area setting unit, and

in a case where the result of the determination by the motion amount determining unit shows that the calculation of the first shake amount fails, the motion amount determining unit is configured to set the first shake amount to a zero value indicating that no displacement is found, provide the zero value to the area setting unit and the motion amount obtaining unit, and cause the motion amount obtaining unit to determine, as the zero value, the first shake amount indicating the displacement amount between the first picture and the second picture.

12. The image processing device according to claim 1, further comprising a post-processing determining unit configured to determine, after the post-processing by the post-processing unit, whether or not the second shake amount is successfully calculated,

wherein, in a case where a result of the determination by the post-processing determining unit shows that the second shake amount is successfully calculated, the post-processing determining unit is configured to provide the second shake amount to the motion amount obtaining unit, and

in a case where the result of the determination by the post-processing determining unit shows that the calculation of the second shake amount fails, the post-processing determining unit is configured to set the second shake amount to a zero value indicating that no displacement is found, and provide the zero value to the motion amount obtaining unit.

13. An image processing method for correcting an amount of shake between pictures obtained by temporally-continuous capturing, the image processing device comprising:

estimating a first shake amount indicating a rotational displacement amount of a second picture with respect to a first picture, the rotational displacement amount being obtained based on a combination of axis rotational directions of mutually perpendicular three axes, and the second picture being captured temporally after the first picture;

setting an area to be used for post-processing for each of the first picture and a first corrected picture, using feature points extracted from the first picture and the second picture, the first corrected picture being generated from the second picture whose displacement is corrected using the estimated first shake amount;

performing the post-processing, by generating a first cutout picture and a second cutout picture and estimating a second shake amount indicating a displacement amount of the second cutout picture with respect to the first cutout picture, the first cutout picture being generated from the area set in the first picture and the second cutout picture being generated from the area set in the first corrected picture;

obtaining a shake amount indicating the amount of shake between the first picture and the second picture, from the estimated first shake amount and the estimated second shake amount; and

correcting the amount of shake between the first picture and the second picture using the shake amount obtained by the obtaining of the shake amount,

wherein the estimating includes;

extracting the feature points from each of the first picture and the second picture;

correcting distortion (i) developed due to optical strain of an optical system used for capturing and (ii) found between the feature points of the first picture and the feature points of the second picture, the feature points being extracted in the extracting; and

matching the feature points of the first picture and the feature points of the second picture with each other to estimate the first shake amount, the first picture and the second picture having the distortion corrected in the correcting.

14. A non-transitory computer-readable recording medium having a program recorded thereon, the program for causing a computer to correct an amount of shake between pictures obtained by temporally-continuous capturing, the computer correcting the amount of shake by executing a method comprising:

estimating a first shake amount indicating a rotational displacement amount of a second picture with respect to a first picture, the rotational displacement amount being obtained based on a combination of axis rotational directions of mutually perpendicular three axes, and the second picture being captured temporally after the first picture;

setting an area to be used for post-processing for each of the first picture and a first corrected picture, using feature points extracted from the first picture and the second picture, the first corrected picture being generated from the second picture whose displacement is corrected using the estimated first shake amount;

performing the post-processing, by generating a first cutout picture and a second cutout picture and estimating a second shake amount indicating a displacement amount of the second cutout picture with respect to the first cutout picture, the first cutout picture being generated from the area set in the first picture and the second cutout picture being generated from the area set in the first corrected picture;

obtaining a shake amount indicating the amount of shake between the first picture and the second picture, from the estimated first shake amount and the estimated second shake amount; and

correcting the amount of shake between the first picture and the second picture using the shake amount obtained by obtaining of the shake amount,

wherein the estimating includes;

extracting the feature points from each of the first picture and the second picture;

correcting distortion (i) developed due to optical strain of an optical system used for capturing and (ii) found between the feature points of the first picture and the feature points of the second picture, the feature points being extracted in the extracting; and

matching the feature points of the first picture and the feature points of the second picture with each other to estimate the first shake amount, the first picture and the second picture having the distortion corrected in the correcting.