PICTURE PROCESSING DEVICE, PICTURE PROCESSING METHOD, PROGRAM FOR PICTURE PROCESSING, AND IMAGING DEVICE

• US 20120281104A1
• Filed: 11/08/2011
• Published: 11/08/2012
• Est. Priority Date: 11/11/2010
• Status: Active Grant

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Ohmiya, Jun, Yamada, Hitoshi
• Granted Patent: US 8,681,230 B2
• Time in Patent Office: Days
• US Class Current: 348/208.6
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

The picture processing device of the present invention includes: a rotational motion distance calculating unit which calculates a rotational motion distance of a second picture with respect to a first picture; a cutout picture generating unit which (i) generates a first cutout picture from the first picture, and (ii) and generates a second cutout picture from a first corrected picture, the first corrected picture being generated from the second picture whose rotational displacement amount, in a combination of directions in three axes, is corrected; a parallel motion amount calculating unit which calculates a parallel motion amount of a second cutout picture with respect to the first cutout picture; and a motion amount determining unit which determines a motion amount of the second picture with respect to the first picture.

Claims

1. A picture processing device which corrects displacement between pictures obtained by temporally-continuous capturing, said picture processing device comprising:

a rotational motion amount calculating unit configured to calculate a rotational motion amount indicating a rotational displacement amount of a second picture with respect to a first picture, the rotational displacement amount being obtained based on a combination of axis rotational directions of mutually perpendicular three axes, and the second picture being captured temporally after the first picture;

a cutout picture generating unit configured to generate a first cutout picture from the first picture and a second cutout picture from the second picture;

a parallel motion amount calculating unit configured to calculate a parallel motion amount indicating a displacement amount of the second cutout picture with respect to the first cutout picture;

a motion amount determining unit configured to determine, based on the calculated rotational motion amount and the calculated parallel motion amount, a motion amount indicating a displacement amount of the second picture with respect to the first picture; and

a correcting unit configured to correct the displacement amount of the second picture with respect to the first picture, based on the motion amount,

wherein said cutout picture generating unit is configured to (i) cut out from the first picture a predetermined first area, and generate the first cutout picture by performing, based on characteristics of an optical system used for capturing, distortion correction on the predetermined first area which is cut out, and (ii) cut out from a first corrected picture a second area corresponding to the predetermined first area, and generate the second cutout picture by performing the distortion correction on the corresponding second area which is cut out, the first corrected picture being generated from the second picture whose rotational displacement amount, in the combination of the three-axis rotational directions, is corrected based on the calculated rotational motion amount.

2. The picture processing device according to claim 1,

wherein said parallel motion amount calculating unit is configured to calculate, based on mutually-perpendicular three axes with respect to the first cutout picture, the parallel displacement amount of the second cutout picture with respect to the first cutout picture so as to obtain the parallel motion amount, the parallel displacement amount extending in at least one of directions in the mutually-perpendicular three axes.

3. The picture processing device according to claim 2,

wherein said parallel motion amount calculating unit is configured to calculate the parallel motion amount of the second cutout picture with respect to the first cutout picture as a value indicating a magnification percentage or a reduction percentage of the second cutout picture with respect to the first cutout picture, using one of the three axes as an axis perpendicular to the first cutout picture, the parallel motion amount extending in a direction of the axis perpendicular to the first cutout picture.

4. The picture processing device according to claim 2,

wherein said parallel motion amount calculating unit is configured to calculate the parallel motion amount of the second cutout picture by one of pattern matching scheme, object matching, and feature point matching between the first cutout picture and the second cutout picture.

5. The picture processing device according to claim 1,

wherein said parallel motion amount calculating unit includes

a first corrected picture generating unit configured to generate the first corrected picture from the second picture whose rotational displacement amount in the combination of the three-axis rotational directions is corrected based on the calculated rotational motion amount, and

said first corrected picture generating unit further includes a coordinate transformation table generating unit configured to generate, based on the calculated rotational motion amount, a coordinate transformation table indicating a correspondence between each of pixels in the first corrected picture and an associated one of the pixels in the second picture, and is configured to (i) cause said coordinate transformation table generating unit to generate the coordinate transformation table, and (ii) generate the first corrected picture using the coordinate transformation table generated by said coordinate transformation table generating unit.

6. The picture processing device according to claim 1,

wherein said cutout picture generating unit is

configured to generate perspective projection pictures for the predetermined first area and for the corresponding second area for performing the distortion correction on the predetermined first area and the corresponding second area so as to generate the first cutout picture and the second cutout picture.

7. The picture processing device according to claim 1, further comprising

a rotational motion amount determining unit configured to determine, before the calculation by said parallel motion amount calculating unit, whether or not the rotational motion amount is successfully calculated,

wherein in the case where a result of the determination shows that the rotational motion amount is successfully calculated, said rotational motion amount determining unit is configured to provide the rotational motion amount to said parallel motion amount calculating unit, and

in the case where the result of the determination shows that calculation of the rotational motion amount fails, said rotational motion amount determining unit is configured to (i) provide, to said parallel motion amount calculating unit and said motion amount determining unit, the failed rotational motion amount as a zero value indicating that no displacement is found in the combination of the three-axis rotational directions, and (ii) cause said motion amount determining unit to determine the zero value as the motion amount indicating the displacement amount of the second picture with respect to the first picture.

8. The picture processing device according to claim 1, further comprising

a parallel motion amount determining unit configured to determine, after the calculation by said parallel motion amount calculating unit, whether or not the parallel motion amount is successfully calculated,

wherein in the case where a result of the determination shows that the parallel motion amount is successfully calculated, said parallel motion amount determining unit is configured to provide the parallel motion amount to said motion amount determining unit, and

in the case where the result of the determination shows that calculation of the parallel motion amount fails, said parallel motion amount determining unit is configured to provide, to said motion amount determining unit, the parallel motion amount as a zero value indicating that no displacement is found in the parallel direction.

9. The picture processing device according to claim 1,

wherein said rotational motion amount calculating unit is configured to calculate the rotational motion amount of the second picture with respect to the first picture through a calculation of (i) a rotation angle of a light axis of the optical system or (ii) a rotation angle of at least one of two axes mutually perpendicular to the light axis of the optical system.

10. The picture processing device according to claim 9,

wherein said rotational motion amount calculating unit includes at least one sensor which measures the rotation angle.

11. The picture processing device according to claim 10,

wherein said sensor is at least one of an angular acceleration sensor, an angular velocity sensor, an acceleration sensor, and a direction sensor.

12. The picture processing device according to claim 1,

wherein said rotational motion amount calculating unit includes:

a feature point extracting unit configured to extract a feature point from each of the first picture and the second picture;

a distortion removing unit configured to remove distortion due to optical strain caused by the optical system and found between the feature point of the first picture and the feature point of the second picture;

a feature point matching unit configured to match the feature point of the first picture with the feature point of the second picture, the first picture and the second picture having the optical distortion removed by said distortion removing unit; and

an error minimizing unit configured to calculate the rotational motion amount based on a pair of (i) the feature point of the first picture and (ii) the feature point, of the second picture, corresponding to the feature point of the first picture, so that said error minimizing unit minimizes an error made when the displacement found in the second picture and extending in the combination of the three-axis rotational directions is to be corrected, the feature point of the first picture and the feature point of the second picture being successfully matched with each other by said feature point matching unit.

13. The picture processing device according to claim 1,

wherein said motion amount determining unit is configured to determine, as the motion amount, a combined motion amount having the calculated rotational motion amount and the calculated parallel motion amount combined with each other.

14. The picture processing device according to claim 13, further comprising:

a corrected picture generating unit configured to generate, based on the motion amount determined by said motion amount determining unit, a corrected picture from the second picture whose displacement is corrected; and

a memory which stores the picture generated by said corrected picture generating unit.

15. The picture processing device according to claim 14, further comprising:

a picture data obtaining unit configured to obtain a coded picture; and

a decoding unit configured to decode the coded picture obtained by said picture data obtaining unit, extract continuous picture data, and sequentially provide the continuous picture data to said first motion amount calculating unit,

wherein when the coded picture includes picture data, audio data, and sensor information, said decoding unit is configured to separate the coded picture into the picture data, the audio data, and the sensor information one another, and hold a synchronous relationship between the extracted continuous picture data and each of the picture data, the audio data, and the sensor information.

16. An imaging device comprising:

said picture processing device according to claim 1;

a lens unit configured to gather light;

an image sensor which records the light gathered by said lens unit and generates a picture from the recorded light, and provides the generated picture to said first motion amount calculating unit in said picture processing device;

a corrected picture generating unit configured to generate a corrected picture from the second picture whose displacement is corrected, the corrected picture being generated based on the motion amount determined by said motion amount determining unit included in said picture processing device;

an encoder which encodes an incoming picture to generate a coded picture; and

a selecting unit configured to select at least one of the picture generated by said image sensor and the picture generated by said corrected picture generating unit, and provide the one selected picture to said encoder.

17. The imaging device according to claim 16, further comprising

a sensor which measures a motion of said imaging device,

wherein said sensor measures a motion of said imaging device by synchronizing the motion with a picture captured by said imaging device, and provides, to said first motion amount calculating unit, the measured motion and the picture corresponding to the measured motion.

18. A picture processing method for correcting displacement between pictures obtained by temporally-continuous capturing, said picture processing method comprising:

calculating a rotational motion amount indicating a rotational displacement amount of a second picture with respect to a first picture, the rotational displacement amount being obtained based on a combination of axis rotational directions of mutually perpendicular three axes, and the second picture being captured temporally after the first picture;

generating a first cutout picture from the first picture and a second cutout picture from the second picture;

calculating a parallel motion amount indicating a displacement amount of the second cutout picture with respect to the first cutout picture;

determining, based on the calculated rotational motion amount and the calculated parallel motion amount, a motion amount indicating a displacement amount of the second picture with respect to the first picture; and

correcting the displacement amount of the second picture with respect to the first picture, based on the determined motion amount, wherein said generating involves (i) cutting out from the first picture a predetermined first area, and generating the first cutout picture by performing, based on characteristics of an optical system used for capturing, distortion correction on the predetermined first area which is cut out, and (ii) cutting out from a first corrected picture a second area corresponding to the predetermined first area, and generating the second cutout picture by performing the distortion correction on the corresponding second area which is cut out, the first corrected picture being generated from the second picture whose rotational displacement amount, in the combination of the three-axis rotational directions, is corrected based on the calculated rotational motion amount.

19. A non-transitory computer-readable recording medium for use in a computer, said recording medium having a computer program recorded thereon, the computer program causing the computer to correct displacement between pictures obtained by temporally-continued capturing, and the program causing the computer to execute:

calculating a rotational motion amount indicating a rotational displacement amount of a second picture with respect to a first picture, the rotational displacement amount being obtained based on a combination of axis rotational directions of mutually perpendicular three axes, and the second picture being captured temporally after the first picture;

generating a first cutout picture from the first picture and a second cutout picture from the second picture;

calculating a parallel motion amount indicating a displacement amount of the second cutout picture with respect to the first cutout picture;

determining, based on the calculated rotational motion amount and the calculated parallel motion amount, a motion amount indicating a displacement amount of the second picture with respect to the first picture; and

correcting the displacement amount of the second picture with respect to the first picture, based on the determined motion amount,

wherein said generating involves (i) cutting out from the first picture a predetermined first area, and generating the first cutout picture by performing, based on characteristics of an optical system used for capturing, distortion correction on the predetermined first area which is cut out, and (ii) cutting out from a first corrected picture a second area corresponding to the predetermined first area, and generating the second cutout picture by performing the distortion correction on the corresponding second area which is cut out, the first corrected picture being generated from the second picture whose rotational displacement amount, in the combination of the three-axis rotational directions, is corrected based on the calculated rotational motion amount.

20. An integrated circuit for correcting displacement between pictures obtained by temporally-continued capturing, said integrated circuit comprising:

a rotational motion amount calculating unit configured to calculate a rotational motion amount indicating a rotational displacement amount of a second picture with respect to a first picture, the rotational displacement amount being obtained based on a combination of axis rotational directions of mutually perpendicular three axes, and the second picture being captured temporally after the first picture;

a cutout picture generating unit configured to generate a first cutout picture from the first picture and a second cutout picture from the second picture;

a parallel motion amount calculating unit configured to calculate a parallel motion amount indicating a displacement amount of the second cutout picture with respect to the first cutout picture;

a motion amount determining unit configured to determine, based on the calculated rotational motion amount and the calculated parallel motion amount, a motion amount indicating a displacement amount of the second picture with respect to the first picture; and

a correcting unit configured to correct the displacement amount of the second picture with respect to the first picture, based on the motion amount,

wherein said cutout picture generating unit is configured to (i) cut out from the first picture a predetermined first area, and generate the first cutout picture by performing, based on characteristics of an optical system used for capturing, distortion correction on the predetermined first area which is cut out, and (ii) cut out from a first corrected picture a second area corresponding to the predetermined first area, and generate the second cutout picture by performing the distortion correction on the corresponding second area which is cut out, the first corrected picture being generated from the second picture whose rotational displacement amount, in the combination of the three-axis rotational directions, is corrected based on the calculated rotational motion amount.