Image processing device, image processing method, and program

• US 8,417,059 B2
• Filed: 06/03/2008
• Issued: 04/09/2013
• Est. Priority Date: 06/28/2007
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Yamada, Hitoshi
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US12/375,726
• Publication Number:
  US 20090290809A1
• Time in Patent Office: 1,771 Days
• Field of Search: 382/190, 382/199, 382/242, 382/236, 382/289, 382/294, 382/297, 382/107, 382282-283, 348154-155, 348/352, 348/699, 348/208.4, 348/208.13, 348/208.16, 348/402.1, 348/413.1, 348/416.1
• US Class Current: 382/283
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations

Abstract

A motion vector calculation unit calculates inter-frame movement amounts. A masked region specification unit separates the entire edge image of frame feature data into (i) an edge image showing relatively large movement amounts and (ii) an edge image showing relatively small movement amounts. The masked region specification unit then specifies the edge image showing relatively large movement amounts as a region to be masked. This way, a correction parameter is detected from a region other than the masked region. When the correction parameter is a slang angle, a slant correction unit performs slant correction on a frame picture obtained by an image sensor.

Claims

1. An image processing device for correcting frame picture data in accordance with a correction parameter, comprising:

a generation unit that generates frame feature data indicating features of an edge image shown in the frame picture data;

a calculation unit that calculates, from each of pixel blocks constituting the frame picture data, a motion vector indicating a movement between the frame picture data and previous frame picture data;

a region specification unit that, in accordance with motion vectors calculated from the pixel blocks, specifies a region to be masked in the frame feature data; and

a detection unit that detects the correction parameter from an unmasked region of the frame feature data,

wherein the region to be masked corresponds to (i) a pixel block whose motion vector indicates a movement amount larger than a predetermined threshold, and (ii) a pixel block whose motion vector indicates a different directional component than a predetermined directional component, and

wherein the predetermined directional component is a direction shown by a pixel block that is proportional to a counterpart pixel block thereof included in the previous frame picture data.

2. The image processing device of claim 1,

wherein the predetermined threshold is an average value of movement amounts that are respectively indicated by the motion vectors.

3. The image processing device of claim 2,

wherein the pixel block whose motion vector indicates the movement amount larger than the predetermined threshold corresponds to a foreground edge image of the frame feature data,

wherein the region specification unit further separates the frame feature data into the foreground edge image and a background edge image in accordance with the movement amounts, and

wherein the region to be masked by the region specification unit includes the foreground edge image and/or the background edge image.

4. The image processing device of claim 1,

wherein the detection unit further (i) generates a histogram for each of a foreground edge image and a background edge image included in the frame feature data, excluding the region to be masked, and (ii) calculates reliability levels in accordance with an average or a distribution of each histogram, each histogram showing frequencies of angles, which are respectively indicated by angular components present in the frame feature data, and ranges of the angles,

wherein each reliability level shows leptokurticity of the frequencies in a corresponding one of the ranges, and

wherein the correction parameter detected by the detection unit is an angle that is selected from one of the ranges that has a highest reliability level of all the ranges.

5. The image processing device of claim 1,

wherein correcting the frame picture data is correcting a defect caused by a shake, which is triggered when taking a frame picture of the frame picture data, by stabilizing a pair of feature points that is commonly present in the frame picture data and the previous frame picture data in accordance with a rotation matrix, and

wherein the correction parameter detected by the detection unit is the rotation matrix indicating a shake amount by which the pair of feature points, which is present in a part of the frame picture data corresponding to the unmasked region, has moved between the frame picture data and the previous frame picture data.

6. The image processing device of claim 5,

wherein the detection unit detects the correction parameter by (i) randomly selecting, from the part of the frame picture data, a plurality of pairs of candidate points as candidates for the pair of feature points, (ii) calculating a pseudo rotation matrix for each pair of candidate points, and (iii) calculating a match level for each pair of candidate points,

wherein each match level shows how the frame picture data and the previous frame picture data would match with each other if the frame picture data were corrected in accordance with the corresponding pseudo rotation matrix, and

wherein the pair of feature points to be used in association with the correction parameter is one of the pairs of candidate points that has a highest match level of all the pairs.

7. The image processing device of claim 1,

wherein a frame picture of the frame picture data is taken by an imaging apparatus having a slant sensor,

wherein correcting the frame picture data is correcting a defect caused by a slant, which is triggered when taking the frame picture, by performing a rotation transformation on the frame picture data,

wherein the detection unit further (i) generates a histogram showing frequencies of angles, which are respectively indicated by angular components present in the unmasked region, and ranges of the angles, and (ii) selects, from the histogram, one of the ranges that falls within a range of angles that is output by the slant sensor, and

wherein the correction parameter is an angle included in the selected range.

8. The image processing device of claim 1, further comprising

a judgment unit that judges, in accordance with a distance between an imaging apparatus and an object to be imaged, whether or not each of the pixel blocks corresponds to a foreground edge image or a background edge image of the frame feature data,

wherein a frame picture of the frame picture data is taken by the imaging apparatus having a sensor that measures the distance between the imaging apparatus and the object to be imaged,

wherein correcting the frame picture data is correcting a defect caused by a slant, which is triggered when taking the frame picture, by performing a rotation transformation on the frame picture data, and

wherein the region to be masked corresponds to a pixel block that has been judged to be corresponding to the foreground edge image.

9. The image processing device of claim 1, further comprising

an estimation unit that (i) selects a pair of feature points from a pixel block that does not correspond to a foreground edge image of the frame feature data, and (ii) estimates a shake amount by which the selected pair of feature points has moved between the frame picture data and the previous frame picture data,

wherein the foreground edge image corresponds to the pixel block whose motion vector indicates the movement amount larger than the predetermined threshold, and

wherein a pixel block showing the predetermined directional component is determined by selecting, in accordance with the estimated shake amount, a pixel block that is proportional to a counterpart pixel block thereof included in the previous frame picture data.

10. The image processing device of claim 1,

wherein correcting the frame picture data is (i) correcting a defect caused by a slant by performing a rotation transformation on the frame picture data, and (ii) correcting a defect caused by a shake by stabilizing backgrounds of the frame picture data and the previous frame picture data, the slant and the shake being triggered when taking a frame picture of the frame picture data.

11. The image processing device of claim 1,

wherein the region specification unit further separates the frame feature data into a foreground edge image and a background edge image in accordance with movement amounts that are respectively indicated by the motion vectors,

wherein the region to be masked includes the foreground edge image,

wherein correcting the frame picture data is correcting a defect caused by a shake, which is triggered when taking a frame picture of the frame picture data, by stabilizing a pair of feature points that is commonly present in the frame picture data and the previous frame picture data in accordance with a rotation matrix, and

wherein the correction parameter detected by the detection unit is the rotation matrix indicating a shake amount by which the pair of feature points present in a part of the frame picture data, which corresponds to the background edge image, has moved between the frame picture data and the previous frame picture data.

12. The image processing device of claim 1, further comprising

an encode unit that encodes the frame picture data that has been corrected in accordance with the correction parameter.

13. The image processing device of claim 1 being connected to a plurality of frame memories,

wherein the frame picture data is stored in one of the frame memories, whereas the previous frame picture data is stored in another one of the frame memories.

14. An image processing method for correcting frame picture data in accordance with a correction parameter, comprising the steps of:

generating frame feature data indicating features of an edge image shown in the frame picture data;

calculating, from each of pixel blocks constituting the frame picture data, a motion vector indicating a movement between the frame picture data and previous frame picture data;

specifying a region to be masked in the frame feature data, in accordance with motion vectors calculated from the pixel blocks; and

detecting the correction parameter from an unmasked region of the frame feature data,

wherein the region to be masked corresponds to (i) a pixel block whose motion vector indicates a movement amount larger than a predetermined threshold, and (ii) a pixel block whose motion vector indicates a different directional component than a predetermined directional component, and

wherein the predetermined directional component is a direction shown by a pixel block that is proportional to a counterpart pixel block thereof included in the previous frame picture data.

15. A non-transitory computer readable recording medium having stored thereon a program for causing a computer to execute processing of correcting frame picture data in accordance with a correction parameter, the processing comprising the steps of:

generating frame feature data indicating features of an edge image shown in the frame picture data;

calculating, from each of pixel blocks constituting the frame picture data, a motion vector indicating a movement between the frame picture data and previous frame picture data;

specifying a region to be masked in the frame feature data, in accordance with motion vectors calculated from the pixel blocks; and

detecting the correction parameter from an unmasked region of the frame feature data,

wherein the region to be masked corresponds to (i) a pixel block whose motion vector indicates a movement amount larger than a predetermined threshold, and (ii) a pixel block whose motion vector indicates a different directional component than a predetermined directional component, and

wherein the predetermined directional component is a direction shown by a pixel block that is proportional to a counterpart pixel block thereof included in the previous frame picture data.