Image processing device, image processing method, and imaging apparatus

• US 7,982,781 B2
• Filed: 06/20/2006
• Issued: 07/19/2011
• Est. Priority Date: 06/21/2005
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Matsui, Akira, Chiba, Takuya, Kosakai, Ryota
• Primary Examiner(s): Tran; Nhan T
• Assistant Examiner(s): Le; Tuan H
• Application Number:
  US11/471,360
• Publication Number:
  US 20070013786A1
• Time in Patent Office: 1,855 Days
• Field of Search: 348/222.1, 348272-283
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

An image processing device includes a correlation value calculating unit calculating plural correlation values indicating degrees of correlation of an image for a pixel of attention to be interpolated on a solid-state imaging device having filters of predetermined color coding; a judging unit judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating unit and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and an interpolating unit applying interpolation processing to the pixel of attention on the basis of information on pixels around the pixel of attention present in the direction judged by the judging unit.

Claims

1. An image processing device, comprising:

correlation value calculating means for calculating plural correlation values indicating degrees of correlation of an image for a pixel of attention to be interpolated on a solid-state imaging device having filters of predetermined color coding;

judging means for judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating means and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and

interpolating means for applying interpolation processing to the pixel of attention on the basis of information on pixels around the pixel of attention present in the direction judged by the judging means,

wherein;

the correlation value calculating means calculates an average of higher order plural correlation values with high correlation among the plural correlation values, and

the judging means judges, on the basis of the average of the higher order plural correlation values, a direction in which interpolation should be performed.

2. An image processing device according to claim 1, wherein the correlation value calculating means calculates the plural correlation values with pixels around the pixel of attention as centers.

3. An image processing device according to claim 1, wherein the correlation value calculating means calculates the plural correlation values from a ratio of a first filter output obtained by applying filtering processing by a first band-pass filter to a first direction and a second filter output obtained by applying filtering processing by a second band-pass filter, which has a same filter characteristic as the first band-pass filter, to a second direction.

4. An image processing device according to claim 1, wherein the different directions are directions shifted from each other by 45°

.

5. An image processing device according to claim 4, wherein the judging means determines, when two correlation values for the different directions are on a correlation line, that reliability of correlation is high and determines, when the two correlation values are not on a correlation line, that reliability of correlation is low.

6. An image processing device according to claim 5, wherein, when the judging means determines that the reliability of correlation is high, the interpolating means performs interpolation using information on pixels around the pixel of attention present in the direction judged by the judging means, and when the judging means determines that the reliability of correlation is low, the interpolating means performs interpolation using an average of the information on the pixels around the pixel of attention.

7. An image processing device according to claim 1, wherein, in the predetermined color coding, with respect to a pixel arrangement in which pixels are arranged in a square lattice shape to be arranged at equal intervals in a horizontal direction and a vertical direction, color components serving as main components in creating a luminance component are arranged to surround respective other color components.

8. An image processing device according to claim 1, wherein, in the predetermined color coding, with respect to an oblique pixel arrangement in which pixels are shifted by ½

of pixel pitches for each row and for each column, color components serving as main components in creating a luminance component are arranged to surround respective other color components.

9. An image processing device according to claim 1, further comprising:

pre-stage interpolation processing means for changing color coding for the oblique pixel arrangement to color coding of a Bayer arrangement according to interpolation processing; and

post-stage interpolation processing means for applying Bayer interpolation processing using interpolation to the color coding of the Bayer arrangement,

wherein the pre-stage interpolation processing means includes;

correlation value calculating means for calculating plural correlation values indicating degrees of correlation of an image for the pixel of attention to be interpolated;

judging means for judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating means, and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and

interpolating means for applying interpolation processing to the pixel of attention on the basis of information on pixels around the pixel of attention present in the direction judged by the judging means.

10. An image processing device, comprising:

a correlation value calculating unit calculating plural correlation values indicating degrees of correlation of an image for a pixel of attention to be interpolated on a solid-state imaging device having filters of predetermined color coding;

a judging unit judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating unit and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and

an interpolating unit applying interpolation processing to the pixel of attention on the basis of information on pixels around the pixel of attention present in the direction judged by the judging unit,

wherein;

the correlation value calculating unit calculates an average of higher order plural correlation values with high correlation among the plural correlation values, and

the judging unit judges, on the basis of the average of the higher order plural correlation values, a direction in which interpolation should be performed.

11. An image processing device according to claim 10, further comprising:

a pre-stage interpolation processing unit changing color coding for the oblique pixel arrangement to color coding of a Bayer arrangement according to interpolation processing; and

a post-stage interpolation processing unit applying Bayer interpolation processing using interpolation to the color coding of the Bayer arrangement,

wherein the pre-stage interpolation processing unit includes;

a correlation value calculating unit calculating plural correlation values indicating degree of correlation of an image for the pixel of attention to be interpolated;

a judging unit judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating unit, and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and

an interpolating unit applying interpolation processing to the pixel of attention on the basis of information on pixels around the pixel of attention present in the direction judged by the judging unit.

12. An image processing method, comprising:

calculating plural correlation values indicating degrees of correlation of an image for a pixel of attention to be interpolated on a solid-state imaging device having filters of predetermined color coding;

judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated in the calculating step and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and

applying interpolation processing to the pixel of attention on the basis of information on pixels around the pixel of attention present in the direction judged in the judging step,

wherein;

the calculating calculates an average of higher order plural correlation values with high correlation among the plural correlation values, and

the judging judges, on the basis of the average of the higher order plural correlation values, a direction in which interpolation should be performed.

13. An imaging apparatus, comprising:

a solid-state imaging device having a filter of predetermined color coding; and

an image processing device processing an output signal of the solid-state imaging device,

wherein the image processing device includes;

correlation value calculating means for calculating plural correlation values indicating degrees of correlation of an image for a pixel of attention to be interpolated on the solid-state imaging device;

judging means for judging a direction in which interpolation should be performed by calculating, in different directions, at least two patterns of the plural correlation values calculated by the correlation value calculating means and comparing the correlation values of at least two patterns with correlation lines obtained by plotting the correlation values with respect to straight lines of various angles; and

interpolating means for applying interpolation processing to the pixel of attention on the basis of information on pixels around the pixel of attention present in the direction judged by the judging means,

wherein;

the correlation value calculating means calculates an average of higher order plural correlation values with high correlation among the plural correlation values, and

the judging means judges, on the basis of the average of the higher order plural correlation values, a direction in which interpolation should be performed.

14. An image processing apparatus for processing an image obtained by an imaging device with a color filter for color coding, the apparatus comprising:

an input unit inputting an image signal;

a correlation value calculating unit calculating a multiplicity of correlation values indicating degrees of correlation of the image;

a selecting unit selecting, from among the multiplicity of correlation values, a first correlation value and a second correlation value having highest reliability;

an identifying unit identifying pixels which are used to interpolate another pixel based on an average of the first correlation value and the second correlation value; and

an interpolating unit interpolating the pixel using the identified pixels.

15. An image processing apparatus according to claim 14, wherein the first correlation value and the second correlation value are calculated based on a ratio of a first filter output obtained by filtering processing by a first band-pass filter and a second filter output obtained by applying filtering processing by a second band-pass filter.

16. An image processing apparatus according to claim 15, wherein the first band-pass filter and the second band-pass filter have directions different from each other.

17. An image processing apparatus according to claim 16, wherein the different directions are directions shifted from each other by 45°

.

18. An image processing apparatus according to claim 16, wherein the identifying unit identifies the pixels which are used to interpolate another pixel by comparing the first correlation value and the second correlation value on the basis of a correlation in 360°

directions.

19. An image processing apparatus according to claim 14, wherein color components serving as main components in creating a luminance component are arranged on the color filter so as to surround respective other color components with respect to a pixel arrangement in which respective pixels are arranged in a square lattice shape such that the pixels are arranged at equal intervals in the horizontal direction and the vertical direction, wherein the color components serving as main components surround respective other color components by a ratio of 8 to 1.

20. An image processing apparatus according to claim 14, wherein color components serving as main components in creating a luminance component are arranged on the color filter so as to surround respective other color components with respect to an oblique pixel arrangement in which pixels are shifted by ½

of pixel pitches for each row and for each column.

21. An image processing apparatus according to claim 20, further comprising:

a pre-stage interpolation processing unit changing color coding for the oblique pixel arrangement to color coding of a Bayer arrangement according to interpolation processing; and

a post-stage interpolation processing unit applying Bayer interpolation processing using interpolation to the color coding of the Bayer arrangement.

22. An image processing apparatus for processing an image obtained by an imaging device with a color filter for color coding, the apparatus comprising:

an input unit inputting an image signal;

a correlation value calculating unit calculating a multiplicity of correlation values wherein each correlation value comprises a ratio of values, each value being determined from at least three pixels;

a reliability calculation unit calculating a reliability of each one of the multiplicity of correlation values;

a selecting unit selecting, from among the multiplicity of correlation values, a first correlation value and a second correlation value having highest reliability; and

an identifying unit identifying pixels which are used to interpolate another pixel based on an average of the first correlation value and the second correlation value; and

an interpolating unit interpolating a pixel by interpolation processing using the selected identified pixels.

23. An image processing apparatus according to claim 22, wherein, when the reliability is outside a predetermined range, the interpolating unit performs interpolation using an average of the information on the pixels around the pixel to be interpolated.