Image processing apparatus and image processing method, and program

• US 20050088550A1
• Filed: 10/25/2004
• Published: 04/28/2005
• Est. Priority Date: 10/23/2003
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Mitsunaga, Tomoo, Nomura, Yoshikuni
• Granted Patent: US 7,548,264 B2
• Time in Patent Office: Days
• US Class Current: 348/272
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements

Abstract

An image processing apparatus is provided which includes: an image obtaining unit for obtaining a mosaic image by an image sensor having at least four kinds of filters with different spectral sensitivities, one of the at least four kinds of filters being used for each pixel; and an image processing unit for generating a color image such that intensity information at a position of each pixel, the intensity information corresponding to a number of colors determined by the spectral sensitivities of the filters, is computed for all pixels, from the mosaic image obtained by the image obtaining unit. In a color filter arrangement of the image sensor of the image obtaining unit, filters possessed by pixels arranged in a checkered manner in half of all the pixels have at least some spectral characteristics having a strong correlation with the spectral characteristic of luminance. The image processing unit calculates estimated intensity values of colors possessed by the pixels arranged in the checkered manner at each pixel and calculates a luminance value of each pixel on a basis of the estimated intensity values.

Claims

1. An image processing apparatus for picking up and processing an image, comprising:

an image obtaining unit for obtaining a mosaic image by an image sensor having at least four kinds of filters with different spectral sensitivities, one of the at least four kinds of filters being used for each pixel of the mosaic image; and

an image processing unit for generating a color image such that intensity information at a position of each pixel, the intensity information corresponding to a plurality of colors determined by the spectral sensitivities of the filters, is computed for all pixels of the mosaic image obtained by the image obtaining unit;

wherein in a color filter arrangement of the image sensor of the image obtaining unit, filters possessed by pixels arranged in a checkered manner in half of all the pixels have at least one spectral characteristic having a strong correlation with a spectral characteristic of luminance; and

wherein the image processing unit calculates estimated intensity values of colors possessed by the pixels arranged in the checkered manner at each pixel and calculates a luminance value of each pixel on a basis of the estimated intensity values.

2. An image processing apparatus as claimed in claim 1, wherein the color filter arrangement of the image sensor of the image obtaining unit further comprises:

first filters corresponding to a first color having a spectral characteristic having a strong correlation with the spectral characteristic of luminance arranged horizontally and vertically in every other line; and

second filters corresponding to a color different from the first color and having a spectral characteristic having a strong correlation with the spectral characteristic of luminance arranged horizontally and vertically in every other line and in lines different from lines of the first filter; and

wherein the image processing unit further comprises;

a first calculating unit for calculating a first estimated intensity value as an estimated intensity value of the first color at a position of a pixel of interest in the mosaic image;

a second calculating unit for calculating a second estimated intensity value as an estimated intensity value of the second color at the position of the pixel of interest; and

a synthesizing unit for synthesizing the first estimated intensity value calculated by the first calculating unit with the second estimated intensity value calculated by the second calculating unit.

3. An image processing apparatus as claimed in claim 2, wherein:

the first calculating unit includes at least one third calculating unit for calculating an estimated intensity value corresponding to a target color at the position of the pixel of interest on a basis of an intensity value corresponding to a known color already obtained at the position of the pixel of interest and a correlation between the known color and the target color desired to be estimated, the correlation being calculated from pixels in proximity to the pixel of interest; and

the at least one third calculating unit calculates, when the pixel of interest is not of the first color, the first estimated intensity value with the color of the filter at the position of the pixel of interest as the known color and the first color as the target color.

4. An image processing apparatus as claimed in claim 3, wherein the third calculating unit further comprises:

a generating unit for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of the pixels corresponding to the target color in proximity to the pixel of interest;

a fourth calculating unit for calculating a center of gravity and a slope of a color distribution between the known color and the target color from the plurality of sets of the pixel intensities of the pixels corresponding to the known color and the pixel intensities of the pixels corresponding to the target color, the plurality of sets being generated by the generating means; and

a fifth calculating unit for calculating the estimated intensity value corresponding to the target color at the position of the pixel of interest on the basis of a center of gravity and a slope of the color distribution calculated by the fourth calculating unit and the intensity value corresponding to the known color at the position of the pixel of interest.

5. An image processing apparatus as claimed in claim 4, wherein the fifth calculating unit calculates the estimated intensity value corresponding to the target color at the position of the pixel of interest using a regression operation.

6. An image processing apparatus as claimed in claim 3, wherein:

the second calculating unit includes a third calculating unit for calculating an estimated intensity value corresponding to a target color at the position of the pixel of interest on a basis of an intensity value corresponding to a known color already obtained at the position of the pixel of interest and a correlation between the known color and the target color desired to be estimated, the correlation being calculated from pixels in proximity to the pixel of interest; and

the at least one third calculating unit calculates, when the pixel of interest if not of the second color, the second estimated intensity value with the color of the filter at the position of the pixel of interest as the known color and the second color as the target color.

7. An image processing apparatus as claimed in claim 6, wherein the at least one third calculating unit further comprises:

a generating unit for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of the pixels corresponding to the target color in proximity to the pixel of interest;

a fourth calculating unit for calculating a center of gravity and a slope of a color distribution between the known color and the target color from the plurality of sets of the pixel intensities of the pixel corresponding to the known color and the pixel intensities of the pixels corresponding to the target color, the plurality of sets being generated by the generating unit; and

a fifth calculating unit for calculating the estimated intensity value corresponding to the target color at the position of the pixel of interest on a basis of the center of gravity and the slope of the color distribution calculated by the fourth calculating unit and the intensity value corresponding to the known color at the position of the pixel of interest.

8. An image processing apparatus as claimed in claim 7, wherein the fifth calculating unit calculates the estimated intensity value corresponding to the target color at the position of the pixel of interest using a regression operation.

9. An image processing apparatus as claimed in claim 3, wherein:

the first calculating unit includes at least two third calculating units for calculating an estimated intensity value corresponding to a target color at the position of the pixel of interest on a basis of an intensity value corresponding to a known color already obtained at the position of the pixel of interest and a correlation between the known color and the target color desired to be estimated, the correlation being calculated from pixels in proximity to the pixel of interest;

a first of the at least one third calculating units calculates, when the pixel of interest is not of the first color, an intensity value corresponding to the second color at the position of the pixel of interest with the color of the filter at the position of the pixel of interest as the known color and the second color as the target color; and

a second of the at least one third calculating units calculates the first estimated intensity value with the second color as the known color and the first color as the target color.

10. An image processing apparatus as claimed in claim 9, wherein the at least one third calculating unit further comprises:

a generating unit for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of the pixels corresponding to the target color;

a fourth calculating unit for calculating the center of gravity and the slope of a color distribution between the known color and the target color from the plurality of sets of the pixel intensities of the pixels corresponding to the known color and the pixel intensities of the pixels corresponding to the target color, the plurality of sets being generated by the generating unit; and

a fifth calculating unit for calculating the estimated intensity value corresponding to the target color at the position of the pixel of interest on the basis of the center of gravity and the slope of the color distribution calculated by the fourth calculating unit and the intensity value corresponding to the known color at the position of the pixel of interest.

11. An image processing apparatus as claimed in claim 10, wherein the fifth calculating unit calculates the estimated intensity value corresponding to the target color at the position of the pixel of interest using a regression operation.

12. An image processing apparatus as claimed in claim 2, wherein:

the second calculating unit includes at least two third calculating units for calculating an estimated intensity value corresponding to a target color at the position of the pixel of interest on a basis of an intensity value corresponding to a known color already obtained at the position of the pixel of interest and a correlation between the known color and the target color desired to be estimated, the correlation being calculated from pixels in proximity to the pixel of interest; and

a first of the at least two third calculating units calculates, when the pixel of interest is not of the second color, an intensity value corresponding to the first color at the position of the pixel of interest with the color of the filter at the position of the pixel of interest as the known color and the first color as the target color; and

a second of the at least two third calculating units calculates the second estimated intensity value with the first color as the known color and the second color as the target color.

13. An image processing apparatus as claimed in claim 12, wherein the at least one third calculating unit further comprises:

a generating unit for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of the pixels corresponding to the target color;

a fourth calculating unit for calculating the center of gravity and the slope of a color distribution between the known color and the target color from the plurality of sets of the pixel intensities of the pixels corresponding to the known color and the pixel intensities of the pixels corresponding to the target color, the plurality of sets being generated by the generating unit; and

a fifth calculating unit for calculating the estimated intensity value corresponding to the target color at the position of the pixel of interest on a basis of the center of gravity and the slope of the color distribution calculated by the fourth calculating unit and the intensity value corresponding to the known color at the position of the pixel of interest.

14. An image processing apparatus as claimed in claim 13, wherein the fifth calculating unit calculates the estimated intensity value corresponding to the target color at the position of the pixel of interest using a regression operation.

15. An image processing apparatus as claimed in claim 2, wherein the first calculating unit further comprises:

a first intensity estimating unit for calculating a first estimated value for the first estimated intensity value;

a second intensity estimating unit for calculating a second estimated value for the first estimated intensity value;

a determining unit for determining a texture direction in proximity of the pixel of interest; and

a selecting unit for selecting a suitable estimated value from the first estimated value estimated by the first intensity estimating unit and the second estimated value estimated by the second intensity estimating unit on a basis of a result of determination by the determining unit;

wherein the first intensity estimating unit includes a first third calculating unit for calculating an estimated intensity value corresponding to a target color at the position of the pixel of interest on a basis of an intensity value corresponding to a known color already obtained at the position of the pixel of interest and a correlation between the known color and the target color desired to be estimated, the correlation being calculated from pixels in proximity to the pixel of interest, such that the first of the third calculating units calculates, when the pixel of interest is not of the first color, the first estimated value for the first estimated intensity value with the color of the filter at the position of the pixel of interest as the known color and the first color as the target color;

wherein the second intensity estimating unit further comprises second and third third calculating units, wherein the second of the third calculating units calculates, when the pixel of interest is not of the first color, an intensity value corresponding to the second color at the position of the pixel of interest with the color of the filter at the position of the pixel of interest as the known color and the second color as the target color, and the third of the third calculating units calculates the second estimated value for the first estimated intensity value with the second color as the known color and the first color as the target color; and

wherein the selecting unit selects the suitable estimated value from the first estimated value estimated by the first intensity estimating unit and the second estimated value estimated by the second intensity estimating unit on a basis of a result of determination by the determining unit and whether filters corresponding to the first color are situated in one of a horizontal direction and a vertical direction with respect to the position of the pixel of interest.

16. An image processing apparatus as claimed in claim 2, wherein the second calculating unit further comprises:

a first intensity estimating unit for calculating a first estimated value for the second estimated intensity value;

a second intensity estimating unit for calculating a second estimated value for the second estimated intensity value;

a determining unit for determining a texture direction in proximity of the pixel of interest; and

a selecting unit for selecting a suitable estimated value from the first estimated value estimated by the first intensity estimating unit and the second estimated value estimated by the second intensity estimating unit on a basis of a result of determination by the determining unit;

wherein the first intensity estimating unit includes a first third calculating unit for calculating an estimated intensity value corresponding to a target color at the position of the pixel of interest on a basis of an intensity value corresponding to a known color already obtained at the position of the pixel of interest and a correlation between the known color and the target color desired to be estimated, the correlation being calculated from pixels in proximity to the pixel of interest, such that the first third calculating unit calculates, when the pixel of interest is not of the second color, the first estimated value for the second estimated intensity value with the color of the filter at the position of the pixel of interest as the known color and the second color as the target color;

wherein the second intensity estimating unit further comprises at least second and third third calculating units, such that the second third calculating unit calculates, when the pixel of interest is not of the second color, an intensity value corresponding to the first color at the position of the pixel of interest with the color of the filter at the position of the pixel of interest as the known color and the first color as the target color, and the third third calculating unit calculates the second estimated value for the second estimated intensity value with the first color as the known color and the second color as the target color; and

wherein the selecting unit selects a suitable estimated value from the first estimated value estimated by the first intensity estimating unit and the second estimated value estimated by the second intensity estimating unit on a basis of the result of determination by the determining unit and whether filters corresponding to the second color are situated in one of a horizontal direction and a vertical direction with respect to the position of the pixel of interest.

17. An image processing apparatus as claimed in claim 2, wherein the color filter arrangement of the image sensor of the image obtaining unit further comprises:

third filters corresponding to a third color different from the first color and the second color, the third filters being arranged horizontally and vertically in every other line and in horizontal lines different from horizontal lines of the first filter, and

fourth filters corresponding to a fourth color different from the first color, the second color and the third color, the fourth filters being arranged horizontally and vertically in every other line and in horizontal lines different from horizontal lines of the second filter.

18. An image processing apparatus as claimed in claim 2, wherein the color filter arrangement of the image sensor of the image obtaining unit further comprises:

third filters corresponding to a third color which is different from the first color and the second color, the third filters being alternately arranged horizontally and vertically in every other line and in horizontal lines different from horizontal lines of the first filter; and

fifth and sixth filters corresponding, respectively, to fifth and sixth colors which are different from the first color, the second color, the third color and the fourth color, the fifth and sixth filters being alternately arranged horizontally and vertically in every other line and in horizontal lines different from horizontal lines of the second filter.

19. An image processing apparatus as claimed in claim 18, wherein:

wherein the third color and the fourth color have spectral characteristics having a high correlation with each other; and

the fifth color and the sixth color have spectral characteristics having a high correlation with each other.

20. An image processing apparatus as claimed in claim 18, wherein the third color and the fourth color are an identical color.

21. An image processing apparatus as claimed in claim 18, wherein the fifth color and the sixth color are an identical color.

22. An image processing apparatus as claimed in claim 1, wherein the color filter arrangement of the image sensor of the image obtaining unit further comprises first filters corresponding to a first color having a spectral characteristic having a strong correlation with a spectral characteristic of luminance arranged in a checkered manner, the image processing unit includes a first calculating unit for calculating an estimated intensity value corresponding to a target color at a position of a pixel of interest on a basis of an intensity value corresponding to a known color already obtained at the position of the pixel of interest and a correlation between the known color and the target color desired to be estimated, the correlation being calculated from pixels in proximity to the pixel of interest;

and

the first calculating unit calculates a first estimated intensity value with the color of the filter at the position of the pixel of interest as the known color and the first color as the target color.

23. An image processing apparatus as claimed in claim 22, wherein the first calculating unit further comprises:

a generating unit for generating a plurality of sets of pixel intensities of pixels corresponding to the known color and pixel intensities of the pixels corresponding to the target color;

a second calculating unit for calculating a center of gravity and a slope of a color distribution between the known color and the target color from the plurality of sets of the pixel intensities of the pixels corresponding to the known color and the pixel intensities of the pixels corresponding to the target color, the plurality of sets being generated by the generating unit; and

a third calculating unit for calculating an estimated intensity value corresponding to the target color at the position of the pixel of interest on a basis of a center of gravity and a slope of the color distribution calculated by the second calculating unit and the intensity value corresponding to the known color at the position of the pixel of interest.

24. An image processing apparatus as claimed in claim 23, wherein the third calculating unit calculates the estimated intensity value corresponding to the target color at the position of the pixel of interest using a regression operation.

25. An image processing apparatus as claimed in claim 22, wherein the color filter arrangement of the image sensor of the image obtaining unit further comprises:

filters of a second color different from the first color of the filters arranged horizontally and vertically in every other line, and

at filter positions where the filters of the first color and the second color are not arranged, filters of a third color and a fourth color different from the first color and the second color so as to form an oblique lattice in every other pixel in an oblique direction.

26. An image processing apparatus as claimed in claim 25, wherein:

the second color is a filter color having a spectral characteristic having sensitivity on a side of a longer wavelength than a wavelength of the first color; and

at least one of the third color and the fourth color is a filter color having a spectral characteristic having sensitivity on a side of a shorter wavelength than the wavelength of the first color.

27. An image processing apparatus as claimed in claim 22, wherein the color filter arrangement of the image sensor of the image obtaining unit further comprises:

filters of a second color, filters of a third color, filters of a fourth color, and filters of a fifth color which are different from the first color, arranged horizontally and vertically in every fourth line so as to form an oblique lattice in every other pixel in an oblique direction, and

wherein the second color and the third color, and the fourth color and the fifth color, respectively, are positioned on horizontal lines and vertical lines different from each other.

28. An image processing apparatus as claimed in claim 27, wherein:

the second color and the third color have spectral. characteristics having a high correlation with each other; and

the fourth color and the fifth color have spectral characteristics having a high correlation with each other.

29. An image processing apparatus as claimed in claim 27, wherein the second color and the third color are an identical color.

30. An image processing apparatus as claimed in claim 27, wherein the fourth color and the fifth color are an identical color.

31. An image processing method of an image processing apparatus for picking up and processing an image, the method comprising:

an obtaining step for obtaining a mosaic image by an image sensor having at least four kinds of filters with different spectral sensitivities, one of the at least four kinds of filters being used for each pixel of the mosaic image; and

an image processing step for generating a color image such that intensity information at a position of each pixel, the intensity information corresponding to a plurality of colors determined by the spectral sensitivities of the filters, is computed for all pixels of the mosaic image obtained by a process of the obtaining step;

wherein in the step of obtaining, the mosaic image is obtained which has a color arrangement such that filters possessed by pixels arranged in a checkered manner in half of all the pixels have at least some spectral characteristics having a strong correlation with the spectral characteristic of luminance; and

wherein in the step of generating, estimated intensity values are calculated of colors possessed by the pixels arranged in the checkered manner at each pixel, and a luminance value is calculated of each pixel on a basis of the estimated intensity values.

32. A program for enabling a computer to perform a process of picking up and processing an image, the program comprising:

an obtaining step for obtaining a mosaic image by an image sensor having at least four kinds of filters with different spectral sensitivities, one of the at least four kinds of filters being used for each pixel; and

an image processing step for generating a color image such that intensity information at a position of each pixel, the intensity information corresponding to a plurality of colors determined by the spectral sensitivities of the filters, is computed for all pixels, from the mosaic image obtained by the obtaining step;

wherein in the process of the obtaining step, the mosaic image which has a color arrangement such that filters possessed by pixels arranged in a checkered manner in half of all the pixels have at least some spectral characteristics having a strong correlation with a spectral characteristic of luminance is obtained; and

the image processing step includes a calculating step for calculating estimated intensity values of colors possessed by the pixels arranged in the checkered manner at each pixel and a calculating step for calculating a luminance value of each pixel on a basis of the estimated intensity values.