Image processing device, image processing method and image processing program

• US 20050276475A1
• Filed: 06/09/2005
• Published: 12/15/2005
• Est. Priority Date: 06/14/2004
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Sawada, Yasuhiro
• Granted Patent: US 7,706,608 B2
• Time in Patent Office: Days
• US Class Current: 382/167
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 1/648 Transmitting or storing the primary (additive or subtractive) colour signals; Compression thereof H04N1/642 - H04N1/646 take precedence
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/133 including elements passing panchromatic light, e.g. filters passing white light
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 25/57 Control of the dynamic range
• CPC Class Codes: H04N 3/155 Control of the image-sensor operation, e.g. image processing within the image-sensor

Abstract

An image processing device generates an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information obtained by an image sensor which has multiple pixels each photoelectrically converting light of a plurality of colors. The device includes: a simulation unit for generating a second image as an estimated image of the sparsely color sampled image; a comparison unit for comparing the second image with the first image, and generating first information thereof; a converting unit for converting the first information into second information corresponding to the color image; a penalty computing unit for computing the correction amount of the input image; and a correction unit for generating a third image by correcting the input image based on the second information and the correcting amount; this third image being used as an output image.

Claims

1. An image processing device for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the image processing device comprising:

a simulation unit configured to generate a second image as an estimated image of the sparsely color sampled image by simulating imaging using the image sensor based on the first image as the color image;

a comparison unit configured to compare the second image with the first image, and to generate first information which shows a comparison result;

a converting unit configured to convert the first information into second information which corresponds to the color image;

a penalty computing unit configured to compute a correction amount of the first image based on an image penalty which is determined corresponding to the first image; and

a correction unit configured to generate a third image as the color image, by correcting the first image based on at least one of the second information and the correction amount;

wherein the third image which is generated by repeating the processing of the correction unit as the input image of the third image, becomes the output image.

2. The image processing device according to claim 1, wherein the simulation unit is configured to generate the second image as the estimated image of the sparsely color sampled image by simulating imaging wherein the image sensor is used, including non-linear conversion, based on the first image of the color image.

3. The image processing device according to claim 1, wherein the first image is acquired through a color filter, the simulation unit is configured to perform a color response conversion wherein the characteristics of the color filter are simulated, and the converting unit is configured to perform color distribution corresponding to the color response conversion.

4. The image processing device according to claim 1, wherein the first image is acquired through an optical low-pass filter, the simulation unit is configured to perform convolution wherein the characteristics of the optical low-pass filter are simulated, and the converting unit is configured to perform inversion convolution as to the results of the convolution.

5. The image processing device according to claim 1, wherein the comparison unit is configured to generate the first information illustrating a difference between the third image and the first image.

6. The image processing device according to claim 1, wherein the comparison unit is configured to generate the first information illustrating a ratio between the third image and the first image.

7. The image processing device according to claim 1, wherein the simulation unit is configured to perform linear conversion and nonlinear conversion which acts upon elements within a same basis space and does not reverse a magnitude correlation of the elements.

8. The image processing device according to claim 1, wherein the correcting amount is a value which decreases the image penalty by adding to the first image.

9. The image processing device according to claim 1, wherein the image penalty is expressed as a sum of pixel penalties capable of calculating by pixel.

10. The image processing device according to claim 1, wherein the image penalty is expressed using a color plane penalty capable of calculating by color plane.

11. The image processing device according to claim 10, wherein the color plane penalty reflects a correlation between the color plane and other color planes.

12. An imaging device comprising:

an image sensor; and

the image processing device according to claim 1.

13. The imaging device according to claim 12, wherein the image sensor is one of an image sensor wherein color filters of four or more colors are arranged, an image sensor wherein color filters of multiple colors are arranged in a cycle greater than 2×

2, or an image sensor wherein the color filters of multiple colors are in an irregular arrangement.

14. An image processing device for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the image processing device comprising:

a simulation unit configured to generate a second image as an estimated image of the sparsely color sampled image, by simulating imaging using the image sensor including nonlinear conversion based on the first image as the color image;

a comparison unit configured to compare the second image with the first image, and to generate first information which shows a comparison result;

a converting unit configured to convert the first image into the second information which corresponds to the color image; and

a correction unit configured to generate a third image as the color image, by correcting the input image based on the second information;

wherein the third image which is generated by repeating the processing of the correction unit as the input image of the third image, becomes the output image.

15. The image processing device according to claim 14, wherein the nonlinear conversion is a nonlinear conversion which acts upon each element within a same basis space and does not reverse a magnitude correlation of the elements.

16. An imaging device comprising:

an image sensor; and

the image processing device according to claim 14.

17. An image processing method for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the method comprising:

a simulation step for generating a second image as an estimated image of the sparsely color sampled image by simulating imaging using the image sensor based on the first image as the color image;

a comparison step for comparing the second image with the first image, and generating first information which shows a comparison result;

a converting step for converting the first information into the second information which corresponds to the color image;

a penalty computing step for computing a correction amount of the input image based on the image penalty which is determined corresponding to the input image; and

a correction step for generating a third image as the color image, by correcting the input image based on at least one of the second information and the correcting amount;

wherein the third image which is generated by repeating the processing of the correction step as the input image of the third image, becomes the output image.

18. An image processing program for causing a computer to execute the image processing method according to claim 17.

19. An image processing method for generating an output image as a color image wherein each pixel has color information, using a first image as a sparsely color sampled image wherein each pixel has monochromatic information, which is obtained by an image sensor which has multiple pixels that photoelectrically convert light of a plurality of colors, the method comprising:

a simulation step for generating a second image as an estimated image of the sparsely color sampled image by simulating imaging using the image sensor including nonlinear conversion based on the first image as the color image;

a comparison step for comparing the second image with the first image and generating first information which shows a comparison result;

a converting step for converting the first information into the second information which corresponds to the color image; and

a correction step for generating a third image as the color image by correcting the input image based on the second information;

wherein the third image which is generated by repeating the processing of the correction step as the input image of the third image, becomes the output image.

20. An image processing program for causing a computer to execute the image processing method according to claim 19.