Method, apparatus, and program for processing an image

• US 20090225200A1
• Filed: 12/16/2008
• Published: 09/10/2009
• Est. Priority Date: 06/23/2003
• Status: Active Grant

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ohki, Mitsuharu
• Granted Patent: US 8,085,313 B2
• Time in Patent Office: Days
• US Class Current: 348/241
• CPC Class Codes: H04N 1/482 using the same detector device sequentially for different colour components
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements

Abstract

The present invention relates to a method, apparatus and program for processing an image, capable of producing a sharp image. A digital camera 1 includes an imaging device 4 formed of a single-plane sensor that provides one color signal for each pixel. A plurality of images are captured at a high shutter speed by the imaging device 4 and stored in a frame memory 22 of a signal processor 7. The signal processor 7 detects a positional relationship among the plurality of images stored in the frame memory 22 and estimates a correct image with three color signals at each pixel, based on the detected positional relationship. The present invention is applicable, for example, to a digital camera.

Claims

1. An image processing method for estimating an output image from a plurality of input images, comprising the steps of:

detecting a positional relationship among the plurality of input images captured by image capture means for capturing an image each pixel of which has n pixel values; and

estimating the output image by identifying observed pixel components that should be taken into consideration from the plurality of input images for each pixel position of the output image, and producing the output image having (n+1) or more pixel values for each pixel based on the identified observed pixel components.

2. An image processing method according to claim 1, wherein in the step of estimating the output image, observed pixel components that should be taken into consideration are selected from the plurality of input images for each pixel position of the output image, and the output image having (n+1) or more pixel values for each pixel is produced based on the selected observed pixel components.

3. An image processing method according to claim 1, wherein in the step of estimating the output image, the output image having (n+1) or more color signals for each pixel is produced from the plurality of input images, based on the positional relationship detected in the detection step.

4. An image processing method according to claim 1, wherein in the step of estimating the output image, the output image is determined taking into account a color correlation at each pixel position and/or in the vicinity of each pixel position of the output image.

5. An image processing method according to claim 1, wherein in the step of estimating the output image, pixel values of pixels of the input images are changed to pixel values depending on the degree of steepness of an edge, and the output image is estimated based on the resultant changed pixel values.

6. An image processing method according to claim 1, wherein

n is equal to 1;

and

in the step of estimating the output image, the output image having three pixel values for each pixel is produced from the plurality of input images, based on the positional relationship detected in the detection step.

7. An image processing method according to claim 1, wherein the image capture means is a single-plane sensor, in which a color signal observed varies depending on the position of each input image.

8. An image processing method according to claim 1, wherein

the image capture means captures the plurality of input images with underexposure;

and

the image processing method further comprises the step of correcting the exposure by gaining up each pixel value of the plurality of input images.

9. An image processing method according to claim 8, wherein the step of estimating the output image comprises the steps of:

determining a corrected position of each pixel depending on the positional relationship detected in the detection step for each pixel of each of the plurality of input images;

producing a light intensity integral constraint formula indicating that the difference between the integral of light intensity in a region centered at the corrected position and a pixel value of a pixel at the corrected position should be less than a predetermined error; and

calculating the light intensity integral constraint formula.

10. An image processing method according to claim 9, wherein when the light intensity integral constraint formula is produced, if the plurality of input images have been gained up, the error is changed depending on the factor of the gaining up.

11. An image processing method according to claim 8, wherein in the step of estimating the output image,

a corrected position of each pixel is determined depending on the positional relationship detected in the detection step for each pixel of each of the plurality of input images;

and

the output image is estimated taking into account a pixel value of a pixel of the input images whose corrected position is located in the vicinity of each pixel position of the output image and also taking into account the error depending on the factor of the gaining up.

12. An image processing method according to claim 1, wherein in the step of estimating the output image,

a corrected position of each pixel is determined depending on the positional relationship detected in the detection step for each pixel of each of the plurality of input images;

the output image is estimated taking into account a pixel value of a pixel of the input images whose corrected position is located in the vicinity of each pixel position of the output image and also taking into account the distance between the corrected position and the position of the corresponding pixel of the output image.

13. An image processing method according to claim 1, wherein in the step of estimating the output image,

a corrected position of each pixel is determined depending on the positional relationship detected in the detection step for each pixel of each of the plurality of input images;

and

the output image is estimated taking into account a pixel value of a pixel of the input images whose corrected position is located in the vicinity of each pixel position of the output image and also taking into account the error depending on the factor of the gaining up.

14. An image processing method according to claim 1, wherein in the step of estimating the output image,

a corrected position of each pixel is determined depending on the positional relationship detected in the detection step for each pixel of each of the plurality of input images;

and

the output image is estimated by determining the weighted sum of pixel values of the input images whose corrected position is located in the vicinity of each pixel position of the output image.

15. An image processing method according to claim 14, wherein in the step of estimating the output image, the correct pixel value of a pixel of interest is estimated by determining the weighted sum of pixel values of the input images whose corrected position is located in the vicinity of each pixel position of the output image, weights of the weighted sum being determined using a function having a lowpass-filter characteristic with respect to the distance between the position of the pixel of interest in the output image and the corrected position.

16. An image processing method according to claim 15, wherein in the step of estimating the output image, the correct pixel value of the pixel of interest is estimated by selectively performing

a first calculation process of determining the weighted sum of pixel values of the input images whose corrected position is located in the vicinity of the position of the pixel of interest;

or

a second calculation process of determining the weighted sum of pixel values of the input images whose corrected position is located in the vicinity of the position of a pixel of the output image located adjacent to the pixel of interest.

17. An image processing method according to claim 16, wherein in the step of estimating the output image,

the second calculation process is selected when the sum of weights in the first calculation process is equal or nearly equal to 0.

18. An image processing method according to claim 15, wherein in the step of estimating the output image, the correct pixel value of the pixel of interest is estimated by selectively performing

a first calculation process of determining the weighted sum of pixel values of the input images whose corrected position is located in a first vicinity of the position of the pixel of interest;

or

a second calculation process of determining the weighted sum of pixel values of the input images whose corrected position is located in a second vicinity of the position of the pixel of interest, the second vicinity being greater than the first vicinity.

19. An image processing method according to claim 18, wherein in the step of estimating the output image, the second calculation process is selected when the sum of weights in the first calculation process is equal or nearly equal to 0.

20. An image processing method according to claim 15, wherein in the step of estimating the output image, the correct pixel value of the pixel of interest is estimated by selectively performing

a first calculation process of determining the sum of pixel values weighted by a first weighting factor of the input images whose corrected position is located in the vicinity of the position of the pixel of interest;

or

a second calculation process of determining the sum of pixel values weighted by a second weighting factor of the input images whose corrected position is located in the vicinity of the position of the pixel of interest.

21. An image processing method according to claim 20, wherein in the step of estimating the output image,

the second calculation process is selected when the sum of weights in the first calculation process is equal or nearly equal to 0.

22. An image processing method according to claim 1, wherein in the detection step,

the positional relationship among the plurality of input images is detected with reference to an input image captured at the exact or nearly exact center of a time during which the plurality of input images were captured.

23. An image processing apparatus for estimating an output image from a plurality of input images, comprising:

image capture means for capturing the plurality of input images having n pixel values for each pixel;

detection means for detecting a positional relationship among the plurality of input images; and

means for estimating the output image by identifying observed pixel components that should be taken into consideration from the plurality of input images for each pixel position of the output image, and producing the output image having (n+1) or more pixel values for each pixel based on the identified observed pixel components.

24. A program for causing a computer to execute a process of estimating an output image from a plurality of input images, the process comprising the steps of:

detecting a positional relationship among the plurality of input images captured by image capture means for capturing an image each pixel of which has n pixel values; and

estimating the output image by identifying observed pixel components that should be taken into consideration from the plurality of input images for each pixel position of the output image, and producing the output image having (n+1) or more pixel values for each pixel based on the identified observed pixel components.