Multi-exposure pattern for enhancing dynamic range of images

• US 20090091645A1
• Filed: 10/03/2007
• Published: 04/09/2009
• Est. Priority Date: 10/03/2007
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Pylkkanen, Tom, Tico, Marius, Trimeche, Mejdi
• Granted Patent: US 7,940,311 B2
• Time in Patent Office: Days
• US Class Current: 348/273
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/583 with different integration times
• CPC Class Codes: H04N 25/585 with pixels having different sensitivities within the sensor, e.g. fast or slow pixels or pixels having different sizes

Abstract

The specification and drawings present a new method, apparatus and software product for enhancing a dynamic range of an image with a multi-exposure pixel pattern taken by an image sensor of a camera for one or more color channels, wherein a plurality of groups of pixels of the image sensor have different exposure times (e.g., pre-selected or adjusted by a user through a user interface using a viewfinder feedback, or adjusted by a user through a user interface after taking and storing RAW image, etc.). Processing of the captured image for constructing an enhanced image of the image for each of the one or more color channels can be performed using weighted combination of exposure times of pixels having different pre-selected exposure times according to a predetermined criterion.

Claims

1. A method comprising:

separating an image, captured with an image sensor with a multi-exposure pixel pattern for one or more color channels with a plurality of groups of pixels of said image sensor having different exposure times, for each color of said one or more color channels and for each exposure time of said different exposure times, into sub-images comprising pixels with exposure values only for one exposure time of said different exposure times and pixels with missing exposure values for exposure times different from said one exposure time;

interpolating exposure values for said pixels with missing exposure values in each sub-image of said sub-images using a predetermined algorithm for providing interpolated images of said sub-images; and

combining said interpolated images for each of said one or more color channels according to a predetermined criterion to provide an enhanced image of said image for each of said one or more color channels for enhancing a dynamic range of said image.

2. The method of claim 1, wherein said different exposure times are selected through a user interface of an electronic device comprising said sensor.

3. The method of claim 1, further comprising:

recombining said enhanced images for color channels of said one or more color channels using said multi-exposure pixel pattern for providing a recombined image with a high dynamic range of said image, wherein a number of said one or more color channels is two or more.

4. The method of claim 3, further comprising:

displaying said recombined image on a viewfinder; and

adjusting said different exposure times through a user interface using said recombined image displayed on said viewfinder as a feedback for optimizing said dynamic range of said recombined image.

5. The method of claim 1, wherein after said capturing said image and before said separating of said image, said different exposure times of said image are adjusted through a user interface using a pre-selected processing procedure.

6. The method of claim 1, wherein said image is a Bayer image.

7. The method of claim 1, wherein said plurality of the groups of said pixels comprises three groups of said pixels with three different exposure times.

8. The method of claim 1, wherein said interpolating is performed by calculating an interpolating value for each pixel of said pixels with missing exposure values using exposure values of one or more neighboring pixels of said each pixel.

9. The method of claim 1, wherein said combining of said interpolated images is performed using weight coefficients for said pixels with exposure values and for said pixels with interpolated exposure values for said pixels with missing exposure values, said weight coefficients being selected using said predetermined criterion.

10. The method of claim 9, wherein said weight coefficients are higher for pixels with longer exposure values in dark image areas and for pixels with shorter exposure values in brighter image areas.

11. A computer program product comprising:

a computer readable storage structure embodying a computer program code thereon for execution by a computer processor with said computer program code, wherein said computer program code comprises instructions for performing the method of claim 1.

12. An apparatus, comprising:

a color/exposure separator, configured to separate an image, captured with an image sensor with a multi-exposure pixel pattern for one or more color channels with a plurality of groups of pixels of said image sensor having different exposure times, for each color of said one or more color channels and for each exposure time of said different exposure times, into sub-images comprising pixels with exposure values only for one exposure time of said different exposure times and pixels with missing exposure values for exposure times different from said one exposure time;

an interpolator, configured to interpolate exposure values for said pixels with missing exposure values in each sub-image of said sub-images using a predetermined algorithm for providing interpolated images of said sub-images; and

a combiner, configured to combine said interpolated images for each of said one or more color channels according to a predetermined criterion to provide enhanced images of said image for each of said one or more color channels for enhancing a dynamic range of said image.

13. The apparatus of claim 12, wherein said color/exposure separator, said interpolator and said combiner are comprised in a processing module removable from said apparatus.

14. The apparatus of claim 12, further comprising:

a recombining and further processing module, configured to recombine said enhanced images for color channels of said one or more color channels using said multi-exposure pixel pattern for providing a high dynamic range of said image, wherein a number of said one or more color channels is two or more.

15. The apparatus of claim 12, further comprising:

an exposure pixel pattern adjustment module, configured to provide an exposure adjustment signal or a further exposure adjustment signal for adjusting said different exposure times in response to a command signal provided through a user interface of said apparatus.

16. The apparatus of claim 15, further comprising:

a viewfinder configured to display said recombined image, such that said exposure pixel pattern adjustment module is configured to provide the exposure adjustment signal in response to said command signal for adjusting said different exposure times using said recombined image displayed on said viewfinder as a feedback for optimizing said dynamic range of said recombined image.

17. The apparatus of claim 12, wherein an integrated circuit comprises all or selected modules of said apparatus.

18. The apparatus of claim 12, wherein said apparatus is an electronic device comprising said sensor for wireless communications.

19. The apparatus of claim 12, wherein said image is a Bayer image.

20. The apparatus of claim 12, wherein said plurality of the groups of said pixels comprises three groups of said pixels with three different exposure times.

21. The apparatus of claim 12, wherein said interpolator is configured to interpolate said exposure values for said pixels with the missing exposure values by calculating an interpolating value for each pixel of said pixels with missing exposure values using exposure values of one or more neighboring pixels of said each pixel.

22. The apparatus of claim 12, wherein said combiner is configured to combine said interpolated images using weight coefficients for said pixels with exposure values and for said pixels with interpolated exposure values for said pixels with missing exposure values, said weight coefficients being selected using said predetermined criterion.

23. The apparatus of claim 12, wherein said image sensor is a complimentary metal oxide semiconductor image sensor.

26. The apparatus of claims 12, comprising said image sensor.

24. An electronic device, comprising:

an image sensor, configured to capture an image with a multi-exposure pixel pattern for one or more color channels, wherein a plurality of groups of pixels of said image sensor have different exposure times;

a processing module, configured to process said image f or constructing an enhanced image of said image for each of said one or more color channels using weighted combination of exposure times of pixels having said different pre-selected exposure times according to a predetermined criterion for enhancing a dynamic range of said image;

a recombining and further processing module, configured to recombine said enhanced images for color channels of said one or more color channels using said multi-exposure pixel pattern for providing a high dynamic range of said image, wherein a number of said one or more color channels is two or more;

a viewfinder configured to display said recombined image; and

an exposure pixel pattern adjustment module, configured to provide adjusting of said different exposure times through a user interface of said electronic device using said recombined image displayed on said viewfinder as a feedback for optimizing said dynamic range of said recombined image.

25. The electronic device of claim 24, wherein said plurality of the groups of said pixels comprises three groups of said pixels with three different exposure times.