INCREASED LOW LIGHT SENSITIVITY FOR IMAGE SENSORS BY COMBINING QUANTUM DOT SENSITIVITY TO VISIBLE AND INFRARED LIGHT

• US 20120200734A1
• Filed: 02/06/2012
• Published: 08/09/2012
• Est. Priority Date: 02/09/2011
• Status: Active Grant

• Current Assignee: Malikie Innovations Limited (Key Patent Innovations Limited)
• Original Assignee: Research In Motion Limited (Blackberry Limited)
• Inventors: Tang, Sui Tong
• Granted Patent: US 8,717,464 B2
• Time in Patent Office: Days
• US Class Current: 348/223.1
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements

Abstract

A camera unit generates a processed digital image by augmenting color image data with infrared image data according to the level of ambient light exposure. The camera has an ambient light sensor that detects the level of ambient light in the camera unit and an image sensor that provides image data. One or more quantum dot layers may be included in the image sensor. A camera controller adapts the camera unit for operation in different modes that are selectable based on the levels of detected ambient light. The image data is processed into a digital image, according to the selected mode of operation for the camera unit, using color image data only when the level of ambient light is high, but augmenting the color image data with infrared image data when the level of ambient light is low to increase the color luminance of the final processed digital image.

Claims

1. A control system for a camera unit to generate a processed digital image represented by a plurality of image pixels, the control system comprising:

an ambient light sensor for generating an ambient light data signal representative of an intensity of ambient light detected;

a camera controller coupled to the ambient light sensor and for adapting the camera unit to a mode of operation selected from a plurality of modes of operation defined for the camera unit in relation to the intensity of the detected ambient light, the plurality of modes of operation comprising a first mode of operation and a second mode of operation; and

a camera sensor coupled to the camera controller to receive a control signal generated by the camera controller indicating the selected mode of operation, the camera sensor comprising;

an image sensor comprising a plurality of pixels for providing image data representing a scene image exposed to the image sensor, the image data comprising color image data and infrared image data; and

an image sensor processor responsive to the control signal to process the image data into the processed digital image, according to the selected mode of operation, by determining a plurality of processed color component values used to represent full colors in the processed digital image, for each image pixel of the processed digital image (i) in the first mode of operation, based on the color image data without using the infrared image data, and (ii) in the second mode of operation, based on the color image data modified by the infrared image data.

2. The control system of claim 1, wherein the image sensor processor is configured to

de-mosaic the image data into pre-processed image data comprising each of a plurality of color component values and an infrared component value associated with each image pixel of the processed digital image;

and

when the second mode of operation is selected, for each image pixel of the processed digital image, determine the plurality of processed color component values by augmenting an achromatic component value, derivable from the associated plurality of color component values, based on the associated infrared component value to provide an augmented achromatic component value.

3. The control system of claim 2, wherein the image sensor processor is configured to determine the plurality of processed color component values, for each image pixel of the processed digital image, by

transforming the associated plurality of color component values into a plurality of equivalent component values comprising the achromatic component value and a plurality of chromatic component values;

combining the achromatic component value and the infrared component value to generate the augmented achromatic component value; and

transforming the augmented achromatic component value and the plurality of chromatic component values into the plurality of processed color component values.

4. The control system of claim 3, wherein the image sensor processor is configured to generate the augmented achromatic component value, for each image pixel of the processed digital image, as a weighted average of the achromatic component value and the infrared component value.

5. The control system of claim 3, wherein the image sensor processor is configured to generate the augmented achromatic component value, for each image pixel of the processed digital image, as a non-linear transformation of the achromatic component value and the infrared component value.

6. The control system of claim 1, wherein the image sensor processor is configured to generate the plurality of processed color component values, according to the selected mode of operation, for each image pixel of the processed digital image, by de-mosaicing the image data (i) in the first mode of operation, based on color component values provided by proximately located sensor pixels on the image sensor without using infrared component values provided by the proximately located sensor pixels, and (ii) in the second mode of operation, based on the color component values provided by the proximately located sensor pixels weighted by at least one infrared component value provided by the proximately located sensor pixels to increase luminance in the processed digital image.

7. The control system of claim 1, wherein the camera controller is coupled to the ambient light sensor to receive the ambient light data signal, and is configured to

monitor a level of the ambient light data signal;

and

automatically select the mode of operation for the camera unit based upon the level of the ambient light data signal in relation to a plurality of ambient light intensity ranges defined for the ambient light data signal, each ambient light intensity range corresponding to a different mode of operation defined for the camera unit.

8. The control system of claim 7, wherein the camera controller is configured to select the first mode of operation for the camera unit when the level of the ambient light data signal is above an upper ambient light intensity threshold.

9. The control system of claim 8, wherein the camera controller is configured to select the second mode of operation for the camera unit when the level of the ambient light data signal is below the upper ambient light intensity threshold.

10. The control system of claim 9, wherein the camera controller is configured to

select the second mode of operation when the level of the ambient light data signal is above a lower ambient light intensity threshold less than the upper ambient light intensity threshold;

and

select a third mode of operation for the camera unit from the plurality of modes of operation when the level of the ambient light data signal is below the lower ambient light intensity threshold; and

wherein the control system comprises a camera flash sub-unit coupled to the camera controller, the camera flash sub-unit responsive to the control signal to generate a camera flash during exposure of the image sensor, when the third mode of operation is selected, to increase the intensity of the ambient light.

11. The control system of claim 1, wherein the ambient light data signal is further representative of a color temperature of the detected ambient light, and wherein the image sensor processor is configured to adjust the infrared component value associated with at least one image pixel based on the detected color temperature of the ambient light.

12. The control system of claim 11, wherein the camera sensor further comprises a database accessible to the image sensor processor for storing a plurality of database entries comprising records associated with a corresponding plurality of stored image objects, and

wherein the image sensor processor is configured to correlate the scene image exposed to the image sensor to one or more of the stored image objects, and process the infrared image data based on the records associated with the one or more stored image objects correlated to the scene image.

13. The control system of claim 12, wherein each database entry comprises a light intensity pattern representing a stored image object in the plurality of stored image objects, a associated color temperature of the stored image object and an associated adjustment factor for the light intensity pattern, and wherein the image sensor processor is configured, when in the second mode of operation, to process the associated infrared component values of the plurality of image pixels based on the detected color temperature of the ambient light by

matching a plurality of the associated infrared component values to the light intensity pattern of the one or more correlated image objects stored in the database and the color temperature of the detected ambient light to the color temperature associated with the one or more correlated image objects;

adjusting at least one of the plurality of associated infrared component values using the associated adjustment factor for the matched light intensity pattern to generate adjusted infrared image data; and

processing the adjusted infrared image data to generate the processed digital image.

14. The control system of claim 1, wherein the camera unit further comprises an input/output interface for graphically representing an intensity of the detected ambient light and for receiving selection input of the mode of operation for the camera unit.

15. A method for controlling a camera unit to generate a processed digital image represented by a plurality of image pixels, the method comprising:

detecting an intensity of ambient light;

receiving image data representing a scene image exposed to an image sensor, the image data comprising color image data and infrared image data;

adapting the camera unit to a mode of operation selected from a plurality of modes of operation defined for the camera unit in relation to the detected ambient light intensity, the plurality of modes of operation comprising a first mode of operation and a second mode of operation; and

processing the image data into the processed digital image, using an image sensor processor, according to the selected mode of operation, by determining a plurality of processed color component values used to represent full colors in the processed digital image, for each image pixel of the processed digital image (i) in the first mode of operation, based on the color image data without using the infrared image data, and (ii) in the second mode of operation, based on the color image data modified by the infrared image data.

16. The method of claim 15, further comprising

de-mosaicing the image data into pre-processed image data comprising each of a plurality of color component values and an infrared component value associated with each image pixel of the processed digital image;

and

when the second mode of operation is selected, for each image pixel of the processed digital image, determining the plurality of processed color component values by augmenting an achromatic component value, derived from the associated plurality of color component values, with the associated infrared component value.

17. The method of claim 16, further comprising determining the plurality of processed color component values, for each image pixel of the processed digital image, by

transforming the associated plurality of color component values into a plurality of equivalent component values comprising the achromatic component value and a plurality of chromatic component values;

combining the achromatic component value and the infrared component value to generate an augmented achromatic component value; and

transforming the augmented achromatic component value and the plurality of chromatic component values into the plurality of processed color component values.

18. The method of claim 17, further comprising generating the augmented achromatic component value, for each image pixel of the processed digital image, as a weighted average of the achromatic component value and the infrared component value.

19. The method of claim 17, further comprising generating the augmented achromatic component value, for each image pixel of the processed digital image, as a non-linear transformation of the achromatic component value and the infrared component value.

20. The method of claim 15, further comprising generating the plurality of processed color component values, according to the selected mode of operation, for each image pixel of the processed digital image, by de-mosaicing the image data (i) in the first mode of operation, based on color component values provided by proximately located sensor pixels on the image sensor without using infrared component values provided by the proximately located sensor pixels, and (ii) in the second mode of operation, based on the color component values provided by the proximately located sensor pixels weighted by at least one infrared component value provided by the proximately located sensor pixels to increase luminance in the processed digital image.

21. The method of claim 15, further comprising

monitoring a level of the ambient light intensity;

and

automatically selecting the mode of operation for the camera unit based upon the monitored level of the ambient light intensity in relation to a plurality of ambient light intensity ranges, each ambient light intensity range corresponding to a different mode of operation defined for the camera unit.

22. The method of claim 21, further comprising selecting the first mode of operation for the camera unit when the monitored level of the ambient light intensity is above an upper ambient light intensity threshold.

23. The method of claim 22, further comprising selecting the second mode of operation for the camera unit when the monitored level of the ambient light intensity is below the upper ambient light intensity threshold.

24. The method of claim 23, further comprising

selecting the second mode of operation when the monitored level of the ambient light intensity is above a lower ambient light intensity threshold less than the upper ambient light intensity threshold;

selecting a third mode of operation for the camera unit from the plurality of modes of operation when the monitored level of the ambient light intensity is below the lower ambient light intensity threshold; and

when the third mode of operation is selected, generating a camera flash during exposure of the image sensor to increase the ambient light intensity.

25. The method of claim 15, further comprising detecting a color temperature of the ambient light, and adjusting the infrared component value associated with at least one image pixel based on the detected color temperature of the ambient light.

26. The method of claim 25, further comprising

storing a plurality of database entries comprising records associated with a corresponding plurality of stored image objects;

correlating the scene image exposed to the image sensor to one or more of the stored image objects; and

processing the infrared image data based on the records associated with the one or more stored image objects correlated to the scene image.

27. The method of claim 26, wherein each database entry comprises a light intensity pattern representing a stored image object in the plurality of stored image objects, a associated color temperature of the stored image object and an associated adjustment factor for the light intensity pattern, and the method further comprises, when the second mode of operation is selected, processing the associated infrared component values of the plurality of image pixels based on the detected color temperature of the ambient light by

matching a plurality of the associated infrared component values to the light intensity pattern of the one or more correlated image objects and the color temperature of the detected ambient light to the color temperature associated with the one or more correlated image objects;

adjusting at least one of the plurality of associated infrared component values using the associated adjustment factor for the matched light intensity pattern to generated infrared adjusted image data; and

processing the infrared adjusted image data to generate the processed digital image.

28. The method of claim 15, further comprising

graphically representing the detected ambient light intensity on an input/output interface of the camera unit;

and

selecting the mode of operation for the camera unit on the input/output interface.

29. A non-transitory computer-readable storage medium storing instructions executable by one or more processors coupled to the storage medium, the stored instructions when executed for programming the one or more processors to control a camera unit to generate a processed digital image represented by a plurality of image pixels, the stored instructions comprising:

detecting an intensity of the ambient light;

receiving image data representing a scene image exposed to an image sensor, the image data comprising color image data and infrared image data;

adapting the camera unit to a mode of operation selected from a plurality of modes of operation defined for the camera unit in relation to the detected ambient light intensity, the plurality of modes of operation comprising a first mode of operation and a second mode of operation; and

processing the image data into the processed digital image, using the one or more processors, according to the selected mode of operation, by determining a plurality of processed color component values used to represent full colors in the processed digital image, for each image pixel of the processed digital image (i) in the first mode of operation, based on the color image data without using the infrared image data, and (ii) in the second mode of operation, based on the color image data modified by the infrared image data.