Sensing images and light sources via visible light filters

• US 9,696,470 B2
• Filed: 03/04/2015
• Issued: 07/04/2017
• Est. Priority Date: 03/04/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cui, Chengwu
• Primary Examiner(s): Tran, Nhan T
• Assistant Examiner(s): NGUYEN, CHAN T H
• Application Number:
  US14/638,302
• Publication Number:
  US 20160259099A1
• Time in Patent Office: 853 Days
• Field of Search: 348272
• CPC Class Codes: G01J 2001/4247 for testing lamps or other light sources
• CPC Class Codes: G01J 3/36 Investigating two or more bands of a spectrum by separate detectors
• CPC Class Codes: G01J 3/505 measuring the colour produced by lighting fixtures other than screens, monitors, displays or CRTs
• CPC Class Codes: G02B 5/204 in which spectral selection is performed by means of a conductive grid or array, e.g. frequency selective surfaces for use with wavelengths longer than the infrared light H01Q15/0006
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 25/702 SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout

Abstract

The description relates to capturing or sensing color images of scenes and information about the type of light (e.g., light source) that illuminated the scene. One example can include an image sensor manifest as an array of pixels. This example can also include a lens configured to focus an image from a scene on the array of pixels. The array of pixels can entail first pixels that are configured to sense the image and second pixels that are configured to capture information about lighting of the scene.

Claims

1. A device, comprising:

an image sensor comprising an array of pixels;

a filter array positioned over the array of pixels;

the filter array comprising a first type of filter positioned over a majority of the pixels of the array and a second type of filter positioned over a minority of the pixels of the array, the first type of filter is configured to provide light filtering so that specific colors of human perceptible light from a visible spectrum are transmitted to underlying pixels and captured as color information of a sensed color image of a scene, and wherein the second type of filter comprises spectral band filters having spectral transmission peaks in other portions of the visible spectrum and that are configured to transmit the other portions of the visible spectrum to different underlying pixels that do not contribute color information to the sensed color image but capture other visible spectrum information; and

,

a processor configured to execute computer-readable instructions comprising;

identifying gray regions of the sensed color image;

identifying whether any of the different underlying pixels are in the gray regions;

determining a ratio of a signal profile of an individual different underlying pixel compared to signal profiles of individual underlying pixels in an individual gray region; and

,

comparing the ratio to known ratios produced by known light source types to identify a type of light source illuminating the scene.

2. The device of claim 1, wherein the spectral band filters have corresponding bandwidths that are indicative of specific types of light sources.

3. The device of claim 1, wherein the specific colors of the first type of filter are associated with transmission wavelengths that overlap transmission wavelengths of the second type of filter.

4. The device of claim 1, wherein the specific colors of the first type of filter are associated with transmission wavelengths that do not overlap transmission wavelengths of the second type of filter.

5. The device of claim 1, wherein the processor is further configured to:

receive information from the image sensor; and

distinguish the other visible spectrum information from the color information of the received information.

6. The device of claim 5, wherein the processor is further configured to:

evaluate a profile of the other visible spectrum information to identify the type of light source illuminating the scene.

7. A device implemented process, comprising:

receiving information from pixels of an image sensor in response to the image sensor sensing a color image of a scene, the sensed color image of the scene based at least upon a set of colors and wherein a majority of the pixels are configured to detect one color of the set of colors;

identifying other pixels that are dedicated to capturing visible light and are not dedicated to detecting one of the set of colors;

organizing the other pixels into groups based at least upon different visible wavelength ranges that the other pixels are configured to sense, wherein the organizing further comprises, within an individual group, identifying pixels that are outliers, discarding the outliers, and evaluating remaining pixels of the individual group;

utilizing the evaluated remaining pixels from the groups to identify a type of light source illuminating the scene;

processing the sensed color image based at least upon the type of light source illuminating the scene to produce an enhanced image; and

,

causing the enhanced image to be displayed for a user.

8. The device implemented process of claim 7, wherein the receiving information comprises receiving signals output from the pixels.

9. The device implemented process of claim 7, wherein the identifying comprises identifying the other pixels from information from the image sensor about relative locations and filter type information of individual pixels on the image sensor.

10. The device implemented process of claim 7, wherein the organizing is performed by a processor.

11. The device implemented process of claim 7, wherein the evaluating comprises evaluating relative intensity of the information from individual remaining pixels at individual wavelengths.

12. The device implemented process of claim 7, wherein the utilizing comprises utilizing the information collectively to generate a spectral profile of visible wavelengths illuminating the scene.

13. The device implemented process of claim 7, wherein the utilizing comprises utilizing the evaluated remaining pixels to detect peaks representing spectral banding.

14. The device implemented process of claim 13, further comprising utilizing a presence of the peaks at specific wavelengths to identify the type of light source.

15. The device implemented process of claim 7, wherein the utilizing further comprises:

identifying gray regions of the sensed color image;

identifying whether any of the other pixels are in the gray regions;

determining a ratio of a signal profile of an individual other pixel compared to signal profiles of individual pixels in an individual gray region; and

comparing the ratio to known ratios produced by known light source types to identify the type of light source.

16. The device implemented process of claim 15, wherein the comparing comprises accessing the known ratios in a data table and determining a similarity between the ratio and individual known ratios.

17. The device implemented process of claim 16, wherein the comparing identifies a match when the similarity exceeds a predefined threshold.

18. A device, comprising:

an image sensor comprising an array of pixels;

first visible light filters having first spectral transmission peaks in the visible light spectrum;

second visible light filters having second spectral transmission peaks in the visible light spectrum;

a lens configured to focus an image from a scene on the array of pixels; and

,

the array of pixels comprising first pixels that are configured to capture the image through the first visible light filters and second pixels that are configured to capture information about visible light of the scene through the second visible light filters; and

,

a processor configured to organize the second pixels into groups based at least upon different visible wavelength ranges that the second pixels are configured to sense, and within an individual group, identify pixels that are outliers, discard the outliers, and evaluate remaining pixels of the individual group.

19. The device of claim 18, wherein the image sensor comprises a charge-coupled device (CCD) or a complementary metal-oxide semiconductor sensor (CMOS).

20. The device of claim 18, manifest as a smart phone, a tablet, a wearable smart device, a single-lens reflex camera, a viewfinder camera, or a consumer device.