Day and night detection based on one or more of illuminant detection, lux level detection, and tiling

• US 9,386,230 B1
• Filed: 06/12/2015
• Issued: 07/05/2016
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Proca, Adrian Mircea, Duran, Bill
• Primary Examiner(s): Moe, Aung S
• Assistant Examiner(s): Cowan, Euel
• Application Number:
  US14/738,225
• Time in Patent Office: 389 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 5/208 for use with infrared or ultraviolet radiation, e.g. for separating visible light from infrared and/or ultraviolet radiation
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/20 for generating image signals from infrared radiation only
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A method, executed at a camera with a Night mode and a Day mode, determines when to switch from Night mode to Day mode using one or more of: illuminant detection (detecting the type of illuminant providing ambient light based on R/G and B/G component ratios of the ambient light), lux detection (detecting the ambient light level), and tiling (performing illuminant detection and/or lux detection for sub-regions of an image sensor array so as to detect localized/narrow beam/point light sources that only illuminate a portion of a scene or are directed at the camera lens). Based on identification of the illuminant type, switching from Night mode to Day mode is done at different lux levels for different types of illuminants. E,g. when the light is due to sunlight or incandescent light switching to Day mode is done at higher lux than when the light is from other sources of visible light.

Claims

1. A method for controlling a camera mode, comprising:

in a camera including a controller, memory storing instructions for execution by the controller, a color sensor array comprising a plurality of sensors, the sensors including first, second and third pixels each having respective peak responses at different respective visible light frequencies, an IR filter, and a lens assembly that is configured to focus light on the sensor array;

when the camera mode is a night mode, wherein in the night mode, the IR filter is not interposed between the lens assembly and the sensor array, and the sensor array is exposed to ambient light via the lens assembly without interference by the IR filter;

detecting a first light component of the ambient light by averaging output signals from the first pixels;

detecting a second light component of the ambient light by averaging output signals from the second pixels;

detecting a third light component of the ambient light by averaging output signals from the third pixels;

determining based on respective values of the first, second and third light components whether the ambient light is due to other than an IR light source;

detecting the ambient light level;

based on a determination that the ambient light that is not filtered by the IR filter is due to other than an IR light source and that the ambient light level exceeds a first lux threshold, initiating a change of the camera mode to a day mode;

based on a determination that the ambient light that is not filtered by the IR filter is due to other than an IR light source and that the ambient light threshold does not exceed the first lux threshold, maintaining the camera in the night mode.

2. The method of claim 1, further comprising:

determining based on values of the first, second and third light components whether the ambient light is due to sunlight or an incandescent light source; and

based on a determination that the ambient light is due to sunlight or an incandescent light source, initiating a change of the camera mode to the day mode only when the ambient light level exceeds a second lux threshold higher than the first lux threshold.

3. The method of claim 2, further comprising:

obtaining a first ratio of the first to the second lighting components of the ambient light;

obtaining a second ratio of the third to the second lighting components of the ambient light; and

obtaining a graph characterizing respective types of light source based on a combination of the first and second ratios associated with the respective types of light sources;

wherein determining whether the ambient light is due to other than an IR light source includes determining based on the graph whether a light source characterized by the first ratio and the second ratio is other than an IR light source.

4. The method of claim 3, wherein the graph identifies a first region defined by specific respective ranges of the first and the second ratios as being associated with sunlight or incandescent light sources;

wherein determining whether the ambient light is due to sunlight or an incandescent light source includes determining based on the graph whether a point defined by the first ratio and the second ratio lies within the first region.

5. The method of claim 1, wherein the first, second and third light components are red, green and blue.

6. The method of claim 3, wherein determining whether the ambient light is due to other than an IR light source includes determining that a point defined by the first ratio and the second ratio is substantially different from (1, 1).

7. The method of claim 1, wherein the camera includes a gain controller that adjusts analog gain of the sensor array based on the ambient light level;

such that detecting the ambient light level comprises obtaining the analog gain of the sensor array.

8. The method of claim 7, wherein the analog gain of the sensor varies with a frame rate of the camera, further comprising:

determining a first frame rate of the camera used to detect the first, second and third light components; and

normalizing the obtained analog gain of the sensor based on a first difference between the first frame rate and a predefined frame rate and predefined associated differences in analog sensor gain based on the first difference.

9. The method of claim 1, wherein the IR filter has a first position in which it is interposed between the lens and the sensor array and a second position in which it is not interposed between the lens and sensor array, further comprising:

in response to initiating the change of the camera mode to the day mode, switching the camera mode to the day mode and causing the IR filter to be moved from the second position to the first position.

10. The method of claim 1, wherein the camera includes an auto white balance processor that provides the obtained first and second ratios.

11. The method of claim 1, wherein:

the color sensor array is one tile of a plurality of tiles in a color sensor array system and the method of claim 1 is performed for each of the tiles, such that a respective mode change signal is generated for a respective tile for which the ambient light at that respective tile is due to other than an IR light source prior to initiating the change of the camera mode to the day mode;

further comprising;

determining a total number of the mode change signals for the color sensor array system;

determining whether the total number of the mode change signals exceeds a predetermined mode change threshold based on a total number of tiles in the color sensor array system; and

when the total number of the mode change signals exceeds the mode change threshold, initiating the change of the camera mode to the day mode.

12. The method of claim 4, wherein the graph is represented using two look-up tables, each addressable by a first index representing one of the first ratios and a second index representing one of the second ratios;

wherein a first one of the lookup tables defines combinations of the first and second ratios associated respectively with IR light sources and other than IR light sources, and a second one of the lookup tables defines combinations of the first and second ratios associated with sunlight and incandescent lights.

13. The method of claim 12, wherein the first lookup table encodes with 1'"'"'s first table locations associated with other than IR light sources and with 0'"'"'s first table locations associated with IR light sources, such that the first lookup table is almost entirely filled with all 1'"'"'s apart from table locations associated with pairs of first and second ratios substantially similar to (1.1).

14. The method of claim 12, wherein the second lookup table corresponds to a higher-resolution version of a sub-region of the first lookup table, wherein the second lookup table encodes with 1'"'"'s second table locations associated with sunlight and incandescent light sources and with 0'"'"'s second table locations associated with other than sunlight and incandescent light sources.

15. A camera mode control system for a camera, comprising:

a controller,

memory storing programs for execution by the controller,

a color sensor array comprising a plurality of sensors, the sensors including first, second and third pixels, each having respective peak responses at different respective visible light frequencies,

a lens assembly that is configured to focus light on the sensor array, and

an IR filter that is not interposed between the lens assembly and the sensor array in a night mode;

the one or more programs including instructions for;

when the camera mode is the night mode and the sensor is exposed to ambient light via the lens assembly without interference by the IR filter;

detecting a first light component of the ambient light by averaging output signals from the first pixels;

detecting a second light component of the ambient light by averaging output signals from the second pixels;

detecting a third light component of the ambient light by averaging output signals from the third pixels;

determining based on respective values of the first, second and third light components whether the ambient light is due to other an IR light source;

detecting the ambient light level;

based on a determination that the ambient light that is not filtered by the IR filter is due to other than an IR light source and that the ambient light level exceeds a first lux threshold, initiating a change of the camera mode to a day mode; and

based on a determination that the ambient light that is not filtered by the IR filter is due to other than an IR light source and that the ambient light threshold does not exceed the first lux threshold, maintaining the camera in the night mode.

16. The system of claim 15, wherein the one or more programs further include instructions for:

determining based on values of the first, second and third light components whether the ambient light is due to sunlight or an incandescent light source; and

based on a determination that the ambient light is due to sunlight or an incandescent light source, initiating a change of the camera mode to the day mode only when the ambient light level exceeds a second lux threshold higher than the first lux threshold.

17. The system of claim 16, wherein the one or more programs further include instructions for:

obtaining a first ratio of the first to the second lighting components of the ambient light;

obtaining a second ratio of the third to the second lighting components of the ambient light; and

obtaining a graph characterizing respective types of light source based on a combination of the first and second ratios associated with the respective types of light sources;

wherein determining whether the ambient light is due to other than an IR light source includes determining based on the graph whether a light source characterized by the first ratio and the second ratio is other than an IR light source.

18. The system of claim 17, wherein the graph identifies a first region defined by specific respective ranges of the first and the second ratios as being associated with sunlight or incandescent light sources;

and

wherein determining whether the ambient light is due to sunlight or an incandescent light source includes determining based on the graph whether a point defined by the first ratio and the second ratio lies within the first region.

19. The system of claim 15, wherein the color sensor array is one tile of a plurality of tiles in a color sensor array system and the instructions of claim 15 are implemented for each of the tiles, such that a respective mode change signal is generated for a respective tile for which the ambient light at that respective tile is due to other than an IR light source prior to initiating the change of the camera mode to the day mode;

and

wherein the one or more programs further include instructions for;

determining a total number of the mode change signals for the color sensor array system;

determining whether the total number of the mode change signals exceeds a predetermined mode change threshold based on a total number of tiles in the color sensor array system; and

when the total number of the mode change signals exceeds the mode change threshold, initiating the change of the camera mode to the day mode.

20. A non-transitory computer readable storage medium storing one or programs for execution by a camera system including a controller, a color sensor array comprising a plurality of sensors, the sensors including first, second and third pixels, each having respective peak responses at different respective visible light frequencies, a lens assembly that is configured to focus light on the sensor array, and an IR filter that is not interposed between the lens assembly and the sensor array in a night mode, the one or more programs comprising instructions for:

when the camera mode is the night mode and the sensor is exposed to ambient light via the lens assembly without interference by the IR filter;

detecting a first light component of the ambient light by averaging output signals from the first pixels;

detecting a second light component of the ambient light by averaging output signals from the second pixels;

detecting a third light component of the ambient light by averaging output signals from the third pixels;

determining based on respective values of the first, second and third light components whether the ambient light is due to other an IR light source;

detecting the ambient light level;

based on a determination that the ambient light that is not filtered by the IR filter is due to other than an IR light source and that the ambient light level exceeds a first lux threshold, initiating a change of the camera mode to a day mode; and

based on a determination that the ambient light that is not filtered by the IR filter is due to other than an IR light source and that the ambient light threshold does not exceed the first lux threshold, maintaining the camera in the night mode.

21. The non-transitory computer readable storage medium of claim 20, the one or more programs further comprising instructions for:

determining based on values of the first, second and third light components whether the ambient light is due to sunlight or an incandescent light source; and

based on a determination that the ambient light is due to sunlight or an incandescent light source, initiating a change of the camera mode to the day mode only when the ambient light level exceeds a second lux threshold higher than the first lux threshold.

22. The non-transitory computer readable storage medium of claim 21, the one or more programs further comprising instructions for:

obtaining a first ratio of the first to the second lighting components of the ambient light;

obtaining a second ratio of the third to the second lighting components of the ambient light; and

obtaining a graph characterizing respective types of light source based on a combination of the first and second ratios associated with the respective types of light sources;

wherein determining whether the ambient light is due to other than an IR light source includes determining based on the graph whether a light source characterized by the first ratio and the second ratio is other than an IR light source.

23. The non-transitory computer readable storage medium of claim 20, wherein the camera includes a gain controller that adjusts analog gain of the sensor array based on the ambient light level, such that detecting the ambient light level comprises obtaining the analog gain of the sensor array.

24. The non-transitory computer readable storage medium of claim 20, wherein the IR filter has a first position in which it is interposed between the lens and the sensor array and a second position in which it is not interposed between the lens and sensor array, the one or more programs further comprising instructions for:

in response to initiating the change of the camera mode to the day mode, switching the camera mode to the day mode and causing the IR filter to he moved from the second position to the first position.

25. The non-transitory computer readable storage medium of claim 20, wherein the color sensor array is one tile of a plurality of tiles in a color sensor array the system and the instructions of claims 20 are implemented on each of the tiles, such that a respective mode change signal is generated for a respective tile for which the ambient light at that respective tile is due to other than an IR light source prior to initiating the change of the camera mode to the day mode;

and

wherein the one or more programs further comprise instructions for;

determining a total number of the mode change signals for the color sensor array system;

determining whether the total number of the mode change signals exceeds a predetermined mode change threshold based on a total number of tiles in the color sensor array system; and

when the total number of the mode change signals exceeds the mode change threshold, initiating the change of the camera mode to the day mode.