Tile-based camera mode switching

• US 10,602,065 B2
• Filed: 07/01/2019
• Issued: 03/24/2020
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Duran, Bill, Proca, Adrian Mircea
• Primary Examiner(s): Selby, Gevell V
• Application Number:
  US16/459,460
• Publication Number:
  US 20190327416A1
• Time in Patent Office: 267 Days
• CPC Class Codes: G02B 5/208 for use with infrared or ultraviolet radiation, e.g. for separating visible light from infrared and/or ultraviolet radiation
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/20 for generating image signals from infrared radiation only
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A method is executed at a camera operating in a night mode in which an IR filter is not interposed between a lens assembly and sensor arrays of the camera. In the night mode, the camera receives ambient light that is not filtered by the IR filter. For each image tile, the camera determines whether the ambient light received at the tile is due to an IR light source or a light source other than an IR light source and generates a respective mode change signal accordingly. A mode change parameter is then determined as a function of a total number of the mode change signals for image tiles of the camera. Based on the mode change parameter, the camera determines whether to continue the operation of the camera in the night mode or switch the operation of the camera from the night mode to a day mode.

Claims

1. A method for controlling a camera mode, comprising:

in a camera including a sensor array system, an IR filter, and a lens assembly that is configured to focus light on the sensor array system, wherein the sensor array system includes a plurality of sensor arrays and a plurality of tiles, and each sensor array corresponds to one of the plurality of tiles;

operating the camera in a night mode, wherein while in the night mode the IR filter is not interposed between the lens assembly and the sensor array system, including;

receiving at the plurality of sensor arrays ambient light that is not filtered by the IR filter;

for each of the plurality of tiles, determining whether the ambient light received at the respective tile is due to an IR light source or a light source other than an IR light source, and generating a respective mode change signal based on a determination that the ambient light received at that respective tile is due to a light source other than an IR light source;

determining a mode change parameter that is a function of a total number of the mode change signals for the plurality of tiles in the sensor array system; and

based on the mode change parameter, determining whether to continue the operation of the camera in the night mode or switch the operation of the camera from the night mode to a day mode.

2. The method of claim 1, further comprising:

determining whether the mode change parameter of the mode change signals exceeds a predetermined mode change threshold; and

in response to a determination that the mode change parameter of the mode change signals does not exceed the mode change threshold, continuing the operation of the camera in the night mode.

3. The method of claim 1, further comprising:

determining whether the mode change parameter of the mode change signals exceeds a predetermined mode change threshold; and

in response to a determination that the mode change parameter of the mode change signals exceeds the mode change threshold, switching the operation of the camera from the night mode to a day mode.

4. The method of claim 1, wherein the camera includes a gain controller configured to adjust analog gain of each sensor array based on a light level of the ambient light received at the respective tile, further comprising:

for each tile, obtaining the analog gain of the corresponding sensor array.

5. The method of claim 4, wherein for each tile, the analog gain of the sensor array varies with a frame rate of the camera, further comprising, for each tile:

determining a first frame rate of the camera used to detect one or more light components; and

normalizing the obtained analog gain of the corresponding sensor array based on at least a first difference between the first frame rate and a predefined frame rate.

6. The method of claim 1, wherein for each tile, the corresponding sensor array includes a plurality of sensors, and the plurality of sensors includes first, second and third pixels each having respective peak responses at different respective visible light frequencies, and wherein determining whether the ambient light received at the respective tile is due to an IR light source or a light source other than an IR light source further comprises, for each tile:

detecting a first light component of the received ambient light by averaging output signals from the first pixels;

detecting a second light component of the received ambient light by averaging output signals from the second pixels;

detecting a third light component of the received ambient light by averaging output signals from the third pixels; and

determining based on respective values of the first, second and third light components whether the received ambient light is due to a light source other than an IR light source.

7. The method of claim 6, further comprising:

obtaining a first ratio of the first to the second light components of the received ambient light;

obtaining a second ratio of the third to the second light components of the received ambient light; and

obtaining a graph characterizing respective types of light source based on a combination of the first and second ratios associated with the respective types of light sources;

wherein determining whether the ambient light received at the respective tile is due to an IR light source or a light source other than an IR light source includes determining based on the graph whether a light source characterized by the first ratio and the second ratio is other than an IR light source.

8. The method of claim 7, wherein the graph identifies a first region defined by specific respective ranges of the first and the second ratios as being associated with sunlight or incandescent light sources, further comprising:

determining whether the received ambient light is due to sunlight or an incandescent light source, including determining based on the graph whether a point defined by the first ratio and the second ratio lies within the first region.

9. The method of claim 7, wherein determining whether the received ambient light is due to a light source other than an IR light source includes determining that a point defined by the first ratio and the second ratio is substantially different from (1, 1).

10. The method of claim 7, wherein the camera includes an auto white balance processor that provides the obtained first and second ratios.

11. A non-transitory computer readable storage medium storing one or programs for execution by a camera system including a controller, a sensor array system, and a lens assembly that is configured to focus light on the sensor array, wherein the sensor array system includes a plurality of sensor arrays and a plurality of tiles, and each sensor array corresponds to one of the plurality of tiles, the one or more programs comprising instructions for:

operating the camera in a night mode, wherein while in the night mode the IR filter is not interposed between the lens assembly and the sensor array system, including;

receiving at the plurality of sensor arrays ambient light that is not filtered by the IR filter;

for each of the plurality of tiles, determining whether the ambient light received at the respective tile is due to an IR light source or a light source other than an IR light source, and generating a respective mode change signal based on a determination that the ambient light received at that respective tile is due to a light source other than an IR light source;

determining a mode change parameter that is a function of a total number of the mode change signals for the plurality of tiles in the sensor array system; and

based on the mode change parameter, determining whether to continue the operation of the camera in the night mode or switch the operation of the camera from the night mode to a day mode.

12. The non-transitory computer readable storage medium of claim 11, the one or more programs further comprising:

determining whether the mode change parameter of the mode change signals exceeds a predetermined mode change threshold; and

in response to a determination that the mode change parameter of the mode change signals does not exceed the mode change threshold, continuing the operation of the camera in the night mode.

13. The non-transitory computer readable storage medium of claim 11, the one or more programs further comprising:

determining whether the mode change parameter of the mode change signals exceeds a predetermined mode change threshold; and

in response to a determination that the mode change parameter of the mode change signals exceeds the mode change threshold, switching the operation of the camera from the night mode to a day mode.

14. The non-transitory computer readable storage medium of claim 11, wherein for each tile, whether the ambient light received at the respective tile is due to a light source other than an IR light source is determined based on respective values of a plurality of color components of the ambient light received at the respective tile.

15. The non-transitory computer readable storage medium of claim 11, the one or more programs further comprising instructions for:

for each tile, determining whether the ambient light received at the respective tile is due to sunlight or an incandescent light source, wherein the respective mode change signal is generated based on a determination that the received ambient light is due to sunlight.

16. A camera mode control system for a camera, comprising:

a sensor array system including a plurality of sensor arrays and a plurality of tiles, each sensor array corresponding to one of the plurality of tiles;

a lens assembly that is configured to focus light on the sensor array;

an IR filter;

a controller; and

memory storing one or more programs for execution by the controller, the one or more programs including instructions for;

operating the camera in a night mode, wherein while in the night mode the IR filter is not interposed between the lens assembly and the sensor array system, including;

receiving at the plurality of sensor arrays ambient light that is not filtered by the IR filter;

for each of the plurality of tiles, determining whether the ambient light received at the respective tile is due to an IR light source or a light source other than an IR light source, and generating a respective mode change signal based on a determination that the ambient light received at that respective tile is due to a light source other than an IR light source;

determining a mode change parameter that is a function of a total number of the mode change signals for the plurality of tiles in the sensor array system; and

based on the mode change parameter, determining whether to continue the operation of the camera in the night mode or switch the operation of the camera from the night mode to a day mode.

17. The system of claim 16, wherein the IR filter has a first position in which it is interposed between the lens and the sensor array system and a second position in which it is not interposed between the lens and sensor array system, and the one or more programs further include instructions for:

in response to initiating a change of a camera mode to the day mode, switching the camera mode to the day mode and causing the IR filter to be moved from the second position to the first position.

18. The system of claim 16, wherein, for each tile:

the corresponding sensor array includes a plurality of sensors;

the plurality of sensors includes first, second and third pixels each having respective peak responses at different respective visible light frequencies;

determining whether the ambient light received at the respective tile is due to an IR light source or a light source other than an IR light source further comprises;

detecting three light component of the received ambient light by averaging output signals from the first pixels, the second pixels, and the third pixels, respectively;

determining two ratios of the three light components; and

obtaining a graph characterizing respective types of light source based on a combination of the two ratios associated with the respective types of light sources.

19. The system of claim 18, wherein the graph is represented using two look-up tables, each addressable by a first index and a second index representing the two ratios, respectively, wherein a first one of the lookup tables defines combinations of the two ratios associated respectively with IR light sources and light sources other than IR light sources, and a second one of the lookup tables defines combinations of the two ratios associated with sunlight and incandescent lights.

20. The system of claim 19, wherein for each tile, the first lookup table encodes with 1'"'"'s to signify first table locations associated with a light source other than IR light sources and with 0'"'"'s to signify first table locations associated with IR light sources, such that the first lookup table is almost entirely filled with all 1'"'"'s apart from table locations associated with pairs of first and second ratios substantially similar to (1, 1).