Using images of a monitored scene to identify windows

• US 10,306,157 B2
• Filed: 02/14/2017
• Issued: 05/28/2019
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Shin, Dongeek, Dixon, Michael
• Primary Examiner(s): Perungavoor, Sathyanaraya V
• Assistant Examiner(s): Hilaire, Clifford
• Application Number:
  US15/432,871
• Publication Number:
  US 20170163910A1
• Time in Patent Office: 833 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10152 Varying illumination
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/514 from specularities
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 2201/121 using special illumination
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19617 Surveillance camera constructional details
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet
• CPC Class Codes: G08B 13/19658 Telephone systems used to communicate with a camera, e.g. PSTN, GSM, POTS
• CPC Class Codes: G08B 17/10 Actuation by presence of smoke or gases , e.g. automatic alarm devices for analysing flowing fluid materials by the use of optical means
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 : for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/56 : provided with illuminating means
• CPC Class Codes: H04N 23/661 : Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/667 : Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/74 : by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/33 : Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A camera system includes memory, image sensors, illuminators, and a processor. The processor operates the illuminators and the image sensors in a first mode to capture a two-dimensional image of the scene using light transmitted by the illuminators and reflected from the scene. The processor operates in a second mode to capture a plurality of images of the scene, including capturing a first image of the scene while one or more of the illuminators are activated and capturing a second image of the scene is while none of the illuminators are activated. The images are transmitted to a remote cloud computing system. The remote system constructs a light intensity map for the scene using the first and second images, and identifies a first region in the light intensity map as a glass surface when the light intensity values for the first region are below a threshold value corresponding to glass.

Claims

1. A camera system, comprising:

a memory configured to store information;

an image sensing element configured to receive light from a scene;

at least one illuminator configured to transmit light; and

a processor, coupled to the image sensing element and the at least one illuminator and configured to;

operate the at least one illuminator and the image sensing element in a first mode to capture a two-dimensional image of the scene using light transmitted by the at least one illuminator and reflected from the scene; and

operate the at least one illuminator and the image sensing element in a second mode to capture a plurality of images of the scene, including capturing a first image of the scene while one or more of the illuminators are activated and capturing a second image of the scene while none of the illuminators are activated; and

a wireless transmitter configured to transmit the plurality of images to a remote cloud computing system, wherein the remote cloud computing system;

constructs a light intensity map for the scene by subtracting light intensity values from the second image from corresponding light intensity values in the first image; and

identifies a first region in the light intensity map as a glass surface when the light intensity values for the first region are below a threshold value corresponding to glass.

2. The camera system of claim 1, wherein the camera system comprises a plurality of illuminators, and wherein in the first mode all of illuminators are activated while capturing the two-dimensional image.

3. The camera system of claim 1, wherein the second mode is a specularity detection mode.

4. The camera system of claim 1, wherein the camera system comprises a plurality of illuminators, and in the second mode a different subset of the illuminators is activated while capturing each of the plurality of images.

5. The camera system of claim 4, wherein each of the different subsets illuminates a different portion of the scene.

6. The camera system of claim 4, wherein each of the different subsets of illuminators is configured at a different angle relative to the image sensing element.

7. A method, comprising:

at a camera system having a memory configured to store information, an image sensing element configured to receive light from a scene, at least one illuminator configured to transmit light, a wireless transmitter, and a processor, coupled to the image sensing element and the at least one illuminator;

operating the at least one illuminator and the image sensing element in a first mode to capture a two-dimensional image of the scene using light transmitted by the at least one illuminator and reflected from the scene; and

operating the at least one illuminator and the image sensing element in a second mode to capture a plurality of images of the scene, including capturing a first image of the scene while one or more of the illuminators are activated and capturing a second image of the scene while none of the illuminators are activated; and

using the wireless transmitter to transmit the plurality of images to a remote cloud computing system, wherein the remote cloud computing system;

constructs a light intensity map for the scene by subtracting light intensity values from the second image from corresponding light intensity values in the first image;

identifies a first region in the light intensity map as a glass surface when the light intensity values for the first region are below a threshold value corresponding to glass.

8. The method of claim 7, wherein identifying the first region further comprises:

using the light intensity map to identify a plurality of pixels whose corresponding intensity values are within a predefined intensity range; and

clustering the identified plurality of pixels into one or more regions including the first region.

9. The method of claim 8, wherein clustering the identified plurality of pixels into one or more regions further comprises using a depth map that is constructed using the plurality of images.

10. The method of claim 8, wherein clustering the identified plurality of pixels into one or more regions further comprises using an RGB image of the scene captured using the image sensing element.

11. The method of claim 8, wherein the predefined intensity range comprises all intensity values below the threshold value corresponding to glass, the method thereby determining that the first region corresponds to a window in the scene.

12. The method of claim 11, wherein determining that the first region corresponds to a window comprises determining that the first region is substantially a quadrilateral.

13. A non-transitory computer-readable storage medium storing one or more programs configured for execution by a camera system having a memory configured to store information, an image sensing element configured to receive light from a scene, at least one illuminator configured to transmit light, a wireless transmitter, and a processor, coupled to the image sensing element and the at least one illuminator, wherein the one or more programs comprise instructions for:

operating the at least one illuminator and the image sensing element in a first mode to capture a two-dimensional image of the scene using light transmitted by the at least one illuminator and reflected from the scene; and

operating the at least one illuminator and the image sensing element in a second mode to capture a plurality of images of the scene, including capturing a first image of the scene while one or more of the illuminators are activated and capturing a second image of the scene while none of the illuminators are activated; and

using the wireless transmitter to transmit the plurality of images to a remote cloud computing system, wherein the remote cloud computing system;

constructs a light intensity map for the scene by subtracting light intensity values from the second image from corresponding light intensity values in the first image;

identifies a first region in the light intensity map as a glass surface when the light intensity values for the first region are below a threshold value corresponding to glass.

14. The non-transitory computer-readable storage medium of claim 13, wherein the one or more programs further comprise instructions for transmitting a video stream of the scene from the camera system to the remote cloud computing system, wherein the remote cloud computing system:

reviews the video stream to detect movement in the scene, wherein the first region is excluded from movement detection; and

generates a motion alert when there is motion detected at the scene outside of the first region.

15. The non-transitory computer-readable storage medium of claim 13, wherein the first and second images are IR images.

16. The non-transitory computer-readable storage medium of claim 13, wherein the light intensity map comprises a plurality of pixels.

17. The non-transitory computer-readable storage medium of claim 13, wherein the first mode is operated at night.