Using a depth map of a monitored scene to identify floors, walls, and ceilings

• US 9,554,064 B2
• Filed: 06/19/2015
• Issued: 01/24/2017
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Dixon, Michael, Shin, Dongeek
• Primary Examiner(s): HALLENBECK-HUBER, JEREMIAH CHARLES
• Application Number:
  US14/745,259
• Publication Number:
  US 20160366348A1
• Time in Patent Office: 585 Days
• Field of Search: 348135-136
• US Class Current: 1/1
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 30/242 Division of the character sequences into groups prior to recognition; Selection of dictionaries
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A camera system includes memory, a lens assembly to direct light from a scene onto an image sensing element, an image sensing element configured to receive light from the scene via the lens assembly, at least one infrared illuminator configured to transmit infrared light, and a processor, coupled to the image sensing element and the at least one infrared illuminator. The processor is configured to operate the illuminators and the image sensing element in a first mode whereby infrared light transmitted by the illuminators and reflected from the scene is used to generate a two-dimensional image of the scene. The processor is also configured to operate the illuminators and the image sensing element in a second mode whereby infrared light transmitted by the illuminators and reflected from the scene is used to identify a planar surface in the scene.

Claims

1. A camera device, comprising:

a memory configured to store information;

a lens assembly arranged to direct light from a scene;

a plurality of image sensing elements configured to receive light from the scene via the lens assembly;

a plurality of infrared illuminators configured to transmit infrared light; and

a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators and configured to;

operate the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and

operate the plurality of infrared illuminators and the image sensing element in a second mode, including;

sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene;

for each subset of illuminators, measuring an intensity of reflected light from the scene for each image sensing element of the plurality of image sensing elements;

for each image sensing element;

generating a light intensity vector based on the measured intensities for the element, wherein the light intensity vector has a component corresponding to each subset of illuminators; and

estimating a depth in the scene using the light intensity vector and a lookup table; and

identifying a planar surface in the scene using the estimated depths.

2. The camera device of claim 1, wherein the first mode is operated at night.

3. The camera device of claim 1, wherein in the first mode all of the infrared illuminators are activated to generate the two-dimensional image.

4. The camera device of claim 1, wherein the second mode is a three-dimensional surface detection mode.

5. The camera device of claim 1, wherein each of the subsets of illuminators is at a different angle relative to the plurality of image sensing elements.

6. The camera device of claim 1, wherein the processor is further configured to:

aggregate the estimated pixel depths into a data structure for identifying planar surfaces in the scene.

7. The camera device of claim 6, wherein identifying planar surfaces in the scene using the estimated pixel depths comprises transmitting the data structure to a planar surface detection module configured to identify planar surfaces using a depth map constructed from the data structure.

8. The camera device of claim 6, wherein identifying planar surfaces in the scene using the estimated pixel depths comprises:

using the data structure to identify a region in the scene; and

fitting a plane to a set of points in the region.

9. The camera device of claim 8, wherein the processor is further configured to:

determine that the region corresponds to a planar surface in the scene when a plane fitting residual error for the fitted plane is less than a predefined value.

10. The camera device of claim 6, wherein the processor is further configured to:

construct a depth map of the scene using the data structure;

use the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map identifies which points in the depth map comprise depth discontinuities; and

select a contiguous region from the binary depth edge map.

11. The camera device of claim 8, wherein the processor is further configured to determine an orientation of the fitted plane.

12. The camera device of claim 11, wherein the orientation of the plane is upwards, and the fitted plane is determined to be a floor.

13. The camera device of claim 11, wherein the orientation of the plane is downwards, and the fitted plane is determined to be a ceiling.

14. The camera device of claim 11, wherein the orientation of the plane is horizontal, and the fitted plane is determined to be a wall.

15. A method, comprising:

at a camera device having a memory configured to store information, a lens assembly arranged to direct light from a scene, a plurality of image sensing elements configured to receive light from the scene via the lens assembly, a plurality of infrared illuminators configured to transmit infrared light, and a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators;

operating the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and

operating the plurality of infrared illuminators and the image sensing element in a second mode, including;

sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene;

for each subset of illuminators, measuring an intensity of reflected light from the scene for each pixel of a plurality of pixels, wherein each pixel corresponds to one or more image sensing elements of the plurality of image sensing elements;

for each pixel;

generating a light intensity vector based on the measured intensities for the pixel, wherein the light intensity vector has a component corresponding to each subset of illuminators; and

estimating a pixel depth in the scene using the light intensity vector and a lookup table; and

identifying a planar surface in the scene using the estimated pixel depths.

16. The method of claim 15, wherein the first mode is configured to operate at night.

17. The method of claim 15, further comprising aggregating the estimated pixel depths into a data structure for identifying planar surfaces in the scene.

18. A non-transitory computer-readable storage medium storing one or more programs configured for execution by a camera device having a memory configured to store information, a lens assembly arranged to direct light from a scene, plurality of image sensing elements configured to receive light from the scene via the lens assembly, a plurality of infrared illuminators configured to transmit infrared light, and a processor, coupled to the plurality of image sensing elements and the plurality of infrared illuminators, wherein the one or more programs comprise instructions for:

operating the plurality of infrared illuminators and the image sensing element in a first mode whereby infrared light transmitted by the plurality of infrared illuminators and reflected from the scene is used to generate a two-dimensional image of the scene; and

operating the plurality of infrared illuminators and the image sensing element in a second mode, including;

sequentially activating a plurality of distinct subsets of the illuminators, each subset illuminating a different portion of the scene;

for each subset of illuminators, measuring an intensity of reflected light from the scene for each pixel of a plurality of pixels, wherein each pixel corresponds to one or more image sensing elements of the plurality of image sensing elements;

for each pixel;

generating a light intensity vector based on the measured intensities for the pixel, wherein the light intensity vector has a component corresponding to each subset of illuminators; and

estimating a pixel depth in the scene using the light intensity vector and a lookup table; and

identifying a planar surface in the scene using the estimated pixel depths.

19. The non-transitory computer-readable storage medium of claim 18, wherein the first mode is configured to operate at night.

20. The non-transitory computer-readable storage medium of claim 18, wherein the one or more programs further comprise instructions for aggregating the estimated pixel depths into a data structure for identifying planar surfaces in the scene.