Using a depth map of a monitored scene to identify floors, walls, and ceilings

• US 9,613,423 B2
• Filed: 06/12/2015
• Issued: 04/04/2017
• Est. Priority Date: 06/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Dixon, Michael, Shin, Dongeek
• Primary Examiner(s): HALLENBECK-HUBER, JEREMIAH CHARLES
• Application Number:
  US14/738,817
• Publication Number:
  US 20160364871A1
• Time in Patent Office: 662 Days
• Field of Search: 348135-136
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 30/242 Division of the character sequences into groups prior to recognition; Selection of dictionaries
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A process identifies large planar surfaces in scenes. The process receives captured IR images of a scene taken by a 2-dimensional array of image sensors of a camera system. Each IR image is captured when a distinct subset of IR illuminators of the camera system is illuminated. The process constructs a depth map of a scene using IR images and uses the depth map to compute a binary depth edge map for the scene. The binary depth edge map identifies which points in the depth map comprise depth discontinuities. The process identifies contiguous components based on the binary depth edge map and determines that a first component of the contiguous components represents a large planar surface in the scene by: fitting a plane to points in the first component; determining the orientation of the plane; and determining that the plane fitting residual error is less than a predefined threshold.

Claims

1. A method of identifying large planar surfaces in scenes, comprising:

at a computing device having one or more processors, and memory storing one or more programs configured for execution by the one or more processors;

receiving a plurality of captured IR images of a scene taken by a 2-dimensional array of image sensors of a camera that has a plurality of IR illuminators proximate to the array of image sensors, wherein each IR image is captured when a distinct subset of the IR illuminators of the camera is illuminated;

constructing a depth map of a scene using the plurality of IR images;

using the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map specifies lines of depth discontinuities in the depth map;

identifying a plurality of image regions enclosed by lines of depth discontinuities in the binary depth edge map;

determining that a first image region of the plurality of image regions represents a large planar surface in the scene by;

fitting a plane to points in the first component;

determining the orientation of the plane; and

determining that the plane fitting residual error is less than a predefined threshold; and

identifying a region in a displayed RGB image of the scene taken by the camera, wherein the region in the RGB image corresponds to the first image region.

2. The method of claim 1, wherein the orientation of the plane is upwards, and the plane is determined to be a floor.

3. The method of claim 1, wherein the orientation of the plane is downwards, and the plane is determined to be a ceiling.

4. The method of claim 1, wherein the orientation of the plane is horizontal, and the plane is determined to be a wall.

5. The method of claim 1, wherein the computing device is a server distinct from the camera system.

6. The method of claim 1, wherein the computing device is included in the camera system.

7. The method of claim 1, wherein the image sensors are partitioned into a plurality of pixels and wherein constructing the depth map comprises:

for each pixel of the plurality of pixels;

forming a respective vector comprising measured IR light intensity at the respective pixel for each of the plurality of captured IR images; and

estimating a depth in the first scene at the respective pixel using the respective vector and a respective lookup table.

8. The method of claim 1, further comprising monitoring a video stream provided by the camera to identify motion events, excluding the region in the RGB image corresponding to the first image region, and generating a motion alert when there is detected motion in the scene outside of the region in the RGB image corresponding to the first image region.

9. The method of claim 2, further comprising estimating a height and tilt angle of the camera relative to the determined floor.

10. A computing device, comprising:

one or more processors;

memory; and

one or more programs stored in the memory configured for execution by the one or more processors, the one or more programs comprising instructions for;

receiving a plurality of captured IR images of a scene taken by a 2-dimensional array of image sensors of a camera that has a plurality of IR illuminators proximate to the array of image sensors, wherein each IR image is captured when a distinct subset of the IR illuminators of the camera is illuminated;

constructing a depth map of a scene using the plurality of IR images;

using the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map specifies lines of depth discontinuities in the depth map;

identifying a plurality of image regions enclosed by lines of depth discontinuities in the binary depth edge map;

determining that a first image region of the plurality of image regions represents a large planar surface in the scene by;

fitting a plane to points in the first component;

determining the orientation of the plane; and

determining that the plane fitting residual error is less than a predefined threshold; and

identifying a region in a displayed RGB image of the scene taken by the camera, wherein the region in the RGB image corresponds to the first image region.

11. The computing device of claim 10, wherein the orientation of the plane is upwards, and the plane is determined to be a floor.

12. The computing device of claim 10, wherein the orientation of the plane is downwards, and the plane is determined to be a ceiling.

13. The computing device of claim 10, wherein the orientation of the plane is horizontal, and the plane is determined to be a wall.

14. The computing device of claim 10, wherein the computing device is a server distinct from the camera system.

15. The computing device of claim 10, wherein the computing device is included in the camera system.

16. The computing device of claim 10, wherein the image sensors are partitioned into a plurality of pixels and wherein the instructions for constructing the depth map comprise instructions for:

for each pixel of the plurality of pixels;

forming a respective vector comprising measured IR light intensity at the respective pixel for each of the plurality of captured IR images; and

estimating a depth in the first scene at the respective pixel using the respective vector and a respective lookup table.

17. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computing device having one or more processors and memory, the one or more programs comprising instructions for:

receiving a plurality of captured IR images of a scene taken by a 2-dimensional array of image sensors of a camera that has a plurality of IR illuminators proximate to the array of image sensors, wherein each IR image is captured when a distinct subset of the IR illuminators of the camera is illuminated;

constructing a depth map of a scene using the plurality of IR images;

using the depth map to compute a binary depth edge map for the scene, wherein the binary depth edge map specifies lines of depth discontinuities in the depth map;

identifying a plurality of image regions enclosed by lines of depth discontinuities in the binary depth edge map;

determining that a first image region of the plurality of image regions represents a large planar surface in the scene by;

fitting a plane to points in the first component;

determining the orientation of the plane; and

determining that the plane fitting residual error is less than a predefined threshold; and

identifying a region in a displayed RGB image of the scene taken by the camera, wherein the region in the RGB image corresponds to the first image region.

18. The computer readable storage medium of claim 17, wherein the orientation of the plane is upwards, and the plane is determined to be a floor.

19. The computer readable storage medium of claim 17, wherein the orientation of the plane is downwards, and the plane is determined to be a ceiling.

20. The computer readable storage medium of claim 17, wherein the computing device is a server distinct from the camera system.

21. The computer readable storage medium of claim 17, wherein the computing device is included in the camera system.

22. The computer readable storage medium of claim 17, wherein the image sensors are partitioned into a plurality of pixels and wherein the instructions for constructing the depth map comprise instructions for:

for each pixel of the plurality of pixels;

forming a respective vector comprising measured IR light intensity at the respective pixel for each of the plurality of captured IR images; and

estimating a depth in the first scene at the respective pixel using the respective vector and a respective lookup table.