System and method for building automation using video content analysis with depth sensing

• US 9,530,060 B2
• Filed: 01/17/2013
• Issued: 12/27/2016
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Fortress Corporation
• Inventors: Zhang, Zhong, Myers, Gary W., Venetianer, Peter L.
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): CESE, KENNY A
• Application Number:
  US13/744,264
• Publication Number:
  US 20130182905A1
• Time in Patent Office: 1,440 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: A61B 2505/07 Home care
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/0046 Arrangements of imaging apparatus in a room, e.g. room provided with shielding or for improved access to apparatus
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1072 measuring distances on the body, e.g. measuring length, height or thickness A61B5/1076 takes precedence
• CPC Class Codes: A61B 5/1073 Measuring volume, e.g. of limbs
• CPC Class Codes: A61B 5/1079 using optical or photographic means
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1117 Fall detection
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/579 : from motion
• CPC Class Codes: G06T 7/62 : of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 : Event detection
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/1365 : Matching; Classification
• CPC Class Codes: G08B 13/19615 : wherein said pattern is defined by the user
• CPC Class Codes: G08B 21/043 : detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0476 : Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A method and system for monitoring buildings (including houses and office buildings) by performing video content analysis based on two-dimensional image data and depth data are disclosed. Occupation and use of such buildings may be monitored with higher accuracy to provide higher energy efficiency usage, to control operation of components therein, and/or provide better security. Height data may be obtained from depth data to provide greater reliability in object detection, object classification and/or event detection.

Claims

1. A method of monitoring a building comprising:

taking a video within a location in the building with a video sensor, the video comprising a plurality of frames, each frame including 2D image data;

for each frame, receiving depth data associated with the 2D image data, the depth data corresponding to one or more distances from the video sensor to features represented by the 2D image data;

analyzing the 2D image data including performing 2D analysis on the 2D image data to detect one or more image blobs in the video;

using the depth data, projecting an image blob of the one or more image blobs onto a plurality of Z-planes, thereby creating a plurality of blob slices;

separating the Z-planes into a ground plane and non-round planes based on a height threshold, so that at least a first blob slice on the ground plane is at or below the height threshold, and a second blob slice and additional blob slices on the non-ground planes are above the height threshold;

creating a refined blob that includes the second blob slice, the additional blob slices, and only a portion of the first blob slice;

performing object detection on the refined blob, to determine that the blob corresponds to a human object, thereby detecting a person in the video;

based on the detected person and one or more additional detected people, counting a number of people at the location in the building; and

controlling a system of the building in response to the number of people counted.

2. The method of claim 1, wherein

controlling a system comprises at least one of turning on lights and turning off lights of the building.

3. The method of claim 1, wherein

controlling a system comprises setting a thermostat temperature of a heating system or a cooling system of the building.

4. The method of claim 1, further comprising providing an alert that at least one of a room or area is utilized when a number of people counted exceeds a predetermined value.

5. The method of claim 4, comprising providing an alert that a conference room of a building is utilized.

6. The method of claim 1, comprising counting a number of people that are sitting.

7. The method of claim 6, comprising detecting that people are sitting in front of a television, and reducing light usage of the building in response to detecting that people are sitting in front of a television.

8. The method of claim 1, further comprising detecting that a person has gone to bed.

9. The method of claim 8, further comprising detecting that a person has gone to bed by determining the person has lied down at a location other than the floor.

10. The method of claim 8, further comprising detecting that a person has gone to bed by detecting a height of the person.

11. The method of claim 8, further comprising reducing light usage of the building in response to detecting that the person has gone to bed.

12. The method of claim 8, further comprising controlling window blinds in response to detecting that the person has gone to bed.

13. The method of claim 1, further comprising detecting that a person has gotten out of bed.

14. The method of claim 13, further comprising, in response to detecting that a persona has gotten out of bed, controlling at least one of:

lights of the building;

window blinds;

heating of the building; and

cooling of the building.

15. The method of claim 1, further comprising:

determining a number of people within an area near elevators by the analyzing the image data and the depth data;

controlling elevator operation in response to the counting of the number of people.

16. The method of claim 15,

wherein the depth data is used to determine heights of detected objects;

and

the determined heights of the detected objects are used to determine if the detected objects are people.

17. The method of claim 1, further comprising:

determining a number of people within an elevator by the analyzing the image data and the depth data;

controlling operation of the elevator in response to the counting of the number of people.

18. The method of claim 17,

wherein the depth data is used to determine heights of detected objects;

and

the determined heights of the detected objects is are used to determine if the detected objects are people.

19. The method of claim 1, further comprising:

determining an action of a detected person by analyzing depth data associated with the person;

controlling a device within the building in response to the detected action.

20. The method of claim 19, wherein determining an action of the detected person comprises detecting that the person is waving an arm.

21. The method of claim 20, wherein controlling a device within the building in response to the detected action comprises controlling at least one of:

light operation;

projector operation; and

blind operation.

22. The method of claim 1, further comprising:

creating the refined blob by;

determining a first region of the first blob slice that overlaps the second blob slice from a top down view; and

including the first region of the first blob slice in the refined blob, without including a second region of the first blob slice in the refined blob.

23. The method of claim 22, wherein the second region of the first blob slice corresponds to a shadow or reflection.

24. A method of monitoring a building comprising:

taking a video within a location in the building, the video comprising a plurality of frames, each frame including 2D image data;

for each frame, receiving depth data associated with the 2D image data, the depth data corresponding to one or more distances from a depth sensor to features represented by the 2D image data;

analyzing the 2D image data to detect at least a first image blob in the video;

using the depth data, projecting the image blob onto a plurality of Z-planes, thereby creating a plurality of blob slices;

based on a height threshold, separating the blob slices into a ground plane blob slice, and non-ground plane blob slices;

creating a refined blob that includes non-ground plane blob slices, and only a portion of the ground plane blob slice;

performing object detection on the refined blob, to determine that the blob corresponds to a human object, thereby detecting a person in the video; and

controlling a system of the building in response to the detected person.

25. The method of claim 24, wherein the ground plane blob slice corresponds to a shadow or reflection.

26. The method of claim 24, further comprising:

creating the refined blob by;

determining a first region of the ground plane blob slice that overlaps the non-ground plane blob slices from a top down view; and

including the first region of the ground plane blob slice in the refined blob, without including a second region of the ground plane blob slice in the refined blob.