System and method for video content analysis using depth sensing

• US 9,805,266 B2
• Filed: 01/26/2016
• Issued: 10/31/2017
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Fortress Corporation
• Inventors: Zhang, Zhong, Myers, Gary W., Venetianer, Peter L.
• Primary Examiner(s): CESE, KENNY A
• Application Number:
  US15/006,117
• Publication Number:
  US 20160140397A1
• Time in Patent Office: 644 Days
• CPC Class Codes: A61B 2505/07 Home care
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/0046 Arrangements of imaging apparatus in a room, e.g. room provided with shielding or for improved access to apparatus
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1072 measuring distances on the body, e.g. measuring length, height or thickness A61B5/1076 takes precedence
• CPC Class Codes: A61B 5/1073 Measuring volume, e.g. of limbs
• CPC Class Codes: A61B 5/1079 using optical or photographic means
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1117 Fall detection
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/579 : from motion
• CPC Class Codes: G06T 7/62 : of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 : Event detection
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/1365 : Matching; Classification
• CPC Class Codes: G08B 13/19615 : wherein said pattern is defined by the user
• CPC Class Codes: G08B 21/043 : detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0476 : Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A method and system for performing video content analysis based on two-dimensional image data and depth data are disclosed. Video content analysis may be performed on the two-dimensional image data, and then the depth data may be used along with the results of the video content analysis of the two-dimensional data for tracking and event detection.

Claims

1. A video content analysis method comprising:

capturing a video sequence that includes a plurality of frames, each frame including a video image;

for each frame, receiving two-dimensional (2D) image data of the video image and also receiving depth data associated with the image data;

analyzing the 2D image data, and based on an analysis of the 2D image data without the depth data detecting one or more objects depicted in the video sequence as potential human beings;

using the depth data along with the one or more detected objects to classify at least a first object of the one or more detected objects as a person to be tracked, wherein a volume of the one or more detected objects is used to classify at least the first object as a person to be tracked;

performing tracking on at least the first classified object; and

performing event detection analysis on the first classified object,

wherein the volume is determined by using the depth data along with the 2D image data to determine a plurality of convex hull slices on different Z-planes, and by summing areas of the plurality of convex hull slices.

2. The video content analysis method of claim 1, further comprising:

using the depth data along with the one or more detected objects to additionally classify at least a second object of the one or more detected objects as an object not to be tracked.

3. The video content analysis method of claim 1, wherein:

classifying the first object as an object to be tracked includes classifying the first object as an object above a predetermined height or volume threshold.

4. The video content analysis method of claim 1, wherein analyzing the image data to detect one or more objects depicted in the video sequence includes detecting at least one blob that corresponds to the one or more objects.

5. The video content analysis method of claim 1, further comprising:

classifying at least the first object of the one or more detected objects as a person to be tracked by using the depth data associated with the one or more detected objects and without analyzing depth data associated with a portion of the video image that is not part of the one or more detected objects.

6. The video content analysis method of claim 1, wherein the depth data is determined by a single depth sensor.

7. The video content analysis method of claim 1, wherein analyzing the 2D image data to detect one or more objects depicted in the video sequence as potential human beings includes performing two-dimensional (2D) analysis on the image data to perform motion and change detection.

8. The video content analysis method of claim 7, wherein:

analyzing the 2D image data to detect one or more objects depicted in the video sequence as potential human beings further includes, based on the motion and change detection, detecting at least one blob that corresponds to the one or more objects; and

using the depth data along with the one or more detected objects to classify at least the first object of the one or more detected objects as a person to be tracked includes classifying only part of the blob as a target to be tracked.

9. The video content analysis method of claim 8, further comprising:

using the depth data with the detected blob to determine that part of the blob does not correspond to the target to be tracked.

10. The video content analysis method of claim 8, wherein using the depth data along with the one or more detected objects to classify at least the first object of the one or more detected objects as a person to be tracked includes using the depth data to determine that the one or more detected objects include two people.

11. A video surveillance system comprising:

one or more sensors that capture two-dimensional (2D) image data and depth data; and

a video content analysis system configured to;

receive a video sequence that includes a plurality of frames, each frame including the 2D image data;

for each frame, receive the 2D image data for that frame, and also receive depth data associated with the video image;

analyze the 2D image data to detect at least a first image blob in the video sequence;

use the depth data to project the first image blob onto a plurality of Z-planes, thereby creating a plurality of blob slices;

based on a height threshold, separate the blob slices into a ground plane blob slice, and non-ground plane blob slices;

create a refined blob that includes non-ground plane blob slices, and only a portion of the ground plane blob slice;

perform object detection on the refined blob, to determine that the blob corresponds to a human object, thereby detecting a person in the video;

perform tracking on the detected person; and

perform event detection analysis on the detected person.

12. The video content analysis method of claim 11, wherein performing tracking includes performing motion and change detection.

13. The video content analysis method of claim 11, wherein performing object detection on the refined blob includes determining that the refined blob includes two people.