SYSTEM AND METHOD FOR VIDEO CONTENT ANALYSIS USING DEPTH SENSING

• US 20130182904A1
• Filed: 01/17/2013
• Published: 07/18/2013
• Est. Priority Date: 01/17/2012
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Zhang, Zhong, Myers, Gary W., Venetianer, Peter L.
• Granted Patent: US 9,247,211 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A61B 2505/07 Home care
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/0046 Arrangements of imaging apparatus in a room, e.g. room provided with shielding or for improved access to apparatus
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1072 measuring distances on the body, e.g. measuring length, height or thickness A61B5/1076 takes precedence
• CPC Class Codes: A61B 5/1073 Measuring volume, e.g. of limbs
• CPC Class Codes: A61B 5/1079 using optical or photographic means
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/1117 Fall detection
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/7282 Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 : from multiple images
• CPC Class Codes: G06T 7/579 : from motion
• CPC Class Codes: G06T 7/62 : of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 20/41 : Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 : Event detection
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/1365 : Matching; Classification
• CPC Class Codes: G08B 13/19615 : wherein said pattern is defined by the user
• CPC Class Codes: G08B 21/043 : detecting an emergency event, e.g. a fall
• CPC Class Codes: G08B 21/0476 : Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

A method and system for performing video content analysis based on two-dimensional image data and depth data are disclosed. Video content analysis may be performed on the two-dimensional image data, and then the depth data may be used along with the results of the video content analysis of the two-dimensional data for tracking and event detection.

Claims

1. A video content analysis method comprising:

capturing a video sequence that includes a plurality of frames, each frame including a video image;

for each frame, receiving image data of the video image and also receiving depth data associated with the video image;

analyzing the image data to detect one or more objects depicted in the video sequence;

using the depth data along with the one or more detected objects to classify at least a first object of the one or more detected objects as an object to be tracked;

performing tracking on at least the first classified object; and

performing event detection analysis on the first classified object.

2. The video content analysis method of claim 1, further comprising:

using the depth data along with the one or more detected objects to additionally classify at least a second object of the one or more detected objects as an object not to be tracked.

3. The video content analysis method of claim 1, wherein:

classifying the first object as an object to be tracked includes classifying the first object as a person.

4. The video content analysis method of claim 1, wherein:

classifying the first object as an object to be tracked includes classifying the first object as an object above a predetermined height or volume threshold.

5. The video content analysis method of claim 1, wherein analyzing the image data to detect one or more objects depicted in the video sequence includes detecting at least one blob that corresponds to the one or more objects.

6. The video content analysis method of claim 1, further comprising:

classifying at least the first object of the one or more detected objects as an object to be tracked by using the depth data associated with the one or more detected objects and without analyzing depth data associated with a portion of the video image that is not part of the one or more detected objects.

7. The video content analysis method of claim 1, wherein the depth data is determined by a single depth sensor.

8. A video content analysis method comprising:

receiving a video sequence that includes a plurality of frames, each frame including a video image;

for each frame, receiving two-dimensional (2D) image data and also receiving depth data;

processing the 2D image data of the video sequence to differentiate foreground data from background data and to detect one or more blobs comprised of the foreground data, the one or more blobs corresponding to one or more objects;

for each detected blob, using the depth data to determine whether at least part of the blob corresponds to at least part of a target; and

after it is determined that at least part of a blob corresponds to at least part of a target, tracking the target and detecting at least one event associated with the target.

9. The method of claim 8, wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target includes:

using the depth data to determine that part or all of a first blob corresponds to a first target.

10. The method of claim 9, wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target includes:

using the depth data to determine that part of the first blob does not correspond to the first target.

11. The method of claim 9, wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target includes:

using the depth data to determine that part of the first blob corresponds to a second target different from the first target.

12. The method of claim 11, wherein a first part of the first blob corresponds to the first target and a second part of the first blob corresponds to a second target, one of the first and second target occluding at least part of the other.

13. The method of claim 9, wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target includes:

using the depth data to determine that a second blob combined with part or all of the first blob correspond to a second target.

14. The method of claim 8, wherein the 2D image data includes RGB data for each pixel in the video image.

15. The method of claim 14, wherein detecting the one or more blobs is performed without using the depth data.

16. The method of claim 15, wherein determining whether at least part of each blob corresponds to at least part of a target is performed without analyzing depth data associated with the background data.

17. The method of claim 8, wherein using the depth data to determine whether at least part of each blob corresponds to at least part of a target comprises:

using the depth data to determine one or more of a height and a volume of the blob; and

using one or more of the height and the volume of the blob to determine whether at least part of the blob corresponds to a target.

18. The method of claim 17, wherein determining whether the blob corresponds to a target includes determining whether the blob is a person.

19. A video content analysis method comprising:

receiving a video sequence that includes a plurality of frames, each frame including a video image;

for each frame, receiving two-dimensional (2D) image data and also receiving depth data;

processing the 2D image data of the video sequence to differentiate foreground data from background data and to detect one or more blobs comprised of the foreground data, the one or more blobs corresponding to one or more objects;

for each detected blob, using the depth data to determine whether to track at least part of the blob as a target; and

after it is determined to track the target, detecting at least one event associated with the target.

20. The method of claim 19, wherein using the depth data to determine whether to track at least part of the blob as a target includes:

determining that part or all of a first blob corresponds to a first target.

21. The method of claim 20, wherein using the depth data to determine whether to track at least part of the blob as a target includes:

determining that part of the first blob does not correspond to the first target.

22. The method of claim 20, wherein using the depth data to determine whether to track at least part of the blob as a target includes:

determining that a part of the first blob corresponds to a second target different from the first target.

23. The method of claim 22, wherein determining that part of the first blob corresponds to the second target includes determining that the first blob corresponds to a first person and the second blob corresponds to a second person, one of the first and second person occluding at least part of the other.

24. The method of claim 20, wherein using the depth data to determine whether to track at least part of the blob as a target includes:

determining that a second blob combined with part or all of the first blob corresponds to a second target.

25. The method of claim 19, wherein the 2D image data includes RGB data for each pixel in the video image.

26. The method of claim 25, wherein detecting the one or more blobs is performed without using the depth data.

27. The method of claim 26, wherein determining whether to track at least part of the blob as a target is performed without analyzing depth data associated with the background data.

28. The method of claim 19, wherein using the depth data to determine whether to track at least part of the blob as a target includes:

using the depth data to determine one or more of a height and a volume of the blob; and

using one or more of the height and volume of the blob to determine whether the blob corresponds to a target.

29. The method of claim 28, wherein determining whether the blob corresponds to a target includes determining whether the blob is a person.

30. A video surveillance system comprising:

one or more sensors that capture two-dimensional (2D) image data and depth data; and

a video content analysis system configured to;

receive a video sequence that includes a plurality of frames, each frame including a video image;

for each frame, receive the 2D image data of the video image for that frame, and also receive depth data associated with the video image;

analyze the image data to detect one or more objects depicted in the video sequence;

use the depth data along with the one or more detected objects to classify at least a first object of the one or more detected objects as an object to be tracked;

perform tracking on at least the first classified object; and

perform event detection analysis on the first classified object.