Methods, devices and systems for detecting objects in a video

• US 9,443,143 B2
• Filed: 03/01/2015
• Issued: 09/13/2016
• Est. Priority Date: 09/12/2012
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Fortress Corporation (Motorola Solutions, Inc.)
• Inventors: Zhang, Zhong, Yin, Weihong, Venetianer, Peter
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US14/634,838
• Publication Number:
  US 20150178571A1
• Time in Patent Office: 562 Days
• Field of Search: 382/100, 382/103, 382106-107, 382117-118, 382/154, 382/155, 382/162, 382/168, 382/173, 382/181, 382189-199, 382/209, 382/219, 382/232, 382/254, 382/274, 382/276, 382/291, 382/305, 382/312, 348/143, 348/142, 348/154, 348/159, 348/86
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Methods, devices and systems for performing video content analysis to detect humans or other objects of interest a video image is disclosed. The detection of humans may be used to count a number of humans, to determine a location of each human and/or perform crowd analyzes of monitored areas.

Claims

1. A method of detecting human objects in a video, comprising:

determining pixels of a video image are foreground pixels, a group of the foreground pixels constituting a foreground blob set of one or more foreground blobs;

for each of N predetermined shapes at corresponding ones of N predetermined locations within the video image, where N is a positive integer greater than one, calculating a first value by comparing an overlap of the corresponding predetermined shape with the foreground blob set, the first value being used to obtain a corresponding probability of a human at the corresponding predetermined location, thereby obtaining N probabilities corresponding to the N locations;

using the N probabilities, determining X humans are represented by the foreground blob set, where X is a whole number; and

providing at least one of a report, an alarm, and an event detection using the determination of the representation of X humans,

wherein a size of the corresponding predetermined shape for each of the N locations is determined in response to calibration of the video system, and

wherein the comparing of the corresponding predetermined shape with the foreground blob set for each of the N predetermined shapes comprises analyzing an amount of an overlapping area of the corresponding predetermined shape with the foreground blob set.

2. The method of claim 1, further comprising using the N probabilities to determine a location of each of the X humans.

3. The method of claim 2, wherein the determined location of each of the X humans is a location within an image plane corresponding to the video image.

4. The method of claim 2, wherein the determined location of each of the X humans is a location with respect to a physical ground plane corresponding to the real world.

5. The method of claim 1,

wherein determining foreground pixels of the video image comprises comparison of a first frame of a video image without foreground objects with comparison of a second frame of the video image containing the foreground objects.

6. The method of claim 1, wherein the predetermined shape is the same for each of the N locations.

7. The method of claim 1, wherein the predetermined shape for at least some of the N locations has a different size.

8. The method of claim 1,

wherein the calibration of the video system comprises determining an image size of a portion of the video image corresponding to an average human size at each of the N locations, and

wherein the size of the predetermined shape for each of the N locations is determined in response to the corresponding image size.

9. The method of claim 1,

further comprising, prior to the determination of the pixels of the video image that are foreground pixels, for each of the N locations, determining the corresponding predetermined shape by estimating a foreground image part to be occupied in the video image when a human exists at the corresponding location.

10. The method of claim 9, wherein estimating the foreground image part for each of the N locations is based on a projection of a model of a human in the real world onto an image plane of the video image.

11. The method of claim 1,

wherein the video image comprises a plurality of image frames, each image frame comprising a two dimensional image having the N locations, each of the N locations identified by a corresponding x, y coordinate pair within the two dimensional image.

12. The method of claim 11,

wherein each of the N locations is associated with a corresponding one of the N predetermined shapes with respect to an image plane corresponding to the video image.

13. The method of claim 1, further comprising, for each of the N locations, calculating a recall ratio of the corresponding predetermined shape and the foreground blob to determine an associated probability.

14. The method of claim 13, wherein for each of the N locations, calculating the recall ratio comprises determining a ratio of (a) an area comprising an overlap of an area occupied by the predetermined shape and the foreground blob and (b) an area of the foreground blob.

15. The method of claim 1, further comprising:

creating a probability map with the N probabilities; and

determining local maximums of probabilities of the probability map.

16. The method of claim 15, further comprising:

selecting a first location of the N locations corresponding to a local maximum of the probability map;

obtaining a first predetermined shape corresponding to the first location; and

analyzing an amount of an overlap of an area occupied by the first predetermined shape and the foreground blob.

17. The method of claim 16, further comprising:

calculating a first ratio of (a) an area comprising an overlap of an area occupied by the first predetermined shape and the foreground blob and (b) an area of the foreground blob,

wherein the first ratio is used to determine that X humans are represented by the foreground blob.

18. The method of claim 15, further comprising calculating a precision value and a recall value for each of m locations of the N locations, m being an integer, each of the m locations corresponding to a local maximum of the probability map.

19. The method of claim 1, further comprising:

selecting a subset of the N predetermined shapes based on the N probabilities; and

analyzing an overlap of an area occupied by selected subset of N predetermined shapes and an area occupied by foreground blob.

20. The method of claim 1, further comprising, sequentially selecting 1 to m ones of the N locations, a selection of an (m−

1)th location excluding selection of a subsequent one of the N locations that falls within a first predetermined distance of the (m−

1)th location.

21. The method of claim 1, further comprising sequentially selecting 1 to m ones of the N locations, wherein the selection of a next location of the N locations comprises selecting a location based on its proximity to a bottom edge of the video image.

22. A method of detecting human objects in a video, comprising:

determining pixels of a video image of a real world scene are foreground pixels, a group of the foreground pixels constituting a foreground blob set of one or more foreground blobs;

for each of N predetermined shapes at corresponding ones of N predetermined locations within the video image, where N is a positive integer greater than one, calculating a first value by comparing an overlap of the corresponding predetermined shape with the foreground blob set, the first value being used to determine X humans are represented by the foreground blob set, where X is whole number and a location of each of the X humans is determined as a location within a horizontal plane of the real world; and

providing at least one of a report, an alarm, and an event detection when the crowd density exceeds a threshold value, using the determination of the representation of X humans,

wherein a size of the corresponding predetermined shape for each of the N locations is determined in response to calibration of the video system,

wherein the video system is used to obtain the video image,

wherein the comparing of the corresponding predetermined shape with the foreground blob for each of the N predetermined shapes comprises analyzing an amount of an overlapping area of the corresponding predetermined shape with the foreground blob set.

23. The method of claim 22, further comprising detecting the existence of a crowd by reviewing at least some of the locations of the X humans.

24. The method of claim 22, further comprising determining an existence of a crowd when it is determined that Y of the X humans are located within an area of the horizontal plane of the real world.

25. The method of claim 24, wherein the area of the horizontal plane of the real word comprises a predetermined geometric shape having a predetermined area size within the real world.

26. The method of claim 24, wherein the area of the horizontal plane of the real world comprises an area defined by a circle.

27. The method of claim 22, further comprising:

determining a first crowd density within a first area corresponding to a first frame of the video image;

determining a second crowd density within the first area corresponding to a second frame of the video image;

determining a crowd gathering event in response to the first crowd density and the second crowd density.

28. The method of claim 22, further comprising:

determining a first crowd density within a first area corresponding to a first frame of the video image;

determining a second crowd density within the first area corresponding to a second frame of the video image;

determining a crowd dispersing event in response to the first crowd density and the second crowd density.

29. A video surveillance system, comprising:

a video source configured to provide a video image of a real world scene; and

a computer, configured to detect foreground pixels of the video image, a group of the foreground pixels constituting a foreground blob set of one or more foreground blobs,

wherein for each of N predetermined shapes at corresponding ones of N predetermined locations within the video image, where N is a positive integer greater than one, configured to calculate a first value by comparing an overlap of the corresponding predetermined shape with the foreground blob set, configured to determine X humans are represented by the foreground blob set using the first value, where X is whole number, configured to compare the corresponding predetermined shape with the foreground blob for each of the N predetermined shapes by analyzing an amount of an overlapping area of the corresponding predetermined shape with the foreground blob set and configured to provide at least one of a report, an alarm, and an event detection when the crowd density exceeds a threshold value, using the determined representation of X humans.