Detection of abnormal crowd behavior

• US 7,558,404 B2
• Filed: 11/28/2005
• Issued: 07/07/2009
• Est. Priority Date: 11/28/2005
• Status: Active Grant

• Current Assignee: Columbia Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Honeywell International Inc.
• Inventors: Bazakos, Michael E., Ma, Yunqian
• Primary Examiner(s): Tabatabai; Abolfazl
• Application Number:
  US11/287,627
• Publication Number:
  US 20070121999A1
• Time in Patent Office: 1,317 Days
• Field of Search: 382/103, 382/107, 382/149, 382/236, 382/287, 348/94, 348/152, 348/154, 348/155, 348/169, 348/170, 348/171, 348/172, 348/208.1, 348/208.2, 348/208.14, 348/347, 348/352, 348/353, 348/356, 348/465, 348/579, 714/39
• US Class Current: 382/103
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A system and method detects the intent and/or motivation of two or more persons or other animate objects in a video scene. In one embodiment, the system forms a blob of the two or more persons, draws a bounding box around said blob, calculates an entropy value for said blob, and compares that entropy value to a threshold to determine if the two or more persons are involved in a fight or other altercation.

Claims

1. A method comprising a computer to perform:

tracking two or more animate objects in a video scene;

forming a single blob by combining into the single blob image data corresponding to said two or more animate objects;

forming a bounding box around said blob; and

calculating a spatial and temporal entropy value for said bounding box.

2. The method of claim 1, further comprising determining whether said entropy value is greater than a threshold.

3. The method of claim 2, wherein said determining whether said entropy value is greater than said threshold indicates that said two or more animate objects are involved in a fight.

4. The method of claim 1, further comprising determining whether said entropy value is within a certain percentage of a threshold.

5. The method of claim 1, further comprising setting a region of interest around said bounding box.

6. The method of claim 5, wherein said region of interest is approximately 25% greater than said bounding region.

7. The method of claim 6, wherein said entropy value is calculated for said region of interest.

8. The method of claim 1, further comprising normalizing said entropy value.

9. The method of claim 8, wherein said normalized entropy is calculated on a per pixel basis as follows:

$D<br><br>\left(x,y,t\right)=\{\begin{array}{ccc}1& \mathrm{if}& <br><br>I<br><br>\left(x,y,t\right)-I<br><br>\left(x,y,t-\mathrm{\tau <br><br>}\right)<br><br>><br><br>\mathrm{\varepsilon <br><br>}\\ 0& & \mathrm{otherwise}\end{array}$

wherein

D(x, y, t) represents a change in intensity of a pixel;

ε

is a statistical variance; and

τ

is a time interval; and

further wherein the normalized entropy comprises;

$\mathrm{entropy}=\frac{1}{T}<br><br>\frac{1}{S_B}<br><br>\underset{t}{\sum <br><br>}<br><br>\underset{\left(x,y\right)\in <br><br>B}{\sum <br><br>}<br><br>D<br><br>\left(x,y,t\right)$

wherein

S_B denotes a size of a region of interest; and

T denotes a temporal duration of the entropy that is calculated after normalization.

10. The method of claim 8, wherein said normalized entropy is calculated on a per sub-window basis as follows:

$m_i<br><br>\left(t\right)=\frac{1}{n_{\mathrm{pixel}}}<br><br>\underset{\left(x,y\right)\in <br><br>\mathrm{Window\_i}}{\sum <br><br>}<br><br>I<br><br>\left(x,y,t\right)$

wherein

m_i(t) is the mean intensity value of a sub-window i at time t; and

further wherein

$D^{*}<br><br>\left(i,t\right)=\{\begin{array}{ccc}1& \mathrm{if}& <br><br>m_i<br><br>\left(t\right)-m_i<br><br>\left(t-\mathrm{\tau <br><br>}\right)<br><br>><br><br>{\mathrm{\varepsilon <br><br>}}^{*}\\ 0& & \mathrm{otherwise}\end{array}$

wherein

ε

is a statistical variance; and

τ

is a time interval; and

further wherein

$\mathrm{entropy}=\frac{1}{T}<br><br>\frac{1}{n_{\mathrm{sub}<br><br>\text{-}<br><br>\mathrm{window}}}<br><br>\underset{i}{\sum <br><br>}<br><br>D^{*}<br><br>\left(i,t\right)$

wherein

T denotes a temporal duration of the entropy that is calculated after normalization; and

n_sub-window denotes the number of sub-windows.

11. The method of claim 1, further comprising determining the real world location of said two or more animate objects in said video scene.

12. The method of claim 1, wherein said entropy value comprises a mean intensity.

13. A system comprising a computer to perform:

a tracking module to track two or more animate objects in a video scene;

a module to form a single blob by combining into the single blob image data corresponding to said two or more animate objects;

a module to form a bounding box around said blob; and

a calculation module to calculate a spatial and temporal entropy value for said bounding box.

14. The system of claim 13, further comprising a module to:

set a region of interest around said blob;

calculate said entropy value for said region of interest; and

normalize said entropy value.

15. The system of claim 13, wherein said module to normalize said entropy value is on a pixel basis and comprises:

$D<br><br>\left(x,y,t\right)=\{\begin{array}{ccc}1& \mathrm{if}& <br><br>I<br><br>\left(x,y,t\right)-I<br><br>\left(x,y,t-\mathrm{\tau <br><br>}\right)<br><br>><br><br>\mathrm{\varepsilon <br><br>}\\ 0& & \mathrm{otherwise}\end{array}$

wherein

D(x, y, t) represents a change in intensity of a pixel;

ε

is a statistical variance; and

τ

is a time interval; and

further wherein the normalized entropy comprises;

$\mathrm{entropy}=\frac{1}{T}<br><br>\frac{1}{S_B}<br><br>\underset{t}{\sum <br><br>}<br><br>\underset{\left(x,y\right)\in <br><br>B}{\sum <br><br>}<br><br>D<br><br>\left(x,y,t\right)$

wherein

S_B denotes a size of a region of interest; and

T denotes a temporal duration of the entropy that is calculated after normalization.

16. The system of claim 13, wherein said module to normalize said entropy value is on a sub-window basis and comprises:

$m_i<br><br>\left(t\right)=\frac{1}{n_{\mathrm{pixel}}}<br><br>\underset{\left(x,y\right)\in <br><br>\mathrm{Window\_i}}{\sum <br><br>}<br><br>I<br><br>\left(x,y,t\right)$

wherein

m_i(t) is the mean intensity value of a sub-window i at time t; and

further wherein

$D^{*}<br><br>\left(i,t\right)=\{\begin{array}{ccc}1& \mathrm{if}& <br><br>m_i<br><br>\left(t\right)-m_i<br><br>\left(t-\mathrm{\tau <br><br>}\right)<br><br>><br><br>{\mathrm{\varepsilon <br><br>}}^{*}\\ 0& & \mathrm{otherwise}\end{array}$

wherein

ε

is a statistical variance; and

τ

is a time interval; and

further wherein

$\mathrm{entropy}=\frac{1}{T}<br><br>\frac{1}{n_{\mathrm{sub}-\mathrm{window}}}<br><br>\underset{i}{\sum <br><br>}<br><br>D^{*}<br><br>\left(i,t\right)$

wherein

T denotes a temporal duration of the entropy that is calculated after normalization; and

n_sub-window denotes the number of sub-windows.

17. A machine readable medium having stored instructions thereon for executing a process comprising:

tracking two or more animate objects in a video scene;

forming a single blob by combining into the single blob image data corresponding to said two or more animate objects;

forming a bounding box around said blob; and

calculating a spatial and temporal entropy value for said bounding box.

18. The machine readable medium of claim 17, further comprising comparing said entropy value to a threshold to determine whether said entropy value is greater than said threshold or said entropy value is within a certain percentage of said threshold.

19. The machine readable medium of claim 17, wherein said entropy value is normalized on a per pixel basis and comprises:

$D<br><br>\left(x,y,t\right)=\{\begin{array}{ccc}1& \mathrm{if}& <br><br>I<br><br>\left(x,y,t\right)-I<br><br>\left(x,y,t-\mathrm{\tau <br><br>}\right)<br><br>><br><br>\mathrm{\varepsilon <br><br>}\\ 0& & \mathrm{otherwise}\end{array}$

wherein

D(x, y, t) represents a change in intensity of a pixel;

ε

is a statistical variance; and

τ

is a time interval; and

further wherein the normalized entropy comprises;

$\mathrm{entropy}=\frac{1}{T}<br><br>\frac{1}{S_B}<br><br>\underset{t}{\sum <br><br>}<br><br>\underset{\left(x,y\right)\in <br><br>B}{\sum <br><br>}<br><br>D<br><br>\left(x,y,t\right)$

wherein

S_B denotes a size of a region of interest; and

T denotes a temporal duration of the entropy that is calculated after normalization.

20. The machine readable medium of claim 17, wherein said entropy value is normalized on a per sub-window basis and comprises:

$m_i<br><br>\left(t\right)=\frac{1}{n_{\mathrm{pixel}}}<br><br>\underset{\left(x,y\right)\in <br><br>\mathrm{Window\_i}}{\sum <br><br>}<br><br>I<br><br>\left(x,y,t\right)$

wherein

m_i(t) is the mean intensity value of a sub-window i at time t; and

further wherein

$D^{*}<br><br>\left(i,t\right)=\{\begin{array}{ccc}1& \mathrm{if}& <br><br>m_i<br><br>\left(t\right)-m_i<br><br>\left(t-\mathrm{\tau <br><br>}\right)<br><br>><br><br>{\mathrm{\varepsilon <br><br>}}^{*}\\ 0& & \mathrm{otherwise}\end{array}$

wherein

ε

is a statistical variance; and

τ

is a time interval; and

further wherein

$\mathrm{entropy}=\frac{1}{T}<br><br>\frac{1}{n_{\mathrm{sub}-\mathrm{window}}}<br><br>\underset{i}{\sum <br><br>}<br><br>D^{*}<br><br>\left(i,t\right)$

wherein

T denotes a temporal duration of the entropy that is calculated after normalization; and

n_sub-window denotes the number of sub-windows.