Method of and system for hierarchical human/crowd behavior detection

• US 20090222388A1
• Filed: 11/17/2008
• Published: 09/03/2009
• Est. Priority Date: 11/16/2007
• Status: Active Grant

• Current Assignee: Axis AB (Canon Inc.)
• Original Assignee: Agilence Incorporated
• Inventors: Hua, Wei, Lu, Juwei, Chen, Xiangrong, Crabb, Ryan, Cook, Jonathan
• Granted Patent: US 8,195,598 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion

Abstract

The present invention is directed to a computer automated method of selectively identifying a user specified behavior of a crowd. The method comprises receiving video data but can also include audio data and sensor data. The video data contains images a crowd. The video data is processed to extract hierarchical human and crowd features. The detected crowd features are processed to detect a selectable crowd behavior. The selected crowd behavior detected is specified by a configurable behavior rule. Human detection is provided by a hybrid human detector algorithm which can include Adaboost or convolutional neural network. Crowd features are detected using textual analysis techniques. The configurable crowd behavior for detection can be defined by crowd behavioral language.

Claims

1. A computer automated method of selectively identifying a behavior of a crowd comprising the steps:

receiving video data of a crowd;

generating hierarchical human and crowd feature data from the video data; and

selectively identifying a behavior of the crowd by processing the hierarchical human and crowd feature data according to a configurable behavior rule wherein the processing is configured to select a behavior.

2. The method of claim 1, further comprising the step:

receiving audio data corresponding to an environment around the crowd;

processing audio data thereby identifying audio characteristics,

wherein the selectively identifying a behavior utilizes the identified audio characteristics.

3. The method of claim 2, further comprising the step:

receiving sensor data corresponding to the environment wherein the sensor data is at least one of a GPS data, location, weather data, date data, and time data;

and wherein the generating hierarchical human and crowd feature data from the video data includes processing the sensor data.

4. The method of claim 3, wherein the processing the audio data includes processing the sensor data to identify audio characteristics.

5. The method of claim 4, further comprising the step of generating the behavior rules using the behavior description language, the graphical tool, or a combination thereof.

6. The method of claim 2, wherein the hierarchical human and crowd feature data includes crowd level features, crowd components, individual person features, and audio features.

7. The method of claim 5, wherein the generating the crowd feature data includes building a multiple reference image background model using day night detection and ground plane calibration.

8. The method of claim 7, wherein the generating crowd level features includes at least one of a hybrid human detector comprising a multiple learning based human detector using an interleaving scanning method, texture analysis using fusion thereby generating crowd density, crowd count, crowd location, crowd size, and optical flow analysis for the crowd dynamics and motion features, thereby providing accurate human localization and crowd density estimation.

9. The method of claim 8, wherein the hybrid human detector comprising multiple learning based human detector comprises at least one of an Adaboost algorithm, a convolutional neural network, or a combination thereof, wherein the Adaboost or convolutional algorithm comprises at least one of a human head detector, a upper body detector, a full body detector, or a combination thereof.

10. The method of claim 9, wherein the individual people feature data includes one or more of coarse level feature data, multiple hypothesis tracking data thereby generating tracking results comprised of people tracking and multiple hypothesis tracking data, and wherein the tracking results are used to get bidirectional count and speed features for user defined counting lines.

11. The method of claim 3, further comprising the step of:

controlling a camera field of view using the behavior, generating a human perceivable indication of the behavior, or a combination thereof.

12. The method of claim 3, further comprising the step of using the identification of an individual person and displaying the path of a person over time within a video scene.

13. One or more processor readable storage devices having processor readable code embodied on the processor readable devices for programming one or more processors to perform the method of claim 1.

14. The one or more devices of claim 13, wherein the readable storage devices are further configured to perform the steps:

receiving audio data corresponding to an environment around the crowd; and

processing audio data thereby identifying audio characteristics,

wherein the selectively identifying a behavior utilizes the identified audio characteristics.

15. The one or more devices 13, wherein the readable storage devices are further configured to perform the steps:

receiving sensor data, wherein the sensor data is at least one of a GPS data, location, weather data, date data, and time data;

and wherein the generating hierarchical human and crowd feature data from the video data includes processing the sensor data.

16. A computer automated method of characterizing a queue of people comprising the steps:

receiving video data of a queue of people comprising a queue entrance region, a queue exit region, and a queue region of interest;

generating crowd features from the queue region of interest;

generating first human features from the queue entrance region;

generating second human features form the queue exit region; and

estimating a queue length, a queue speed, a queue wait-time from at least one of the crowd features, the first human features, and the second human features.