System architecture and process for assessing multi-perspective multi-context abnormal behavior

• US 8,169,481 B2
• Filed: 05/05/2008
• Issued: 05/01/2012
• Est. Priority Date: 05/05/2008
• Status: Expired due to Fees

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Ozdemir, Hasan Timucin, Li, Hongbing, Liu, Lipin, Lee, Kuo Chu, Joo, Namsoo
• Primary Examiner(s): Nguyen, Thu
• Assistant Examiner(s): TRAN, NAM T
• Application Number:
  US12/034,181
• Publication Number:
  US 20090276705A1
• Time in Patent Office: 1,457 Days
• Field of Search: None
• US Class Current: 348/155
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A multi-perspective context sensitive behavior assessment system includes an adaptive behavior model builder establishing a real-time reference model that captures intention of motion behavior. It operates by modeling outputs of multiple user defined scoring functions with respect to multiple references of application specific target areas of interest. The target areas have criticality values representing a user'"'"'s preference regarding the target areas with respect to one another. The outputs of the scoring functions are multiplied by the critically values to form high level sequences of representation that are communicated to the user.

Claims

1. A multi-perspective context sensitive behavior assessment system comprising:

an adaptive behavior model builder, executed by a processor, establishing a real-time reference model capturing intention of motion behavior by modeling outputs of multiple user defined scoring functions with respect to multiple references of application specific target areas of interest,

wherein at least two of the target areas have criticality values representing a user'"'"'s preference regarding at least one of the target areas with respect to at least one other of the target areas, and

wherein the outputs of the scoring functions are multiplied by the critically values to form high level sequences of representation that are communicated to the user, further comprising a behavior assessment module that performs behavior assessment by;

using the scoring functions associated with application specific target areas to calculate an intention score, and the scoring functions assign a score to a current sample point of a trajectory by aggregating previous scores of the trajectory in a moving time window;

normalizing the intention score by using a score assessment model that is continuously updated for each of plural application contexts;

calculating a target-intention matrix by using the normalized intention scores with the criticality value of each target area; and

assigning an application specific high level semantic label to a moving object by using values in the target-intention matrix and application context descriptions.

2. The system of claim 1, further comprising a set of distributed behavior assessment and detection components, wherein said distributed behavior assessment and detection components describe behaviors of trajectories as high level behaviors defined by temporal and spatial properties of the outputs of the scoring functions for each trajectory with respect to user defined target areas and user defined criticality of each target area for an application context, wherein the set of distributed behavior assessment and detection components comprises:

a score aggregation module producing a target-score aggregation result;

said adaptive behavior model builder that accumulates statistical behavior of motion trajectories and scores for each target area and application context;

a set of dynamically created behavior assessment application instances that collaborate using a subscription and publication event channels;

a knowledge warehouse storing multiple copies of low level models and behavior scoring models that are learned adaptively; and

a multi-dimensional database managing sequences of behavior mean max scores, trajectories, and behavior models.

3. The system of claim 2, wherein said set of components further comprises a GUI notify the user of behavior assessment events and collecting feedback from the user to accumulate patterns into an application specific multidimensional indexing database.

4. The system of claim 1, wherein said model builder transforms trajectory data in real-time for each of plural application contexts by aggregating and summarizing scores obtained from a user defined set of intention detection functions and user assigned criticality weights of application specific target areas to an application specific context aware representation, wherein the application specific context aware representation is a semantic representation of time stamped events corresponding to each sample point of a motion trajectory and a set of the semantic events is decided by a specific application context.

5. The system of claim 4, wherein said model builder utilizes observations from different sensors by applying error bounds for obtaining a decreased measurement error.

6. The system of claim 4, wherein a notification of a desired application context specific behavior detection is subjected to a multidimensional fast search in the multidimensional indexing database to decide whether the notification should be filtered out.

7. The system of claim 6, wherein the multidimensional fast search employs user defined distance functions to judge the similarity.

8. The system of claim 1, wherein said model builder performs feature extraction for a trajectory associated with a deployed application by:

(a) obtaining target specific score values by applying user selected intention functions to target areas specified in an application context;

(b) normalizing the target specific score values by applying reference score assessment models that are continuously updated;

(c) weighting each of the target specific score values by their respective criticality values, wherein the criticality values and the target areas user-defined;

(d) finding an element having a maximum score value in a target-score evaluation matrix; and

(e) determining an application specific semantic event by using an element-specific target area and its intention function, wherein the element-specific target area is associated with the element having the maximum score value.

9. The system of claim 8, wherein said model builder performs said feature extraction for each deployed application to obtain application specific feature vectors of motion trajectory, thereby transforming a single set of motion trajectory data into application specific behavior assessments for each application context supporting multi-perspective behavior assessment of a moving object.

10. The system of claim 1, further comprising a configuration manager implementing a configuration and deployment process for multiple user applications sharing a same sensor infrastructure, wherein the configuration and deployment process includes:

selecting a camera;

defining an application identifier for one of the applications;

creating a record in a configuration manager for the application;

configuring application specific target areas of interest in a field of view of the camera;

recording descriptions of the target areas of interest in the configuration manager and associations thereof with application contexts;

producing a configuration of application specific intention functions;

recording the intention functions and their configuration in the configuration manager;

associating the intention functions with the application contexts; and

deploying the application into a moving object behavior assessment system.

11. The system of claim 1, wherein said model builder adaptively builds and stores a statistical score assessment model for each of plural application contexts with respect to the application specific intention functions and target areas in an application context description, and score assessment models are built for each field of view of plural cameras.

12. The system of claim 1, wherein said model builder builds and stores measurement error models with respect to target areas of applications in different sensors, and utilizes the error models to select more reliable input that is used during an intentions calculation.

13. The system of claim 1, wherein said model builder checks data inconsistency before performing an intention evaluation, thereby suppressing erroneous observations, reducing false behavior assessments, and increasing accuracy of spatio-temporal object detection event models.

14. The system of claim 1, wherein said model builder collects and stores application specific high level behavior data obtained by transforming observed motion data for a query and report generation process, thereby supporting application specific forensic usage of the data in the system.

15. The system of claim 1, wherein said behavior assessment module is instantiated for each of the application contexts for high level interpretation of a same motion trajectory for different applications sharing a same surveillance infrastructure, thereby providing multi-perspective evaluation of the motion trajectory.

16. The system of claim 1, wherein said behavior assessment module receives a scoring evaluation of motion trajectories observed by multiple cameras for a same application specific target area and selects a most accurate evaluation by utilizing error models associated with fields of view of the multiple cameras.

17. The system of claim 1, further comprising a multidimensional indexing module supporting fast access to multidimensional score assessment models, error models, and normal behavior models.

18. The system of claim 1, further comprising a graphical user interface supplying a summary view of a real time global severity ranked alert summary report providing statistical summary data based on abnormal behavior assessment aggregated scores that are dynamically aggregated from at least one of:

one or multiple distributed scoring sources;

or one or more distributed data warehouses, thereby providing real time top ranked severity alert incidents in a graphical chart to security operation users.

19. The system of claim 1 further comprising a graphic user interface displaying a summary view of a real time global highest score dashboard report that provides statistical summary data of a max score at each time point based on abnormal behavior assessment aggregated scores that are dynamically aggregated from at least one of:

one or multiple distributed scoring sources;

or one or more distributed data warehouse, thereby providing a real time max score curve to provide security operation users with situational awareness summary view in terms of incident severity.

20. The system of claim 1 further comprising a graphic user interface displaying a real time view of a max score gauge dashboard for selected individual scoring function types, thereby providing a real time max score gauge of selected individual scoring function types at a current time point, and permitting users to select what scoring function types to be displayed.

21. The system of claim 20, wherein said graphic user interface also displays a statistical distribution of historical score data that is displayed contemporaneously with the gauge, and dynamically calculates a score scaling function from a learned score data model for global score normalization.

22. The system of claim 21, wherein said graphic user interface further displays a global ranking alert report that is based on global ranking of normalized scores that are compared to multiple scoring sources of same scoring function types and also to multiple scoring sources of different scoring function types.