System architecture and process for automating intelligent surveillance center operations

• US 7,840,515 B2
• Filed: 02/16/2007
• Issued: 11/23/2010
• Est. Priority Date: 02/16/2007
• Status: Expired due to Fees

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Ozdemir, Hasan Timucin, Lee, Kuo Chu, Li, Hongbing, Liu, Lipin
• Primary Examiner(s): Sparks; Donald
• Assistant Examiner(s): OLUDE AFOLABI, OLATOYOSI
• Application Number:
  US11/676,043
• Publication Number:
  US 20080201277A1
• Time in Patent Office: 1,376 Days
• Field of Search: 706/11, 706/12, 706/13, 706/47, 348/143, 348/150, 434/236
• US Class Current: 706/47
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 31/00 Predictive alarm systems characterised by extrapolation or other computation using updated historic data

Abstract

The intelligent, automated surveillance system collects the interactions between the security personal and the surveillance system during the handling of an alarm. Each alarm is modeled as a “transaction” and each operation/action that a security personal executes modeled as an “event” within the transaction. The collected events within the transaction are in partial order. Furthermore, the system provides a scoring system for a security manager to evaluate the performance of the security guard. The score of the sequence of actions that the security guard performed manually and the system performed automatically for each set of dependent alarms are used to decide future sequence of operation. Security guards can overwrite the automatic sequencing of actions with manual sequence of operations.

Claims

1. An automated surveillance system, comprising:

one or more media capture devices that capture media, and that generate media metadata and alarms;

a graphical user interface communicating the alarms and the captured media to security personnel together with alarm handling rules; and

a learning module that iteratively learns operation sequences executed by the security personnel during the handling of a set of alarms, extracts alarm handling rules from the operation sequences, and presents the alarm handling rules to the security personnel via said graphical user interface, wherein said learning module presents the alarm handling rules to the security personnel as a possible automation, and the system uses the rules to automate at least part of the alarm handling process when the security personnel expresses agreement with the automation via said graphical user interface.

2. The system of claim 1, wherein said learning module adds the rules to a profile of security personnel in response to the security personnel expressing agreement with the automation.

6. The system of claim 1, wherein said learning module extracts the alarm handling rules by discovering an alarm handling process model in the form of a weighted, directed graph.

7. The system of claim 6, further comprising a feedback mechanism for presenting at least part of the alarm handling process model to the security personnel, whereby the security personnel can accept, reject, or update the alarm handling process model.

8. The system of claim 6, further comprising a data mining query mechanism setting data mining conditions on specified data as opposed to an entirety of operation logs.

9. The system of claim 1, further comprising a scoring engine that handles incoming alarms to automatically execute control commands, including PTZ controls, recording controls, monitor controls, and snap shot controls.

3. An automated surveillance system, comprising:

one or more media capture devices that capture media, and that generate media metadata and alarms;

a graphical user interface communicating the alarms and the captured media to security personnel together with alarm handling rules; and

a learning module that iteratively learns operation sequences executed by the security personnel during the handling of a set of alarms, extracts alarm handling rules from the operation sequences, and presents the alarm handling rules to the security personnel via said graphical user interface, wherein said learning module includes an alarm sequence learning module that learns the sequence of input alarms that interest the security personnel, together with the media capture device, media content, and media metadata for each generated alarm.

4. The system of claim 3, wherein said learning module includes an action sequence learning module that learns a sequence of actions taken by the security personnel reacting to a sequence of the alarms.

5. The system of claim 4, wherein said learning module includes a rule maintenance learning module that learns the sequence of input alarms, the sequence of actions taken by the security personnel, and a score assigned by a security manager evaluating the security personnel'"'"'s response.

10. A multi-camera, distributed surveillance system, comprising:

a surveillance metadata mining system;

a real-time decision making and control system;

distributed data mining agents that interact with multiple sub-function modules in the surveillance system to automate an analysis, learning, optimization, and control process; and

an alarm handling process logging component that creates alarm records, assigns a unique session identifier for each stored alarm incident, stores actions taken by a security operator during the handling of an alarm, and extracts alarm handling rules from the stored actions.

11. The system of claim 10, wherein said metadata mining system collects and constructs a context sensitive metadata model for multiple sensors and operators, loads service processing function modules, and creates a repository of alarm and behavior statistical function models in a metadata knowledge warehouse.

12. The system of claim 11, wherein said data mining system observes and controls dynamic association and loading of said distributed data mining agents, said service processing function modules, and said statistical models for a subset of sensors and operators to automate a sequence of surveillance operations.

13. The system of claim 12, wherein the surveillance system comprises interfaces and an execution environment for said data mining agents to interact with said service processing function modules, including alarm handling process logging, alarm handling automation, alarm handling process mining, action rules, and configuration model storage components.

14. The system of claim 10, wherein said alarm handling process logging component stores each alarm with alarm type, alarm time, camera, location within field of view of camera, and assigning a unique identifier for each stored alarm, receives actions taken by the operator from an alarm handling automation component, stores each action information that comprises of alarm identifier, operator identifier, time stamp, action name, input parameter values, and, output parameter values;

and sends events when an alarm record is created and when an action record is created.

15. The system of claim 10, further comprising an alarm handling automation component that is connected to alarm handling process logging component to store the actions taken by the security operator, that executes matching alarm handling automation rules, that interacts with the operator to display at least part of a discovered alarm handling process model and obtain feedback of the operator, that is connected to an alarm handling process mining module to start building of alarm handling process models and to retrieve discovered alarm handling process models, and that is connected to a rule storage component to store and retrieve alarm handling automation rules.

16. The system of claim 15, wherein said alarm handling automation component maintains an alarm handling session by creating an alarm handling session object when a new alarm is received, wherein the alarm handling session object records alarm type, alarm time, camera identifier, alarm location, operator identifier, and context information that includes actions taken and input and output parameter values of executed actions during an alarm handling process.

17. The system of claim 15, wherein said alarm handling automation component collects alarm handling process action data by:

(1) collecting data associated with executed actions during handling of a particular alarm by using a unique identifier, wherein the actions include at least one of actions initiated by the security guard and automatically executed actions enacted according to automation rules, and data associated with an executed action includes an action/method name, input parameter values, and output parameter values;

(2) sending a message to an alarm handling process logging component, wherein the sent message comprises of alarm identifier, operator identifier, time stamp, action name, input parameter values, and output parameter values; and

(3) sending a message to the alarm handling process logging component, the sent message being indicative of termination of an alarm handling process for a particular alarm by using a unique identifier of the alarm.

18. The system of claim 15, wherein said alarm handling automation component executes matching rules by:

(1) employing a rule matching component that can be configured with a set of rules and supports a rule matching function and a function to re-normalize strength values of a set of rules;

(2) retrieving alarm handling automation rules from rule storage associated with the operator and loading the retrieved rules into the rule matching component;

(3) retrieving the matched rules from a rule engine by using data collected in an alarm handling session object, including alarm type, alarm time, camera identifier, alarm location, operator identifier, and sequence of actions taken so far, wherein each item in the sequence records action name, values of input, and output parameters;

(4) if only one rule is matched, executing actions associated with the matched rule and incrementing a strength attribute of the rule;

(5) if more than one rule is matched, listing the rules based on their strength values and asking the user to select which one to execute; and

(6) if the user selects one of the rules, then executing actions associated with the selected rule and incrementing its strength attribute.

19. The system of claim 15, wherein said alarm handling automation component creates a new automation rule by:

(1) submitting alarm handling process discovery requests to an alarm handling process mining component for ad-hoc alarm handling process mining, the request comprising of selection criteria regarding at least one of alarm type, time interval, or operator identifier;

(2) submitting an alarm handling process model retrieval request to an alarm handling process mining component for continuous alarm handling process mining, the request comprising selection criteria regarding at least one of alarm type, time interval, or operator identifier;

(3) displaying the discovered alarm handling process model with a threshold on confidence value, with matching execution paths highlighted, and with rules already in effect rendered in a distinctive manner;

(4) interacting with a security operator in order to automate one or more execution paths in the discovered alarm handling process model;

(5) sending selected steps to an alarm handling process mining process to discover default values for input parameter of actions;

(6) displaying discovered default parameter values;

(7) collecting selections from the operator regarding whether to automate the assignments by accepting or modifying the values;

(8) sending the selections to the alarm handling process mining module to obtain a rule;

(9) displays the new rule to the user for final confirmation; and

(10) if the user accepts the new rule, saving the new rule to a rule storage component, and loading the new rule to a rule engine.

20. The system of claim 10, further comprising an alarm handling process mining component that discovers an alarm handling process model from alarm handling process log data, processes on-going alarm handling process mining requests and stores associated models, returns discovered alarm handling process models associated with alarm handling process mining requests, and extracts rules from discovered alarm handling process models.

21. The system of claim 20, wherein said alarm handling process mining component:

handles ad-hoc alarm handling process discovery by;

(1) selecting alarm handling log data from an alarm handling process logging component by using selection criteria including at least one of selection by alarm type, selection by using time interval, or selection by using operator identifier;

(2) executing alarm handling process discovery to build the alarm handling process model; and

(3) returns the alarm handling process model;

handles continuous alarm handling process discovery for continuous model building by;

(1) storing continuous alarm handling process models;

(2) receiving events published by alarm handling process logging component for events generated in response to creation of alarm records and action records; and

(3) returns the current, discovered alarm handling process model;

discovers default parameter values for alarm handling automation steps by;

(1) selecting alarm handling log data from the alarm handling process logging component by using the selection criteria;

(2) using log data to find an association between values of input attributes of task and values of attributes in context of the previous tasks in an execution path; and

(3) returns discovered results containing a conditional instruction on threshold value of an attribute, and a copy of a value of an attribute in a context of a previous task in the execution path to the input parameter;

generates a rule by;

(1) building a condition part of the rule by using the values of attributes in the context of the previous tasks in the execution path; and

(2) building an action part of the rule by using a discovered or configured default parameter handling instructions and task execution sequences, wherein the action part is expressed in script language and specifies at least one of execution of a single task without operator input, execution of a single task with some input from the operator, execution of a series of tasks without operator input, or execution of a series of tasks with some input from the operator;

handles queries on the alarm handling process model to;

(1) produce a model from an alarm handling process model based on query selection criteria including at least one of selection by using operator via operator identifier, selection by alarm type, or selection by using time interval; and

(2) produce a difference model between at least two models specified by the query selection criteria for the two or more models.

22. The system of claim 20, wherein the alarm handling process model:

(1) is based on graph representation in which nodes in the graph;

(a) represent tasks, actions, and methods; and

(b) contain action name, input parameters, output parameters, and context, wherein context in a start node includes alarm type identifier, alarm time, camera identifier, location of alarm in field of view of camera, and operator identifier, and context in intermediate nodes include action name, values of input parameters, values of output parameters, and context of a previous node;

(2) contains branching rules conditioned on attribute values in the context of a node, wherein conditions include alarm type, time stamp, camera identifier, location of alarm, operator identifier, previous action name, input, and output parameters, and each edge in the graph corresponds to a branching rule from one node to another and contains a confidence measure obtained from data generated by a model building algorithm; and

(3) is serialized in metadata.

23. The system of claim 10, further comprising a rule storage component that supports maintenance of the alarm handling automation rules for each security operator.

24. The system of claim 23, wherein said rule storage component stores alarm handling automation rules for each operator and returns automation rules upon query.

25. The system of claim 10, further comprising a configuration storage component that maintains alarm types, action descriptions, alarm handling process mining algorithms, algorithms to extract rules from alarm handling process models, algorithms for discovery of default parameter assignments, and schema descriptions of alarm handling process models.

26. The system of claim 25, further wherein said configuration storage component:

(1) stores alarm types, wherein each record includes a unique alarm type identifier, an alarm semantic description, and types of devices capable of producing such an alarm;

(2) stores action descriptions, wherein each record includes a method name, specification of input and output parameters with type description;

(3) stores alarm handling process mining algorithms, wherein each record includes algorithm name, executable algorithm implementation code, and an alarm handling process model specification;

(4) stores algorithms to extract rules from alarm handling process models, wherein each record includes algorithm name and executable implementation code;

(5) stores algorithms for discovery of default parameter assignments, wherein each record includes algorithm name and executable implementation code;

(6) stores schema descriptions of alarm handling process models, wherein each record includes model name, schema definition for the model, an identifier for a model building algorithm, an identifier for a rule extraction algorithm, and an identifier for a default parameter assignment discovery algorithm.