Method and system for scoring surveillance system footage

• US 7,667,596 B2
• Filed: 02/16/2007
• Issued: 02/23/2010
• Est. Priority Date: 02/16/2007
• Status: Active Grant

• Current Assignee: I-Pro Company Limited
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Kibey, Sameer, Joo, Namsoo, Liu, Lipin, Mosali, Supraja, Ozdemir, Hasan Timucin, Lee, Kuo Chu, Yu, Juan, Li, Hongbing
• Primary Examiner(s): Previl; Daniel
• Application Number:
  US11/676,127
• Publication Number:
  US 20080201116A1
• Time in Patent Office: 1,103 Days
• Field of Search: 340/541, 34053922-53926, 340/539.32, 340/568.1, 340/572.1, 706 45- 47
• US Class Current: 340/541
• CPC Class Codes: G08B 13/196 using television cameras
• CPC Class Codes: G08B 21/0423 detecting deviation from an expected pattern of behaviour or schedule
• CPC Class Codes: G08B 31/00 Predictive alarm systems characterised by extrapolation or other computation using updated historic data

Abstract

A surveillance system generally includes a data capture module that collects sensor data. A scoring engine module receives the sensor data and computes at least one of an abnormality score and a normalcy score based on the sensor data, at least one dynamically loaded learned data model, and a learned scoring method. A decision making module receives the at least one of the abnormality score and the normalcy score and generates an alert message based on the at least one of the abnormality score and the normalcy score and a learned decision making method to produce progressive behavior and threat detection.

Claims

1. A surveillance system, comprising:

a data capture module collecting sensor data, wherein the sensor data includes video surveillance footage of an object being monitored;

a scoring engine module receiving the sensor data and computing at least one of an abnormality score and a normalcy score based on the sensor data, at least one dynamically loaded learned data model, and a plurality of learned scoring methods, wherein each learned scoring method generates a subscore used to compute the at least one of the abnormality score and the normalcy score, and wherein the abnormality score and the normalcy score indicate an amount of divergence or adherence to the at least one dynamically loaded learned data model by the object being monitored; and

a decision making module receiving the at least one of the abnormality score and the normalcy score and generating an alert message based on the at least one of the abnormality score and the normalcy score and a learned decision making method to produce progressive behavior and threat detection.

2. The surveillance system of claim 1 further comprising a device configuration module automatically loading the learned scoring methods, the learned decision making methods, and the learned model to at least one of the scoring engine module and the decision making module.

3. The surveillance system of claim 1 further comprising a model builder module adaptively learning the model and wherein the scoring engine module computes the at least one of the abnormality score and the normalcy score based on the adaptively learned models.

4. The surveillance system of claim 1 further comprising a model builder module building the model based on at least one of a simulation of the sensor data and accumulated sensor data.

5. The surveillance system of claim 4 further comprising a graphical user interface accepting parameters from a user to generate the simulation.

6. The surveillance system of claim 1 wherein the learned scoring method calculates an observed property of objects in motion against the model stored in a data cube to obtain a set of scores representing at least one of similarity and difference scores between an object in motion and the learned model.

7. The surveillance system of claim 6 wherein the at least one of the similarity and difference scores are accumulated and normalized for the object in motion, to represent the at least on of normalcy and abnormality scores.

8. The surveillance system of claim 1 further comprising a learning module adaptively learning at least one of the scoring methods, the decision making methods, and the learned model.

9. The surveillance system of claim 1 further comprising an alarm handling module receiving the alert message and generates an alarm message based on a further examination of the alert message.

10. The surveillance system of claim 1 wherein the data capture module collects sensor data from an image sensor and extracts object data from the sensor data, and wherein the scoring engine module computes the at least one of the abnormality score and the normalcy score based on the object data.

11. The surveillance system of claim 1 wherein the decision making module receives at least one of an abnormality score and a normalcy score generated from other sensor data and generates an alert message based on the at least one of the abnormality score and the normalcy score generated from the other sensor data.

12. The surveillance system of claim 1 wherein each learned scoring method has a weight associated therewith, and wherein the scoring engine computes a weighted average of the subscores using the weights associated with each learned scoring method.

13. The surveillance system of claim 1 wherein each learned scoring method analyzes a different type of behavior of the object being monitored, wherein a score associated with a particular scoring method indicates a degree of normalcy or abnormality given the type of behavior being analyzed.

14. A surveillance system, comprising:

a plurality of image sensing devices, wherein the image sensing devices each include;

a data capture module collecting sensor data, wherein the sensor data includes video surveillance footage of an object being monitored;

a scoring engine module receiving the sensor data and computing at least one of an abnormality score and a normalcy score based on the sensor data, at least one dynamically loaded data model, and a plurality of learned scoring methods, wherein each learned scoring method generates a subscore used to compute the at least one of the abnormality score and the normalcy score, and wherein the abnormality score and the normalcy score indicate an amount of divergence or adherence to the at least one dynamically loaded learned data model by the object being monitored, and wherein each learned scoring method analyzes a different type of behavior of the object being monitored; and

a decision making module receiving the at least one of the abnormality score and the normalcy score and generating an alert message based on the at least one of the abnormality score and the normalcy score and a learned decision making method to produce progressive behavior and threat detection.

15. The surveillance system of claim 14 wherein the decision making module of a first image sensing device receives the at least one of the abnormality score and the normalcy score from a second image sensing device, and wherein the decision making module of the first image sensing device generates the alert message based on the at least one of the abnormality score and the normalcy score from the second image sensing device.

16. The surveillance system of claim 14 further comprising a model builder module adaptively learning the predetermined models.

17. The surveillance system of claim 14 wherein the image sensing devices each further include a device configuration automatically loading updated scoring methods, decision making methods, and the learned models to the image sensing device.

18. The surveillance system of claim 14 further comprising a model builder module building models based on a simulation of the sensor data and accumulated real sensor data.

19. The surveillance system of claim 18 further comprising a graphical user interface accepting motion parameters from a user to generate the simulation.

20. The surveillance system of claim 14 further comprising a learning module adaptively learning a decision making method and wherein the decision making method is selectively loaded to at least one of the plurality of image sensing devices.

21. The surveillance system of claim 14 further comprising an alarm handling module receiving the alert messages from the plurality of image sensing devices and generating an alarm message based on a further examination of the alert messages.

22. A surveillance method, comprising:

receiving sensor data, wherein the sensor data includes video surveillance footage of an object being monitored;

dynamically loading data models;

computing at least one of an abnormality score and a normalcy score based on the sensor data, a plurality of learned scoring methods, and the dynamically loaded data models, wherein each learned scoring method generates a subscore used to compute the at least one of the abnormality score and the normalcy score, and wherein the abnormality score and the normalcy score indicate an amount of divergence or adherence to the at least one dynamically loaded learned data model by the object being monitored, and wherein each learned scoring method analyzes a different type of behavior of the object being monitored; and

generating an alert message based on the at least one of the abnormality score and the normalcy score.

23. The surveillance method of claim 22 further comprising selectively loading at least one of scoring methods and decision making methods to be used by at least one of the computing and the generating.

24. The surveillance method of claim 22 further comprising:

adaptively learning the data models, and

wherein the computing comprises computing the at least one of the abnormality score and the normalcy score based on the adaptively learned data models.

25. The surveillance method of claim 22 further comprising building the model based on a simulation of the sensor data.

26. The surveillance method of claim 22 further comprising:

adaptively learning a decision making method, and

wherein the generating comprises generating the alert message based on the adaptively learned decision making method.

27. The surveillance method of claim 22 further comprising:

further examining the alert message; and

generating an alarm message based on the further examining.

28. The surveillance method of claim 22 wherein the receiving comprises receiving sensor data from an image sensor.

29. The surveillance method of claim 28 further comprising:

extracting object data from the sensor data, and

wherein the computing further comprises computing the at least one of the abnormality score and the normalcy score based on the object data.