Active and adaptive intelligent video surveillance system

• US 8,649,594 B1
• Filed: 06/03/2010
• Issued: 02/11/2014
• Est. Priority Date: 06/04/2009
• Status: Active Grant

• Current Assignee: Axis AB (Canon Inc.)
• Original Assignee: Agilence Incorporated
• Inventors: Hua, Wei, Lu, Juwei, Kang, Jinman, Cook, Jon, Gu, Haisong
• Primary Examiner(s): Lazaro, David
• Assistant Examiner(s): DRABIK, SARAH E
• Application Number:
  US12/802,265
• Time in Patent Office: 1,349 Days
• Field of Search: 348/143, 348/159, 382/155, 382/159, 382/190, 706/14
• US Class Current: 382/159
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06V 10/7784 based on feedback from supervisors
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method for assessing events detected by a surveillance system includes assessing the likelihood that the events correspond to events being monitored from feedback in response to a condition set by a user. Classifiers are created for the events from the feedback. The classifiers are applied to allow the surveillance system improve its accuracy when processing new video data.

Claims

1. A surveillance system for classifying detected events into one or more event types and utilizing user feedback as to whether an event of the detected events is a true event to improve accuracy, comprising:

a video analytics engine embodied on a computer processor configured to;

receive video data from at least one video camera;

generate an on-line data set from received video data, where the on-line data set comprises events and feature data; and

utilize event classifiers as tuning parameters to classify events;

a feedback collection engine embodied on a computer processor configured to present an event to a user and receive feedback from the user as to whether the event is a true event that matches an event type being monitored by the surveillance system;

an active learning engine embodied on a computer processor configured to;

generate event classifiers using an on-line feature data set and feedback received from a user, where the event classifiers can be used to classify detected events into one or more event types;

apply an event classifier to an event and calculate a confidence score for the event representing the confidence of classifying a positive sample;

normalize a plurality of confidence scores of event classifiers by mapping outputs of event classifiers to a common domain;

duplicate the on-line feature data set to K groups with different partitions where each group includes a training set and a validation set with no overlap;

perform iterations of ensemble classifier learning to train a classifier for each of the K groups, where each iteration comprises;

training a classifier for each group with a classification error for each group returned from the previous iteration;

computing a classification error for each of the K groups by applying the classifier trained for each group to the corresponding validation set within the group;

aggregating the computed classification errors to obtain an overall error and determining if a learning stop criterion is satisfied by the overall error;

stopping if the learning stop criterion is satisfied; and

continuing to the next iteration if the learning stop criterion is not satisfied; and

output K ensemble classifiers; and

a surveillance system manager embodied on a computer processor configured to apply an event classifier to a second on-line data set generated by the video analytics engine.

2. The surveillance system of claim 1 wherein the feature data includes images captured by the at least one video camera and the active learning engine is further configured to extract additional features from the images for classifier learning.

3. The surveillance system of claim 2 where the additional features include features extracted using rule-based classification, Harr-like features, and local binary pattern features.

4. The surveillance system of claim 1 wherein the event classifiers comprise the event types of human detection and vehicle detection.

5. The surveillance system of claim 1 wherein an event classifier operates to filter out unwanted events that are determined not to match the event type being monitored.

6. The surveillance system of claim 1 wherein an event classifier operates to sort, classify, and label events into one or more event types.

7. The surveillance system of claim 1 wherein the active learning engine is further configured to utilize off-line feature data from an off-line data set that includes only positive patterns collected from environments other than the present environment to generate event classifiers.

8. The surveillance system of claim 1 wherein the active learning engine is further configured to utilize off-line feature data that includes model data that represents a default average detection system such that the surveillance system maintains average performance for true events that have not been seen by utilizing the model data in generating event classifiers and improves accuracy by using the events that have been seen.

9. The surveillance system of claim 8 wherein the off-line feature data includes representative positive samples of human detection events and vehicle detection events.

10. The surveillance system of claim 1 wherein the event classifiers are generated based on annotation labels that describe the event type.

11. The surveillance system of claim 1 wherein the active learning engine is further configured to factor in expected error detection objectives into a generated event classifier.

12. A method for classifying detected events into one or more event types and utilizing user feedback as to whether an event of the detected events is a true event to improve accuracy using a video surveillance system, the method comprising:

receiving video data from at least one video camera;

generating an on-line data set from received video data, where the on-line data set comprises events and feature data;

utilizing event classifiers as tuning parameters to classify events;

presenting an event to a user and receiving feedback from the user as to whether the event is a true event that matches an event type being monitored by the surveillance system;

generating event classifiers using an on-line feature data set and feedback received from a user, where the event classifiers can be used to classify detected events into one or more event types;

applying an event classifier to an event and calculating a confidence score for the event representing the confidence of classifying a positive sample;

normalizing a plurality of confidence scores of event classifiers by mapping outputs of event classifiers to a common domain;

duplicating the on-line feature data set to K groups with different partitions where each group includes a training set and a validation set with no overlap;

performing iterations of ensemble classifier learning to train a classifier for each of the K groups, where each iteration comprises;

training a classifier for each group with a classification error for each group returned from the previous iteration;

computing a classification error for each of the K groups by applying the classifier trained for each group to the corresponding validation set within the group;

aggregating the computed classification errors to obtain an overall error and determining if a learning stop criterion is satisfied by the overall error;

stopping if the learning stop criterion is satisfied; and

continuing to the next iteration if the learning stop criterion is not satisfied;

outputting K ensemble classifiers; and

applying an event classifier to a second on-line data set generated by the video analytics engine.

13. The method of claim 12 wherein the feature data includes images captured by the at least one video camera and the method further comprises extracting additional features from the images for classifier learning.

14. The method of claim 13 where the additional features include features extracted using rule-based classification, Harr-like features, and local binary pattern features.

15. The method of claim 12 wherein the event classifiers comprise the event types of human detection and vehicle detection.

16. The method of claim 12 wherein an event classifier operates to filter out unwanted events that are determined not to match the event type being monitored.

17. The method of claim 12 wherein an event classifier operates to sort, classify, and label events into one or more event types.

18. The method of claim 12 wherein generating event classifiers using an on-line feature data set and feedback received from a user further comprises utilizing off-line feature data from an off-line data set that includes only positive patterns collected from environments other than the present environment to generate event classifiers.

19. The method of claim 12 wherein generating event classifiers using an on-line feature data set and feedback received from a user further comprises utilizing off-line feature data that includes model data that represents a default average detection system such that the surveillance system maintains average performance for true events that have not been seen by utilizing the model data in generating event classifiers and improves accuracy by using the events that have been seen.

20. The method of claim 19 wherein the off-line feature data includes representative positive samples of human detection events and vehicle detection events.

21. The method of claim 12 wherein the event classifiers are generated based on annotation labels that describe the event type.

22. The method of claim 12 wherein generating event classifiers using an on-line feature data set and feedback received from a user further comprises factoring in expected error detection objectives into a generated event classifier.