Unsupervised learning of events in a video sequence

• US 7,606,425 B2
• Filed: 09/09/2004
• Issued: 10/20/2009
• Est. Priority Date: 09/09/2004
• Status: Expired due to Fees

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Bazakos, Michael E., Ma, Yunqian, Morellas, Vassilios
• Primary Examiner(s): Chawan; Sheela C
• Application Number:
  US10/938,244
• Publication Number:
  US 20060053342A1
• Time in Patent Office: 1,867 Days
• Field of Search: 382/103, 382/141, 382/224, 382/225, 435/7.24, 379/265.02, 700/94, 348/150, 348/161, 348/E7.085, 348/143, 348/155, 348/144, 348/148, 348/149, 348/154, 348/169, 348/170, 348/171, 348/172, 348/586, 348/700, 348/699, 714/26, 714/25, 707/104.1, 707/E17.009, 707/3, 707/255, 707/256, 709/224, 709/223, 436/536, 436/805, 436/800, 436/172, 715/256, 704/E15.026, 704/E15.045, 324/402, 324/379, 395/200.11, 395/800, 395/200.08, 395/850, 395/200.01, 702/179, 702/180, 702/181, 702/182, 702187-188, 702/194, 702/196, 717/125, 717/127, 717/132, 717/134, 717/155
• US Class Current: 382/224
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19606 Discriminating between target movement or movement in an area of interest and other non-signicative movements, e.g. target movements induced by camera shake or movements of pets, falling leaves, rotating fan
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19673 Addition of time stamp, i.e. time metadata, to video stream
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: G08B 21/0423 detecting deviation from an expected pattern of behaviour or schedule
• CPC Class Codes: G08B 21/0476 Cameras to detect unsafe condition, e.g. video cameras

Abstract

Methods and systems for the unsupervised learning of events contained within a video sequence, including apparatus and interfaces for implementing such systems and methods, are disclosed. An illustrative method in accordance with an exemplary embodiment of the present invention may include the steps of providing a behavioral analysis engine, initiating a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object located within an image sequence, and then analyzing the feature vector to determine a number of possible event candidates. The behavioral analysis engine can be configured to prompt the user to confirm whether an event candidate is a new event, an existing event, or an outlier. Once trained, a testing/operational phase mode of the behavioral analysis engine can be further implemented to detect the occurrence of one or more learned events, if desired.

Claims

1. A method of learning events contained within a video image sequence, the method comprising:

providing a computing system that is configured to receive the image sequence, the computing system programmed to;

provide a behavioral analysis engine that is configured to learn new events contained within the image sequence;

initiate a training phase mode within the behavioral analysis engine and obtain a feature vector including one or more parameters relating to an object disposed within the image sequence;

identify one or more clusters for at least some of the one or more parameters, at least some of the one or more clusters corresponding to possible event candidates;

display an identifier for at least some of the possible event candidates on a display; and

allow a user to select one or more of the possible event candidates, and to include the selected one or more of the possible event candidates into an event library.

2. The method of claim 1, wherein the behavioral analysis engine of the computing system includes and/or accesses an actor feature database and/or a frame feature database.

3. The method of claim 1, wherein the one or more clusters are within a feature space of the image sequence.

4. The method of claim 1, wherein the feature vector is a multi-dimensional vector.

5. The method of claim 4, wherein said multi-dimensional vector is a three-dimensional vector.

6. The method of claim 1, wherein the feature vector includes a number of spatial parameters.

7. The method of claim 1, wherein the feature vector includes a number of motion parameters.

8. The method of claim 1, wherein the feature vector includes a number of temporal parameters.

9. The method of claim 1, wherein the computing system is further programmed to prompt a user to confirm that a selected possible event candidate is a new event before including the selected possible event candidate into the event library.

10. The method of claim 9, wherein the said step of prompting is accomplished using a graphical user interface on the display.

11. The method of claim 9, wherein the computing system is further programmed to store the selected possible event candidate within the event library.

12. The method of claim 9, wherein said step of prompting a user to confirm that a selected possible event candidate is performed by the user.

13. The method of claim 1, further including the step of applying a time-consistency filtering routine to the image sequence prior to said identifying step.

14. The method of claim 1, wherein said identifying step is performed automatically by the behavioral analysis engine of the computing system.

15. The method of claim 1, wherein said identifying step is performed using an unsupervised learning algorithm.

16. The method of claim 15, wherein said unsupervised learning algorithm includes a clustering routine.

17. The method of claim 1, further including the step of initiating a testing phase mode within the behavioral analysis engine of the computing system, said testing phase mode is configured to detect the occurrence of one or more learned events.

18. The method of claim 17, wherein, during the testing phase mode of the behavioral analysis engine, the computing device is programmed to:

compare features contained within the feature vector against one or more learned events;

determine the probability that an event has occurred; and

output a response to the user if the probability that the event has occurred is greater than a confidence threshold value.

19. The method of claim 1, wherein said behavioral analysis engine is a software engine running on the computing system.

20. A method of learning events contained within a video image sequence, the method comprising:

providing a computing system that is configured to receive the image sequence, the computing system programmed to;

provide a behavioral analysis engine that is configured to learn new events contained within the image sequence, the behavioral analysis engine including an actor feature database, a frame feature database, and a programmable event library stored in a memory;

initiate a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object disposed within the image sequence;

analyze the feature vector to determine a number of possible event candidates;

prompt a user to confirm whether a detected event candidate is a newly identified event; and

store the new event within the event library if the detected event candidate is confirmed by the user.

21. The method of claim 20, wherein the feature vector includes a cluster of points disposed within a feature space of the image sequence.

22. The method of claim 20, wherein the feature vector is a multi-dimensional vector.

23. The method of claim 22, wherein said multi-dimensional vector is a three-dimensional vector.

24. The method of claim 20, wherein the feature vector includes a number of spatial parameters.

25. The method of claim 20, wherein the feature vector includes a number of motion parameters.

26. The method of claim 20, wherein the feature vector includes a number of temporal parameters.

27. The method of claim 20, wherein the computing system is programmed to prompt a user to confirm whether a detected event candidate is a newly identified event via a graphical user interface on a display.

28. The method of claim 20, wherein the computing system is further programmed to store an event within the event library subsequent to the step of confirming that a detected event candidate is a newly identified event.

29. The method of claim 20, wherein the computing system is further programmed to apply a time-consistency filtering routine to the image seguence prior to said step of analyzing the feature vector to determine a number of possible event candidates.

30. The method of claim 20, wherein said step of analyzing the feature vector to determine a number of possible event candidates is performed automatically by the behavioral analysis engine.

31. The method of claim 20, wherein said step of analyzing the feature vector to determine a number of possible event candidates is performed using an unsupervised learning algorithm.

32. The method of claim 31, wherein said unsupervised learning algorithm includes a clustering routine.

33. The method of claim 20, wherein the computing system is further programmed to initiate a testing phase mode within the behavioral analysis engine, said testing phase mode configured to detect the occurrence of one or more learned events.

34. The method of claim 33, wherein, during the training phase mode of the behavioral analysis engine, the computing system is programmed to:

compare features contained within the feature vector against one or more learned events;

determine the probability that an event has occurred; and

output a response to the user if the probability that the event has occurred is greater than a confidence threshold value.

35. The method of claim 20, wherein said behavioral analysis engine is a software engine running on the computing system.

36. A method of learning events contained within a video image sequence, the method comprising:

providing a computing system that is configured to receive the image sequence, the computing system programmed to;

provide a behavioral analysis engine that is configured to learn new events contained within the image sequence;

allow a user to initiate a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object disposed within the image sequence;

apply a time-consistency filtering routine to the image sequence;

analyze the feature vector to determine a number of possible event candidates;

prompt a user to confirm whether a detected event candidate is a newly identified event; and

store the newly identified event within an event library if the detected event candidate is confirmed by the user.

37. A method of learning events contained within a video image sequence, the method comprising:

providing a computing system that is configured to receive the image sequence, the computing system programmed to;

provide a behavioral analysis engine that is configured to learn new events contained within the image sequence;

initiate a testing phase mode within the behavioral analysis engine;

detect the occurrence of one or more possible event candidates;

receive confirmation from a user whether a detected event candidate is a newly identified event; and

store the newly identified event within an event library if the detected event candidate is confirmed by the user.

38. The method of claim 37, wherein said step of detecting the occurrence of one or more possible event candidates is performed in real-time.

39. The method of claim 37, wherein said step of detecting the occurrence of one or more possible event candidates is performed using previously recorded video.

40. A method of learning events contained within a video image sequence, the method comprising the steps of:

providing a computing system that is configured to receive the image sequence, the computing system programmed to;

provide a behavioral analysis engine that is configured to learn new events contained within the image sequence;

detect the occurrence of one or more events learned by the behavioral analysis engine;

determine the probability that an event has occurred; and

output a response to a user if the probability that the event has occurred is greater than a confidence threshold value.