Unsupervised learning of events in a video sequence

• US 20060053342A1
• Filed: 09/09/2004
• Published: 03/09/2006
• Est. Priority Date: 09/09/2004
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Bazakos, Michael E., Ma, Yunqian, Morellas, Vassilios
• Granted Patent: US 7,606,425 B2
• Time in Patent Office: Days
• US Class Current: 714/37
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19606 Discriminating between target movement or movement in an area of interest and other non-signicative movements, e.g. target movements induced by camera shake or movements of pets, falling leaves, rotating fan
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19673 Addition of time stamp, i.e. time metadata, to video stream
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19693 using multiple video sources viewed on a single or compound screen
• CPC Class Codes: G08B 21/0423 detecting deviation from an expected pattern of behaviour or schedule
• CPC Class Codes: G08B 21/0476 Cameras to detect unsafe condition, e.g. video cameras

Abstract

Methods and systems for the unsupervised learning of events contained within a video sequence, including apparatus and interfaces for implementing such systems and methods, are disclosed. An illustrative method in accordance with an exemplary embodiment of the present invention may include the steps of providing a behavioral analysis engine, initiating a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object located within an image sequence, and then analyzing the feature vector to determine a number of possible event candidates. The behavioral analysis engine can be configured to prompt the user to confirm whether an event candidate is a new event, an existing event, or an outlier. Once trained, a testing/operational phase mode of the behavioral analysis engine can be further implemented to detect the occurrence of one or more learned events, if desired.

Claims

1. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence;

initiating a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object disposed within the image sequence; and

analyzing the feature vector to determine a number of possible event candidates.

2. The method of claim 1, wherein the behavioral analysis engine includes an actor feature database, a frame feature database, and/or an event library.

3. The method of claim 1, wherein the feature vector includes a cluster of points disposed within a feature space of the image sequence.

4. The method of claim 1, wherein the feature vector is a multi-dimensional vector.

5. The method of claim 4, wherein said multi-dimensional vector is a three-dimensional vector.

6. The method of claim 1, wherein the feature vector includes a number of spatial parameters.

7. The method of claim 1, wherein the feature vector includes a number of motion parameters.

8. The method of claim 1, wherein the feature vector includes a number of temporal parameters.

9. The method of claim 1, further including the step of prompting a user to confirm whether a detected event candidate is a new event subsequent to the step of analyzing the feature vector to determine a number of possible event candidates.

10. The method of claim 9, wherein the said step of prompting a user to confirm whether a detected event candidate is a new event is accomplished using a graphical user interface.

11. The method of claim 9, further including the step of storing the event within an event library subsequent to the step of confirming that a detected event candidate is a new event.

12. The method of claim 1, further including the step of applying a time-consistency filtering routine to the image sequence prior to said step of analyzing the feature vector to determine a number of possible event candidates.

13. The method of claim 1, wherein said step of analyzing the feature vector to determine a number of possible event candidates is performed automatically by the behavioral analysis engine.

14. The method of claim 1, wherein said step of analyzing the feature vector to determine a number of possible event candidates is performed using an unsupervised learning algorithm.

15. The method of claim 14, wherein said unsupervised learning algorithm includes a clustering routine.

16. The method of claim 9, wherein said step of confirming a detected event candidate is performed by the user.

17. The method of claim 1, further including the step of initiating a testing phase mode within the behavioral analysis engine, said testing phase mode adapted to detect the occurrence of one or more learned events.

18. The method of claim 17, wherein, during the testing phase mode of the behavioral analysis engine, the method further includes the steps of:

comparing features contained within the feature vector against one or more learned events;

determining the probability that an event has occurred; and

outputting a response to a user if the probability that the event has occurred is greater than a confidence threshold value.

19. The method of claim 1, wherein said behavioral analysis engine is a software engine.

20. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence, the behavioral analysis engine including an actor feature database, a frame feature database, and a programmable event library;

initiating a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object disposed within the image sequence;

analyzing the feature vector to determine a number of possible event candidates;

prompting a user to confirm whether a detected event candidate is a new event; and

storing the new event within the event library if the detected event candidate is confirmed by the user.

21. The method of claim 20, wherein the feature vector includes a cluster of points disposed within a feature space of the image sequence.

22. The method of claim 20, wherein the feature vector is a multi-dimensional vector.

23. The method of claim 22, wherein said multi-dimensional vector is a three-dimensional vector.

24. The method of claim 20, wherein the feature vector includes a number of spatial parameters.

25. The method of claim 20, wherein the feature vector includes a number of motion parameters.

26. The method of claim 20, wherein the feature vector includes a number of temporal parameters.

27. The method of claim 20, wherein the said step of prompting a user to confirm whether a detected event candidate is a new event is accomplished using a graphical user interface.

28. The method of claim 20, further including the step of step of storing an event within the event library subsequent to the step of confirming that a detected event candidate is a new event.

29. The method of claim 20, further including the step of applying a time-consistency filtering routine to the image sequence prior to said step of analyzing the feature vector to determine a number of possible event candidates.

30. The method of claim 20, wherein said step of analyzing the feature vector to determine a number of possible event candidates is performed automatically by the behavioral analysis engine.

31. The method of claim 20, wherein said step of analyzing the feature vector to determine a number of possible event candidates is performed using an unsupervised learning algorithm.

32. The method of claim 31, wherein said unsupervised learning algorithm includes a clustering routine.

33. The method of claim 20, wherein said step of confirming a detected event candidate is performed by the user.

34. The method of claim 20, further including the step of initiating a testing phase mode within the behavioral analysis engine, said testing phase mode adapted to detect the occurrence of one or more learned events.

35. The method of claim 34, wherein, during the training phase mode of the behavioral analysis engine, the method further includes the steps of:

comparing features contained within the feature vector against one or more learned events;

determining the probability that an event has occurred; and

outputting a response to the user if the probability that the event has occurred is greater than a confidence threshold value.

36. The method of claim 20, wherein said behavioral analysis engine is a software engine.

37. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence;

initiating a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object disposed within the image sequence; and

employing a clustering routine to analyze the feature vector to determine a number of possible event candidates.

38. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence;

initiating a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object disposed within the image sequence;

applying a time-consistency filtering routine to the image sequence; and

analyzing the feature vector to determine a number of possible event candidates.

39. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence;

initiating a testing phase mode within the behavioral analysis engine; and

detecting the occurrence of one or more events learned by the behavioral analysis engine.

40. The method of claim 39, wherein said step of detecting the occurrence of one or more evens learned by the behavioral analysis engine is performed in real-time.

41. The method of claim 39, wherein said step of detecting the occurrence of one or more events learned by the behavioral analysis engine is performed using previously recorded video.

42. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence;

initiating a testing phase mode within the behavioral analysis engine; and

detecting the occurrence of one or more learned events selected by a user.

43. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence;

initiating a testing phase mode within the behavioral analysis engine;

detecting the occurrence of one or more events learned by the behavioral analysis engine;

determining the probability that an event has occurred; and

outputting a response to a user if the probability that the event has occurred is greater than a confidence threshold value.

44. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence;

initiating a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object disposed within the image sequence;

analyzing the feature vector to determine a number of possible event candidates;

prompting a user to confirm whether a detected event candidate is a new event;

storing the new event within an event library if the detected event candidate is confirmed by the user;

initiating a testing phase mode within the behavioral analysis engine;

comparing features contained within the feature vector against one or more learned events contained within the event library;

determining the probability that an event has occurred; and

outputting a response to the user if the probability that the event has occurred is greater than a confidence threshold value.

45. A method of learning events contained within a video sequence, the method comprising the steps of:

providing a behavioral analysis engine adapted to learn new events contained within an image sequence;

initiating a training phase mode within the behavioral analysis engine and obtaining a feature vector including one or more parameters relating to an object disposed within the image sequence;

analyzing the feature vector to determine a number of possible event candidates; and

wherein said step of analyzing the feature vector to determine a number of possible event candidates is performed automatically using a clustering routine employed by the behavioral analysis engine.

46. A video surveillance system, comprising:

at least one image sensor adapted to produce an image sequence within a field of view; and

a behavioral analysis engine adapted to learn new events contained within the image sequence.

47. The video surveillance system of claim 46, wherein said behavioral analysis engine includes an actor feature database, a frame feature database, and/or an event library.

48. The video surveillance system of claim 46, further including a means for configuring events within the behavioral analysis engine.

49. The video surveillance system of claim 48, wherein said means for configuring events within the behavioral analysis engine includes a graphical user interface.

50. The video surveillance system of claim 46, wherein the behavioral analysis engine includes an unsupervised learning algorithm for automatically learning new events contained within the image sequence.

51. The video surveillance system of claim 50, wherein said unsupervised learning algorithm includes a clustering routine.

52. The video surveillance system of claim 46, wherein said behavioral analysis engine is a software engine.

53. A video surveillance system, including:

at least one image sensor adapted to produce an image sequence within a field of view;

a computer configured to run a behavioral analysis engine for automatically learning new events contained within the image sequence; and

a graphical user interface.