Video monitoring system employing hierarchical hidden markov model (HMM) event learning and classification

• US 20030058340A1
• Filed: 06/27/2002
• Published: 03/27/2003
• Est. Priority Date: 09/27/2001
• Status: Active Grant

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Lin, Yun-Ting, Brodsky, Tomas, Gutta, Srinivas, Philomin, Vasanth
• Granted Patent: US 7,076,102 B2
• Time in Patent Office: Days
• US Class Current: 348/159
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 21/0476 Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method and apparatus are disclosed for automatically learning and identifying events in image data using hierarchical HMMs to define and detect one or more events. The hierarchical HMMs include multiple paths that encompass variations of the same event. Hierarchical HMMs provide a framework for defining events that may be exhibited in various ways. Each event is modeled in the hierarchical HMM with a set of sequential states that describe the paths in a high-dimensional feature space. These models can then be used to analyze video sequences to segment and recognize each individual event to be recognized. The hierarchical HMM is generated during a training phase, by processing a number of images of the event of interest in various ways, typically observed from multiple viewpoints.

Claims

1. A method for representing an event in a computer vision system, comprising:

processing a number of images of said event; and

generating a hierarchical HMM to represent said event, said hierarchical HMM including a plurality of paths each representing a variation of said event.

2. The method of claim 1, wherein said images include images of said event manifested in various ways.

3. The method of claim 1, wherein said images include images of said event observed from multiple viewpoints.

4. The method of claim 1, wherein said generating step further comprises the steps of initializing said hierarchical HMM with two states and iteratively training said hierarchical HMM parameters to add additional states, until an overall likelihood criterion is satisfied.

5. The method of claim 1, wherein said generating step further comprises the step of splitting a state in said hierarchical HMM into two states if the state model is not sufficiently representative to create a branch from the split state.

6. The method of claim 1, wherein said generating step further comprises the step of reducing overlapping states in said hierarchical HMM model.

7. The method of claim 1, wherein said variation of said event is a variation in terms of a speed of said event.

8. The method of claim 1, wherein said variation of said event is a variation in terms of a direction of said event.

9. The method of claim 1, wherein said variation of said event is a variation in terms of intermediate states of a body posture associated with said event.

10. The method of claim 1, further comprising the step of comparing an image sequence to said hierarchical HMM to recognize said event.

11. A method for representing an event in a computer vision system, comprising:

comparing a sequence of input images to a hierarchical HMM representing said event; and

detecting said event if said comparing step exceeds a predefined criteria.

12. The method of claim 11, wherein said detecting step further comprises the step of finding a path in said hierarchical HMM that matches said sequence of input images.

13. The method of claim 11, wherein said hierarchical HMM includes a plurality of paths each representing a variation of said event.

14. The method of claim 13, wherein said variation of said event is a variation in terms of a speed of said event.

15. The method of claim 13, wherein said variation of said event is a variation in terms of a direction of said event.

16. The method of claim 13, wherein said variation of said event is a variation in terms of intermediate states of a body posture associated with said event.

17. The method of claim 11, further comprising the step of evaluating context information associated with said event.

18. A system for representing an event in a computer vision system, comprising:

a memory that stores computer-readable code; and

a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to;

process a number of images of said event; and

generate a hierarhical HMM to represent said event, said hierarchical HMM including a plurality of paths each representing a variation of said event.

19. The system of claim 18, wherein said processor is further configured to initialize said hierarchical HMM with two states and iteratively training said hierarchical HMM parameters to add additional states, until an overall likelihood criterion is satisfied.

20. The system of claim 18, wherein said processor is further configured to split a state in said hierarchical HMM into two states if the state model is not sufficiently representative to create a branch from the split state.

21. The system of claim 18, wherein said processor is further configured to reduce overlapping states in said hierarchical HMM model.

22. The system of claim 18, wherein said processor is further configured to compare an image sequence to said hierarchical HMM to recognize said event.

23. A system for representing an event in a computer vision system, comprising:

a memory that stores computer-readable code; and

a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to;

compare a sequence of input images to a hierarchical HMM representing said event; and

detect said event if said comparing step exceeds a predefined criteria.

24. The system of claim 23, wherein said processor is further configured to find a path in said hierarchical HMM that matches said sequence of input images.

25. The system of claim 23, wherein said hierarchical HMM includes a plurality of paths each representing a variation of said event.

26. An article of manufacture for representing an event in a computer vision system, comprising:

a computer-readable medium having computer-readable code means embodied thereon, said computer-readable program code means comprising;

a step to process a number of images of said event; and

a step to generate a hierarhical HMM to represent said event, said hierarchical HMM including a plurality of paths each representing a variation of said event.

27. An article of manufacture for representing an event in a computer vision system, comprising:

a computer-readable medium having computer-readable code means embodied thereon, said computer-readable program code means comprising;

a step to compare a sequence of input images to a hierarchical HMM representing said event; and

a step to detect said event if said comparing step exceeds a predefined criteria.