Video surveillance system with rule-based reasoning and multiple-hypothesis scoring

• US 7,499,571 B1
• Filed: 09/22/2006
• Issued: 03/03/2009
• Est. Priority Date: 11/17/2003
• Status: Active Grant

• Current Assignee: Axis AB (Canon Inc.)
• Original Assignee: Vidient Systems, Inc. (Agilence Incorporated)
• Inventors: Tao, Hai, Gong, Yihong, Han, Mei
• Primary Examiner(s): Tucker; Wesley
• Application Number:
  US11/525,795
• Time in Patent Office: 893 Days
• Field of Search: 382/100, 382/103, 348/143, 348152-155, 348169-172, 340/541
• US Class Current: 382/103
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position

Abstract

A video surveillance system uses rule-based reasoning and multiple-hypothesis scoring to detect predefined behaviors based on movement through zone patterns. Trajectory hypothesis spawning allows for trajectory splitting and/or merging and includes local pruning to managed hypothesis growth. Hypotheses are scored based on a number of criteria, illustratively including at least one non-spatial parameter. Connection probabilities computed during the hypothesis spawning process are based on a number of criteria, illustratively including object size. Object detection and probability scoring is illustratively based on object class.

Claims

1. A method for use in an electronic surveillance apparatus comprising a processor and a memory, the memory storing instructions which when executed cause the processor to perform the method comprising the following steps:

a) establishing a plurality of hypotheses associated with a particular point in time, the plurality of hypotheses including at least first and second hypotheses, each of the at least first and second hypotheses comprising a set of objects hypothesized to be in an area under surveillance by the system at the particular point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the hypothesized objects,

b) detecting objects present in the area under surveillance at a subsequent point in time,

c) establishing a first plurality of extended hypotheses associated with the subsequent point in time, the first plurality of extended hypotheses including at least a first extended hypothesis and one or more other extended hypotheses,

i) the first extended hypothesis in the first plurality comprising

     1) at least ones of the trajectories of the first hypothesis,

     2) one or more of the detected objects, and

     3) a particular set of connections extending at least ones of the trajectories of the first hypothesis to the one or more of the detected objects, the first extended hypothesis in the first plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

ii) the one or more other extended hypotheses in the first plurality each comprising

     1) at least ones of the trajectories of the first hypothesis,

     2) one or more of the detected objects, and

     3) a particular set of connections extending at least ones of the trajectories of the first hypothesis to the one or more of the detected objects, each of the one or more other extended hypotheses in the first plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

iii) the first extended hypothesis of the first plurality and at least one of said one or more other extended hypotheses of the first plurality being different from one another,

d) establishing a second plurality of extended hypotheses associated with said subsequent point in time, the second plurality of extended hypotheses including at least a first extended hypothesis and one or more other extended hypotheses,

i) the first extended hypothesis in the second plurality comprising

     1) at least ones of the trajectories of the second hypothesis,

     2) one or more of the detected objects, and

     3) a particular set of connections extending at least ones of the trajectories of the second hypothesis to the one or more of the detected objects, the first extended hypothesis in the second plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at said subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

ii) the one or more other extended hypotheses in the second plurality each comprising

     1) at least ones of the trajectories of the second hypothesis,

     2) so one or more of the detected objects, and

     3) a particular set of connections extending at least ones of the trajectories of the second hypothesis to the one or more of the detected objects, each of the one or more other extended hypothesis in the second plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

iii) the first extended hypothesis of the second plurality and at least one of said one or more other extended hypotheses of the second plurality being different from one another,

e) identifying at least one of the plurality of extended hypotheses associated with said subsequent point in time as being more likely than others of the extended hypotheses associated with that point in time as comprising the actual trajectories of the actual objects in the area under surveillance at that point in time, and

f) utilizing the identified extended hypothesis to determine whether at least one predefined alert condition has occurred within the area under surveillance.

2. The method of claim 1 wherein the identifying includes computing a likelihood for each of the extended hypotheses, the likelihood being a function of connection probabilities each associated with a respective trajectory of that hypothesis, each connection probability being a measure of the probability that an object that terminates the associated trajectory is, in fact, an object in the area under surveillance that is on that trajectory.

3. The method of claim 2 wherein the identified extended hypothesis is the one having the largest computed likelihood.

4. The method of claim 1 wherein the objects are people and wherein the alert condition includes a predetermined pattern of movement of at least one person relative to a fixed object in the area under surveillance.

5. The method of claim 1 wherein the objects are people and wherein the alert condition includes a predetermined pattern of movement that includes at least one time criterion.

6. The method of claim 1 wherein the objects are people and wherein the alert condition includes a predetermined pattern of movement between two or more people.

7. The method of claim 6 wherein the alert condition further includes a predetermined pattern of movement between at least one of the people and a fixed object in the area under surveillance.

8. The method of claim 1 wherein the objects are people and wherein the alert condition includes a predetermined pattern of movement of one or more people relative to at least one of a) at least one other person, b) one or more fixed objects, or c) one or more predefined zones within the area under surveillance.

9. The method of claim 8 wherein the predetermined pattern includes at least one time criterion.

10. An electronic surveillance apparatus comprising a processor and a memory, the memory storing instructions which when executed cause the processor to perform the method comprising:

a) establishing a plurality of hypotheses associated with a particular point in time, the plurality of hypotheses including at least first and second hypotheses, each of the at least first and second hypotheses comprising a set of objects hypothesized to be in an area under surveillance by the system at the particular point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the hypothesized objects,

b) detecting objects present in the area under surveillance at a subsequent point in time,

c) establishing a first plurality of extended hypotheses associated with the subsequent point in time, the first plurality of extended hypotheses including at least a first extended hypothesis and one or more other extended hypotheses,

i) the first extended hypothesis in the first plurality comprising

     1) at least ones of the trajectories of the first hypothesis,

     2) one or more of the detected objects, and

     3) a particular set of connections extending at least ones of the trajectories of the first hypothesis to the one or more of the detected objects, the first extended hypothesis in the first plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

ii) the one or more other extended hypotheses in the first plurality each comprising

     1) at least ones of the trajectories of the first hypothesis,

     2) one or more of the detected objects, and

     3) a particular set of connections extending at least ones of the trajectories of the first hypothesis to the one or more of the detected objects, each of the one or more other extended hypotheses in the first plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

iii) the first extended hypothesis of the first plurality and at least one of said one or more other extended hypotheses of the first plurality being different from one another,

d) establishing a second plurality of extended hypotheses associated with said subsequent point in time, the second plurality of extended hypotheses including at least a first extended hypothesis and one or more other extended hypotheses,

i) the first extended hypothesis in the second plurality comprising

     1) at least ones of the trajectories of the second hypothesis,

     2) one or more of the detected objects, and

     3) a particular set of connections extending at least ones of the trajectories of the second hypothesis to the one or more of the detected objects, the first extended hypothesis in the second plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at said subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

ii) the one or more other extended hypotheses in the second plurality each comprising

     1) at least ones of the trajectories of the second hypothesis,

     2) one or more of the detected objects, and

     3) a particular set of connections extending at least ones of the trajectories of the second hypothesis to the one or more of the detected objects, each of the one or more other extended hypothesis in the second plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising, hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

iii) the first extended hypothesis of the second plurality and at least one of said one or more other extended hypotheses of the second plurality being different from one another,

e) identifying at least one of the plurality of extended hypotheses associated with said subsequent point in time as being more likely than others of the extended hypotheses associated with that point in time as comprising the actual trajectories of the actual objects in the area under surveillance at that point in time, and

f) utilizing the identified extended hypothesis to determine whether at least one predefined alert condition has occurred within the area under surveillance.

11. A computer readable medium on which are stored instructions that are executable by a processor to carry out the method comprising

a) establishing a plurality of hypotheses associated with a particular point in time, the plurality of hypotheses including at least first and second hypotheses, each of the at least first and second hypotheses comprising a set of objects hypothesized to be in an area under surveillance be the system at the particular point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the hypothesized objects,

b) detecting objects present in the area under surveillance at a subsequent point in time,

c) establishing a first plurality of extended hypotheses associated with the subsequent point in time, the first plurality of extended hypotheses including at least a first extended hypothesis and one or more other extended hypotheses,

i) the first extended hypothesis in the first plurality comprising 1) at least ones of the trajectories of the first hypothesis, 2) one or more of the detected objects, and 3) a particular set of connections extending at least ones of the trajectories of the first hypothesis to the one or more of the detected objects, the first extended hypothesis in the first plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

ii) the one or more other extended hypotheses in the first plurality each comprising 1) at least ones of the trajectories of the first hypothesis, 2) one or more of the detected objects, and 3) a particular set of connections extending at least ones of the trajectories of the first hypothesis to the one or more of the detected objects, each of the one or more other extended hypotheses in the first plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

iii) the first extended hypothesis of the first plurality and at least one of said one or more other extended hypotheses of the first plurality being different from one another,

d) establishing a second plurality of extended hypotheses associated with said subsequent point in time, the second plurality of extended hypotheses including at least a first extended hypothesis and one or more other extended hypotheses,

i) the first extended hypothesis in the second plurality comprising 1) at least ones of the trajectories of the second hypothesis, 2) one or more of the detected objects, and 3) a particular set of connections extending at least ones of the trajectories of the second hypothesis to the one or more of the detected objects, the first extended hypothesis in the second plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at said subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

ii) the one or more other extended hypotheses in the second plurality each comprising 1) at least ones of the trajectories of the second hypothesis, 2) one or more of the detected objects, and 3) a particular set of connections extending at least ones of the trajectories of the second hypothesis to the one or more of the detected objects, each of the one or more other extended hypothesis in the second plurality thereby comprising a set of objects hypothesized to be in the area under surveillance at the subsequent point in time and further comprising hypothesized prior trajectories through the area under surveillance for at least ones of the set of hypothesized objects,

iii) the first extended hypothesis of the second plurality and at least one of said one or more other extended hypotheses of the second plurality being different from one another,

e) identifying at least one of the plurality of extended hypotheses associated with said subsequent point in time as being more likely than others of the extended hypotheses associated with that point in time as comprising the actual trajectories of the actual objects in the area under surveillance at that point in time, and

f) utilizing the identified extended hypothesis to determine whether at least one predefined alert condition has occurred within the area under surveillance.