ANOMALOUS OBJECT INTERACTION DETECTION AND REPORTING

• US 20150003671A1
• Filed: 06/28/2013
• Published: 01/01/2015
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Behavioral Recognition Systems Incorporated
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: SAITWAL, Kishor Adinath, URECH, Dennis G., COBB, Wesley Kenneth
• Granted Patent: US 9,911,043 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: G06N 3/0409 Adaptive resonance theory [ART] networks
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Techniques are disclosed for analyzing a scene depicted in an input stream of video frames captured by a video camera. The techniques include evaluating sequence pairs representing segments of object trajectories. Assuming the objects interact, each of the sequences of the sequence pair may be mapped to a sequence cluster of an adaptive resonance theory (ART) network. A rareness value for the pair of sequence clusters may be determined based on learned joint probabilities of sequence cluster pairs. A statistical anomaly model, which may be specific to an interaction type or general to a plurality of interaction types, is used to determine an anomaly temperature, and alerts are issued based at least on the anomaly temperature. In addition, the ART network and the statistical anomaly model are updated based on the current interaction.

Claims

1. A computer-implemented method for analyzing a scene depicted in an input stream of video frames captured by a video camera, the method comprising:

receiving two or more sequences, wherein each sequence corresponds to a segment of a trajectory taken by a respective object through the scene;

determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects; and

if the objects interact;

mapping each sequence of the two or more sequences to a sequence cluster, and

determining a rareness value for the sequence clusters to which the sequences map based on a learned joint probability of the sequence clusters.

2. The method of claim 1, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, further comprising:

normalizing the received sequences to n-dimensional vectors; and

normalizing the position and time components of the received sequences.

3. The method of claim 1, wherein the sequence clusters are clusters of an adaptive resonance theory (ART) network, and wherein the ART network is updated based on observed sequences.

4. The method of claim 1, wherein the learned joint probability is retrieved from an ngram trie whose nodes store joint probabilities of sequence cluster pairs, and wherein the ngram trie is updated based on observed sequence cluster pairs.

5. The method of claim 1, wherein the objects are deemed to interact if the objects pass within a given spatial neighborhood of each other in the scene within a given time period.

6. The method of claim 1, further comprising:

determining, using a statistical anomaly model, an anomaly temperature for the rareness value, wherein the anomaly temperature indicates a frequency of occurrence of the rareness value;

determining whether one or more reporting criteria are met based at least on the anomaly temperature; and

if the one or more reporting criteria are met, reporting the interaction.

7. The method of claim 6, further comprising, updating the statistical anomaly model based on observed interactions.

8. The method of claim 6, further comprising:

determining, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects,

wherein the statistical anomaly model used is specific to the type of interaction.

9. The method of claim 8, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction.

10. The method of claim 6, wherein the interaction is reported to a user interface and includes interaction properties (xInt,yInt,tInt), where xInt and yInt indicate x and y positions of the interaction and tInt indicates a time difference of the objects'"'"' trajectories at a spatial intersection of the trajectories, and wherein the user is permitted to create an alert directive to publish another alert if another interacting sequence pair occurs within a given spatial neighborhood (dx,dy) of (xInt,yInt) and temporal neighborhood dt of tInt.

11. The method of claim 1, wherein the rareness value is given by

12. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations for analyzing a scene depicted in an input stream of video frames captured by a video camera, the operations comprising:

receiving two or more sequences, wherein each sequence corresponds to a segment of a trajectory taken by a respective object through the scene;

determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects; and

if the objects interact;

mapping each sequence of the two or more sequences to a sequence cluster, and

determining a rareness value for the sequence clusters to which the sequences map based on a learned joint probability of the sequence clusters.

13. The computer-readable storage medium of claim 12, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, the operations further comprising:

normalizing the received sequences to n-dimensional vectors; and

normalizing the position and time components of the received sequences.

14. The computer-readable storage medium of claim 12, wherein the sequence clusters are clusters of an adaptive resonance theory (ART) network, and wherein the ART network is updated based on observed sequences.

15. The computer-readable storage medium of claim 12, wherein the learned joint probability is retrieved from an ngram trie whose nodes store joint probabilities of sequence cluster pairs, and wherein the ngram trie is updated based on observed sequence cluster pairs.

16. The computer-readable storage medium of claim 12, wherein the objects are deemed to interact if the objects pass within a given spatial neighborhood of each other in the scene within a given time period.

17. The computer-readable storage medium of claim 12, the operations further comprising:

determining, using a statistical anomaly model, an anomaly temperature for the rareness value, wherein the anomaly temperature indicates a frequency of occurrence of the rareness value;

determining whether one or more reporting criteria are met based at least on the anomaly temperature; and

if the one or more reporting criteria are met, reporting the interaction.

18. The computer-readable storage medium of claim 17, the operations further comprising, updating the statistical anomaly model based on observed interactions.

19. The computer-readable storage medium of claim 17, the operations further comprising:

determining, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects,

wherein the statistical anomaly model used is specific to the type of interaction.

20. The computer-readable storage medium of claim 19, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction.

21. The computer-readable storage medium of claim 17, wherein the interaction is reported to a user interface and includes interaction properties (xInt,yInt,tInt), where xInt and yInt indicate x and y positions of the interaction and tInt indicates a time difference of the objects'"'"' trajectories at a spatial intersection of the trajectories, and wherein the user is permitted to create an alert directive to publish another alert if another interacting sequence pair occurs within a given spatial neighborhood (dx,dy) of (xInt,yInt) and temporal neighborhood dt of tInt.

22. The computer-readable storage medium of claim 19, wherein the rareness value is given by

23. A system, comprising:

a processor; and

a memory, wherein the memory includes an application program configured to perform operations for analyzing a scene depicted in an input stream of video frames captured by a video camera, the operations comprising;

receiving two or more sequences, wherein each sequence corresponds to a segment of a trajectory taken by a respective object through the scene,

determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects, and

if the objects interact;

mapping each sequence of the two or more sequences to a sequence cluster; and

determining a rareness value for the sequence clusters to which the sequences map based on a learned joint probability of the sequence clusters.

24. The system of claim 23, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, the operations further comprising:

normalizing the received sequences to n-dimensional vectors; and

normalizing the position and time components of the received sequences.

25. The system of claim 23, wherein the learned joint probability is retrieved from an ngram trie whose nodes store joint probabilities of sequence cluster pairs, and wherein the ngram trie is updated based on observed sequence cluster pairs.

26. The system of claim 23, the operations further comprising:

determining, using a statistical anomaly model, an anomaly temperature for the rareness value, wherein the anomaly temperature indicates a frequency of occurrence of the rareness value;

determining whether one or more reporting criteria are met based at least on the anomaly temperature; and

if the one or more reporting criteria are met, reporting the interaction.

27. The system of claim 26, further comprising:

determining, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects,

wherein the statistical anomaly model used is specific to the type of interaction.

28. The system of claim 27, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction.