Anomalous object interaction detection and reporting

• US 9,911,043 B2
• Filed: 06/28/2013
• Issued: 03/06/2018
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Intellective Ai, Inc.
• Original Assignee: Omni AI, Inc.
• Inventors: Saitwal, Kishor Adinath, Urech, Dennis G., Cobb, Wesley Kenneth
• Primary Examiner(s): Ge, Jin
• Application Number:
  US13/931,058
• Publication Number:
  US 20150003671A1
• Time in Patent Office: 1,712 Days
• Field of Search: 345474
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: G06N 3/0409 Adaptive resonance theory [ART] networks
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Techniques are disclosed for analyzing a scene depicted in an input stream of video frames captured by a video camera. The techniques include evaluating sequence pairs representing segments of object trajectories. Assuming the objects interact, each of the sequences of the sequence pair may be mapped to a sequence cluster of an adaptive resonance theory (ART) network. A rareness value for the pair of sequence clusters may be determined based on learned joint probabilities of sequence cluster pairs. A statistical anomaly model, which may be specific to an interaction type or general to a plurality of interaction types, is used to determine an anomaly temperature, and alerts are issued based at least on the anomaly temperature. In addition, the ART network and the statistical anomaly model are updated based on the current interaction.

Claims

1. A computer-implemented method for analyzing a scene depicted in an input stream of video frames captured by a video camera, the method comprising:

receiving at least two sequences, wherein each sequence of the at least two sequences corresponds to a segment of a trajectory taken by a respective object through the scene;

determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects; and

if the objects interact;

mapping each sequence of the at least two sequences to a respective sequence cluster,

retrieving, from an ngram trie, a learned joint probability indicating a likelihood of a given sequence cluster pair of a plurality of sequence cluster pairs occurring in the scene, the ngram trie including a plurality of nodes, the given sequence cluster pair being represented jointly by a first node in a first layer of the ngram trie and a second node in a second layer of the ngram trie, the first node and the second node representing sequence clusters in the given sequence cluster pair, and

determining a rareness value for each sequence cluster pair based on the learned joint probability and a frequency of a most frequently observed sequence cluster pair from the scene, the rareness value given by

2. The method of claim 1, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, further comprising:

normalizing the received sequences to n-dimensional vectors; and

normalizing the position and time components of the received sequences.

3. The method of claim 1, wherein the sequence clusters are clusters of an adaptive resonance theory (ART) network, and wherein the ART network is updated based on observed sequences.

4. The method of claim 1, wherein the ngram trie is updated based on observed sequence cluster pairs.

5. The method of claim 1, wherein the objects are deemed to interact if the objects pass within a given spatial neighborhood of each other in the scene within a given time period.

6. The method of claim 1, further comprising, updating the statistical anomaly model based on observed interactions.

7. The method of claim 1, further comprising:

determining, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects, wherein the statistical anomaly model used is specific to the type of interaction.

8. The method of claim 7, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction.

9. The method of claim 1, wherein the interaction is reported to a user interface and includes interaction properties (xInt, yInt, tInt), where xInt and yInt indicate x and y positions of the interaction and tInt indicates a time difference of the objects'"'"' trajectories at a spatial intersection of the trajectories, and wherein the user is permitted to create an alert directive to publish another alert if another interacting sequence pair occurs within a given spatial neighborhood (dx, dy) of (xInt, yInt) and temporal neighborhood dt of tInt.

10. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations for analyzing a scene depicted in an input stream of video frames captured by a video camera, the instructions comprising instructions to:

receive at least two sequences, each sequence of the at least two sequences corresponding to a segment of a trajectory taken by a respective object through the scene;

determine, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects; and

if the objects interact;

map each sequence of the at least two sequences to a sequence cluster,

retrieve, from an ngram trie, a learned joint probability indicating a likelihood of a given sequence cluster pair from a plurality of sequence cluster pairs occurring in the scene, the ngram trie including a plurality of nodes, the (liven sequence cluster pair being represented jointly by a first node in a first layer of the ngram trie and a second node in a second layer of the ngram trie, the first node and the second node representing sequence clusters in the (liven sequence cluster pair, and

determine a rareness value for each sequence cluster pair of the plurality of sequence cluster pairs based on learned joint probabilities of sequence cluster pairs and a frequency of a most frequently observed sequence cluster pair, the rareness value based on

11. The computer-readable storage medium of claim 10, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, the instructions further comprising instructions to:

normalize the received sequences to n-dimensional vectors; and

normalize the position and time components of the received sequences.

12. The computer-readable storage medium of claim 10, wherein the sequence clusters are clusters of an adaptive resonance theory (ART) network, and wherein the ART network is updated based on observed sequences.

13. The computer-readable storage medium of claim 10, wherein the ngram trie is updated based on observed sequence cluster pairs.

14. The computer-readable storage medium of claim 10, wherein the objects are deemed to interact if the objects pass within a given spatial neighborhood of each other in the scene within a given time period.

15. The computer-readable storage medium of claim 10, the instructions further comprising instructions to:

update the statistical anomaly model based on observed interactions.

16. The computer-readable storage medium of claim 10, the instructions further comprising instructions to:

determine, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects,

wherein the statistical anomaly model used is specific to the type of interaction.

17. The computer-readable storage medium of claim 16, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction.

18. The computer-readable storage medium of claim 10, wherein the interaction is reported to a user interface and includes interaction properties (xInt, yInt, tInt), where xInt and yInt indicate x and y positions of the interaction and tInt indicates a time difference of the objects'"'"' trajectories at a spatial intersection of the trajectories, and wherein the user is permitted to create an alert directive to publish another alert if another interacting sequence pair occurs within a given spatial neighborhood (dx, dy) of (xInt, yInt) and temporal neighborhood dt of tInt.

19. A system, comprising:

a processor; and

a memory, wherein the memory includes an application program configured to perform operations for analyzing a scene depicted in an input stream of video frames captured by a video camera, the operations comprising;

receiving at least two sequences, wherein each sequence of the at least two sequences corresponds to a segment of a trajectory taken by a respective object through the scene,

determining, via one or more processors, whether the objects interact based on a spatio-temporal proximity of the objects, and

if the objects interact;

mapping each sequence of the at least two sequences to a sequence cluster;

retrieving, from an ngram trie, learned joint probabilities of sequence cluster pairs, a learned joint probability indicating a likelihood of a sequence cluster pair occurring in the scene, the ngram trie including a plurality of nodes, the sequence cluster pair being represented jointly by a first node in a first layer of the ngram trie and a second node in a second layer of the ngram trie, the first node and the second node representing sequence clusters in the sequence cluster pair, and

determining a rareness value based on the learned joint probabilities and a frequency of a most frequently observed sequence cluster pair, the rareness value based on

20. The system of claim 19, wherein the received sequences each include one or more data points, each data point representing a position of the respective object at a given point in time, the operations further comprising:

normalizing the received sequences to n-dimensional vectors; and

normalizing the position and time components of the received sequences.

21. The system of claim 19, wherein the ngram trie is updated based on observed sequence cluster pairs.

22. The system of claim 19, the operations further comprising:

determining, based on the sequence clusters to which the sequences map and/or spatio-temporal properties of the sequences, a type of interaction between the objects, wherein the statistical anomaly model used is specific to the type of interaction.

23. The system of claim 22, wherein the type of interaction is one of jail-breaking, collision/scattering, an extended interaction, an anomalous temporal interaction, an anomalous special interaction, and an anomalous spatio-temporal interaction.