INTRA-TRAJECTORY ANOMALY DETECTION USING ADAPTIVE VOTING EXPERTS IN A VIDEO SURVEILLANCE SYSTEM

• US 20110043626A1
• Filed: 08/18/2009
• Published: 02/24/2011
• Est. Priority Date: 08/18/2009
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Saitwal, Kishor Adinath, COBB, WESLEY KENNETH, Friedlander, David Samuel
• Granted Patent: US 8,379,085 B2
• Time in Patent Office: Days
• US Class Current: 348/143
• CPC Class Codes: G06F 18/2137 based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

A sequence layer in a machine-learning engine configured to learn from the observations of a computer vision engine. In one embodiment, the machine-learning engine uses the voting experts to segment adaptive resonance theory (ART) network label sequences for different objects observed in a scene. The sequence layer may be configured to observe the ART label sequences and incrementally build, update, and trim, and reorganize an ngram trie for those label sequences. The sequence layer computes the entropies for the nodes in the ngram trie and determines a sliding window length and vote count parameters. Once determined, the sequence layer may segment newly observed sequences to estimate the primitive events observed in the scene as well as issue alerts for inter-sequence and intra-sequence anomalies.

Claims

1. A computer-implemented method for analyzing a scene depicted in an input stream of video frames captured by a video camera of a video surveillance system, the method comprising:

receiving a first sequence storing an ordered string of labels, wherein each label in the string corresponds to a cluster in an adaptive resonance theory (ART) network, wherein the string of labels is generated by mapping kinematic data vectors generated for a foreground object detected in the input stream of video frames to nodes of a self-organizing map (SOM) and clustering the nodes of the SOM using the ART network;

identifying one or more segments in the first sequence, wherein each segment includes a subsequence of the ordered string of labels in the first sequence;

determining a probability of observing each of the one or more segments relative to a probability distribution generated from an ngram trie, wherein the ngram trie is generated from a plurality of previously observed sequences, wherein each sequence of the plurality stores an ordered string of labels assigned to clusters in the ART network;

upon determining a first one of the segments has a probability of being observed below a specified threshold, issuing an alert to a user of the video surveillance system.

2. The computer-implemented method of claim 1, wherein identifying one or more segments in the first sequence, comprises:

evaluating the subsequences in the first sequence using a sliding window of length (L), wherein an internal entropy expert and a boundary entropy expert each vote a position in each sliding window subsequence at which to split the sliding window sequence; and

inducing a segment boundary at each position in the first sequence that receives at least a number of votes satisfying a vote count threshold (V).

3. The computer-implemented method of claim 2, wherein:

the internal entropy expert votes to minimize internal entropy by selecting a position which minimizes internal entropy in each respective sliding window sequence, based on the determined entropies; and

the boundary entropy expert votes by selecting a position in each respective sliding window that maximizes entropy between segments in the sliding window subsequence.

4. The computer-implemented method of claim 2, further comprising, determining the sliding window length (L) as

5. The computer-implemented method of claim 2, wherein the vote count threshold (v) is determined as follows

6. The computer-implemented method of claim 1, wherein the probability distribution generated from the ngram trie is an exponential probability distribution function, and wherein the specified threshold is a mean (μ

_H) of the exponential probability distribution function.

7. The computer-implemented method of claim 1, wherein the subsequence in the first one of the segments is determined to include a label not present in the ngram trie.

8. The computer-implemented method of claim 1, wherein the subsequence in the first one of the segments is determined to include a string of labels not present in the ngram trie.

9. The computer-implemented method of claim 1, further comprising:

determining a segment count M as a number of segments induced in the plurality of previously observed sequences;

determining a count m of segments identified in the first sequence; and

upon determining a ratio m/M falls below a specified threshold, issuing an alert to a user of the video surveillance system.

10. A computer-readable storage medium containing a program, which when executed on a processor, performs an operation for analyzing a scene depicted in an input stream of video frames captured by a video camera of a video surveillance system, the operation comprising:

receiving a first sequence storing an ordered string of labels, wherein each label in the string corresponds to a cluster in an adaptive resonance theory (ART) network, wherein the string of labels is generated by mapping kinematic data vectors generated for a foreground object detected in the input stream of video frames to nodes of a self-organizing map (SOM) and clustering the nodes of the SOM using the ART network;

identifying one or more segments in the first sequence, wherein each segment includes a subsequence of the ordered string of labels in the first sequence;

determining a probability of observing each of the one or more segments relative to a probability distribution generated from an ngram trie, wherein the ngram trie is generated from a plurality of previously observed sequences, wherein each sequence of the plurality stores an ordered string of labels assigned to clusters in the ART network;

upon determining a first one of the segments has a probability of being observed below a specified threshold, issuing an alert to a user of the video surveillance system.

11. The computer-readable storage medium of claim 10, wherein identifying one or more segments in the first sequence, comprises:

evaluating the subsequences in the first sequence using a sliding window of length (L), wherein an internal entropy expert and a boundary entropy expert each vote a position in each sliding window subsequence at which to split the sliding window sequence; and

inducing a segment boundary at each position in the first sequence that receives at least a number of votes satisfying a vote count threshold (V).

12. The computer-readable storage medium of claim 11, wherein:

the internal entropy expert votes to minimize internal entropy by selecting a position which minimizes internal entropy in each respective sliding window sequence, based on the determined entropies; and

the boundary entropy expert votes by selecting a position in each respective sliding window that maximizes entropy between segments in the sliding window subsequence.

13. The computer-readable storage medium of claim 11, further comprising, determining the sliding window length (L) as

14. The computer-readable storage medium of claim 11, wherein the vote count threshold (v) is determined as follows

15. The computer-readable storage medium of claim 10, wherein the probability distribution generated from the ngram trie is an exponential probability distribution function, and wherein the specified threshold is a mean (μ

_H) of the exponential probability distribution function.

16. The computer-readable storage medium of claim 10, wherein the subsequence in the first one of the segments is determined to include a label not present in the ngram trie.

17. The computer-readable storage medium of claim 10, wherein the subsequence in the first one of the segments is determined to include a string of labels not present in the ngram trie.

18. A video surveillance system, comprising:

a video input source configured to provide an input stream of video frames captured by a video camera, each depicting a scene;

a processor; and

a memory containing a program, which, when executed on the processor is configured to perform an operation for analyzing the scene depicted in the input stream of video frames, the operation comprising;

receiving a first sequence storing an ordered string of labels, wherein each label in the string corresponds to a cluster in an adaptive resonance theory (ART) network, wherein the string of labels is generated by mapping kinematic data vectors generated for a foreground object detected in the input stream of video frames to nodes of a self-organizing map (SOM) and clustering the nodes of the SOM using the ART network,

identifying one or more segments in the first sequence, wherein each segment includes a subsequence of the ordered string of labels in the first sequence,

determining a probability of observing each of the one or more segments relative to a probability distribution generated from an ngram trie, wherein the ngram trie is generated from a plurality of previously observed sequences, wherein each sequence of the plurality stores an ordered string of labels assigned to clusters in the ART network,

upon determining a first one of the segments has a probability of being observed below a specified threshold, issuing an alert to a user of the video surveillance system.

19. The system of claim 18, wherein identifying one or more segments in the first sequence, comprises:

evaluating the subsequences in the first sequence using a sliding window of length (L), wherein an internal entropy expert and a boundary entropy expert each vote a position in each sliding window subsequence at which to split the sliding window sequence; and

inducing a segment boundary at each position in the first sequence that receives at least a number of votes satisfying a vote count threshold (V).

20. The system of claim 19, wherein:

the internal entropy expert votes to minimize internal entropy by selecting a position which minimizes internal entropy in each respective sliding window sequence, based on the determined entropies; and

the boundary entropy expert votes by selecting a position in each respective sliding window that maximizes entropy between segments in the sliding window subsequence.

21. The system of claim 19, further comprising, determining the sliding window length (L) as

22. The system of claim 19, wherein the vote count threshold (v) is determined as follows

23. The system of claim 18, wherein the probability distribution generated from

the ngram trie is an exponential probability distribution function, and wherein the specified threshold is a mean (μ

_H) of the exponential probability distribution function.

24. The system of claim 18, wherein the subsequence in the first one of the segments is determined to include a label not present in the ngram trie.

25. The system of claim 18, wherein the subsequence in the first one of the segments is determined to include a string of labels not present in the ngram trie.