Detecting anomalous events using a long-term memory in a video analysis system

• US 8,126,833 B2
• Filed: 12/16/2008
• Issued: 02/28/2012
• Est. Priority Date: 09/11/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Cobb, Wesley Kenneth, Seow, Ming-Jung, Xu, Gang
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Hill, Stanley K
• Application Number:
  US12/336,354
• Publication Number:
  US 20100061624A1
• Time in Patent Office: 1,169 Days
• Field of Search: None
• US Class Current: 706/46
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Techniques are described for detecting anomalous events using a long-term memory in a video analysis system. The long-term memory may be used to store and retrieve information learned while a video analysis system observes a stream of video frames depicting a given scene. Further, the long-term memory may be configured to detect the occurrence of anomalous events, relative to observations of other events that have occurred in the scene over time. A distance measure may used to determine a distance between an active percept (encoding an observed event depicted in the stream of video frames) and a retrieved percept (encoding a memory of previously observed events in the long-term memory). If the distance exceeds a specified threshold, the long-term memory may publish the occurrence of an anomalous event for review by users of the system.

Claims

1. A computer-implemented method for detecting anomalous events using a long- term memory in a video analysis system configured to observe patterns of behavior depicted in a sequence of video frames, comprising:

receiving an active percept, wherein the active percept comprises a sub-graph of a neural network excited by a semantic symbol stream, wherein the semantic symbol stream describes objects depicted in the sequence of video frames;

querying the long-term memory using the active percept as an input stimulus;

receiving, in response to the querying, a retrieved percept from the long-term memory and an occurrence frequency (F) of the retrieved percept, wherein the retrieved percept encodes a pattern of behavior previously observed by the video analysis system;

determining a distance (d) between the active percept and the retrieved percept; and

upon determining the distance (d) exceeds a specified threshold, publishing an alert notification indicating the occurrence of an anomalous event, as represented by the active percept.

2. The method of claim 1, further comprising:

querying an episodic memory using the active percept as an input stimulus, wherein the episodic memory stores percepts encoding instance specific details regarding events observed to have occurred in the sequence of video frames;

receiving a response percept from the episodic memory; and

storing the response percept in the long-term memory.

3. The method of claim 1, wherein the distance d is defined by:

d=1−

[α

S+(1−

α

)F]

wherein S is a cosine similarity determined between the active percept and the retrieved percept, wherein F is the occurrence frequency of the retrieved percept in the long-term memory, and wherein a is α

heuristic weighting coefficient between 0 and 1.

4. The method of claim 1, wherein the distance d is defined by:

d=1−

(S√

{square root over (F)}−

√

{square root over (1−

S²)}√

{square root over (1−

F)})²

wherein S is a cosine similarity determined between the active percept and the retrieved percept, wherein F is the occurrence frequency of the retrieved percept in the long-term memory.

5. The method of claim 1, wherein the distance d is defined by:

6. The method of claim 1, wherein publishing the alert notification comprises:

decoding the active percept to generate a semantic symbol stream corresponding to the active percept; and

generating, from the semantic symbol stream, a linguistic description of a pattern of behavior represented by the semantic symbol stream.

7. The method of claim 1, wherein the long-term memory includes an adaptive resonance theory (ART) network and a sparse distributed memory.

8. The method of claim 1, wherein the long-term memory is configured to store a plurality of percepts, wherein each percept represents a pattern of behavior of observed to have occurred in a sequence of video frames.

9. The method of claim 8, wherein the retrieved percept is composed from two or more percepts retrieved in response to querying the long-term memory using the active percept as an input stimulus.

10. The method of claim 1, further comprising, prior to publishing the alert notification indicating the occurrence of the anomalous event, determining that the long-term memory is in a mature state.

11. A computer-readable storage medium containing a program which, when executed by a processor, performs an operation for detecting anomalous events using a long-term memory in a video analysis system configured to observe patterns of behavior depicted in a sequence of video frames, the operation comprising:

receiving an active percept, wherein the active percept comprises a sub-graph of a neural network excited by a semantic symbol stream, wherein the semantic symbol stream describes objects depicted in the sequence of video frames;

querying the long-term memory using the active percept as an input stimulus;

receiving, in response to the querying, a retrieved percept from the long-term memory and an occurrence frequency (F) of the retrieved percept, wherein the retrieved percept encodes a pattern of behavior previously observed by the video analysis system;

determining a distance (d) between the active percept and the retrieved percept; and

upon determining the distance (d) exceeds a specified threshold, publishing an alert notification indicating the occurrence of an anomalous event, as represented by the active percept.

12. The computer-readable storage medium of claim 11, wherein the operation further comprises:

querying an episodic memory using the active percept as an input stimulus, wherein the episodic memory stores percepts encoding instance specific details regarding events observed to have occurred in the sequence of video frames;

receiving a response percept from the episodic memory; and

storing the response percept in the long-term memory.

13. The computer-readable storage medium of claim 11, wherein the distance d is defined by:

d=1−

[α

S+(1−

α

)F]

wherein S is a cosine similarity determined between the active percept and the retrieved percept, wherein F is the occurrence frequency of the retrieved percept in the long-term memory, and wherein α

is a heuristic weighting coefficient between 0 and 1.

14. The computer-readable storage medium of claim 11, wherein the distance d is defined by:

d=1−

(S√

{square root over (F)}−

√

{square root over (1−

S²)}√

{square root over (1−

F)})²

wherein S is a cosine similarity determined between the active percept and the retrieved percept, wherein F is the occurrence frequency of the retrieved percept in the long-term memory.

15. The computer-readable storage medium of claim 11, wherein the distance d is defined by:

16. The computer-readable storage medium of claim 11, wherein publishing the alert notification comprises:

decoding the active percept to generate a semantic symbol stream corresponding to the active percept; and

generating, from the semantic symbol stream, a linguistic description of a pattern of behavior represented by the semantic symbol stream.

17. The computer-readable storage medium of claim 11, wherein the long-term memory includes an adaptive resonance theory (ART) network and a sparse distributed memory.

18. The computer-readable storage medium of claim 11, wherein the long-term memory is configured to store a plurality of percepts, wherein each percept represents a pattern of behavior of observed to have occurred in a sequence of video frames.

19. The computer-readable storage medium of claim 18, wherein the retrieved percept is composed from two or more percepts retrieved in response to querying the long-term memory using the active percept as an input stimulus.

20. A system, comprising:

a video input source configured to provide a sequence of video frames, each depicting a scene;

a processor; and

a memory containing a machine learning application which when executed by the processor is configured to perform an operation for detecting anomalous events using a long-term memory in a video analysis system configured to observe patterns of behavior depicted in a sequence of video frames, the operation comprising;

receiving an active percept, wherein the active percept comprises a sub-graph of a neural network excited by a semantic symbol stream, wherein the semantic symbol stream describes objects depicted in the sequence of video frames;

querying the long-term memory using the active percept as an input stimulus;

receiving, in response to the querying, a retrieved percept from the long-term memory and an occurrence frequency (F) of the retrieved percept, wherein the retrieved percept encodes a pattern of behavior previously observed by the video analysis system;

determining a distance (d) between the active percept and the retrieved percept; and

upon determining the distance (d) exceeds a specified threshold, publishing an alert notification indicating the occurrence of an anomalous event, as represented by the active percept.

21. The system of claim 20, wherein the operation further comprises:

querying an episodic memory using the active percept as an input stimulus, wherein the episodic memory stores percepts encoding instance specific details regarding events observed to have occurred in the sequence of video frames;

receiving a response percept from the episodic memory; and

storing the response percept in the long-term memory.

22. The system of claim 20, wherein the distance d is defined by:

d=1−

[α

S+(1−

α

)F]

wherein, S is a cosine similarity determined between the active percept and the retrieved percept, wherein F is the occurrence frequency of the retrieved percept in the long-term memory, and wherein α

is a heuristic weighting coefficient between 0 and 1.

23. The system of claim 20, wherein the distance d is defined by:

d=1−

(S√

{square root over (F)}−

√

{square root over (1−

S²)}√

{square root over (1−

F)})²

wherein, S is a cosine similarity determined between the active percept and the retrieved percept, wherein F is the occurrence frequency of the retrieved percept in the long-term memory.

24. The system of claim 20, wherein the distance d is defined by:

25. The system of claim 20, wherein publishing the alert notification comprises:

decoding the active percept to generate a semantic symbol stream corresponding to the active percept; and

generating, from the semantic symbol stream, a linguistic description of a pattern of behavior represented by the semantic symbol stream.