LONG-TERM MEMORY IN A VIDEO ANALYSIS SYSTEM

• US 20100063949A1
• Filed: 09/11/2008
• Published: 03/11/2010
• Est. Priority Date: 09/11/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: EATON, JOHN ERIC, Cobb, Wesley Kenneth, Xu, Gang, Seow, Ming-Jung, Friedlander, David Samuel
• Granted Patent: US 8,121,968 B2
• Time in Patent Office: Days
• US Class Current: 706/14
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/0409 Adaptive resonance theory [ART] networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/84 using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A long-term memory used to store and retrieve information learned while a video analysis system observes a stream of video frames is disclosed. The long-term memory provides a memory with a capacity that grows in size gracefully, as events are observed over time. Additionally, the long-term memory may encode events, represented by sub-graphs of a neural network. Further, rather than predefining a number of patterns recognized and manipulated by the long-term memory, embodiments of the invention provide a long-term memory where the size of a feature dimension (used to determine the similarity between different observed events) may grow dynamically as necessary, depending on the actual events observed in a sequence of video frames.

Claims

1. A computer-implemented method for storing an encoded percept in a long-term memory, comprising:

initializing the long-term memory, wherein the long-term memory includes a set of hard-locations, a weight matrix, and an activation function;

receiving a percept, wherein the percept comprises a sub-graph of a neural network excited by an input stimulus;

encoding the percept as a bit pattern;

generating a variable length entry in the set of hard locations

storing the encoded percept as a hard location in the set of hard locations; and

generating an entry in the weight matrix corresponding to the encoded percept.

2. The method of claim 1, further comprising, compressing the bit pattern using an encoding/decoding scheme that enables varying length features.

3. The method of claim 1, wherein generating the entry in the weight matrix corresponding to the encoded percept comprises copying the bit pattern of the encoded percept to the entry in the weight matrix.

4. The method of claim 1, further comprising:

receiving a second percept, wherein the second percept comprises a sub-graph of a neural network excited by an input stimulus;

encoding the second percept as a second bit pattern;

determining, using an adaptive resonance theory (ART) network, at least one hard location within a specified measure of similarity to the second bit pattern;

reinforcing the entry in the weight matrix corresponding to the at least one hard location.

5. The method of claim 4, wherein reinforcing the entry in the weight matrix corresponding to the at least one hard location comprises:

converting each 0 value in the bit pattern to a −

1; and

adding each respective value in the converted bit pattern to a corresponding value in the weight matrix entry corresponding to the identified at least one hard location.

6. The method of claim 1, further comprising:

receiving a request to return a percept from the long-term memory, wherein the request includes a second percept;

encoding the second percept as a second bit pattern;

evaluating the second bit pattern using a choice function and a vigilance function specified for an adaptive resonance theory (ART) network to identify one or more hard locations within a specified measure of similarity to the second bit pattern;

retrieving a weight matrix entry corresponding to each identified hard location;

adding the weight matrix entries to determine a final activation;

converting the final activation to a third bit pattern;

decoding the third bit pattern;

generating from the decoded third bit pattern, a third percept; and

returning, as a response to the request, the third percept.

7. The method of claim 6, wherein converting the final activation to the third bit pattern comprises converting each positive value in the final activation to a 1 and each negative value to a 0.

8. The method of claim 1, wherein the input stimulus comprises a primitive event symbol stream supplied by a computer vision engine configured to analyze a sequence of video frames.

9. A computer-readable storage medium containing a program which, when executed by a processor, performs an operation for storing an encoded percept in a long-term memory, the operation comprising:

initializing the long-term memory, wherein the long-term memory includes a set of hard-locations, a weight matrix, and an activation function;

receiving a percept, wherein the percept comprises a sub-graph of a neural network excited by an input stimulus;

encoding the percept as a bit pattern;

generating a variable length entry in the set of hard locations

storing the encoded percept as a hard location in the set of hard locations; and

generating an entry in the weight matrix corresponding to the encoded percept.

10. The computer-readable storage medium of claim 9, wherein the operation further comprises, compressing the bit pattern using an encoding/decoding scheme that enables varying length features.

11. The computer-readable storage medium of claim 9, wherein generating the entry in the weight matrix corresponding to the encoded percept comprises copying the bit pattern of the encoded percept to the entry in the weight matrix.

12. The computer-readable storage medium of claim 9, wherein the operation further comprises:

receiving a second percept, wherein the second percept comprises a sub-graph of a neural network excited by an input stimulus;

encoding the second percept as a second bit pattern;

determining, using an adaptive resonance theory (ART) network, at least one hard location within a specified measure of similarity to the second bit pattern;

reinforcing the entry in the weight matrix corresponding to the at least one hard location.

13. The computer-readable storage medium of claim 12, wherein reinforcing the entry in the weight matrix corresponding to the at least one hard location comprises:

converting each 0 value in the bit pattern to a −

1; and

adding each respective value in the converted bit pattern to a corresponding value in the weight matrix entry corresponding to the identified at least one hard location.

14. The computer-readable storage medium of claim 9, wherein the operation further comprises:

receiving a request to return a percept from the long-term memory, wherein the request includes a second percept;

encoding the second percept as a second bit pattern;

evaluating the second bit pattern using a choice function and a vigilance function specified for an adaptive resonance theory (ART) network to identify one or more hard locations within a specified measure of similarity to the second bit pattern;

retrieving a weight matrix entry corresponding to each identified hard location;

adding the weight matrix entries to determine a final activation;

converting the final activation to a third bit pattern;

decoding the third bit pattern;

generating from the decoded third bit pattern, a third percept; and

returning, as a response to the request, the third percept.

15. The computer-readable storage medium of claim 14, wherein converting the final activation to the third bit pattern comprises converting each positive value in the final activation to a 1 and each negative value to a 0.

16. The computer-readable storage medium of claim 9, wherein the input stimulus comprises a primitive event symbol stream supplied by a computer vision engine configured to analyze a sequence of video frames.

17. A system comprising,

a video input source configured to provide a sequence of video frames, each depicting a scene;

a processor; and

a memory containing a machine learning application which when executed by the processor is configured to perform an operation for storing an encoded percept in a long-term memory, the operation comprising;

initializing the long-term memory, wherein the long-term memory includes a set of hard-locations, a weight matrix, and an activation function,

receiving a percept, wherein the percept comprises a sub-graph of a neural network excited by an input stimulus,

encoding the percept as a bit pattern,

generating a variable length entry in the set of hard locations

storing the encoded percept as a hard location in the set of hard locations, and

generating an entry in the weight matrix corresponding to the encoded percept.

18. The system of claim 17, wherein the operation further comprises, compressing the bit pattern using an encoding/decoding scheme that enables varying length features.

19. The system of claim 17, wherein generating the entry in the weight matrix corresponding to the encoded percept comprises copying the bit pattern of the encoded percept to the entry in the weight matrix.

20. The system of claim 17, wherein the operation further comprises:

receiving a second percept, wherein the second percept comprises a sub-graph of a neural network excited by an input stimulus;

encoding the second percept as a second bit pattern;

determining, using an adaptive resonance theory (ART) network, at least one hard location within a specified measure of similarity to the second bit pattern;

reinforcing the entry in the weight matrix corresponding to the at least one hard location.

21. The system of claim 20, wherein reinforcing the entry in the weight matrix corresponding to the at least one hard location comprises:

converting each 0 value in the bit pattern to a −

1; and

adding each respective value in the converted bit pattern to a corresponding value in the weight matrix entry corresponding to the identified at least one hard location.

22. The system of claim 17, wherein the operation further comprises:

receiving a request to return a percept from the long-term memory, wherein the request includes a second percept;

encoding the second percept as a second bit pattern;

evaluating the second bit pattern using a choice function and a vigilance function specified for an adaptive resonance theory (ART) network to identify one or more hard locations within a specified measure of similarity to the second bit pattern;

retrieving a weight matrix entry corresponding to each identified hard location;

adding the weight matrix entries to determine a final activation;

converting the final activation to a third bit pattern;

decoding the third bit pattern;

generating from the decoded third bit pattern, a third percept; and

returning, as a response to the request, the third percept.

23. The system of claim 22, wherein converting the final activation to the third bit pattern comprises converting each positive value in the final activation to a 1 and each negative value to a 0.

24. The system of claim 17, wherein the input stimulus comprises a primitive event symbol stream supplied by a computer vision engine configured to analyze a sequence of video frames