COGNITIVE MODEL FOR A MACHINE-LEARNING ENGINE IN A VIDEO ANALYSIS SYSTEM

• US 20090016600A1
• Filed: 07/09/2008
• Published: 01/15/2009
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Friedlander, David S., Risinger, Lon W., Solum, David M., Urech, Dennis G., Gottumukkal, Rajkiran K., Yang, Tao, Cobb, Wesley Kenneth, Saitwal, Kishor Adinath, Xu, Gang, Seow, Ming-Jung, EATON, JOHN ERIC
• Granted Patent: US 8,189,905 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A machine-learning engine is disclosed that is configured to recognize and learn behaviors, as well as to identify and distinguish between normal and abnormal behavior within a scene, by analyzing movements and/or activities (or absence of such) over time. The machine-learning engine may be configured to evaluate a sequence of primitive events and associated kinematic data generated for an object depicted in a sequence of video frames and a related vector representation. The vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and the streams describe actions of the objects depicted in the sequence of video frames.

Claims

1. A method for processing data generated from a sequence of video frames, the method comprising:

receiving, as a trajectory for a first object, a series of primitive events associated with a path of the first object depicted in the sequence of video frames as the first object moves through the scene, wherein each primitive event includes at least an object type and a set of one or more kinematic variables associated with the second object;

after receiving the trajectory for the first object, receiving a first vector representation generated for the first object, wherein the first vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and wherein the streams describe actions of at least the first object depicted in the sequence of video frames;

exciting one or more nodes of a perceptual associative memory using the trajectory and the first vector representation;

identifying, based on the one or more excited nodes, a percept;

copying the percept to a workspace;

in response to copying the percept to the workspace, selecting a codelet, wherein the codelet includes an executable sequence of instructions; and

invoking execution of the codelet.

2. The method of claim 1, further comprising, receiving a series of primitive events associated with a path of a second object interacting with the first object in the sequence of video frames, wherein each primitive event includes at least an object type and a set of one or more kinematic variables associated with the second object.

3. The method of claim 1, wherein the set of kinematic variables include at least a time variable, a position variable, and a velocity variable, etc., and, in the case where it is interacting with one or more other objects, the same data for those objects.

4. The method of claim 1, further comprising:

prior to invoking execution of the codelet, identifying, based on the percept, one or more memories stored in a long-term memory data structure; and

copying the identified memories to the workspace.

5. The method of claim 4, wherein the one or more long-term memories are identified based on a measure of similarity used to evaluate the memories stored in the long-term memory and the percept.

6. The method of claim 4, wherein the codelet is configured to reinforce one or more aspects of the identified memories, based on the percept.

7. The method of claim 4, wherein the long term memory data structure is implemented as a sparse distributed memory.

8. The method of claim 1, wherein the codelet is configured to generate a long-term memory from the percept;

and wherein the method further comprises, storing the long-term memory in a long-term memory data structure.

9. The method of claim 1, wherein the codelet is configured to generate an episodic memory from the percept, and wherein the method further comprises, storing the episodic memory data structure in long-term memory.

10. The method of claim 1, wherein the primitive event symbol stream includes a sequence of primitive event symbols, wherein each primitive event symbol represents a behavior engaged in by at least one of the objects depicted in the sequence of video.

11. The method of claim 1, wherein the first vector representation is generated by applying a singular value decomposition (SVD) to a second first vector representation.

12. The method of claim 1, wherein the perceptual associative memory includes a network of nodes, each node representing a behavior observed to have occurred in the scene and wherein links between nodes represents a relationship between two behaviors.

13. A computer-readable storage medium containing a program, which, when executed on a processor is configured to perform an operation for processing data generated from a sequence of video frames, the operation comprising:

receiving, as a trajectory for a first object, a series of primitive events associated with a path of the first object depicted in the sequence of video frames as the first object moves through the scene, wherein each primitive event includes at least an object type and a set of one or more kinematic variables associated with the second object;

after receiving the trajectory for the first object, receiving a first vector representation generated for the first object, wherein the first vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and wherein the streams describe actions of at least the first object depicted in the sequence of video frames;

exciting one or more nodes of a perceptual associative memory using the trajectory and the first vector representation;

identifying, based on the one or more excited nodes, a percept;

copying the percept to a workspace;

in response to copying the percept to the workspace, selecting a codelet, wherein the codelet includes an executable sequence of instructions; and

invoking execution of the codelet.

14. The computer-readable storage medium of claim 13, wherein the operation further comprises:

prior to invoking execution of the codelet, identifying, based on the percept, one or more memories stored in a long-term memory data structure; and

copying the identified memories to the workspace.

15. The computer-readable storage medium of claim 14, wherein the one or more long-term memories are identified based on a measure of similarity used to evaluate the memories stored in the long-term memory and the percept.

16. The computer-readable storage medium of claim 14, wherein the codelet is configured to reinforce one or more aspects of the identified memories, based on the percept.

17. The computer-readable storage medium of claim 14, wherein the long term memory data structure is implemented as a sparse distributed memory.

18. The computer-readable storage medium of claim 13, wherein the codelet is configured to generate a long-term memory from the percept;

and wherein the operation further comprises, storing the long-term memory in a long-term memory data structure.

19. The computer-readable storage medium of claim 13, wherein the codelet is configured to generate an episodic memory from the percept, and wherein the operation further comprises, storing the episodic memory data structure in long-term memory.

20. The computer-readable storage medium of claim 13, wherein the primitive event symbol stream includes a sequence of primitive event symbols, wherein each primitive event symbol represents a behavior engaged in by at least one of the objects depicted in the sequence of video.

21. The computer-readable storage medium of claim 13, wherein the first vector representation is generated by applying a singular value decomposition (SVD) to a second first vector representation.

22. The computer-readable storage medium of claim 13, wherein the perceptual associative memory includes a network of nodes, each node representing a behavior observed to have occurred in the scene and wherein links between nodes represents a relationship between two behaviors.

23. A system, comprising:

a video input source;

a processor; and

a memory storing a machine learning engine, wherein the machine learning engine is configured to;

receive, as a trajectory for a first object, a series of primitive events associated with a path of the first object depicted in the sequence of video frames as the first object moves through the scene, wherein each primitive event includes at least an object type and a set of one or more kinematic variables associated with the second object;

after receiving the trajectory for the first object, receive a first vector representation generated for the first object, wherein the first vector representation is generated from a primitive event symbol stream and a phase space symbol stream, and wherein the streams describe actions of at least the first object depicted in the sequence of video frames;

excite one or more nodes of a perceptual associative memory using the trajectory and the first vector representation;

identify, based on the one or more excited nodes, a percept;

copy the percept to a workspace;

in response to copying the percept to the workspace, select a codelet, wherein the codelet includes an executable sequence of instructions; and

invoke execution of the codelet.

24. The system of claim 23, wherein the machine learning engine is further configured to:

prior to invoking execution of the codelet, identify, based on the percept, one or more memories stored in a long-term memory data structure; and

copy the identified memories to the workspace.

25. The system of claim 24, wherein the one or more long-term memories are identified based on a measure of similarity used to evaluate the memories stored in the long-term memory and the percept.

26. The system of claim 24, wherein the codelet is configured to reinforce one or more aspects of the identified memories, based on the percept.

27. The system of claim 23, wherein the perceptual associative memory includes a network of nodes, each node representing a behavior observed to have occurred in the scene and wherein links between nodes represents a relationship between two behaviors.