Method for online learning and recognition of visual behaviors

• US 8,948,499 B1
• Filed: 12/07/2010
• Issued: 02/03/2015
• Est. Priority Date: 12/07/2010
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Medasani, Swarup, Allen, David L., Chelian, Suhas E., Owechko, Yuri
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Choi, Timothy
• Application Number:
  US12/962,548
• Time in Patent Office: 1,519 Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats

Abstract

Described is a system for object and behavior recognition which utilizes a collection of modules which, when integrated, can automatically recognize, learn, and adapt to simple and complex visual behaviors. An object recognition module utilizes a cooperative swarm algorithm to classify an object in a domain. A graph-based object representation module is configured to use a graphical model to represent a spatial organization of the object within the domain. Additionally, a reasoning and recognition engine module consists of two sub-modules: a knowledge sub-module and a behavior recognition sub-module. The knowledge sub-module utilizes a Bayesian network, while the behavior recognition sub-module consists of layers of adaptive resonance theory clustering networks and a layer of a sustained temporal order recurrent temporal order network. The described invention has applications in video forensics, data mining, and intelligent video archiving.

Claims

1. A multi-modular system for object and behavior recognition, the system comprising one or more processors and a memory having instructions such that when the instructions are executed, the one or more processors perform operations of:

operating a plurality of software agents configured to operate as a cooperative swarm to classify an object in a domain with an object recognition module;

representing a spatial organization of the object in the domain using a graphical model with a graph-based object representation module;

storing a set of known object behaviors to allow the system to recognize the set of known object behaviors with a knowledge sub-module of a reasoning and recognition engine module;

learning both the set of known object behaviors from the knowledge sub-module and a set of novel object behaviors with a behavior recognition sub-module of the reasoning and recognition engine module, immediately from the graph-based object representation module;

proposing the set of learned novel object behaviors from the behavior recognition sub-module back to the knowledge sub-module as a set of new behaviors derived from online learning; and

outputting a behavior classification for the object, wherein the behavior classification for the object is classified as a known object behavior or as a novel object behavior based on comparison to a predetermined threshold value.

2. The multi-modular system for object and behavior recognition as set forth in claim 1, wherein the knowledge sub-module is a Bayesian network.

3. The multi-modular object and behavior recognition system as set forth in claim 2, wherein fuzzy attributed relational graphs provide an input for the reasoning and recognition engine module.

4. The multi-modular object and behavior recognition system as set forth in claim 3, wherein the Bayesian network is a Hidden Markov Model.

5. The multi-modular object and behavior recognition system as set forth in claim 1, wherein the knowledge sub-module is a Hidden Markov Model.

6. A computer-implemented method for object and behavior recognition, comprising an act of:

causing a data processor to perform operations of;

operating a plurality of software agents configured to operate as a cooperative swarm to classify an object in a domain with an object recognition module;

representing a spatial organization of the object in the domain using a graphical model with a graph-based object representation module;

storing a set of known object behaviors to allow the system to recognize the set of known object behaviors with a knowledge sub-module of a reasoning and recognition engine module;

learning both the set of known object behaviors from the knowledge sub-module and a set of novel object behaviors with a behavior recognition sub-module of the reasoning and recognition engine module, immediately from the graph-based object representation module;

proposing the set of learned novel object behaviors from the behavior recognition sub-module back to the knowledge sub-module as a set of new behaviors derived from online learning; and

outputting a behavior classification for the object, wherein the behavior classification for the object is classified as a known object behavior or as a novel object behavior based on comparison to a predetermined threshold value.

7. The method for object and behavior recognition as set forth in claim 6, wherein the knowledge sub-module is a Bayesian network.

8. The method for object and behavior recognition as set forth in claim 7, wherein fuzzy attributed relational graphs provide an input for the reasoning and recognition engine module.

9. The method for object and behavior recognition as set forth in claim 8, wherein the Bayesian network is a Hidden Markov Model.

10. The method for object and behavior recognition as set forth in claim 6, wherein the knowledge sub-module is a Hidden Markov Model.

11. A computer program product for object and behavior recognition, the computer program product comprising:

computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of;

operating a plurality of software agents configured to operate as a cooperative swarm to classify an object in a domain with an object recognition module;

representing a spatial organization of the object in the domain using a graphical model with a graph-based object representation module;

storing a set of known object behaviors to allow the system to recognize the set of known object behaviors with a knowledge sub-module of a reasoning and recognition engine module;

learning both the set of known object behaviors from the knowledge sub-module and a set of novel object behaviors with a behavior recognition sub-module of the reasoning and recognition engine module, immediately from the graph-based object representation module;

proposing the set of learned novel object behaviors from the behavior recognition sub-module back to the knowledge sub-module as a set of new behaviors derived from online learning; and

outputting a behavior classification for the object, wherein the behavior classification for the object is classified as a known object behavior or as a novel object behavior based on comparison to a predetermined threshold value.

12. The computer program product for object and behavior recognition as set forth in claim 11, wherein the knowledge sub-module is a Bayesian network.

13. The computer program product for object and behavior recognition as set forth in claim 12, wherein fuzzy attributed relational graphs provide an input for the reasoning and recognition engine module.

14. The computer program product for object and behavior recognition as set forth in claim 13, wherein the Bayesian network is a Hidden Markov Model.

15. The computer program product for object and behavior recognition as set forth in claim 11, wherein the knowledge sub-module is a Hidden Markov Model.