Behavior recognition using cognitive swarms and fuzzy graphs

• US 8,589,315 B2
• Filed: 05/03/2007
• Issued: 11/19/2013
• Est. Priority Date: 08/14/2004
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Medasani, Swarup, Owechko, Yuri
• Primary Examiner(s): RIFKIN, BEN M
• Application Number:
  US11/800,264
• Publication Number:
  US 20070263900A1
• Time in Patent Office: 2,392 Days
• Field of Search: 706/62, 706/1
• US Class Current: 706/1
• CPC Class Codes: G06F 18/2111 by using evolutionary computational techniques, e.g. genetic algorithms
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Described is a behavior recognition system for detecting the behavior of objects in a scene. The system comprises a semantic object stream module for receiving a video stream having at least two frames and detecting objects in the video stream. Also included is a group organization module for utilizing the detected objects from the video stream to detect a behavior of the detected objects. The group organization module further comprises an object group stream module for spatially organizing the detected objects to have relative spatial relationships. The group organization module also comprises a group action stream module for modeling a temporal structure of the detected objects. The temporal structure is an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects.

Claims

1. A behavior recognition system for detecting the behavior of objects in a scene, the system comprising a processor and a memory coupled with the processor, wherein the processor includes:

a semantic object stream module for receiving a video stream having at least two frames and detecting objects in the video stream; and

a group organization module for utilizing the detected objects from the video stream to detect a behavior of the detected objects, including both static and moving objects, where the group organization module detects the behavior using a hierarchical event detection scheme further comprising;

a spatial organization layer comprising an object group stream module, the object group stream module being configured to spatially organize the detected objects to have relative spatial relationships; and

a temporal organization layer comprising a group action stream module, the group action stream module being configured to model a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects; and

wherein the group organization module is operative for using fuzzy attributed relational graphs (FARGs) for modeling spatial and temporal aspects of the behavior of the object to generate a scene FARG from the detected objects in the scene;

wherein the FARGs include nodes and edges, with the nodes representing different objects in the scene and the edges representing relationships between the objects;

wherein the processor is further configured to receive a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior;

wherein the processor is further configured to associate the user query with a behavior FARG, with the behavior FARG modeling the spatial and temporal aspects of a behavior the user is searching for, and once associated, the processor matches the scene FARG with a behavior FARG to identify an object exhibiting the desired behavior; and

wherein the scene FARG is matched with the behavior FARG by determining compatibilities between nodes in the FARGs and using the compatibilities in a fuzzy graph matching (FGM) algorithm to match the scene FARG with the behavior FARG.

2. A behavior recognition system as set forth in claim 1, wherein the processor is further configured to receive the user query in a form selected from a group consisting of a linguistic query, a graphical query, and a markup language query.

3. A behavior recognition system as set forth in claim 1, wherein the semantic object stream module further comprises a plurality of agents that are configured to detect objects in the video stream.

4. A computer program product for behavior recognition, the computer program product comprising computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer for causing the computer to perform operations of:

receiving a video stream having at least two frames and detecting objects in the video stream; and

utilizing the detected objects from the video stream to detect a behavior of the detected objects, including both static and moving objects, using a hierarchical event detection scheme further comprising operations of;

spatially organizing the detected objects to have relative spatial relationships; and

modeling a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects; and

using fuzzy attributed relational graphs (FARGs) to model spatial and temporal aspects of the behavior of the object to generate a scene FARG from the detected objects in the scene;

generating the FARGs to include nodes and edges, with the nodes representing different objects in the scene and the edges representing relationships between the objects;

receiving a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior;

associating the user query with a behavior FARG, with the behavior FARG modeling the spatial and temporal aspects of a behavior the user is searching for, and once associated, the system matches the scene FARG with a behavior FARG to identify an object exhibiting the desired behavior; and

wherein the scene FARG is matched with the behavior FARG by determining compatibilities between nodes in the FARGs and using the compatibilities in a fuzzy graph matching (FGM) algorithm to match the scene FARG with the behavior FARG.

5. A computer program product as set forth in claim 4, further comprising instruction means for causing a computer to receive the user query in a form selected from a group consisting of a linguistic query, a graphical query, and a markup language query.

6. A computer program product as set forth in claim 4, further comprising instruction means for causing a computer to detect objects in the video stream using a plurality of agents.

7. A computer implemented method for behavior recognition, comprising acts of:

receiving, in a processor, a video stream having at least two frames and detecting objects in the video stream; and

utilizing the detected objects from the video stream to detect a behavior of the detected objects, including both static and moving objects, using a hierarchical event detection scheme further comprising act of;

spatially organizing the detected objects to have relative spatial relationships; and

modeling a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects; and

using fuzzy attributed relational graphs (FARGs) to model spatial and temporal aspects of the behavior of the object to generate a scene FARG from the detected objects in the scene;

generating the FARGs to include nodes and edges, with the nodes representing different objects in the scene and the edges representing relationships between the objects;

receiving a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior;

associating the user query with a behavior FARG, with the behavior FARG modeling the spatial and temporal aspects of a behavior the user is searching for, and once associated, the system matches the scene FARG with a behavior FARG to identify an object exhibiting the desired behavior; and

wherein the scene FARG is matched with the behavior FARG by determining compatibilities between nodes in the FARGs and using the compatibilities in a fuzzy graph matching (FGM) algorithm to match the scene FARG with the behavior FARG.

8. A computer implemented method as set forth in claim 7, further comprising an act of receiving a user query in a form selected from a group consisting of a linguistic query, a graphical query, and a markup language query.

9. A method computer implemented as set forth in claim 7, further comprising an act of detecting objects in the video stream using a plurality of agents.