Behavior recognition using cognitive swarms and fuzzy graphs

• US 20070263900A1
• Filed: 05/03/2007
• Published: 11/15/2007
• Est. Priority Date: 08/14/2004
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Owechko, Yuri, Medasani, Swarup
• Granted Patent: US 8,589,315 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/2111 by using evolutionary computational techniques, e.g. genetic algorithms
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Described is a behavior recognition system for detecting the behavior of objects in a scene. The system comprises a semantic object stream module for receiving a video stream having at least two frames and detecting objects in the video stream. Also included is a group organization module for utilizing the detected objects from the video stream to detect a behavior of the detected objects. The group organization module further comprises an object group stream module for spatially organizing the detected objects to have relative spatial relationships. The group organization module also comprises a group action stream module for modeling a temporal structure of the detected objects. The temporal structure is an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects.

Claims

1. A behavior recognition system for detecting the behavior of objects in a scene, the system comprising:

a semantic object stream module for receiving a video stream having at least two frames and detecting objects in the video stream; and

a group organization module for utilizing the detected objects from the video stream to detect a behavior of the detected objects.

2. A behavior recognition system as set forth in claim 1, wherein the group organization module further comprises an object group stream module, the object group stream module being configured to spatially organize the detected objects to have relative spatial relationships.

3. A behavior recognition system as set forth in claim 2, wherein the group organization module further comprises a group action stream module configured to model a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects.

4. A behavior recognition system as set forth in claim 3, wherein the group organization module is operative for using fuzzy attributed relational graphs (FARGs) for modeling spatial and temporal aspects of the behavior of the object to generate a scene FARG from the detected objects in the scene.

5. A behavior recognition system as set forth in claim 4, wherein the FARGs include nodes and edges, with the nodes representing different objects in the scene and the edges representing relationships between the objects.

6. A behavior recognition system as set forth in claim 5, wherein the system is further configured to receive a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior.

7. A behavior recognition system as set forth in claim 6, wherein the system is further configured to associate the user query with a behavior FARG, with the behavior FARG modeling the spatial and temporal aspects of a behavior the user is searching for, and once associated, the system matches the scene FARG with a behavior FARG to identify an object exhibiting the desired behavior.

8. A behavior recognition system as set forth in claim 7, wherein the system is configured to match the scene FARG with the behavior FARG using a fuzzy graph matching (FGM) algorithm.

9. A behavior recognition system as set forth in claim 1, wherein the system is further configured to receive a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior.

10. A behavior recognition system as set forth in claim 9, wherein the system is further configured to receive the user query in a form selected from a group consisting of a linguistic query, a graphical query, and a markup language query.

11. A behavior recognition system as set forth in claim 1, wherein the semantic object stream module further comprises a plurality of agents that are configured to detect objects in the video stream.

12. A behavior recognition system as set forth in claim 1, wherein the group organization module further comprises a group action stream module configured to model a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects.

13. A computer program product for behavior recognition, the computer program product comprising computer-readable instruction means stored on a computer-readable medium that are executable by a computer for causing the computer to perform operations of:

receiving a video stream having at least two frames and detecting objects in the video stream; and

utilizing the detected objects from the video stream to detect a behavior of the detected objects.

14. A computer program product as set forth in claim 13, further comprising instruction means for causing a computer to spatially organize the detected objects to have relative spatial relationships.

15. A computer program product as set forth in claim 14, further comprising instruction means for causing a computer to model a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects.

16. A computer program product as set forth in claim 15, further comprising instruction means for causing a computer to use fuzzy attributed relational graphs (FARGs) to model spatial and temporal aspects of the behavior of the object to generate a scene FARG from the detected objects in the scene.

17. A computer program product as set forth in claim 16, further comprising instruction means for causing a computer to generate the FARGs to include nodes and edges, with the nodes representing different objects in the scene and the edges representing relationships between the objects.

18. A computer program product as set forth in claim 17, further comprising instruction means for causing a computer to receive a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior.

19. A computer program product as set forth in claim 18, further comprising instruction means for causing a computer to associate the user query with a behavior FARG, with the behavior FARG modeling the spatial and temporal aspects of a behavior the user is searching for, and once associated, the system matches the scene FARG with a behavior FARG to identify an object exhibiting the desired behavior.

20. A computer program product as set forth in claim 19, further comprising instruction means for causing a computer to match the scene FARG with the behavior FARG using a fuzzy graph matching (FGM) algorithm.

21. A computer program product as set forth in claim 13, further comprising instruction means for causing a computer to receive a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior.

22. A computer program product as set forth in claim 21, further comprising instruction means for causing a computer to receive the user query in a form selected from a group consisting of a linguistic query, a graphical query, and a markup language query.

23. A computer program product as set forth in claim 13, further comprising instruction means for causing a computer to detect objects in the video stream using a plurality of agents.

24. A computer program product as set forth in claim 13, further comprising instruction means for causing a computer to model a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects.

25. A method for behavior recognition, comprising acts of:

receiving a video stream having at least two frames and detecting objects in the video stream; and

utilizing the detected objects from the video stream to detect a behavior of the detected objects.

26. A method as set forth in claim 25, further comprising an act of spatially organizing the detected objects to have relative spatial relationships.

27. A method as set forth in claim 26, further comprising an act of modeling a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects.

28. A method as set forth in claim 27, further comprising an act of using fuzzy attributed relational graphs (FARGs) to model spatial and temporal aspects of the behavior of the object to generate a scene FARG from the detected objects in the scene.

29. A method as set forth in claim 28, further comprising an act of generating the FARGs to include nodes and edges, with the nodes representing different objects in the scene and the edges representing relationships between the objects.

30. A method as set forth in claim 29, further comprising an act of receiving a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior.

31. A method as set forth in claim 30, further comprising an act of associating the user query with a behavior FARG, with the behavior FARG modeling the spatial and temporal aspects of a behavior the user is searching for, and once associated, the system matches the scene FARG with a behavior FARG to identify an object exhibiting the desired behavior.

32. A method as set forth in claim 31, further comprising an act of matching the scene FARG with the behavior FARG using a fuzzy graph matching (FGM) algorithm.

33. A method as set forth in claim 25, further comprising an act of receiving a user query to detect a desired behavior, and based on the query and objects in the video stream, identify if the objects exhibit the desired behavior.

34. A method as set forth in claim 33, further comprising an act of receiving a user query in a form selected from a group consisting of a linguistic query, a graphical query, and a markup language query.

35. A method as set forth in claim 25, further comprising an act of detecting objects in the video stream using a plurality of agents.

36. A method as set forth in claim 25, further comprising an act of modeling a temporal structure of the detected objects, the temporal structure being an action of the detected objects between the two frames, whereby through detecting, organizing and modeling actions of objects, a user can detect the behavior of the objects.