Event structure system and controlling method and medium for the same

• US 20090208111A1
• Filed: 07/03/2008
• Published: 08/20/2009
• Est. Priority Date: 02/18/2008
• Status: Active Grant

• Current Assignee: Institute Of Automation Chinese Academy Of Sciences, Samsung Electronics Co. Ltd.
• Original Assignee: Institute Of Automation Chinese Academy Of Sciences, Samsung Electronics Co. Ltd.
• Inventors: Huang, Kaiqi, Zhang, Zhang, Park, Gyu-tae, Jeong, Moon-Sik
• Granted Patent: US 8,620,022 B2
• Time in Patent Office: Days
• US Class Current: 382/190
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

Disclosed are an event structure system and a method and medium for controlling the event structure system. The method includes recognizing multiple-person interaction primitives, composing an event by inference based on temporal relations using the multiple-person interaction primitive, and determining a final event by either eliminating an unnecessary event from the composed event, or adding a new event in the composed event.

Claims

1. A method for controlling an event structure system, the method comprising:

recognizing multiple-person interaction primitives from an image, which is displayed on a display screen;

composing an event by inference based on temporal relations using the multiple-person interaction primitive; and

determining a final event by either eliminating an unnecessary event from the composed event, or adding a new event in the composed event.

2. The method of claim 1, wherein at least one of a trajectory and optical flow of a single person is used to recognize multiple-person interaction primitives.

3. The method of claim 1, wherein the recognizing of multiple-person interaction primitives comprises:

transforming an image plane including multiple persons into relative trajectory coordinates of a reference object corresponding to any single person of multiple persons in the image plane;

performing trajectory clustering in the relative trajectory coordinates; and

recognizing the multiple-person interaction primitives by each trajectory cluster obtained by performing the trajectory clustering.

4. The method of claim 3, wherein the transforming comprises:

performing a trajectory projection from the image plane to a ground plane; and

performing another trajectory projection from the ground plane to the relative trajectory coordinates of the reference object.

5. The method of claim 3, wherein the performing of the trajectory clustering comprises:

segmenting a relative trajectory interval by the origin of the relative trajectory coordinates; and

performing a trajectory clustering algorithm based on similarity to obtain the trajectory clusters.

6. The method of claim 3, wherein the recognizing of the multiple-person interaction primitives by each of the trajectory clusters recognizes the multiple-person interaction primitives using a Hidden Markov Model (HMM) for modeling results obtained by performing the trajectory clustering.

7. The method of claim 1, wherein the determining of the final event comprises:

eliminating an unnecessary event by Multi-Thread Parsing (MTP);

computing a start point distance and an end point distance between two events to thereby infer temporal relations;

re-generating an event disregarded due to errors; and

adding the re-generated event in the composed event.

8. The method of claim 1, wherein an expansion of an Early-Stolcke parser model including the temporal relations is used to eliminate the unnecessary event.

9. The method of claim 7, wherein the eliminating of the unnecessary event further comprises determining whether the recognized two events are combined.

10. The method of claim 7, wherein the eliminating of the unnecessary event comprises:

generating all the possible combinations of the event;

limiting a temporal interval of the event;

comparing a predetermined event rule and an event rule of the event; and

eliminating an unnecessary event using event related results computed by the temporal interval constraint of the event and the combination of the event.

11. The method of claim 10, wherein the eliminating of the unnecessary event further comprises requiring a set of identifications (IDs) of an event, where the IDs do not overlap each other.

12. The method of claim 11, wherein the set of IDs is a set of the multiple-person interaction primitives composing the event.

13. At least one computer-readable recording medium storing a program for implementing the method of claim 1.

14. A system for controlling an event structure, the system comprising:

a multiple-person interaction primitives recognizing unit to recognize multiple-person interaction primitives from an image, which is displayed on a display screen; and

a multi-thread parser to compose an event by inference based on temporal relations using the multiple-person interaction primitives, and to determine a final event by either eliminating an unnecessary event from the composed event or adding a new event in the composed event.

15. The system of claim 14, further comprising at least one of:

a trajectory extracting unit to extract a trajectory of multiple persons; and

an optical flow extracting unit to extract features related with behaviors of multiple persons based on a speed in which a pixel on the display screen displaying the person is moved to the next position within a predetermined period.

16. The system of claim 14, wherein the multiple-person interaction primitives recognizing unit comprises:

a relative trajectory coordinates transforming unit to transform an image plane including the multiple persons into relative trajectory coordinates of a reference object corresponding to any single person of the multiple persons;

a trajectory clustering unit to perform trajectory clustering in the relative trajectory coordinates; and

a recognizing unit to recognize the multiple-person interaction primitives by each trajectory cluster obtained by performing the trajectory clustering via the trajectory clustering unit.

17. The system of claim 16, wherein the relative trajectory coordinates transforming unit performs a trajectory projection from the image plane to a ground plane, and performs another trajectory projection from the ground plane to the relative coordinates of the reference object.

18. The system of claim 16, wherein the trajectory clustering unit segments a relative trajectory interval by the origin of the relative trajectory coordinates, and performs a trajectory clustering algorithm based on similarity for the purpose of obtaining the trajectory clusters.

19. The system of claim 16, wherein the recognizing unit recognizes the multiple-person interaction primitives using a HMM for modeling results obtained by performing the trajectory clustering.

20. The system of claim 14, wherein the multi-thread parser comprises:

a constraint fulfillment determining unit to eliminate an unnecessary event by multi-thread parsing;

a temporal relations inferring unit to compute a start point distance and an end point distance between two events to thereby infer temporal relations; and

an error prediction generating unit to re-generate an event disregarded due to errors, and to add the re-generated event in the composed event.

21. The system of claim 14, wherein the multi-thread parser eliminates the unnecessary event using an expansion of an Early-Stolcke parser model including the temporal relations.

22. The system of claim 14, wherein the multi-thread parser determines whether the recognized two events are combined.

23. The system of claim 20, wherein the constraint fulfillment determining unit comprises:

an ID set constraint unit to generate all the possible combinations of the event;

a temporal interval constraint unit to constrain a temporal interval of the event;

a comparing unit to compare a predetermined event rule and an event rule of the event; and

a maximum error constraint unit to eliminate an unnecessary event using event related results computed by the ID set constraint unit and the temporal interval constraint unit.