VIDEO ACCESS SYSTEM AND METHOD BASED ON ACTION TYPE DETECTION

• US 20140105573A1
• Filed: 08/29/2013
• Published: 04/17/2014
• Est. Priority Date: 10/12/2012
• Status: Active Grant

• Current Assignee: Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek Tno
• Original Assignee: Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek Tno
• Inventors: Hanckmann, Patrick, Schutte, Klamer, de Penning, Leo, Burghouts, Gerardus Johannes
• Granted Patent: US 9,554,081 B2
• Time in Patent Office: Days
• US Class Current: 386/241
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 5/91 Television signal processing therefor

Abstract

Descriptions of video segments for use to search or select video segments are generated by using a combination of video based action type detection and image based object detection. Action type detection results in detection of action types from a predetermined set of action types, with little or no information about actors involved in the action. Object detection results in detection of individual objects in the images, with little or no information actions. A set of rules is used that define one or more roles associated with action types and conditions on objects that can fill these roles. Detected action types are used to select rules and detected objects are assigned to roles defined the selected rule. Score values are computed for different assignments of detected objects to the role, as a function of the attribute values of the assigned objects. A combination of the detected action type and the detected objects is selected on the basis of the score values. A description of the video segment is generated from the selected combination.

Claims

1. A video access system for generating a description of the video segment based on a predetermined set of action describing rules, each rule associated with a respective action type from a predetermined set of detectable action types, each of the rules in the set defining at least one role in the action, and based on predetermined conditions defined for the rules, for computing score values dependent on the attribute values of an object or objects assigned to the at least one role, the system comprising

an action type detector coupled to a source of video data;

an object detector coupled to the source of video data;

a description generator coupled to the action type detector and the object detector, and configured to

select a rule dependent on an action type detected from the video data for a video segment by the action type detector, the rule being selected from a predetermined set of rules;

assign detected objects, which have been detected from the video data for a video segment by the object detector, to the at least role of the selected rule;

compute score values for assignments of different detected objects to the role, each score value being computed dependent on the attribute values of the assigned object according to the predetermined conditions defined for the rule;

select a combination of the detected action type and at least one assigned detected objects on the basis of the score values;

generate a description of the video segment from the selected combination.

2. A video access system according to claim 1, the system comprising

a data base for storing records associating identifications of video segments with descriptions,

a query processor configured to search the data base for descriptions that match a received query;

the description generator being configured to store the generated description in association with an identification of the video segment in the data base.

3. A video access system according to claim 1, comprising a query processor configured to compare the description with a predetermined query when the description is generated, and to activate a signal in response to detection that the description matches the query.

4. A video access system according to claim 1, wherein the description generator is configured to compute score values as a function of score values generated for the action type by the action type detector and/or for the detected object by the object detector.

5. A video access system according to claim 4, wherein the description generator is configured to compute score values according to

score value=w0*R(A,t)+Sum{w(i)*R(condition,t(i))}

wherein the sum is taken over one or more conditions defined by the selected rule, “

i”

indexing the conditions, w0 and w(i) being predetermined weight factors, R(A,t) being a score value for an action type A at time t generated by the action type detector, R(condition, t(i)) being a score value or score values generated by the object detector for attribute values that are defined by the rule.

6. A video access system according to claim 1, wherein the description generator is configured to generate the description in the form of a string of text.

7. A video access system according to claim 1, wherein the description generator is configured to

assign combinations of the detected objects to a combination of roles defined by the selected rule;

compute the score values as a function of the attribute values of the assigned objects in the combination according to predetermined conditions defined for the rule;

select a combination of the detected action type and a combination of the detected objects on the basis of the score values;

generate a description of the video segment from the selected combination of the detected action type and a combination of the detected objects.

8. A video access system according to claim 1, wherein the description generator is configured to

select a plurality of rules dependent on a plurality of action types detected for the video segment;

select combinations of different ones of the detected action types and detected objects on the basis of the score values;

generate a plurality of descriptions of the video segment from the selected combinations.

9. A video access system according to claim 1, wherein the action type detector is configured to detect spatiotemporally localized image sequence features in the video segment;

construct a histogram of feature values of the detected spatiotemporally localized image sequence features; and

determine the detected action type based on a comparison of the histogram with reference histograms.

10. A video segment selection method, the method comprising

determining a detected action type for a video segment, and/or a likelihood value of the detected action type, the detected action type being selected from a predetermined set of action types dependent on features derived from the video segment;

determining detections and/or likelihood values for detected objects in the video segment and attribute values of the detected objects;

assigning the detected objects to a role defined in a predetermined rule associated with the detected action type;

computing score values for different assignments of detected objects to the role, each as a function of the attribute values of the assigned objects according to predetermined conditions defined for the rule;

selecting a combination of the detected action type and the detected objects on the basis of the score values;

generating a description of the video segment from the selected combination.

11. A method according to claim 10, wherein the description is a textual description.

12. A method according to claim 10, determining a plurality of detected action types for the video segment, and/or likelihood values of the detected action types, the detected action types being selected from the predetermined set of action types;

assigning the detected objects to roles of respective predetermined rules associated with respective ones of the plurality of detected action types;

computing score values for different assignments of detected objects to the roles, each as a function of the attribute values of the assigned objects according to predetermined conditions defined for the respective predetermines rules;

selecting combinations of the detected action types and the detected objects on the basis of the score values;

generating a plurality of descriptions of the video segment from the selected combinations.

13. A method according to claim 10, wherein determining the detected action type comprises

detecting spatiotemporally localized image sequence features in the video segment;

constructing a histogram of feature values of the detected spatiotemporally localized image sequence features;

determining the detected action type based on a comparison of the histogram with reference histograms.

14. A computer readable medium, comprising a program of machine executable instructions for a programmable computer system that, when executed by the programmable computer system, will cause the programmable computer system to execute the method of claim 10.