Tree structured CRF with unary potential function using action unit features of other segments as context feature

• US 10,445,582 B2
• Filed: 12/20/2016
• Issued: 10/15/2019
• Est. Priority Date: 12/20/2016
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Mehrseresht, Nagita
• Primary Examiner(s): Potts, Ryan P
• Application Number:
  US15/385,291
• Publication Number:
  US 20180173955A1
• Time in Patent Office: 1,029 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

A method of determining a composite action from a video clip, using a conditional random field (CRF), the method includes determining a plurality of features from the video clip, each of the features having a corresponding temporal segment from the video clip. The method may continue by determining, for each of the temporal segments corresponding to one of the features, an initial estimate of an action unit label from a corresponding unary potential function, the corresponding unary potential function having as ordered input the plurality of features from a current temporal segment and at least one other of the temporal segments. The method may further include determining the composite action by jointly optimizing the initial estimate of the action unit labels.

Claims

1. A method of determining a classification of a composite action including a plurality of action units in a video clip, the method comprising:

extracting a plurality of features from the video clip;

determining a corresponding feature in the plurality of features for each of temporal segments of the video clip;

determining an initial estimate of an action unit for each of the temporal segments using a potential function for each segment modeling dependency between a concatenation of features and a classification of a corresponding action unit by inputting a feature from a current temporal segment and a feature from at least one of preceding temporal segments or subsequent temporal segments as the concatenation of features;

aggregating the potential functions into a probability distribution; and

determining the classification of the composite action using the probability distribution by jointly inferring the classification of the composite action and classifications of the action units of the temporal segments based on the initial estimate of each action unit for each of the temporal segments.

2. The method according to claim 1, wherein the plurality of features are semantic features.

3. The method according to claim 1, wherein the plurality of features are low level features.

4. The method according to claim 1, further comprising:

classifying at least one contextual object in at least one other of the temporal segments preceding the current segment,

the at least one contextual object being independent of any action units of interest in the at least one other of the temporal segments preceding the current segment; and

determining an action unit of interest in the current segment of the video clip, the action unit of interest being performed with the classified at least one contextual object and the determination of the action unit of interest in the current segment being based on the classification of the at least one contextual object wherein the current segment and the other segment preceding the current segment are disjoint.

5. The method according to claim 1, wherein the probability distribution is a conditional random field (CRF) probability distribution.

6. The method according to claim 5, wherein the CRF has a tree structure.

7. The method according to claim 5, wherein the CRF is in log-linear form.

8. A non-transitory computer readable medium having a computer program recorded on the computer readable medium, the computer program being executable by a computer system to perform a method of determining a classification of a composite action including a plurality of action units in a video clip, the method comprising:

extracting a plurality of features from the video clip;

determining a corresponding feature in the plurality of features for each of temporal segments of the video clip;

determining an initial estimate of an action unit for each of the temporal segments using a potential function for each segment modeling dependency between a concatenation of features and a classification of a corresponding action unit by inputting a feature from a current temporal segment and a feature from at least one of preceding temporal segments or subsequent temporal segments as the concatenation of features;

aggregating the potential functions into a probability distribution; and

determining the classification of the composite action using the probability distribution by jointly inferring the classification of the composite action and classifications of the action units of the temporal segments based on the initial estimate of each action unit for each of the temporal segments.

9. A computer system, comprising:

a processor;

a memory having a computer program recorded thereon, the memory being in communication with the processor;

the processor executing the computer program to perform a method of determining a classification of a composite action including a plurality of action units in a video clip, the method comprising;

extracting a plurality of features from the video clip;

determining a corresponding feature in the plurality of features for each of temporal segments of the video dip;

determining an initial estimate of an action unit for each of the temporal segments using a potential function for each segment modeling dependency between a concatenation of features and a classification of a corresponding action unit by inputting a feature from a current temporal segment and a feature from at least one of preceding temporal segments or subsequent temporal segments as the concatenation of features;

aggregating the potential functions into a probability distribution; and

determining the classification of the composite action using the probability distribution by jointly inferring the classification of the composite action and classifications of the action units of the temporal segments based on the initial estimate of each action unit for each of the temporal segments.