Continuous Linear Dynamic Systems

• US 20120219186A1
• Filed: 02/27/2012
• Published: 08/30/2012
• Est. Priority Date: 02/28/2011
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Wang, Jinjun, Xiao, Jing
• Granted Patent: US 8,917,907 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Aspects of the present invention include systems and methods for segmentation and recognition of action primitives. In embodiments, a framework, referred to as the Continuous Linear Dynamic System (CLDS), comprises two sets of Linear Dynamic System (LDS) models, one to model the dynamics of individual primitive actions and the other to model the transitions between actions. In embodiments, the inference process estimates the best decomposition of the whole sequence into continuous alternating between the two set of models, using an approximate Viterbi algorithm. In this way, both action type and action boundary may be accurately recognized.

Claims

1. A computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes steps for recognizing a sequence of actions comprising:

segmenting input sensor data into time frames;

generating a feature for each time frame;

for a time frame, selecting an action associated with a maximized value of an objective function that comprises a first set of feature-dependent models for intra-action transitions and a second set of feature-dependent models for inter-action transitions.

2. The computer-readable medium or media of claim 1 wherein the feature-dependent models are based upon linear dynamic system models.

3. The computer-readable medium or media of claim 2 wherein the objective function models a hidden state at a current time frame as being directly dependent upon a hidden state for a sequence of one or more time frames prior to the current time frame, an estimated action for the sequence of one or more time frames prior to the current time frame, and an estimated action of the current time frame.

4. The computer-readable medium or media of claim 1 wherein the second set of feature-dependent models for inter-action transitions comprise at least one model for a transition between a same action.

5. The computer-readable medium or media of claim 1 wherein the input sensor data is a video and the process of segmenting input sensor data into time frames comprises:

segmenting the video data into image frames.

6. The computer-readable medium or media of claim 5 wherein the feature for each image frame is an embedded optical flow feature formed by pooling a set of sparse coding vectors extracted from the image frame based upon a weighting of a distribution of a set of local motion features in the image frame.

7. The computer-readable medium or media of claim 6 wherein extracting spare coding vectors from the image frame comprises:

obtaining the set of local motion features for the image frame;

generating the set of sparse coding vectors by generating a sparse coding vector for each local motion feature from the set of local motion features; and

forming the embedded optical flow feature that represents the image frame by pooling the set of sparse coding vectors based upon a weighting of the distribution of the set of local motion features and normalizing the pooled sparse coding vector.

8. The computer-readable medium or media of claim 7 wherein pooling the set of sparse coding vectors based upon a weighting of the distribution of the set of local motion features comprises:

pooling the set of sparse coding vectors for the image frame by weighting each sparse coding vector by an inverse proportional to a square root of a posterior of a local motion feature corresponding to the sparse coding vector.

9. A computer-implement method for recognizing a sequence of actions comprising:

modeling actions comprising a plurality of action dynamic models, wherein each dynamic model comprises a label associated with an action that it models;

modeling at least one transition between actions comprising at least one transition dynamic model, wherein each of the least one transition dynamic model comprises a label associated with a transition between actions that it models; and

for a feature representing a set of input sensor data for a time frame, selecting an action associated with the dynamic model that maximizes an objective function, wherein a transition probability from one action to another action is non-scalar and is calculated using at least one transition dynamic model.

10. The computer-implement method of claim 9 wherein the transition probability of a hidden state at a current time frame is directly dependent upon a hidden state for a sequence of one or more time frames prior to the current time frame, an estimated action for the sequence of one or more time frames prior to the current time frame, and an estimated action of the current time frame.

11. The computer-implement method of claim 9 wherein input sensor data is a video and the method further comprises:

segmenting the video data into image frames;

obtaining a set of optical flow features for the image frame;

generating the set of sparse coding vectors by generating a sparse coding vector for each local motion feature from the set of optical flow features; and

forming the feature that represents the image frame by pooling the set of sparse coding vectors based upon a weighting of the distribution of the set of optical flow features and normalizing the pooled sparse coding vector.

12. The computer-implement method of claim 11 wherein pooling the set of sparse coding vectors based upon a weighting of the distribution of the set of local motion features comprises:

pooling the set of sparse coding vectors for the image frame by weighting each sparse coding vector by an inverse proportional to a square root of a posterior of a local motion feature corresponding to the sparse coding vector.

13. The computer-implement method of claim 9 further comprising:

using example sequences to train the action dynamic models and the at least one transition dynamic model.

14. The computer-implement method of claim 13 wherein using example sequences to train the action dynamic models and the at least one transition dynamic model comprises:

jointly training the action dynamic models and the at least one transition dynamic model using expectation-maximization (EM) methodology, wherein optimal decomposition of the example sequences into intra-action and inter-action transitions are obtained in an expectation step, and linear dynamic system model parameters are learned by maximum a posteriori estimation.

15. A computer-implemented method for recognizing a sequence of actions comprising:

segmenting input sensor data into time frames;

generating a feature for each time frame;

for a time frame, selecting an action associated with a maximized value of an objective function that comprises a first set of feature-dependent models for intra-action transitions and a second set of feature-dependent models for inter-action transitions.

16. The computer-implemented method of claim 15 wherein the feature-dependent models are based upon linear dynamic system models.

17. The computer-implemented method of claim 16 wherein the objective function models a hidden state at a current time frame as being directly dependent upon a hidden state for a sequence of one or more time frames prior to the current time frame, an estimated action for the sequence of one or more time frames prior to the current time frame, and an estimated action of the current time frame.

18. The computer-implemented method of claim 15 wherein the second set of feature-dependent models for inter-action transitions comprise at least one model for a transition between a same action.

19. The computer-implemented method of claim 15 wherein input sensor data is a video and the process of segmenting input sensor data into time frames comprises:

segmenting the video data into image frames.

20. The computer-implemented method of claim 19 wherein the feature for each image frame is an embedded optical flow feature formed by pooling a set of sparse coding vectors extracted from the image frame by weighting each sparse coding vector obtained from a local motion feature extracted from the image frame by an inverse proportional to a square root of a posterior of the local motion feature corresponding to the sparse coding vector.