Video-based motion capture and adaptation

• US 9,322,653 B2
• Filed: 01/16/2013
• Issued: 04/26/2016
• Est. Priority Date: 01/16/2013
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Hodgins, Jessica K., Sigal, Leonid, Vondrak, Marek
• Primary Examiner(s): Shah, Utpal
• Assistant Examiner(s): TRAN, MAI H
• Application Number:
  US13/742,780
• Publication Number:
  US 20140198948A1
• Time in Patent Office: 1,196 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 11/04 Interpretation of pictures
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

The disclosure provides an approach for estimating a state-space controller from a set of video frames depicting a motion of an entity. The approach includes incrementally optimizing parameters of the state-space controller and changing a structure of the state-space controller based on expanding subsets of the set of video frames. In one embodiment, a controller-estimation application greedily selects, at every stage of the incremental optimization, structure and parameters of the controller which minimize an objective function. In another embodiment, the controller-estimation application re-optimizes, after the incremental optimization, all parameters of the state-space controller based on all of the video frames. In yet a further embodiment, the controller-estimation application alters the structure of the state-space controller for robustness and compactness by adding cycles in the state-space controller and enforcing constraints on the structure of the state-space controller and adding and modifying state transition types, as appropriate.

Claims

1. A computer-implemented method for estimating a state-space controller from a set of video frames, comprising:

receiving the set of video frames, wherein the video frames include two-dimensional (2D) images which depict a motion of an entity;

initializing, via one or more processors, the state-space controller, wherein the state-space controller includes states each corresponding to an atomic controller and transitions between the states; and

incrementally optimizing parameters of the state-space controller and changing a structure of the state-space controller to approximate through simulation the motion as depicted in an initial subset of the video frames and in expanded subsets of the video frames,

wherein the parameters include at least one of representations of target poses for angular configurations of a body and control and balance parameters,

wherein parameters of a last one or more states of the state-space controller are optimized at each stage of the incremental optimization, and

wherein changing the structure of the state-space controller includes adding one or more additional states and transitions between states to the state space controller.

2. The method of claim 1, further comprising, pre-processing the video frames to extract one or more image features from each video frame.

3. The method of claim 2, wherein the image features include at least one of foreground silhouettes, edges, color, texture, and motion capture marker locations.

4. The method of claim 1, further comprising re-optimizing the parameters of the state-space controller based on all of the video frames.

5. The method of claim 1, further comprising altering the structure of the state-space controller for robustness and compactness by at least one of adding cycles in the structure, enforcing constraints on the structure of the state-space controller, adding a state transition type, and modifying a state transition type.

6. The method of claim 5, wherein the incremental optimization minimizes an objective function with respect to state-space controller structure and parameters, initial pose, and initial velocity.

7. The method of claim 1, wherein the incremental optimization is greedy.

8. The method of claim 1, wherein the video frames include one or more depth maps.

9. The method of claim 1, where the incremental optimization is implemented using a genetic search algorithm.

10. The method of claim 1, where the incremental optimization is implemented using a sampling-based optimization.

11. A non-transitory computer-readable storage media storing instructions, which when executed by a computer system, perform operations for estimating a state-space controller from a set of video frames, the operations comprising:

receiving the set of video frames, wherein the video frames include two-dimensional (2D) images which depict a motion of an entity;

initializing the state-space controller, wherein the state-space controller includes states each corresponding to an atomic controller and transitions between the states; and

incrementally optimizing parameters of the state-space controller and changing a structure of the state-space controller to approximate through simulation the motion as depicted in an initial subset of the video frames and in expanded subsets of the video frames,

wherein the parameters include at least one of representations of target poses for angular configurations of a body and control and balance parameters,

wherein parameters of a last one or more states of the state-space controller are optimized at each stage of the incremental optimization, and

wherein changing the structure of the state-space controller includes adding one or more additional states and transitions between states to the state space controller.

12. The computer-readable storage media of claim 11, the operations further comprising, pre-processing the video frames to extract one or more image features from each video frame.

13. The computer-readable storage media of claim 12, wherein the image features include at least one of foreground silhouettes, edges, color, texture, and motion capture marker locations.

14. The computer-readable storage media of claim 11, the operations further comprising, re-optimizing the parameters of the state-space controller based on all of the video frames.

15. The computer-readable storage media of claim 11, the operations further comprising altering the structure of the state-space controller for robustness and compactness by at least one of adding cycles in the state-space controller, enforcing constraints on the structure of the state-space controller, adding a state transition type, and modifying a state transition type.

16. The computer-readable storage media of claim 15, wherein the incremental optimization minimizes an objective function with respect to state-space controller structure and parameters, initial pose, and initial velocity.

17. The computer-readable storage media of claim 11, wherein the video frames include one or more depth maps.

18. The computer-readable storage media of claim 11, where the incremental optimization is implemented using a genetic search algorithm.

19. The computer-readable storage media of claim 11, where the incremental optimization is implemented using a sampling-based optimization.

20. A system, comprising:

a processor; and

a memory, wherein the memory includes an application program configured to perform operations for estimating a state-space controller from a set of video frames, the operations comprising;

receiving the set of video frames, wherein the video frames include two-dimensional (2D) images which depict a motion of an entity,

initializing the state-space controller, wherein the state-space controller includes states each corresponding to an atomic controller and transitions between the states, and

incrementally optimizing parameters of the state-space controller and changing a structure of the state-space controller to approximate through simulation the motion as depicted in an initial subset of the video frames and in expanded subsets of the video frames;

wherein the parameters include at least one of representations of target poses for angular configurations of a body and control and balance parameters;

wherein parameters of a last one or more states of the state-space controller are optimized at each stage of the incremental optimization; and

wherein changing the structure of the state-space controller includes adding one or more additional states and transitions between states to the state space controller.