DEEP-LEARNING MOTION PRIORS FOR FULL-BODY PERFORMANCE CAPTURE IN REAL-TIME

• US 20180096259A1
• Filed: 09/30/2016
• Published: 04/05/2018
• Est. Priority Date: 09/30/2016
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Andrews, Sheldon, Huerta Casado, Ivan, Mitchell, Kenneth J., Sigal, Leonid
• Granted Patent: US 10,902,343 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Training data from multiple types of sensors and captured in previous capture sessions can be fused within a physics-based tracking framework to train motion priors using different deep learning techniques, such as convolutional neural networks (CNN) and Recurrent Temporal Restricted Boltzmann Machines (RTRBMs). In embodiments employing one or more CNNs, two streams of filters can be used. In those embodiments, one stream of the filters can be used to learn the temporal information and the other stream of the filters can be used to learn spatial information. In embodiments employing one or more RTRBMs, all visible nodes of the RTRBMs can be clamped with values obtained from the training data or data synthesized from the training data. In cases where sensor data is unavailable, the input nodes may be unclamped and the one or more RTRBMs can generate the missing sensor data.

Claims

1. A method for motion capture, the method being implemented by a processor configured to execute machine-readable instructions, the method comprising:

obtaining a deep learning model;

obtaining training data and training the deep learning model using the training data, the training data including temporal and spatial information regarding one or more actors'"'"' motion captured previously;

receiving, from one or more motion capture sensors, real-time motion data for an actor'"'"'s movement; and

estimating motion information regarding the actor'"'"'s motion based on the received motion data using the deep learning model.

2. The method of claim 1, wherein the deep learning model includes a convolutional neural network (CNN), and wherein the training of the deep learning model includes receiving the spatial information in a first stream to learn a spatial relationship of the one or more actors'"'"' motion captured previously.

3. The method of claim 2, wherein the training of the motion model further includes receiving the temporal information in a second stream to learn a temporal relationship of the deep learning model, wherein the second stream is received independent of the first stream.

4. The method of claim 3, wherein the training of the motion model further includes connecting the first and second streams using a recurrent neural net (RNN).

5. The method of claim 4, wherein the training of the motion model further includes training the motion model using a stochastic gradient decent with RELU activation.

6. The method of claim 1, wherein estimating missing motion information from the received motion data using the motion model includes estimating a reference pose for a frame.

7. The method of claim 1, wherein the deep learning model includes a recurrent temporal restricted Boltzmann machine (RTRBM), and wherein training of the motion model includes receiving raw sensor data and presenting the sensor data to a network in a last visible layer of the RTRBM.

8. The method of claim 7, wherein training of the motion model includes synthesizing the raw sensor data and clamping all visible nodes with the raw sensor data and/or the synthesized sensor data.

9. The method of claim 7, wherein estimating missing motion information from the received motion data using the motion model includes estimating a reference pose for a frame and combining the received motion data with the estimated reference pose.

10. A system for motion capture, the system comprising one or more of a processor configured to execute machine-readable instructions such that when the machine-readable instructions are executed, the process is caused to perform:

obtaining a deep learning model;

obtaining training data and training the deep learning model using the training data, the training data including temporal and spatial information regarding one or more actors'"'"' motion captured previously;

receiving, from one or more motion capture sensors, real-time motion data for an actor'"'"'s movement; and

estimating motion information regarding the actor'"'"'s motion based on the received motion data using the deep learning model.

11. The system of claim 10, wherein the deep learning model includes a convolutional neural network (CNN), and wherein the training of the deep learning model includes receiving the spatial information in a first stream to learn a spatial relationship of the one or more actors'"'"' motion captured previously.

12. The system of claim 11, wherein the training of the motion model further includes receiving the temporal information in a second stream to learn a temporal relationship of the deep learning model, wherein the second stream is received independent of the first stream.

13. The system of claim 12, wherein the training of the motion model further includes connecting the first and second streams using a recurrent neural net (RNN).

14. The system of claim 13, wherein the training of the motion model further includes training the motion model using a stochastic gradient decent with RELU activation.

15. The system of claim 10, wherein estimating missing motion information from the received motion data using the motion model includes estimating a reference pose for a frame.

16. The system of claim 10, wherein the deep learning model includes a recurrent temporal restricted Boltzmann machine (RTRBM), and wherein training of the motion model includes receiving raw sensor data and presenting the sensor data to a network in a last visible layer of the RTRBM.

17. The system of claim 16, wherein training of the motion model includes synthesizing the raw sensor data and clamping all visible nodes with the raw sensor data and/or the synthesized sensor data.

18. The system of claim 17, wherein estimating missing motion information from the received motion data using the motion model includes estimating a reference pose for a frame and combining the received motion data with the estimated reference pose.