Tracking rigged smooth-surface models of articulated objects

• US 10,186,081 B2
• Filed: 12/29/2015
• Issued: 01/22/2019
• Est. Priority Date: 12/29/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Taylor, Jonathan James, Cashman, Thomas Joseph, Fitzgibbon, Andrew William, Sharp, Toby, Shotton, Jamie Daniel Joseph
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): Delaney, Molly
• Application Number:
  US14/982,878
• Publication Number:
  US 20170186165A1
• Time in Patent Office: 1,120 Days
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models

Abstract

A tracker is described which comprises an input configured to receive captured sensor data depicting an object. The tracker has a processor configured to access a rigged, smooth-surface model of the object and to compute values of pose parameters of the model by calculating an optimization to fit the model to data related to the captured sensor data. Variables representing correspondences between the data and the model are included in the optimization jointly with the pose parameters.

Claims

1. A tracker comprising:

an input controller configured to receive captured sensor data depicting an object; and

a processor configured to access a rigged, smooth-surface model of the object;

the processor configured to compute values of pose parameters of the model by calculating an optimization to fit the model to data related to the captured sensor data, where variables representing correspondences between the data and the model are included in the optimization jointly with the pose parameters; and

the processor further configured to repeatedly compute the optimization for instances of the captured sensor data, and to reinitialize the optimization using data from another source.

2. The tracker of claim 1 where the processor is configured to use a gradient-based optimization process to calculate the optimization.

3. The tracker of claim 1 where the pose parameters comprise global pose parameters and one or more joint positions.

4. The tracker of claim 1 where the captured sensor data comprises a depth image and the processor is configured to back project the depth image to compute a three dimensional point cloud.

5. The tracker of claim 1 where the processor is arranged to extract a region of interest from the captured sensor data, and to use the region of interest and not the rest of the captured sensor data during the optimization.

6. The tracker of claim 1 where the processor is arranged to obtain initial values of the variables representing correspondences from a trained machine learning system and to begin the optimization using the initial values.

7. The tracker of claim 1 where the processor is arranged to obtain initial values of the variables representing correspondences by choosing, for individual points of the captured sensor data, a closest point on the model, and to begin the optimization using the initial values.

8. The tracker of claim 1 where the processor is arranged to obtain initial values of the variables representing correspondences by random selection and to begin the optimization using the initial values.

9. The tracker of claim 1 where the processor is arranged to obtain initial values of global pose parameters which are part of the pose parameters, either from a trained machine learning system or from previous values of the pose parameters, and to begin the optimization using the initial values of the global pose parameters.

10. The tracker of claim 1 where the variables representing the correspondences are values specifying a point on the rigged smooth-surface model.

11. The tracker of claim 10 where the tracker comprises a defining function which generates a three dimensional world point corresponding to a value of a correspondence variable.

12. The tracker of claim 1 where the rigged smooth-surface model comprises a surface defined by repeatedly subdividing a mesh model.

13. The tracker of claim 1 which is arranged to reinitialize the optimization using data from an independent tracker.

14. A tracker comprising:

an input controller configured to receive captured sensor data depicting an object and

a processor configured to access a rigged, smooth-surface model of the object;

the processor configured to compute values of pose parameters of the model by using non-linear optimization to fit the model to data related to the captured sensor data, where variables representing correspondences between the data and the model are included in the optimization jointly with the pose parameters; and

the processor further configured to repeatedly compute the optimization for instances of the captured sensor data, and to reinitialize the optimization using data from another source.

15. A computer-implemented method comprising:

receiving captured sensor data depicting an object;

accessing a rigged, smooth-surface model of the object; and

computing, at a processor, values of pose parameters of the model by calculating an optimization to fit the model to data related to the captured sensor data, where variables representing correspondences between the data and the model are included in the optimization jointly with the pose parameters; and

computing, repeatedly, the optimization for instances of the captured sensor data, and reinitializing the optimization using data from another source.

16. The method of claim 15 comprising obtaining initial values of the variables representing correspondences from a trained machine learning system and beginning the optimization using the initial values.

17. The method of claim 15 where the variables representing the correspondences are values specifying a point on the rigged smooth-surface model.

18. The method of claim 15 comprising using a defining function which generates a three dimensional world point corresponding to a value of a correspondence variable.

19. The method of claim 15 where the rigged smooth-surface model comprises a surface defined by repeatedly subdividing a mesh model of the object.

20. The method of claim 15 further comprising reinitializing, by the processor, the optimization using data from an independent tracker.