Tracking rigged polygon-mesh models of articulated objects

• US 10,565,791 B2
• Filed: 02/26/2018
• Issued: 02/18/2020
• Est. Priority Date: 12/29/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shen, Jingjing, Cashman, Thomas Joseph, Hutton, Timothy James
• Primary Examiner(s): Vu, Kim Y
• Assistant Examiner(s): Delaney, Molly
• Application Number:
  US15/905,740
• Publication Number:
  US 20180182166A1
• Time in Patent Office: 722 Days
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06T 7/75 involving models

Abstract

A tracker is described which comprises a processor configured to receive captured sensor data depicting an object. The processor is configured to access a rigged polygon mesh model of the object and to compute a plurality of approximate surface normals of a limit surface of the rigged polygon mesh. The processor is configured to compute values of pose parameters of the model by calculating an optimization to fit the model to the captured sensor data where the optimization uses an evaluation function based on the plurality of approximate surface normals.

Claims

1. A tracker comprising:

a processor configured to receive captured sensor data depicting an object;

the processor configured to access a rigged polygon mesh model of the object;

the processor configured to compute a plurality of approximate surface normals; and

the processor configured to compute a plurality of control vertices of the rigged polygon mesh model by, for each of the plurality of the control vertices, projecting a vertex onto a limit surface corresponding to the rigged polygon mesh model to compute the plurality of control vertices on the limit surface.

2. The tracker of claim 1 where the processor is configured to compute values of pose parameters of the rigged polygon mesh model by calculating an optimization to fit the rigged polygon mesh model to the captured sensor data, where an evaluation metric of the optimization uses the plurality of approximate surface normal.

3. The tracker of claim 2 where the processor is configured to compute, for each control vertex, a control vertex normal of the control vertex with respect to the limit surface.

4. The tracker of claim 3 where the processor is configured to compute the approximate surface normals as interpolated values, interpolated between the control vertex normals.

5. The tracker of claim 4 where the processor is configured to access the rigged polygon mesh model in a neutral pose, and is configured to apply a specified pose to the rigged polygon mesh model and then to apply the specified pose to the control vertex normals before the interpolation is computed.

6. The tracker of claim 4 where the processor is configured to access the rigged polygon mesh model in a neutral pose, and is configured to apply a specified pose to the rigged polygon mesh before the control vertices and the control vertex normals are computed.

7. The tracker of claim 2 where the processor is configured to use a gradient-based optimization process to calculate the optimization.

8. The tracker of claim 2 where the pose parameters comprise global pose parameters and one or more joint positions.

9. The tracker of claim 1 where the captured sensor data comprises a depth image and the processor is configured to back project the depth image to compute a three dimensional point cloud.

10. The tracker of claim 2 where the processor is arranged to extract a region of interest from the captured sensor data, and to use the region of interest and not the rest of the captured sensor data during the optimization.

11. The tracker of claim 2 where the processor is arranged to obtain initial values of global pose parameters which are part of the pose parameters, either from a trained machine learning system or from previous values of the pose parameters, and to begin the optimization using the initial values of the global pose parameters.

12. The tracker of claim 2 which is arranged to repeatedly compute the optimization for instances of the captured sensor data, and to reinitialize the optimization using data from another source.

13. The tracker of claim 12 which is arranged to reinitialize the optimization using data from an independent tracker.

14. The tracker of claim 2 which is configured to send the computed values of the pose parameters to any one or more of:

an augmented reality system, a natural user interface, a game system, medical equipment.

15. A tracker comprising:

a processor configured to receive captured sensor data depicting an object; and

the processor configured to access a rigged polygon mesh model of the object in a neutral pose and to apply a specified pose to the rigged polygon mesh model;

the processor configured to compute control vertices projected from vertices of the rigged polygon mesh model in the specified pose and to compute a plurality of approximate surface normals of a limit surface of the rigged polygon mesh by interpolating between the control vertex normals; and

the processor configured to compute values of pose parameters of the model by calculating an optimization to fit the model to the captured sensor data, where the optimization uses an evaluation function based on the plurality of approximate surface normals.

16. A computer-implemented method comprising:

receiving captured sensor data depicting an object;

accessing a rigged polygon mesh model of the object; and

computing, at a processor, a plurality of approximate surface normals of a limit surface of the rigged polygon mesh model; and

for individual ones of vertices of the rigged polygon mesh model, projecting a vertex onto the limit surface of the rigged polygon mesh model to compute a plurality of control vertices on the limit surface.

17. The method of claim 16 comprising, computing, at the processor, values of pose parameters of the rigged polygon mesh model by calculating an optimization to fit the rigged polygon mesh model to the captured sensor data, where an evaluation metric of optimization uses the plurality of approximate surface normals.

18. The method of claim 17 comprising, for individual control vertices, computing a control vertex normal of the control vertex with respect to the limit surface.

19. The method of claim 18 comprising computing the approximate surface normals as interpolated values, interpolated between the control vertex normals and control vertices.

20. The method of claim 19 comprising accessing the rigged polygon mesh model in a neutral pose and applying a specified pose to the rigged polygon mesh model and then applying the specified pose to the control vertex normals before the interpolation is computed.