Calibrating object shape

• US 10,037,624 B2
• Filed: 12/29/2015
• Issued: 07/31/2018
• Est. Priority Date: 12/29/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cashman, Thomas Joseph, Tan, David Joseph New, Shotton, Jamie Daniel Joseph, Fitzgibbon, Andrew William, Khamis, Sameh, Taylor, Jonathan James, Sharp, Toby, Tarlow, Daniel Stefan
• Primary Examiner(s): McDowell, Jr., Maurice L
• Assistant Examiner(s): Ricks, Donna J
• Application Number:
  US14/982,568
• Publication Number:
  US 20170186226A1
• Time in Patent Office: 945 Days
• Field of Search: None
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

Examples describe an apparatus for calibrating a three dimensional (3D) mesh model of an articulated object. The articulated object is an instance of a specified object class. The apparatus comprises an input configured to receive captured sensor data depicting the object. The apparatus has a calibration engine configured to compute values of shape parameters of the 3D mesh model which indicate which member of the object class is depicted in the captured sensor data, in order to calibrate the 3D mesh model. The calibration engine is configured to compute the values of the shape parameters with an optimization process to find at least one potential local or global minimum of an energy function, the energy function expressing a degree of similarity between data rendered from the model and the received sensor data.

Claims

1. An apparatus for calibrating a three dimensional mesh model of an articulated object which is an instance of a specified object class, the apparatus comprising:

a processor configured to act as;

an input configured to receive captured sensor data depicting the object;

a calibration engine configured to compute values of shape parameters of the three dimensional mesh model which indicate which member of the object class is depicted in the captured sensor data, in order to calibrate the three dimensional mesh model;

the calibration engine configured to compute the values of the shape parameters with an optimization process to find at least one potential local or global minimum of an energy function, the energy function expressing a degree of similarity between data rendered from the three dimensional mesh model and the received sensor data; and

the calibration engine configured to compute the optimization process by computing derivatives of the energy function, wherein the optimization process includes a back-tracking search.

2. The apparatus of claim 1 where the shape parameters are included in the energy function jointly with pose parameters.

3. The apparatus of claim 1 where the calibration engine is further configured to calibrate the three dimensional mesh model, the calibrated three dimensional mesh model representing an individualized three dimensional mesh model that corresponds to the object;

and

wherein the processor is further configured to act as a tracker configured to track the object using the calibrated three dimensional mesh model.

4. The apparatus of claim 1 where the calibration engine is configured to compute the optimization process where the energy function is non-smooth.

5. The apparatus of claim 1 where the calibration engine is configured to compute the optimization process using finite differences.

6. The apparatus of claim 1 where the calibration engine is configured to use a differentiable renderer.

7. The apparatus of claim 1 where the calibration engine is configured to minimize the energy function where the energy function includes a pose prior energy.

8. The apparatus of claim 1 where the calibration engine is configured to minimize the energy function where the energy function includes a sum of squared differences penalty.

9. The apparatus of claim 1 where the input is configured to receive a plurality of instances of captured sensor data depicting the object in different poses and wherein the calibration engine is configured to jointly infer the shape parameters across the plurality of instances of the captured sensor data.

10. The apparatus of claim 1 where the mesh model comprises a combination of an articulated skeleton and a mapping from shape parameters to mesh vertices.

11. The apparatus of claim 1 where the mesh model comprises a model of a human hand.

12. The apparatus of claim 1 further comprising a tracker arranged to compute pose parameters of a rigged smooth-surface model of the articulated object, corresponding to the three dimensional mesh model, and using the shape parameters.

13. The apparatus of claim 12 the tracker configured to compute the pose parameters by fitting the captured sensor data to the rigged smooth-surface model after the shape parameters have been applied to the rigged smooth-surface model.

14. The apparatus of claim 12 the calibration engine configured to operate during operation of the tracker.

15. The apparatus of claim 1 comprising an output arranged to display a request for calibration data, being captured sensor data of the articulated entity, and where the calibration engine is configured to use the calibration data to compute the shape parameters.

16. A method of calibrating a three dimensional mesh model of an articulated object which is an instance of a specified object class comprising:

receiving captured sensor data depicting the object;

computing values of shape parameters which indicate which member of the object class is represented, in order to calibrate the three dimensional mesh model;

wherein computing the values of the shape parameters comprises using an optimization process to find at least one potential local or global minimum of an energy function, the energy function expressing a degree of similarity between data rendered from the three dimensional mesh model and the received sensor data; and

computing the optimization process by computing derivatives of the energy function, wherein the optimization process includes a back-tracking search.

17. The method of claim 16 comprising including shape parameters in the energy function jointly with pose parameters.

18. The method of claim 16 comprising:

calibrating the three dimensional mesh model, the calibrated three dimensional mesh model representing an individualized three dimensional mesh model that corresponds to the object; and

tracking the object using the calibrated three dimensional mesh model.

19. The method of claim 16 comprising computing the optimization process where the energy function is non-smooth.

20. An apparatus for calibrating a three dimensional mesh model of an articulated object which is an instance of a specified object class, the apparatus comprising:

a processor configured to act as;

an input configured to receive captured sensor data depicting the object;

a calibration engine configured to compute values of shape parameters of the three dimensional mesh model which indicate which member of the object class is depicted in the captured sensor data, in order to calibrate the three dimensional mesh model;

the calibration engine configured to compute the values of the shape parameters and values of pose parameters of the model with an optimization process to find at least one potential local or global minimum of an energy function, the energy function expressing a degree of similarity between data rendered from the three dimensional mesh model and the received sensor data;

the calibration engine further configured to compute the optimization process by computing derivatives of the energy function, wherein the optimization process includes a back-tracking search.