Avatar construction using depth camera

• US 9,001,118 B2
• Filed: 08/14/2012
• Issued: 04/07/2015
• Est. Priority Date: 06/21/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Molyneaux, David, Tong, Xin, Liu, Zicheng, Chang, Eric, Yang, Fan, Kapur, Jay, Yang, Emily, Liu, Yang, Wu, Hsiang-Tao
• Primary Examiner(s): Faulk, Devona
• Assistant Examiner(s): BEUTEL, WILLIAM A
• Application Number:
  US13/585,540
• Publication Number:
  US 20130342527A1
• Time in Patent Office: 966 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/754 involving a deformation of the sample pattern or of the reference pattern; Elastic matching
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method for constructing an avatar of a human subject includes acquiring a depth map of the subject, obtaining a virtual skeleton of the subject based on the depth map, and harvesting from the virtual skeleton a set of characteristic metrics. Such metrics correspond to distances between predetermined points of the virtual skeleton. In this example method, the characteristic metrics are provided as input to an algorithm trained using machine learning. The algorithm may be trained using a human model in a range of poses, and a range of human models in a single pose, to output a virtual body mesh as a function of the characteristic metrics. The method also includes constructing a virtual head mesh distinct from the virtual body mesh, with facial features resembling those of the subject, and connecting the virtual body mesh to the virtual head mesh.

Claims

1. A method for constructing an avatar of a human subject, the method comprising:

acquiring a depth map of the human subject;

obtaining a virtual skeleton of the human subject based on the depth map;

harvesting from the virtual skeleton a set of characteristic metrics corresponding to distances between predetermined points of the virtual skeleton;

providing the characteristic metrics as input to an algorithm trained using machine learning, the algorithm having been trained using at least one single human being in a range of poses and a range of human beings in a single pose, to output a virtual body mesh as a function of the characteristic metrics;

refining the virtual body mesh by varying the characteristic metrics to decrease distance between points on the depth map and corresponding points of the virtual body mesh;

constructing a virtual head mesh, distinct from the virtual body mesh, with facial features resembling those of the human subject; and

connecting the virtual body mesh to the virtual head mesh.

2. The method of claim 1 further comprising repositioning the virtual skeleton and computing a deformation of the virtual body mesh based on the repositioned virtual skeleton.

3. The method of claim 2 wherein computing the deformation includes applying a linear skinning model to the virtual body mesh.

4. The method of claim 2 wherein computing the deformation includes applying a mechanical skinning simulation to the virtual body mesh.

5. The method of claim 2 further comprising repeating said repositioning and said computing to animate the avatar.

6. The method of claim 5 further comprising using the animated avatar as a character in a video game.

7. The method of claim 1 wherein the set of characteristic metrics are related to a size and shape of a body of the human subject.

8. The method of claim 1 further comprising applying a rotation, a body-pose transform, and a body shape transform to a virtual body-mesh template to obtain the virtual body mesh.

9. The method of claim 1 wherein the depth map is a first depth map, and wherein the virtual head mesh is constructed based on a second depth map, in which one or more facial features of the human subject are more finely resolved than in the first depth map.

10. The method of claim 9 further comprising deforming a virtual head-mesh template to minimize distance between points on the second depth map and corresponding points on the virtual head mesh.

11. The method of claim 1 further comprising augmenting the virtual body mesh with a body texture synthesized to match a face and skin texture of the human subject.

12. The method of claim 1 further comprising augmenting the virtual body mesh with virtual clothing modeled to resemble actual clothing for sale to the human subject.

13. The method of claim 12 further comprising presenting the avatar so constructed and so clothed for review by the human subject.

14. A method for constructing an avatar of a human subject, the method comprising:

acquiring a depth map of the human subject;

obtaining a virtual skeleton of the human subject based on the depth map;

harvesting from the virtual skeleton a set of characteristic metrics corresponding to distances between predetermined points of the virtual skeleton;

providing the characteristic metrics as input to an algorithm trained using machine learning, the algorithm having been trained using at least one single human being in a range of poses and a range of human beings in a single pose, to output a virtual body mesh as a function of the characteristic metrics;

refining the virtual body mesh by varying the characteristic metrics to minimize a distance between points on the depth map and corresponding points of the virtual body mesh;

constructing a virtual head mesh, distinct from the virtual body mesh, with facial features resembling those of the human subject; and

connecting the virtual body mesh to the virtual head mesh.

15. The method of claim 14 wherein the virtual body mesh is refined by applying an iterative closest-point algorithm.

16. A system for constructing an avatar of a human subject, the system comprising

a logic subsystem;

a storage subsystem holding instructions executable by the logic subsystem to;

obtain a virtual skeleton of the human subject based on a depth map,

harvest from the virtual skeleton a set of characteristic metrics corresponding to distances between predetermined points of the virtual skeleton,

provide the characteristic metrics as input to an algorithm trained using machine learning, the algorithm having been trained using at least one single human being in a range of poses and a range of human beings in a single pose, to output a virtual body mesh as a function of the characteristic metrics,

refine the virtual body mesh by varying the characteristic metrics to decrease distance between points on the depth map and corresponding points of the virtual body mesh;

augment the virtual body mesh with virtual clothing;

construct a virtual head mesh, distinct from the virtual body mesh, with facial features resembling those of the human subject, and

connect the virtual body mesh to the virtual head mesh.

17. The system of claim 16 further comprising:

a depth camera operatively coupled to the logic subsystem and configured to acquire a depth map of the human subject; and

a color camera operatively coupled to the logic subsystem and configured to acquire a color image of a face of the human subject;

wherein the instructions cause the logic subsystem to augment the virtual body mesh with one or more of a skin color and a skin texture derived from the color image.

18. The system of claim 16 wherein the characteristic metrics include one or more of a height, a leg length, an arm length, a shoulder width, a chest radius, a waist radius, a buttocks radius, an arm radius, and a leg radius.

19. The system of claim 16 wherein the range of poses includes one or more standing poses and one or more sitting poses.

20. The system of claim 16 wherein the range of human beings includes male and female human beings of varying body height and of varying body weight.