Generation of avatar reflecting player appearance

• US 9,013,489 B2
• Filed: 11/16/2011
• Issued: 04/21/2015
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Clark, Justin Avram, Osborn, Dan, Evertt, Jeffrey Jesus, Middleton, Zachary Tyler, Puls, Matthew J, Mihelich, Mark Thomas, Campbell, Andrew R, Martin, Charles Everett, Hill, David M
• Primary Examiner(s): Faulk, Devona
• Assistant Examiner(s): Ge, Jin
• Application Number:
  US13/297,890
• Publication Number:
  US 20120309520A1
• Time in Patent Office: 1,252 Days
• Field of Search: 345/473, 345/418
• US Class Current: 345/473
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Systems, methods, and computer media for generating an avatar reflecting a player'"'"'s current appearance. Data describing the player'"'"'s current appearance is received. The data includes a visible spectrum image of the player, a depth image including both the player and a current background, and skeletal data for the player. The skeletal data indicates an outline of the player'"'"'s skeleton. Based at least in part on the received data, one or more of the following are captured: a facial appearance of the player; a hair appearance of the player; a clothing appearance of the player; and a skin color of the player. A 3D avatar resembling the player is generated by combining the captured facial appearance, hair appearance, clothing appearance, and/or skin color with predetermined avatar features.

Claims

1. Computer hardware devices having embodied thereon computer-executable instructions that, when executed by a computing device, perform a method of generating an avatar reflecting a player'"'"'s current appearance, the method comprising:

receiving data describing the player'"'"'s current appearance, the data including;

a visible spectrum image of the player, a depth image including both the player and a current background, and skeletal data for the player, the skeletal data indicating an outline of the player'"'"'s skeleton;

capturing a facial appearance of the player based at least in part on the received data by;

identifying the player'"'"'s face,

identifying facial features of the player'"'"' s face, and

warping a portion of the received visible spectrum image that includes the player'"'"'s face into a face texture map based on the identified facial features, wherein the face texture map is a UV space map of the portion of the received visible spectrum image that includes the player'"'"'s face;

capturing a hair appearance of the player based at least in part on the received data by;

identifying an extent of the player'"'"' s hair,

matching the identified extent of the player'"'"'s hair to a predetermined hairstyle template, and

identifying a color of the player'"'"' s hair;

capturing a clothing appearance of the player based at least in part on the received data by;

using the received depth image including both the player and the current background and the received visible spectrum image of the player, determining an outer clothing boundary using edge detection,

using the received visible spectrum image of the player, determining at least one inner clothing boundary by distinguishing clothing from skin based on a determined color gradient, and

identifying player clothing as the area bounded at least in part by the outer clothing boundary and the at least one inner clothing boundary; and

wherein the generated 3D avatar has clothing resembling the identified player clothing; and

generating a 3D avatar reflecting the captured facial appearance and hair appearance of the player such that the 3D avatar resembles the player by combining the face texture map, the hairstyle template that matches the identified extent of the player'"'"'s hair, and the identified player hair color with predetermined avatar features.

2. The devices of claim 1, wherein identifying the player'"'"'s face comprises:

identifying the player'"'"'s head by analyzing the received skeletal data for the player,

identifying a head portion of the received visible spectrum image that corresponds to the player'"'"'s head, and

applying a face detection algorithm to the head portion of the received visible spectrum image.

3. The devices of claim 1, wherein capturing the facial appearance of the player further comprises identifying face alignment points corresponding to the identified facial features, and

wherein warping a portion of the received visible spectrum image that includes the player'"'"'s face into the face texture map based on the identified facial features comprises:

matching the identified face alignment points to destination points on a template texture map such that the face texture map includes the face alignment points corresponding to the identified facial features of the player'"'"'s face mapped to the destination points of the template texture map.

4. The devices of claim 3, wherein generating the 3D avatar comprises creating a morphed 3D head model by:

analyzing the mapped face alignment points of the face texture map,

for a plurality of the identified facial features corresponding to the mapped face alignment points, calculating the size of the plurality of identified facial features, and

based on the face texture map and the calculated size of the plurality of identified facial features, combining a plurality of predetermined head models to create the morphed 3D head model.

5. The devices of claim 1, wherein identifying the extent of the player'"'"'s hair comprises:

using the received depth image including both the player and the current background, determining an outer hair boundary by distinguishing the depth of the background from the depth of the player'"'"'s hair, and

using the received visible spectrum image of the player, determining an inner hair boundary by distinguishing the player'"'"'s skin from the player'"'"'s hair.

6. The devices of claim 5, wherein determining the inner hair boundary comprises:

distinguishing the player'"'"'s hair from clothing worn by the player, and

using the facial features identified in capturing the facial appearance of the player, determining the extent of the player'"'"'s forehead.

7. The devices of claim 5, wherein matching the identified extent of the player'"'"'s hair to a predetermined hairstyle template comprises:

identifying the player'"'"'s hair as the area bounded at least in part by the outer and inner hair boundaries,

identifying a plurality of hair attributes of the player'"'"'s hair, each of the plurality of hair attributes describing a portion or characteristic of the player'"'"'s hair, and

comparing the identified plurality of hair attributes to a library of predetermined hairstyle templates.

8. The devices of claim 1, wherein the edge detection distinguishes the depth of the background from the depth of the player in the received depth image and distinguishes the color of clothes from the color of the background in the received visible spectrum image.

9. The devices of claim 1, wherein capturing the clothing appearance of the player further comprises:

using the received skeletal data, identifying body features of the player, and

determining the outer clothing boundary and the at least one inner clothing boundary by analyzing areas of the received visible spectrum image and areas of the received depth image corresponding to identified body features of the player where clothes are typically worn.

10. The devices of claim 1, wherein capturing the clothing appearance of the player further comprises:

rescaling the skeletal data for the player to a predetermined skeleton size to form a scaled player skeleton,

reposing the scaled player skeleton to a predetermined pose, and

creating a clothing texture map based on the reposed, scaled player skeleton and the identified player clothing; and

wherein a 3D body model generated from the clothing texture map is used in generating the 3D avatar.

11. The devices of claim 1, wherein capturing the clothing appearance of the player further comprises:

using the received skeletal data, identifying areas corresponding to the arms and legs of the player, and

analyzing the color of the received visible spectrum image of the player along the identified areas corresponding to the arms and legs of the player to detect at least one of;

(1) a shirt sleeve, (2) a pants or shorts leg, and (3) a skirt or dress hem.

12. The devices of claim 1, wherein capturing the clothing appearance of the player further comprises:

using the received skeletal data, identifying areas corresponding to the legs of the player, and

using the received depth image including both the player and the current background, analyzing the area between the identified areas corresponding to the legs of the player to detect a skirt or dress edge by distinguishing background depth from player depth and identifying a substantially horizontal edge separating the background depth from the player depth.

13. The devices of claim 1, further comprising capturing a skin color of the player based at least in part on the received data by:

sampling the color of one or more areas of the received visible spectrum image in areas corresponding to the player'"'"'s face, and

blending the sampled colors of the one or more areas to determine a player skin color; and

wherein the generated 3D avatar has a skin color reflecting the determined player skin color.

14. One or more computer hardware devices having a system embodied thereon including computer-executable instructions that, when executed, perform a method for providing query-dependent audio and video clip previews, the system comprising:

a data acquisition component that receives data describing the player'"'"'s current appearance, the data including;

a visible spectrum image of the player, a depth image including both the player and a current background, and skeletal data for the player, the skeletal data indicating an outline of the player'"'"'s skeleton;

a face capture component that captures a facial appearance of the player based at least in part on the received data, wherein capturing the facial appearance of the player comprises warping a portion of the received visible spectrum image comprising the player'"'"'s face into a face texture map, wherein the face texture map is a UV space map of the portion of the received visible spectrum image that includes the player'"'"'s face;

a hair capture component that captures a hair appearance of the player based at least in part on the received data;

a clothing capture component that captures a clothing appearance of the player based at least in part on the received data by;

using the received depth image including both the player and the current background and the received visible spectrum image of the player, determining an outer clothing boundary using edge detection,

using the received visible spectrum image of the player, determining at least one inner clothing boundary by distinguishing clothing from skin based on a determined color gradient, and

identifying player clothing as the area bounded at least in part by the outer clothing boundary and the at least one inner clothing boundary;

an avatar generation component that generates a 3D avatar resembling the player by combining the captured facial appearance, hair appearance, and clothing appearance with predetermined avatar features.

15. The devices of claim 14, further comprising:

a sleeve detection component that identifies, based at least in part on the received data, at least one of;

(1) a shirt sleeve, (2) a pants or shorts leg, and (3) a skirt or dress hem;

a skin color capture component that identifies a skin color of the player based at least in part on the received data; and

wherein the avatar generation component combines the identified at least one of;

(1) a shirt sleeve, (2) a pants or shorts leg, and (3) a skirt or dress hem and the identified skin color to generate the 3D avatar resembling the player.

16. A computer-implemented method for generating an avatar reflecting a player'"'"'s current appearance, the method comprising:

receiving data describing the player'"'"'s current appearance, the data including;

a visible spectrum image of the player, a depth image including both the player and a current background, and skeletal data for the player, the skeletal data indicating an outline of the player'"'"'s skeleton;

capturing a facial appearance of the player based at least in part on the received data by;

identifying the player'"'"'s face,

identifying facial features of the player'"'"'s face, and

warping a portion of the received visible spectrum image that includes the player'"'"'s face into a face texture map based on the identified facial features, wherein the face texture map is a UV space map of the portion of the received visible spectrum image that includes the player'"'"'s face;

capturing a hair appearance of the player based at least in part on the received data by;

identifying an extent of the player'"'"'s hair,

matching the identified extent of the player'"'"'s hair to a predetermined hairstyle template, and

identifying a color of the player'"'"'s hair;

capturing a clothing appearance of the player based at least in part on the received data by;

using the received depth image including both the player and the current background and the received visible spectrum image of the player, determining an outer clothing boundary using edge detection,

using the received visible spectrum image of the player, determining at least one inner clothing boundary by distinguishing clothing from skin based on a determined color gradient, and

identifying player clothing as the area bounded at least in part by the outer clothing boundary and the at least one inner clothing boundary;

capturing a skin color of the player based at least in part on the received data by;

sampling the color of one or more areas of the received visible spectrum image in areas corresponding to the player'"'"'s face, and

blending the sampled colors of the one or more areas to determine a player skin color; and

using a processor of a computing device to generate a 3D avatar reflecting the captured facial appearance, hair appearance, clothing appearance, and skin color of the player such that the 3D avatar resembles the player by combining the face texture map, hairstyle template that matches the identified extent of the player'"'"'s hair, and identified player hair color with predetermined avatar features.

17. The method of claim 16, wherein capturing the clothing appearance of the player further comprises:

using the received skeletal data, identifying areas corresponding to the arms and legs of the player, and

analyzing the color gradient of the received visible spectrum image of the player along the identified areas corresponding to the arms and legs of the player to detect at least one of;

(1) shirt sleeve, (2) pants or shorts leg, and (3) skirt or dress hem.

18. The method of claim 16, wherein capturing the clothing appearance of the player further comprises:

using the received skeletal data, identifying areas corresponding to the legs of the player, and

using the received depth image including both the player and the current background, analyzing the area between the identified areas corresponding to the legs of the player to detect a skirt or dress edge by distinguishing background depth from player depth and identifying a substantially horizontal edge separating the background depth from the player depth.

19. The method of claim 16, wherein identifying the extent of the player'"'"'s hair comprises:

using the received depth image including both the player and the current background, determining an outer hair boundary by distinguishing the depth of the background from the depth of the player'"'"'s hair,

using the received visible spectrum image of the player, determining an inner hair boundary by distinguishing the player'"'"'s skin from both the player'"'"'s hair and from clothing worn by the player,

using the facial features identified in capturing the facial appearance of the player, determining the extent of the player'"'"'s forehead; and

wherein matching the identified extent of the player'"'"'s hair to a predetermined hairstyle template comprises;

identifying the player'"'"'s hair as the area bounded at least in part by the outer and inner hair boundaries,

identifying a plurality of hair attributes of the player'"'"'s hair, each of the plurality of hair attributes describing a portion or characteristic of the player'"'"'s hair, and

comparing the identified plurality of hair attributes to a library of predetermined hairstyle templates.