Semantic rigging of avatars

• US 9,734,637 B2
• Filed: 12/06/2010
• Issued: 08/15/2017
• Est. Priority Date: 12/06/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Winchester, Royal D.
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US12/960,898
• Publication Number:
  US 20120139899A1
• Time in Patent Office: 2,444 Days
• Field of Search: 345419, 345473
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/52 Parallel processing
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

Techniques are described herein that are capable of generating a virtual rig for animation of a virtual 3D representation (e.g., an avatar) of an object. Semantic identifiers are associated with vertices of the virtual 3D representation. Each of the semantic identifiers specifies a respective feature of the object. The virtual rig is generated based on association of the semantic identifiers with the respective vertices. The virtual rig is capable of maneuvering the mesh to multiple configurations to animate the virtual 3D representation. In an example, the semantic identifiers may be used to generate and/or place virtual items in combination with the mesh. In another example, the semantic identifiers may be used to define reference elements. The reference elements serve as references for maneuvering respective portions of the mesh and/or virtual items that are associated with those portions.

Claims

1. A method for generating a virtual rig to animate a virtual three-dimensional representation of a human face, the method comprising:

receiving a mesh that includes a plurality of vertices to provide a virtual three-dimensional representation of a human face, the plurality of vertices defining a plurality of polygons that define a virtual three-dimensional representation of a surface of the human face;

associating a plurality of semantic identifiers with the plurality of respective vertices, each semantic identifier specifying a respective physical feature of the human face;

generating a virtual rig that is capable of maneuvering the mesh to a plurality of configurations, using one or more processors, based at least in part on association of the plurality of semantic identifiers with the plurality of respective vertices to animate the virtual three-dimensional representation of the human face;

determining that a virtual item is to be combined with the mesh to represent a corresponding item of the human face based at least in part on a designated subset of the plurality of semantic identifiers; and

generating the virtual item based at least in part on a subset of the plurality of vertices that is associated with the designated subset of the plurality of semantic identifiers and further based at least in part on whether the virtual item is a first virtual eyeball that represents a first physical eyeball of the human face or a second virtual eyeball that represents a second physical eyeball of the human face;

wherein at least one of a position, a size, or a shape of the first virtual eyeball and a respective at least one of a position, a size, or a shape of the second virtual eyeball are asymmetric with respect to the mesh to represent that at least one of a position, a size, or a shape of the first physical eyeball and a respective at least one of a position, a size, or a shape of the second physical eyeball are asymmetric with respect to the human face; and

wherein generating the virtual rig comprises;

defining a reference element based at least in part on the designated subset of the plurality of semantic identifiers and further based at least in part on a thickness of a virtual layer that is coincident with an outer surface of the mesh to serve as a reference for maneuvering the virtual item, the virtual layer being a virtual representation of skin that is coincident with the surface of the human face.

2. The method of claim 1, wherein the plurality of vertices is associated with a plurality of respective vertex values;

wherein the method further comprises;

changing at least one vertex value of the plurality of vertex values to be at least one respective changed vertex value on which the virtual three-dimensional representation is based, based at least in part on an image of the human face, to increase accuracy of the virtual three-dimensional representation of the human face; and

wherein generating the virtual rig comprises;

generating the virtual rig based at least in part on the at least one changed vertex value.

3. The method of claim 2, further comprising:

capturing the image; and

analyzing the image to determine the at least one changed vertex value.

4. The method of claim 1, wherein defining the reference element comprises:

defining a plurality of reference elements based on a plurality of respective subsets of the plurality of semantic identifiers, each reference element serving as a reference for maneuvering a respective portion of the mesh that corresponds to the subset on which that reference element is based.

5. The method of claim 1,

wherein the virtual rig is capable of maneuvering the virtual item to a second plurality of configurations based on the reference element to animate the virtual three-dimensional representation of the human face.

6. The method of claim 1, wherein defining the reference element comprises:

determining at least one of a location or a position of the reference element based at least in part on the thickness of the virtual layer.

7. The method of claim 1, wherein generating the virtual rig further comprises:

automatically creating one or more supplementary reference elements that aid in determining at least one of a location or a position of the reference element; and

modifying at least one of an initial location or an initial orientation of the reference element based at least in part on the one or more supplementary reference elements to provide at least one of a revised location or a revised orientation of the reference element.

8. A system for generating a virtual rig to animate a virtual three-dimensional representation of a human face, the system comprising:

at least one element including at least one of (a) one or more processors or (b) hardware logic/electrical circuitry;

a semantic identifier module, implemented using the at least one element, configured to associate a plurality of semantic identifiers with a plurality of respective vertices of a mesh that provides a virtual three-dimensional representation of a human face, the plurality of vertices defining a plurality of polygons that define a virtual three-dimensional representation of a surface of the human face, each semantic identifier specifying a respective physical feature of the human face;

a virtual rig module, implemented using the at least one element, configured to generate a virtual rig that is capable of maneuvering the mesh to a plurality of configurations based at least in part on association of the plurality of semantic identifiers with the plurality of respective vertices to animate the virtual three-dimensional representation of the human face; and

a virtual item module, implemented using the at least one element, configured to determine that a virtual item is to be combined with the mesh to represent a corresponding item of the human face based at least in part on a designated subset of the plurality of semantic identifiers, the virtual item module further configured to generate the virtual item based at least in part on a subset of the plurality of vertices that is associated with the designated subset of the plurality of semantic identifiers and further based at least in part on whether the virtual item is a first virtual eyeball that represents a first physical eyeball of the human face or a second virtual eyeball that represents a second physical eyeball of the human face;

wherein at least one of a position, a size, or a shape of the first virtual eyeball and a respective at least one of a position, a size, or a shape of the second virtual eyeball are asymmetric with respect to the mesh to represent that at least one of a position, a size, or a shape of the first physical eyeball and a respective at least one of a position, a size, or a shape of the second physical eyeball are asymmetric with respect to the human face;

wherein the virtual rig module comprises;

a reference element module, implemented using the at least one element, configured to define a reference element based at least in part on the designated subset of the plurality of semantic identifiers and further based at least in part on a thickness of a virtual layer that is coincident with an outer surface of the mesh to serve as a reference for maneuvering the virtual item; and

wherein the virtual layer is a virtual representation of skin that is coincident with the surface of the human face.

9. The system of claim 8, wherein the plurality of vertices is associated with a plurality of respective vertex values;

wherein the system further comprises;

a mesh module, implemented using the at least one element, configured to change at least one vertex value of the plurality of vertex values to be at least one respective changed vertex value on which the virtual three-dimensional representation of the human face is based, based at least in part on an image of the human face, to increase accuracy of the virtual three-dimensional representation of the human face; and

wherein the virtual rig module is configured to generate the virtual rig based at least in part on the at least one changed vertex value.

10. The system of claim 8,

wherein the virtual rig is further configured to maneuver the virtual item to a second plurality of configurations based at least in part on the reference element to animate the virtual three-dimensional representation of the object.

11. The system of claim 8, wherein the reference element module is configured to determine at least one of a location or a position of the reference element based at least in part on the thickness of the virtual layer.

12. The system of claim 8, wherein the reference element module is configured to define a plurality of reference elements based on a plurality of respective subsets of the plurality of semantic identifiers, each reference element serving as a reference for maneuvering a respective portion of the mesh that corresponds to the subset on which that reference element is based.

13. The system of claim 8, wherein the virtual rig module is configured to create one or more supplementary reference elements to aid in a determination of at least one of a location or a position of the reference element;

and

wherein the virtual rig module is configured to modify at least one of an initial location or an initial orientation of the reference element based at least in part on the one or more supplementary reference elements to provide at least one of a revised location or a revised orientation of the reference element.

14. The system of claim 9, further comprising:

a camera to capture the image; and

a mesh module, implemented using the at least one element, configured to analyze the image to determine the at least one changed vertex value.

15. The system of claim 13, wherein the virtual rig module is configured to automatically create the one or more supplementary reference elements.

16. A computer program product comprising a computer-readable medium, which is not a signal, having instructions recorded thereon for enabling a processor-based system to generate a virtual rig to animate a virtual three-dimensional representation of a human head, the instructions comprising:

first instructions for enabling the processor-based system to associate a plurality of semantic identifiers with a plurality of respective vertices of a mesh that provides a virtual three-dimensional representation of a human head, the plurality of vertices defining a plurality of polygons that define a virtual three-dimensional representation of a face of the human head, each semantic identifier specifying a respective feature of the human head;

second instructions for enabling the processor-based system to generate a virtual rig that is capable of maneuvering the mesh to a plurality of configurations based at least in part on association of the plurality of semantic identifiers with the plurality of respective vertices to animate the virtual three-dimensional representation of the human head;

third instructions for enabling the processor-based system to determine that first and second virtual eyeballs are to be combined with the mesh to represent corresponding first and second physical eyeballs of the human head based at least in part on respective first and second designated subsets of the plurality of semantic identifiers; and

fourth instructions for enabling the processor-based system to generate the first and second virtual eyeballs based at least in part on respective first and second subsets of the plurality of vertices that are associated with the respective first and second designated subsets of the plurality of semantic identifiers;

wherein at least one of a position, a size, or a shape of the first virtual eyeball and a respective at least one of a position, a size, or a shape of the second virtual eyeball are asymmetric with respect to the mesh to represent that at least one of a position, a size, or a shape of the first physical eyeball and a respective at least one of a position, a size, or a shape of the second physical eyeball are asymmetric with respect to the human face; and

wherein the second instructions comprise;

instructions for enabling the processor-based system to define respective first and second reference elements based at least in part on the respective first and second designated subsets of the plurality of semantic identifiers and further based at least in part on a thickness of a virtual layer that is coincident with an outer surface of the mesh to serve as references for maneuvering the respective first and second virtual eyeballs, wherein the virtual layer is a virtual representation of skin that is coincident with the surface of the human head.

17. The computer program product of claim 16, wherein the plurality of vertices is associated with a plurality of respective vertex values;

wherein the instructions further comprise;

fifth instructions for enabling the processor-based system to change at least one vertex value of the plurality of vertex values to be at least one respective changed vertex value on which the virtual three-dimensional representation of the human head is based, based at least in part on an image of the human head, to increase accuracy of the virtual three-dimensional representation of the human head; and

wherein the second instructions include instructions for enabling the processor-based system to generate the virtual rig based at least in part on the at least one changed vertex value.

18. The computer program product of claim 16,

wherein the instructions further comprise:

fifth instructions for enabling the processor-based system to maneuver the first and second virtual eyeballs to respective configurations based at least in part on the respective first and second reference elements.

19. The computer program product of claim 16, wherein the fifth instructions comprise:

instructions for enabling the processor-based system to determine at least one of a location or a position of at least one of the first reference element or the second reference element based at least in part on the thickness of the virtual layer.

20. The computer program product of claim 17, further comprising:

sixth instructions for enabling the processor-based system to capture the image; and

seventh instructions for enabling the processor-based system to analyze the image to determine the at least one changed vertex value.