Facial animation method

• US 7,289,124 B2
• Filed: 07/11/2001
• Issued: 10/30/2007
• Est. Priority Date: 07/13/2000
• Status: Expired due to Fees

• Current Assignee: Orange S.A.
• Original Assignee: Orange S.A.
• Inventors: Breton, Gaspard, Pele, Danielle, Bouville, Christian
• Primary Examiner(s): Zimmerman; Mark
• Assistant Examiner(s): Yang; Andrew
• Application Number:
  US10/258,879
• Publication Number:
  US 20030160791A1
• Time in Patent Office: 2,302 Days
• Field of Search: 345/420
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation

Abstract

A 3D imaged head consisting of a set of meshes is animated by modelled facial muscles. For natural animation of the lips in speech, the lower and upper lips are distinguished in a set of lip mesh based on the nodes of the mesh on a boundary of the lip corner and mouth opening, and lower and upper zones of the head are divided by the boundary. Modelled muscles attached to the upper lip, with intersecting zones of influence in the upper zone, and attached to the lower lip, with intersecting zones of influence in the lower zone, displace nodes of mesh respectively in the upper lip and under the nose and in the lower lip and the chin. Other muscles have zones of influence in the lower and upper zones.

Claims

1. A method of animating, by use of modelled facial muscles, a head imaged in three dimensions and acquired in a data processing arrangement in the form of sets of meshes including a set of lips, said method including the steps of:

distinguishing a lower lip and an upper lip in said set of lips as a function of mesh nodes at the level of a lip corner and mouth opening boundary,

the distinguishing step including;

(a) identifying mesh nodes at the periphery of said set of lips to define an interior contour and an exterior contour of said lips,

(b) detecting laterally outermost mesh nodes on said interior contour and exterior contour,

(c) marking mesh nodes below one of lip corner segments defined by the nodes previously detected as belonging to said lower lip,

(d) recursively marking all the unmarked mesh nodes, starting with nodes near a lower node of said lower lip, as belonging to said lower lip, and

(e) marking all the other unmarked nodes in said set of lips not belonging to said lower lip as belonging to said upper lip,

determining an upper zone and a lower zone of said head substantially shared by at least a lip corner and mouth opening boundary between the mesh nodes belonging to said lower lip and the mesh nodes belonging to said upper lip, and

providing a first group of modelled muscles extending toward said upper lip and having zones of influence intersecting at least two by two situated in said upper zone for displacing mesh nodes in said upper lip and under a nose of said head, a second group of modelled muscles extending toward said lower lip and having zones of influence intersecting at least two by two situated in said lower zone for displacing mesh nodes in said lower lip and in a chin of said head, and a last group of modelled muscles having zones of influence for displacing each of the mesh nodes in said upper zone and lower zone in order to stretch and contract said lips.

2. A method according to claim 1, wherein said last group includes modelled muscles extending respectively up to the corners of said lips and having independent zones of influence for respectively and independently moving mesh nodes in opposite directions toward the exterior of said corners of said lips.

3. A method according to claim 1, wherein said last group includes modelled muscles attached to the middle of the mouth and having independent zones of influence for respectively and independently displacing mesh nodes toward said corners of said lips and toward the interior of said mouth.

4. A method according to claim 1, wherein said determining step includes the following steps:

determining the middle of a segment between external nodes as laterally outermost nodes detected on said exterior contour of said lips,

determining jaw pivot axis nodes,

marking all the nodes situated under a broken line joining said jaw pivot axis nodes through said segment between said external nodes as belonging to the lower zone, and

marking all the other mesh nodes of said head as belonging to the upper zone.

5. A method according to claim 1, wherein a modelled muscle is defined by a conical zone of influence having a cone angle α

, and a mesh node inside said zone of influence is subjected to a displacement vector that is directed toward the apex of said zone of influence at an angle β

to the axis of said zone of influence and which is proportional to an angular attenuation coefficient δ

_A defined as follows;

6. A method according to claim 1, wherein a modelled muscle is defined by a vector having an origin A, an oriented extremity I in a frustoconical zone of influence having an apex dA, and two elliptical aperture angles, a mesh node S_j situated within said zone of influence and at a radial distance S′

_jS_j from the axis of said zone of influence in an elliptical section of radius r passing through said mesh node S_j is subjected to a displacement vector which is directed toward the node dA of said zone of influence and which is proportional to an angular attenuation coefficient δ

_eA and to a radial attenuation coefficient δ

_eR defined as follows;

δ

_eA=(r²−

∥

S′

_jS_j∥

/r²

δ

_eR=(∥

AI∥

/∥

dAI∥

)cos[((∥

dAS_J∥

−

∥

dAt′

∥

)/(∥

dAS′

_j∥

−

∥

dAt′

∥

))(π

/2)]

and δ

_e=δ

_eA=∥

AI∥

/∥

dAI∥ and

δ

_eR=1

if ∥

dAa∥

≦

∥

dAS_j∥

≦

∥

dAt∥

where t′

is the intersection of segment dAS_j with an intermediate base between a smaller base and a larger base of said zone of influence.

7. A method according to claim 1, wherein, when a mesh node is situated in a plurality of intersecting zones of influence of muscles, and further including displacing said mesh node by a displacement vector determined by adding displacement vectors resulting from the action of said muscles in order to obtain a resultant vector and then truncating said resultant vector so that said resultant vector has a modulus equal to the highest modulus of said displacement vectors of said muscles.

8. A method according to claim 1, wherein there is included a group of modelled frontal muscles respectively extending downward substantially as far as external, middle and internal extremities of eyebrows of said head and to middles of eyelids of said head, and having zones of influence intersecting at least two by two to displace mesh nodes at least in said eyebrows and said eyelids.

9. A method according to claim 1, including a step of determining parameters for animating at least one of the following organs:

an eye, an eyelid, a lower jaw, a neck in said head.

10. A method according to claim 1, including a step of determining a rotation center of an eye of said head as the middle of a segment whose extremities are two farthest apart mesh nodes in a set of meshes relating to a white of said eye.

11. A method according to claim 1, including a step of determining a rotation axis of an eyelid of said head passing through two farthest apart mesh nodes in a set of meshes relating to said eyelid.

12. A method according to claim 1, including the steps of:

determining mesh nodes as jaw pivot axis nodes, a chin node and a head—

neck transition node to define an angular articulation sector AM of jaw,

determining a predetermined angular sector α

_M less than and contained in said articulation sector AM and having a lower side common therewith, and

marking mesh nodes substantially contained within said articulation sector AM, in order for a rotation of the jaw to operate completely on mesh nodes situated within an angular sector AM—

α

_M and in an attenuated manner on mesh nodes within said angular sector α

_M so that, if angular distance α

_i between a mesh node and said common lower side decreases, said mesh node is displaced as a function of an attenuation coefficient defined as follows;

13. A method according to claim 1, including the steps of:

determining upper and lower anterior nodes and upper and lower posterior nodes ●

of a neck of said head;

marking mesh nodes substantially contained between lateral planes passing through said upper and lower nodes as belonging to said neck;

marking all the mesh nodes in said head that do not belong to said neck;

determining a center of rotation of said head consisting of the center of a volume encompassing mesh nodes belonging to said neck; and

determining minimum and maximum ordinates Y_min and Y_max of said upper and lower nodes of said neck, in order for a subsequent rotation of said neck to operate completely on mesh nodes situated above a lateral plane passing through said upper nodes and in an attenuated fashion on mesh nodes within said neck so that the closer the mesh node with the ordinate Y_si to a lateral plane passing through said lower nodes, the greater the angular displacement of said mesh node with said ordinate Y_si decreases as a function of a ratio defined as follows;

14. A client-server system, comprising a client data processing arrangement having a three-dimensional model of a head in the form of sets of meshes including a set of lips and parameters, and a processing server for converting responses read in a database addressed by requests corresponding to messages transmitted by the client into messages and animation commands transmitted to the client for thereby animating said head as a function of said animation commands in said client data processing arrangement,

said client data processing arrangement being arranged for:

(a) distinguishing a lower lip and an upper lip in said set of lips as a function of mesh nodes at the level of a lip corner and mouth opening boundary, the client data processing arrangement being arranged for distinguishing the lower and upper lip by steps including;

(i) identifying mesh nodes at the periphery of said set of lips to define an interior contour and an exterior contour of said lips,

(ii) detecting laterally outermost mesh nodes on said interior contour and exterior contour,

(iii) marking mesh nodes below one of lip corner segments defined by the nodes previously detected as belonging to said lower lip,

(iv) recursively marking all the unmarked mesh nodes, starting with nodes near a lower node of said lower lip, as belonging to said lower lip, and

(v) marking all the other unmarked nodes in said set of lips not belonging to said lower lip as belonging to said upper lip,

(b) determining an upper zone and a lower zone of said head substantially shared by at least a lip corner and mouth opening boundary between the mesh nodes belonging to said lower lip and the mesh nodes belonging to said upper lip, and

(c) providing a first group of modelled muscles extending toward said upper lip and having zones of influence intersecting at least two by two situated in said upper zone for displacing mesh nodes in said upper lip and under a nose of said head, a second group of modelled muscles extending toward said lower lip and having zones of influence intersecting at least two by two situated in said lower zone for displacing mesh nodes in said lower lip and in a chin of said head, and a last group of modelled muscles having zones of influence for displacing each of the mesh nodes in said upper zone and lower zone in order to stretch and contract said lips.

15. A client-server system according to claim 14, wherein said sets of meshes are arranged to be selected and acquired from a predetermined server.

16. A client-server system according to claim 14, wherein said parameters define at least animations of a jaw, a neck and muscles of said head.

17. A client-server system according to claim 14, wherein said animation commands relate to at least the following facial animations of said head:

turning said head, turning an eye of said head, turning an eyelid of said head, enunciating a text, enunciating a sound, choosing a neutral expression of the face of said head, portraying an emotion.

18. A client-server system comprising a database server and a client data processing arrangement having a head modelled in three dimensions in the form of sets of meshes including a set of lips and parameters for thereby animating said head as a function of animation commands,

said client data processing arrangement being arranged to provide head animation and simulated conversation, said simulated conversation being arranged for converting responses read in said database server addressed by requests of said simulated conversation corresponding to messages transmitted by said head animation into messages and animation commands transmitted to said head animation,

said head animation being arranged for:

(a) distinguishing a lower lip and an upper lip in said set of lips as a function of mesh nodes at the level of a lip corner and mouth opening boundary, the head animation being arranged for distinguishing the lower lip from upper lip by steps including;

(i) identifying mesh nodes at the periphery of said set of lips to define an interior contour and an exterior contour of said lips,

(ii) detecting laterally outermost mesh nodes on said interior contour and exterior contour,

(iii) marking mesh nodes below one of lip corner segments defined by the nodes previously detected as belonging to said lower lip,

(iv) recursively marking all the unmarked mesh nodes, starting with nodes near a lower node of said lower lip, as belonging to said lower lip, and

(v) marking all the other unmarked nodes in said set of lips not belonging to said lower lip as belonging to said upper lip,

(b) determining an upper zone and a lower zone of said head substantially shared by at least a lip corner and mouth opening boundary between the mesh nodes belonging to said lower lip and the mesh nodes belonging to said upper lip, and

(c) providing a first group of modelled muscles extending toward said upper lip and having zones of influence intersecting at least two by two situated in said upper zone for displacing mesh nodes in said upper lip and under a nose of said head, a second group of modelled muscles extending toward said lower lip and having zones of influence intersecting at least two by two situated in said lower zone for displacing mesh nodes in said lower lip and in a chin of said head, and a last group of modelled muscles having zones of influence for displacing each of the mesh nodes in said upper zone and lower zone in order to stretch and contract said lips.

19. A client-server system according to claim 18, wherein said sets of meshes are selected and acquired from a predetermined server.

20. A client-server system according to claim 18, wherein said parameters are arranged for defining at least animations of a jaw, a neck and of muscles of said head.

21. A client-server system according to claim 18, wherein said animation commands relate to at least the following facial animations of said head:

turning said head, turning an eye of said head, turning an eyelid of said head, enunciating a text, enunciating a sound, choosing a neutral expression of the face of said head, portraying an emotion.

22. A memory for a client-server system including a database server and a client data processing arrangement having a head modelled in three dimensions in the form of sets of meshes including a set of lips and parameters for thereby animating said head as a function of animation commands, the memory being arranged to cause the client data processing arrangement to provide head animation and simulated conversation, said simulated conversation being arranged for converting responses read in said database server addressed by requests of said simulated conversation corresponding to messages transmitted by said head animation into messages and animation commands transmitted to said head animation,

said head animation being arranged for:

(a) distinguishing a lower lip and an upper lip in said set of lips as a function of mesh nodes at the level of a lip corner and mouth opening boundary, the head animation being arranged for distinguishing the lower lip from upper lip by steps including;

(i) identifying mesh nodes at the periphery of said set of lips to define an interior contour and an exterior contour of said lips,

(ii) detecting laterally outermost mesh nodes on said interior contour and exterior contour,

(iii) marking mesh nodes below one of lip corner segments defined by the nodes previously detected as belonging to said lower lip,

(iv) recursively marking all the unmarked mesh nodes, starting with nodes near a lower node of said lower lip, as belonging to said lower lip, and

(v) marking all the other unmarked nodes in said set of lips not belonging to said lower lip as belonging to said upper lip,

(b) determining an upper zone and a lower zone of said head substantially shared by at least a lip corner and mount opening boundary between the mesh nodes belonging to said lower lip and the mesh nodes belonging to said upper lip, and

(c) providing a first group of modelled muscles extending toward said upper lip and having zones of influence intersecting at least two by two situated in said upper zone for displacing mesh nodes in said upper lip and under a nose of said head, a second group of modelled muscles extending toward said lower lip and having zones of influence intersecting at least two by two situated in said lower zone for displacing mesh nodes in said lower lip and in a chin of said head, and a last group of modelled muscles having zones of influence for displacing each of the mesh nodes in said upper zone and lower zone in order to stretch and contract said lips.

23. The memory of claim 22, wherein said sets of meshes are selected and acquired from a predetermined server.

24. The memory of claim 22, wherein said parameters define at least animations of a jaw, a neck and of muscles of said head.

25. The memory of claim 22, wherein said animation commands relate to at least the following facial animations of said head:

turning said head, turning an eye of said head, turning an eyelid of said head, enunciating a text, enunciating a sound, choosing a neutral expression of the face of said head, portraying an emotion.