Training system and methods for dynamically injecting expression information into an animated facial mesh

• US 8,581,911 B2
• Filed: 10/02/2009
• Issued: 11/12/2013
• Est. Priority Date: 12/04/2008
• Status: Expired due to Fees

• Current Assignee: Cubic Corporation
• Original Assignee: Intific, Inc. (Cubic Corporation)
• Inventors: Becker, Michael J., Copenhagen, Keith, Taylor, Murray
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US12/573,003
• Publication Number:
  US 20100141663A1
• Time in Patent Office: 1,502 Days
• Field of Search: None
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites

Abstract

A system and method for modifying facial animations to include expression and microexpression information is disclosed. Particularly, a system and method for applying actor-generated expression data to a facial animation, either in realtime or in storage is disclosed. Present embodiments may also be incorporated into a larger training program, designed to train users to recognize various expressions and microexpressions.

Claims

1. A system for training a trainee to recognize facial expressions from animations of computer-simulated faces, the system comprising:

a trainee interface configured to display a facial animation;

a non-transitory machine-readable repository of facial expression overlays each corresponding to an emotion;

a non-transitory computer-readable storage medium encoded with a computer program with instructions for combining one of the expression overlays into a facial animation to form a combined animation, by mapping at least a portion of the expression overlay to at least a portion of the facial animation, the computer program further including instructions to provide the combined animation for display in the trainee interface;

a non-transitory computer-readable storage medium encoded with a computer program with instructions for executing a trainee evaluation module for evaluating a trainee'"'"'s performance in correctly recognizing displayed facial expressions in the combined animation, the trainee evaluation module configured to receive from the trainee a response to the combined animation displayed in the trainee interface; and

a computer system operative to execute at least one of the trainee interface, the computer program in the computer-readable storage medium, and the trainee evaluation module, the computer system also operative to access the repository.

2. The system of claim 1, wherein the trainee evaluation module is configured to use the received response to determine a performance level indicative of an extent to which the trainee has correctly interpreted the one or more combined animations.

3. The system of claim 2, wherein the trainee evaluation model is configured to report the performance level by displaying it in the trainee interface or in another interface.

4. The system of claim 1, wherein the computer program further includes instructions to:

provide a selection of emotions for display in a trainer interface;

receive a trainer'"'"'s selection of one of the emotions after the emotions are displayed in the trainer interface;

obtain, from the repository, a facial expression overlay corresponding to the emotion selection; and

form said combined animation by combining at least a portion of the obtained expression overlay into at least a portion of the facial animation.

5. The system of claim 4, wherein the computer program further includes instructions to receive an intensity level, and also to adjust, within the combined animation, an intensity of a facial expression emotion corresponding to the trainer'"'"'s emotion selection.

6. The system of claim 4, wherein the computer program further includes instructions to:

receive from the trainer selections of a plurality of the emotions after the emotions are displayed in the trainer interface;

obtain, from the repository, a plurality of facial expression overlays each corresponding to one of the emotion selections; and

form said combined animation by combining at least portions of the obtained overlays together into the facial animation.

7. The system of claim 1, wherein the facial expression overlays comprise facial microexpression overlays each corresponding to an emotion and each being less than 0.5 seconds in duration, wherein a microexpression is a minute involuntary physical change on a face.

46. The method of claim 1, wherein the non-transitory computer-readable storage medium encoded with the computer program with instructions for combining is the same computer-readable storage medium as the non-transitory computer-readable storage medium encoded with the computer program with instructions for executing the trainee evaluation module.

47. The method of claim 1, wherein the non-transitory computer-readable storage medium encoded with the computer program with instructions for combining is a different computer-readable storage medium as the non-transitory computer-readable storage medium encoded with the computer program with instructions for executing the trainee evaluation module.

48. The method of claim 7, wherein the microexpression has a duration of less than 0.25 seconds.

8. A method for training a trainee to recognize facial expressions from animations of computer-simulated faces, the method comprising:

accessing at least a portion of a facial expression overlay from a non-transitory machine-readable repository of facial expression overlays each corresponding to an emotion;

combining the accessed expression overlay into a facial animation to form a combined animation, by mapping at least a portion of the accessed expression overlay to at least a portion of the facial animation;

providing the combined animation for display in a trainee interface;

receiving a trainee'"'"'s response to the combined animation shown in the trainee interface; and

using the received response to determine a performance level indicative of an extent to which the trainee has correctly interpreted the combined animation;

wherein at least a portion of the method is performed by one or more computing devices.

9. The method of claim 8, further comprising reporting the performance level by displaying it in the trainee interface or in another interface.

10. A non-transitory computer-readable storage medium having computer-executable modules, for training a trainee to recognize facial expressions from computer-displayed facial animations, comprising:

an expression injection module operative to;

access at least a portion of a facial expression overlay from a non-transitory machine-readable repository of facial expression overlays each corresponding to an emotion;

combine the accessed expression overlay into a facial animation to form a combined animation, by mapping at least a portion of the accessed expression overlay to a digital mesh of the facial animation; and

provide the combined animation for display in a trainee interface configured to display facial animations of an avatar; and

a trainee evaluation module operative to receive a trainee'"'"'s response to the combined animation displayed in the trainee interface.

11. The computer-readable storage medium of claim 10, wherein the trainee evaluation module is further operative to:

use the received response to determine a performance level indicative of an extent to which the trainee has correctly interpreted the combined animation; and

report the performance level by displaying it in the trainee interface or in another interface.

12. A system for combining at least a portion of a microexpression into a facial animation, wherein a microexpression is a minute involuntary physical change on a face, the system comprising:

a non-transitory machine-readable repository of facial microexpression overlays each corresponding to a facial microexpression;

a non-transitory computer-readable medium encoded with a computer program with instructions to combine at least a portion of one of the microexpression overlays into a facial animation; and

a computer system operative to execute the computer-readable medium and access the repository.

13. The system of claim 12, further comprising a user interface configured to receive selections of a plurality of microexpression overlays, the computer program further including instructions to:

form the combined animation by mapping the plurality of microexpression overlays together to a digital mesh of the facial animation, so that facial microexpressions associated with the plurality of microexpression overlays appear simultaneously in the combined animation, and

provide the combined animation for display after the user interface receives the selections of the plurality of microexpression overlays, the computer system being operative to execute the user interface.

14. The system of claim 12, further comprising a user interface configured to present a set of emotions and to receive a selection of one of the emotions by a user, wherein:

each of the facial microexpression overlays in the repository corresponds to one of the emotions;

the computer program further including instructions to;

form the combined animation by combining at least a portion of one of the microexpression overlays corresponding to the user-selected emotion into the facial animation, and by mapping the microexpression overlay to a digital mesh of the facial animation,

and provide the combined animation for display after the user interface receives the user-selected emotion; and

the computer system is operative to execute the user interface.

15. The system of claim 14, wherein the user interface is configured to receive selections of first and second ones of the emotions from the user, the repository containing first and second microexpression overlays respectively corresponding to the first and second emotions, the computer program further including instructions to:

form the combined animation by mapping the first and second microexpression overlays together to the digital mesh of the facial animation, so that facial microexpressions associated with the first and second microexpression overlays appear simultaneously in the combined animation, and

provide the combined animation for display after the user interface receives the selected first and second emotions.

16. The system of claim 12, further comprising a user interface configured to receive an intensity level from a user, the computer program further including instructions to adjust an intensity of the combined microexpression in the combined animation in accordance with the received intensity level, the computer system being operative to execute the user interface.

17. The system of claim 12, wherein the computer program further including instructions to limit an intensity of at least a portion of the combined microexpression in the combined animation based on speech data associated with the facial animation.

18. The system of claim 12, wherein each of the facial microexpression overlays is no longer than 0.5 seconds in duration.

19. The system of claim 12, wherein each of the microexpression overlays is formed by subtracting a digital image of a non-expressive facial pose of an actor from an animation of the actor'"'"'s performance of a facial microexpression.

20. The system of claim 12, wherein the facial animation and each of the microexpression overlays comprise vertex displacements over time, the computer program further including instructions to combine a microexpression overlay with the facial animation by adding together vertex displacements of the overlay with vertex displacements of the animation.

21. The system of claim 20, wherein the computer program further including instructions to display an animated texture bump map on a digital mesh of the combined animation.

22. The system of claim 21, wherein the animated texture bump map displays a portion of a facial microexpression represented by the microexpression overlay.

23. The system of claim 22, wherein the texture bump map is obtained from an actor'"'"'s performance of the facial microexpression.

49. The method of claim 12, wherein the microexpression has a duration of less than 0.25 seconds.

24. A method for combining at least a portion of a microexpression into a facial animation, wherein a microexpression is a minute involuntary physical change on a face, the method comprising:

combining a facial microexpression overlay into a facial animation to form a combined animation, by mapping at least a portion of the microexpression overlay to at least a portion of a digital mesh of the facial animation; and

providing the combined animation for display;

wherein the method is performed by one or more computing devices.

25. The method of claim 24, further comprising, prior to said combining:

displaying a set of emotions; and

receiving a selection of at least one of the emotions by a user;

wherein combining the facial microexpression overlay comprises combining an overlay that corresponds to the user selected emotion.

26. The method of claim 25, further comprising selecting the facial microexpression overlay from a non-transitory machine-readable repository of facial microexpression overlays and emotions associated with the microexpression overlays, each microexpression overlay in the repository being no longer than 0.5 seconds in duration, wherein selecting the microexpression overlay comprises:

finding an emotion in the repository that matches the emotion selection received from the user; and

finding, in the repository, a microexpression overlay that is associated with said emotion that matches the emotion selection received from the user.

27. The method of claim 25, wherein said providing the combined animation for display occurs after said receiving the user-selected emotion.

50. The method of claim 24, wherein the microexpression has a duration of less than 0.25 seconds.

28. A method for receiving a facial animation with a combined microexpression, wherein a microexpression is a minute involuntary physical change on a face, the method comprising:

sending an emotion identifier to a non-transitory computer-readable medium storage medium encoded with a computer program configured to instruct a computer device to access at least a portion of a facial expression overlay from a non-transitory machine-readable repository of facial expression overlays each corresponding to an emotion, and to combine the accessed expression overlay into a facial animation to form a combined animation; and

receiving a combined animation from the computing device, the combined animation comprising a combination of at least a portion of a facial microexpression overlay corresponding to the emotion identifier with at least a portion of a facial animation, in which the microexpression overlay is mapped to a digital mesh of the facial animation;

wherein the method is performed by one or more computing devices.

29. The method of claim 28, further comprising, prior to said sending:

receiving a set of emotions; and

selecting one of the emotions;

wherein the emotion identifier corresponds to the selected emotion.

30. The method of claim 28, wherein said receiving the combined animation occurs after said sending.

31. The method of claim 28, wherein the facial microexpression overlay is no longer than ⅕

of a second in duration.

51. The method of claim 28, wherein the microexpression has a duration of less than 0.25 seconds.

32. A non-transitory computer-readable storage medium storing a computer-readable program code adapted to be executed by hardware to perform a method comprising:

accessing a facial microexpression overlay from a non-transitory machine-readable repository of facial microexpression overlays each corresponding to a facial microexpression wherein a microexpression is a minute involuntary physical change on a face;

combining the accessed microexpression overlay into a facial animation to form a combined animation, by mapping at least a portion of the accessed microexpression overlay to at least a portion of the facial animation; and

providing the combined animation for display in an interface.

52. The method of claim 32, wherein the microexpression has a duration of less than 0.25 seconds.

33. A non-transitory computer-readable storage medium having computer-executable modules, executed by hardware, for combining a facial microexpression into a facial animation, wherein a microexpression is a minute involuntary physical change on a face, the storage medium comprising:

a user interface operative to;

present a set of emotions; and

receive a selection of one of the emotions by a user; and

an expression injection module operative to;

select a facial microexpression overlay corresponding to the user-selected emotion from a non-transitory machine-readable repository of facial microexpression overlays each corresponding to an emotion;

combine the selected microexpression overlay into a facial animation to form a combined animation, by mapping at least a portion of the microexpression overlay to at least a portion of a digital mesh of the facial animation; and

provide the combined animation for display after the user interface receives the user-selected emotion.

53. The method of claim 33, wherein the microexpression has a duration of less than 0.25 seconds.

34. A method of inserting microexpression information into a facial animation, comprising:

converting an actor'"'"'s performance of a facial microexpression into a digital animation;

subtracting a digital image of a non-expressive facial pose of the actor from the animation to create an overlay of the microexpression; and

mapping the overlay to a digital mesh of a facial animation sequence;

wherein the method is performed by one or more computing devices.

35. The insertion method of claim 34, wherein the digital animation and the overlay each comprise temporal displacement of vertices on the mesh.

36. The insertion method of claim 35, wherein said mapping comprises adding together said vertex displacements of the digital animation and the overlay.

37. The insertion method of claim 34, further comprising applying an animated texture bump map on the surface of the mesh.

38. The insertion method of claim 37, wherein the texture bump map is extracted from the actor'"'"'s performance of the facial microexpression.

39. The insertion method of claim 37, wherein the texture bump map represents at least dermal wrinkles.

40. The insertion method of claim 34, wherein said mapping comprises using a spring-based, dermal elasticity model for the mesh.

41. A method comprising:

displaying an avatar in a virtual environment accessible to a plurality of users, said avatar corresponding to a first user;

receiving an instruction from the first user, for controlling a facial animation of the avatar in the virtual environment;

determining an emotion of the avatar associated with the instruction;

accessing a facial microexpression overlay corresponding to the determined emotion, wherein a microexpression is a minute involuntary physical change on a face; and

combining the microexpression overlay into the facial animation of the avatar to form a combined animation, by mapping the microexpression overlay to the facial animation;

wherein at least a portion of the method is performed by one or more computing devices.

42. The method of claim 41, wherein determining the emotion comprises determining truth or falsity of the instruction.

43. The method of claim 41, wherein determining the emotion comprises comparing the instruction to a profile of the first user.

44. The method of claim 41, wherein the instruction comprises voice audio information.

45. The method of claim 41, wherein a degree of combining of the microexpression overlay is determined by emergent behavior arising from the virtual environment and/or the user.

54. The method of claim 41, wherein the microexpression has a duration of less than 0.25 seconds.