Apparatus and method for efficient animation of believable speaking 3D characters in real time

• US 20030137515A1
• Filed: 09/06/2002
• Published: 07/24/2003
• Est. Priority Date: 01/22/2002
• Status: Active Grant

• Current Assignee: DG Holdings Inc. (DAZ Productions, Inc.)
• Original Assignee: 3DME, Inc.
• Inventors: Griffith, Todd, Cederwall, Chris, Rom, Hillel
• Granted Patent: US 7,663,628 B2
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2213/12 Rule based animation

Abstract

An apparatus for animating a moving and speaking enhanced-believability, character in real time, comprising a plurality of behavior generators, each for defining a respective aspect of facial behavior, a unifying scripter, associated with the behavior generators, the scripter operable to combine the behaviors into a unified animation script, and a renderer, associated with the unifying scripter, the renderer operable to render the character in accordance with the script, thereby to enhance believability of the character.

Claims

1. An apparatus for animating a moving and speaking enhanced-believability character in real time, comprising:

i. a plurality of behavior generators each for defining a respective aspect of facial behavior;

ii. a unifying scripter, associated with said behavior generators, said scripter operable to combine said behaviors into a unified animation script; and

iii. a renderer, associated with said unifying scripter, said renderer operable to render said character in accordance with said script, thereby to enhance believability of said character.

2. An apparatus according to claim 1 wherein said behavior generator is operable to generate said behavior over a defined time.

3. An apparatus according to claim 1 wherein said behavior is maintainable in an active behavior pool in accordance with said defined time.

4. An apparatus according to claim 2 wherein said behavior comprises a set of behavior elements.

5. An apparatus according to claim 4 wherein said behavior elements comprise at least one chosen from a list comprising:

geometric;

orientation; and

appearance elements.

6. An apparatus according to claim 4 wherein said behavior elements are operable upon a plurality of polygonal surfaces comprising a visual appearance of said character.

7. An apparatus according to claim 3 wherein one of said behavior generators is a facial speech behavior generator operable to generate facial expressions for said character based upon character speech.

8. An apparatus according to claim 7 wherein said character speech is at least one chosen from a list comprising a text source and an audio signal.

9. An apparatus according to claim 8 wherein said character speech is transformable by a phoneme recognizer, said phoneme recognizer producing pairs of respective phonemes and phoneme weights.

10. An apparatus according to claim 9 wherein said phoneme weights effect a relative emphasis of respective phonemes based upon at least one phoneme attribute.

11. An apparatus according to claim 10 wherein said phoneme attribute is chosen from a list comprising:

amplitude, pitch, and emphasis.

12. An apparatus according to claim 8 wherein text from said text source is convertible into a speech audio signal using a text to speech converter.

13. An apparatus according to claim 12 wherein said audio signal is forwarded to an audio device for synchronized playback with facial speech behavior of said character.

14. An apparatus according to claim 3 wherein one of said behavior generators is an eye gaze direction behavior generator operable to determine eye gaze direction on an animation frame-by-frame basis.

15. An apparatus according to claim 14 wherein inputs for said eye gaze direction behavior generator comprise:

a primary gaze default direction, probabilities for eye movement, and a maximum time for resetting gaze to said primary default direction.

16. An apparatus according to claim 15 wherein said eye gaze direction behavior generator further comprises a primary thresholder operable upon elapsed time since last gaze change versus said maximum time, indicating a gaze change to said primary gaze default direction if elapsed time exceeds said maximum time or indicating no gaze change if said maximum time is not exceeded.

17. An apparatus according to claim 16 wherein said eye gaze direction behavior generator further comprises a secondary random thresholder operable in association with said primary thresholder, whereby a random threshold is generated based on said probabilities for eye movement and whereby elapsed time since said last gaze change against said random threshold is compared, said secondary random thresholder thresholding a gaze change by a specific amount in a random direction if elapsed time exceeds said random threshold or no gaze change if said random threshold is not exceeded.

18. An apparatus according to claim 17 further comprising a saccade accumulator to simulate a gaze saccade operable in association with said amount of gaze change, said accumulator accumulating said amount of gaze change with an accumulated gaze change amount.

19. An apparatus according to claim 18 further comprising a saccade thresholder operable upon said accumulated gaze change amount to determine an actual eye movement equal to said accumulated gaze change amount and to subsequently reset said accumulated gaze change amount to zero when a saccade threshold value is exceeded, and to determine an actual eye movement equal to zero when said saccade threshold value is not exceeded.

20. An apparatus according to claim 19 wherein said saccade threshold value is operable upon a maximum up-down and a maximum left-right actual eye movement.

21. An apparatus according to claim 20 wherein said maximum up-down and said maximum left-right actual eye movement values are 3 degrees and 7 degrees, respectively.

22. An apparatus according to claim 15 wherein a mood and a personality of said character are useable to set said probabilities.

23. An apparatus according to claim 3 wherein one of said behavior generators is an eye blink behavior generator, operable to effect eye blink behavior.

24. An apparatus according to claim 23 wherein inputs for eye blink behavior generator comprise:

an inherent blink rate probability, determinable in association with a personality of said character, speech behavior, from which a speech-related blink probability is generatable;

mood, from which a mood-related blink probability is generatable; and

eye movement, from which an eye movement related blink probability is generatable.

25. An apparatus according to claim 24 wherein said eye blink behavior generator further comprises a summer, operable to sum respectively generated blink probabilities to yield a summed blink probability larger than zero and smaller than one.

26. An apparatus according to claim 25 wherein said eye blink behavior generator further comprises a thresholder, operable in association with said summer, to indicate an occurrence of a blink.

27. An apparatus according to claim 26 wherein said thresholder comprises:

i. a random number generator wherein a random number larger than zero and smaller than one is generated;

ii. a random thresholder for comparing said summed blink probability against said random number;

a blink being indicated if said summed blink probability is larger than said random number, and no blink being indicated if said summed blink probability is smaller than said random number; and

iii. an elapsed time thresholder operable in association with said random thresholder indicating a blink status, so that elapsed time since a last blink is compared against a blink threshold time, said elapsed time thresholder indicating a blink if said elapsed time is larger than said blink threshold, or no blink indicated if elapsed time is smaller than said blink threshold.

28. An apparatus according to claim 3 wherein one of said behavior generators is a mood behavior generator comprised of:

i. inputs chosen from a list comprising;

direct user input, text analysis, and audio analysis;

ii. a mood selector for using said inputs in the selection of a mood; and

iii. a behavior element producer for generation of at least one behavior element for defining behavior in accordance with said selected mood.

29. An apparatus according to claim 28 wherein said direct user input is effectible by means of a graphical input device wherein a plurality of mood selectors are arrayed on a circumference of a circular dial with an adjustable radial hand, said adjustable radial hand being user operable to indicate at least one mood selector and a respective relative mood intensity.

30. Apparatus according to claim 29 wherein said adjustable radial hand is user pointable to a desired mood selector.

31. Apparatus according to claim 29 wherein said adjustable radial hand is user pointable between two adjacent mood selectors, thereby indicating a relative combination of both moods, based on the relative positioning of said adjustable radial hand between said two adjacent moods.

32. Apparatus according to claim 29 wherein the length of said adjustable radial hand is indicative of said relative mood intensity.

33. An apparatus according to claim 3 wherein one of said behavior generators is a head orientation behavior generator operable to effect head orientation behavior on an animation frame-by-frame basis.

34. An apparatus according to claim 33 wherein said head orientation generator is operable to select from user input and other behavior generator output to use in said effecting of head orientation behavior.

35. An apparatus according to claim 33 wherein said head orientation is chosen from a list comprising head turn, nod, and tilt.

36. An apparatus according to claim 2 wherein said unifying scripter is operable to order said behaviors according to respective behavior elements and respective defined times, thereby creating said unified animation script.

37. An apparatus according to claim 2 wherein a behavior weighter is operable to weigh respective behavior elements using respective weights, to vary respective behavior element influence upon said character.

38. An apparatus according to claim 37 wherein said respective weight is defined as:

Weight=V₀+f((T−

T₀)/D)*(V₁−

V₀) where;

T=a current time;

T₀=start time, relative to other behavior elements in said set of behavior elements;

D=time duration of said behavior element;

V₁=weight value when said function evaluates to 1;

V₀=weight value when the function evaluates to 0; and

f is a function returning a value ranging from 1 to 0.

39. Apparatus according to claim 38 wherein said function is chosen from a list of functions comprising:

trigonometric, wedge, and linear.

40. An apparatus according to claim 6 wherein said geometric behavior element comprises changes of position of a plurality of vertices of said polygonal surfaces.

41. An apparatus according to claim 6 wherein said orientation behavior element comprises changes to said character resulting from transformations to vertices of said polygonal surfaces.

42. An apparatus according to claim 41 wherein said transformations comprise a rigid transformation applied to a plurality of vertices of said polygonal surfaces of at least one part of said character.

43. An apparatus according to claim 41 wherein said transformations comprise tapered transformations operable upon a plurality of points of said character in accordance with respective stickiness factors.

44. An apparatus according to claim 43 wherein respective stickiness factors are defined as:

s_ν

=1−

k_ν

where;

s_ν

is said stickiness factor for a rotation of a given vertex within a body of vertices; and

k_ν

is an angular coefficient for said vertex, k_ν

further being defined in;

θ

<sub>ν

,α</sub>

=k_ν

α

where;

a is a rotation of said body of vertices; and

θ

<sub>ν

,α</sub>

is a resultant rotation at said vertex.

45. An apparatus according to claim 44 further comprising a graphics library rotation mechanism to effect a sticky rotation, resembling a standard rigid rotation, said mechanism comprising:

i. a sorter for sorting said vertices according to respective s_ν

values;

ii. a rotation definer for defining a local rotation, based upon said rotation angle a around an axis W, using library commands;

iii. a rotation matrix computer, in association with said sorter and said rotation definer, operable to compute a rigid rotation matrix with angle −

s_ν

a for said vertices when s_ν

is not equal to zero.

iv. a vertex multiplier, in association with said thresholder, for multiplying respective vertices with a non-zero s_ν

value by said rigid rotation matrix, thereby yielding a negative rotation over said angle a.

46. An apparatus according to claim 5 wherein said appearance behavior element comprises at least one appearance aspect chosen from a list comprised of:

color;

texture; and

lighting.

47. An apparatus according to claim 46 wherein a material is associated with a respective appearance aspect.

48. A method for animating a moving and speaking enhanced-believability character in real time, comprising the steps of:

i. defining respective aspects of facial behavior in order to generate respective facial behaviors;

ii. unifying said behaviors into a unified animation script; and

iii. rendering said character in accordance with said script, thereby to enhance believability of said character.

49. A method according to claim 48 wherein said generation of respective behaviors is performed substantially off line.

50. A method according to claim 49 wherein said generation of respective behaviors is for a respective defined time.

51. A method according to claim 49 wherein said generation of respective behaviors includes generation of facial speech behavior, whereby facial expressions for said character are generated, based upon character speech, comprising the steps of:

i. using a text source or audio signal as said character speech; and

ii. transforming said character speech into phonemes.

52. A method according to claim 51 wherein text from said text source is converted into a speech audio signal.

53. A method according to claim 52 wherein said audio signal is forwarded for synchronized playback with said facial speech behavior.

54. A method according to claim 49 wherein said generation of respective behaviors includes generation of eye gaze direction behavior to determine eye gaze direction on an animation frame-by-frame basis, comprises the steps of:

i. thresholding elapsed time since last gaze change versus an inputted maximum time;

ii. indicating a gaze change to an inputted primary gaze default direction if elapsed time exceeds said maximum time;

or, if no gaze change is indicated;

iii. thresholding elapsed time since last gaze change versus a randomly generated threshold value, based upon probabilities for eye movement;

iv. indicating no gaze change if said maximum time is not exceeded; and

v. indicating a gaze change by a small amount in a random direction if said maximum time is exceeded.

55. A method according to claim 54 further comprising the steps of:

i. summing said gaze change amount to an accumulated gaze change value;

ii. thresholding said accumulated gaze change thereby simulating a gaze saccade effect, so that an actual gaze change is set equal to said accumulated gaze change and said accumulated gaze change is subsequently set to zero if said accumulated gaze change exceeds a saccade threshold value; and

iii. setting an actual gaze change to zero if said accumulated gaze change does not exceed said saccade threshold value.

56. A method according to claim 55 wherein said saccade threshold is definable for an up-down and a left-right direction gaze change.

57. A method according claim 54 wherein a mood and a personality of said character are used to set said probabilities.

58. A method according to claim 49 wherein said generation of respective behaviors includes generation of eye blink behavior comprising the steps of:

i. generating a plurality of blink probabilities;

ii. summing said blink probabilities to yield a value between zero and one;

iii. thresholding said summed blink probability against a randomly generated number between zero and one;

iv. indicating a blink if said summed blink probability is larger than said number;

or, if no blink is indicated;

v. thresholding an elapsed time since last blink against a blink threshold time;

vi. indicating a blink if said elapsed time is larger than said blink threshold time.

59. A method according to claim 49 wherein said generation of respective behaviors includes generation of mood behavior comprising using inputs to select said a mood and said mood intensity.

60. A method according to claim 49 wherein said generation of respective behaviors includes generation of head orientation behavior on an animation frame-by-frame basis, comprising selecting from user input and other behavior generation to effect head orientation behavior.