System and method of providing conversational visual prosody for talking heads

• US 7,136,818 B1
• Filed: 06/17/2002
• Issued: 11/14/2006
• Est. Priority Date: 05/16/2002
• Status: Active Grant

• Current Assignee: Interactions, LLC
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Graf, Hans Peter, Cosatto, Eric, Strom, Volker Franz
• Primary Examiner(s): {hacek over (S)}mits, Talivaldis Ivars
• Assistant Examiner(s): SAINT CYR, LEONARD
• Application Number:
  US10/173,184
• Time in Patent Office: 1,611 Days
• Field of Search: 704/270.1, 704/275, 704/276, 348/14.01, 358/2.1, 382/100
• US Class Current: 704/275
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A system and method of controlling the movement of a virtual agent while the agent is speaking to a human user during a conversation is disclosed. The method comprises receiving speech data to be spoken by the virtual agent, performing a prosodic analysis of the speech data, selecting matching prosody patterns from a speaking database and controlling the virtual agent movement according to the selected prosody patterns.

Claims

1. A method of controlling a virtual agent movement when speaking to a user, the method comprising:

receiving speech data to be spoken by the virtual agent to the user;

performing a prosodic analysis of the speech data; and

controlling the virtual agent movement according to the prosodic analysis.

2. The method of claim 1, wherein the prosodic analysis further comprises selecting matching prosodic patterns from the speaking database.

3. The method of claim 1, wherein the prosodic analysis further comprises using a model to determine what virtual agent movement to associate with the speech data.

4. The method of claim 1, wherein the prosody analysis includes analyzing speech intonations comprising pauses.

5. The method of claim 1, wherein the prosody analysis includes analyzing speech intonations comprising pitch.

6. The method of claim 1, wherein the prosody analysis includes analyzing speech intonations comprising timing effects.

7. The method of claim 1, wherein the prosody analysis includes analyzing speech intonations comprising loudness.

8. The method of claim 1, wherein the prosody analysis includes analyzing speech intonations comprising accent.

9. The method of claim 1, wherein the movement of the virtual agent is controlled to be approximately simultaneous with the received speech data that triggers the movement.

10. The method of claim 1, wherein the virtual agent movement is at least one of:

raising a head of the virtual agent, raising at least one eyebrow of the virtual agent, nodding the head of the virtual agent.

11. The method of claim 1, further comprising:

determining a culture associated with the user; and

modifying the virtual agent movement according to the determined culture of the user.

12. The method of claim 1, further comprising:

performing a syntactic analysis of the speech data; and

wherein controlling the virtual agent movement further comprises controlling the virtual agent movement according to the prosodic analysis and the syntactic analysis.

13. A method of controlling a virtual agent movement when speaking to a user, the method comprising:

receiving speech data to be spoken by the virtual agent to the user;

selecting segments of matching visual prosody patterns from an audio-visual database of recorded speech, where both audio and video are recorded of a person speaking text; and

controlling the virtual agent movements according to the movements of the person in selected audio-visual recorded speech segments.

14. A method of controlling movement of a animated entity while the animated entity is speaking to a user, the method comprising:

performing a prosodic analysis of speech data to be spoken by the animated entity; and

controlling movement of the animated entity according to the prosody analysis.

15. The method of claim 14, further comprising:

selecting matching prosody patterns from a speaking database for use in controlling the movement of the animated entity.

16. The method of claim 14, wherein the movement of the animated entity is controlled to be approximately simultaneous with the received speech data that triggers the movement.

17. The method of claim 14, wherein the prosody analysis includes analyzing speech intonations comprising pauses.

18. The method of claim 14, wherein the prosody analysis includes analyzing speech intonations comprising pitch.

19. The method of claim 14, wherein the prosody analysis includes analyzing speech intonations comprising timing effects.

20. The method of claim 14, wherein the prosody analysis includes analyzing speech intonations comprising loudness.

21. The method of claim 14, wherein the prosody analysis includes analyzing speech intonations comprising accent.

22. An apparatus for controlling movement of a virtual agent while speaking to a user, the apparatus comprising:

a speaking module that receives speech to be spoken by the virtual agent and performs a prosodic analysis on the speech input;

a selection module that selects virtual agent control data from a speaking database; and

a rendering module for controlling the movement of the virtual agent while it is speaking to the user according to the prosodic analysis.

23. The apparatus of claim 22, wherein the movements of the virtual agent are approximately simultaneous with the received speech.

24. A method of controlling a virtual agent movement while speaking to a user, the method comprising:

at a client device, performing a prosodic analysis of speech data to be spoken by the virtual agent; and

controlling the virtual agent movement according to the prosodic analysis while the virtual agent speaks to the user.

25. The method of claim 24, wherein the prosody analysis includes analyzing speech intonations comprising pauses.

26. The method of claim 24, wherein the prosody analysis includes analyzing speech intonations comprising pitch.

27. The method of claim 24, wherein the prosody analysis includes analyzing speech intonations comprising timing effects.

28. The method of claim 24, wherein the prosody analysis includes analyzing speech intonations comprising loudness.

29. The method of claim 24, wherein the prosody analysis includes analyzing speech intonations comprising accent.

30. The method of claim 24, wherein the movement of the virtual agent is controlled to be approximately simultaneous with the received speech data that triggers the movement.

31. The method of claim 24, further comprising:

at the client device, after the virtual agent finishes speaking to the user, receiving speech data from the user and transmitting the user speech data to a server for speech processing in order to generate a response by the virtual agent to the user.

32. The method of claim 31, further comprising:

at the client device, after the user finishes a speech segment, receiving response speech from the server for generating a virtual agent response to the user.

33. The method of claim 32, wherein the movement of the virtual agent is controlled to be approximately simultaneous with user speech data or virtual agent speech data that triggers the movement.

34. A method of controlling a virtual agent movement when speaking to a user, the method comprising, at a client device:

receiving speech data to be spoken by the virtual agent;

receiving the speech data over a network from a server for generating virtual agent movement data based on a prosodic analysis of the speech data; and

receiving the virtual agent movement data from the server for controlling the virtual agent movement while the virtual agent speaks to the user.

35. The method of claim 34, wherein the network is the Internet.

36. The method of claim 34, wherein the network is a packet network.

37. The method of claim 34, wherein the network is a wireless network.

38. The method of claim 34, wherein the network is an Internet Protocol network.

39. A method of controlling virtual agent movement on a client device while speaking to a user, the method comprising, at a server:

transmitting speech data to be spoken by the virtual agent to the client device over a network;

generating virtual agent movement data based on a prosodic analysis of the user speech data; and

transmitting the virtual agent movement data to the client device over the network for controlling the virtual agent movement while speaking to the user.

40. The method of claim 39, wherein the network is the Internet.

41. The method of claim 39, wherein the network is a packet network.

42. The method of claim 39, wherein the network is a wireless network.

43. The method of claim 39, wherein the network is an Internet Protocol network.

44. A method of controlling movements of a virtual animated entity during a transition from speaking to listening, the method comprising:

as the virtual animated entity is concluding a speaking segment, selecting transition movement data from a transition movement database; and

controlling the movement of the virtual animated entity between the time the animated entity has approximately finished speaking until the animated entity stops speaking according to the transition movement data.

45. A method of controlling movements of a virtual animated entity during a transition from talking to listening, the method comprising:

approximately at the end of the virtual animated entity talking, selecting transition movement data from a transition database; and

controlling the movement of the virtual animated entity to indicate that the virtual animated entity is approximately finished talking and will soon listen for speech data from a user.