System and method of providing conversational visual prosody for talking heads

• US 7,076,430 B1
• Filed: 06/17/2002
• Issued: 07/11/2006
• Est. Priority Date: 05/16/2002
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Isaacson, Thomas M., Strom, Volker Franz, Cosatto, Eric, Graf, Hans Peter
• Primary Examiner(s): HARPER, V PAUL
• Application Number:
  US10/173,341
• Time in Patent Office: 1,485 Days
• Field of Search: 704/271, 704/272, 704/270.1, 704/275, 704/276
• US Class Current: 704/275
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

A system and method of controlling the movement of a virtual agent while the agent is listening to a human user during a conversation is disclosed. The method comprises receiving speech data from the user, performing a prosodic analysis of the speech data and controlling the virtual agent movement according to the prosodic analysis.

Claims

1. A method of controlling a virtual agent movement when listening to a user, the method comprising:

receiving speech data from the user;

performing a prosodic analysis of the speech data by selecting matching prosody patterns from a listening database; and

controlling the virtual agent movement according to the prosodic analysis.

2. The method of claim 1, wherein the prosody analysis further comprises using a model to determine what virtual agent movement to associate with the speech data.

3. The method of claim 1, wherein the prosody analysis further comprises analyzing speech intonations comprising pauses.

4. The method of claim 3, wherein the prosody analysis includes analyzing speech intonations comprising pitch.

5. The method of claim 4, wherein the prosody analysis includes analyzing speech intonations comprising timing effects.

6. The method of claim 5, wherein the prosody analysis includes analyzing speech intonations comprising loudness.

7. The method of claim 6, wherein the prosody analysis includes analyzing speech intonations comprising accent.

8. The method of claim 1, wherein the movement of the virtual agent is controlled by to approximately simultaneous with the received speech data that triggers the movement.

9. The method of claim 1, wherein the prosody analysis includes analyzing speech intonations comprising pauses.

10. A method of controlling movement of a virtual animated entity while the animated entity is listening to a user, the method comprising:

performing a prosodic analysis of speech data received from the user by selecting matching prosody patterns form a listening database for use in controlling the movement of the animated entity; and

controlling movement of the animated entity according to the prosody analysis.

11. The method of claim 10, further comprising:

using a model to associate the movement of the animated entity based on prosodic of the speech data.

12. The method of claim 10, wherein the movement of the virtual agent is controlled to be approximately simultaneous with the received speech data that triggers the movement.

13. The method of claim 10, wherein the prosody analysis includes analyzing speech intonations comprising pauses.

14. The method of claim 13, wherein the prosody analysis includes analyzing speech intonations comprising pitch.

15. The method of claim 14, wherein the prosody analysis includes analyzing speech intonations comprising timing effects.

16. The method of claim 15, wherein the prosody analysis includes analyzing speech intonations comprising loudness.

17. The method of claim 16, wherein the prosody analysis includes analyzing speech intonations comprising accent.

18. The method of claim 17, wherein the prosody analysis includes analyzing speech intonations comprising pauses.

19. An apparatus for controlling movement of a virtual agent while listening to a user, the apparatus comprising:

a listening module that receives speech input from the user and performs a prosodic analysis on the speech input, the prosodic analysis including selecting matching prosody patterns from a listening database;

a selection module that selects virtual agent control data from a listening database; and

a rendering module for controlling the movement of the virtual agent while it is listening to the user.

20. The apparatus of claim 19, wherein the movements of the virtual agent are approximately simultaneous with the received speech input from the user.

21. The method of claim 19, wherein the movement of the virtual agent is controlled by to approximately simultaneous with the received speech data that triggers the movement.

22. The method of claim 19, wherein the prosody analysis includes analyzing speech intonations comprising pauses.

23. The method of claim 19, further comprising:

at the client device, transmitting speech data to a server for speech processing in order to generate a response to the user.

24. The method of claim 23, further comprising:

at the client device, after the user finishes a speech segment, receiving from the server response speech and response virtual agent movements for generating a virtual agent response to the user.

25. The method of claim 24, wherein the movement of the virtual agent is controlled to be approximately simultaneous with the received speech data that triggers the movement.

26. A method of controlling a virtual agent movement when listening to a user, the method comprising:

at a client device, performing a prosodic analysis of user speech data, the prosodic analysis including selecting matching prosody patterns from a listening database; and

controlling the virtual agent movement according to the prosodic analysis while receiving the user speech data.

27. The method of claim 26, wherein the prosody analysis includes analyzing speech intonations comprising pauses.

28. The method of claim 27, wherein the prosody analysis includes analyzing speech intonations comprising pitch.

29. The method of claim 28, wherein the prosody analysis includes analyzing speech intonations comprising timing effects.

30. The method of claim 29, wherein the prosody analysis includes analyzing speech intonations comprising loudness.

31. The method of claim 30, wherein the prosody analysis includes analyzing speech intonations comprising accent.

32. A method of controlling a virtual agent movement when listening to a user, the method comprising, at a client device:

receiving speech data from the user;

transmitting the speech data over a network to a server for generating virtual agent movement data based on a prosodic analysis of the speech data, the prosodic analysis further comprising selecting matching prosody patterns from a listening database; and

receiving the virtual agent movement data from the server for controlling the virtual agent movement while receiving speech data from the user.

33. The method of claim 32, wherein the network is the Internet.

34. The method of claim 32, wherein the network is a packet network.

35. The method of claim 32, wherein the network is a wireless network.

36. The method of claim 32, wherein the network is an Internet Protocol network.

37. A method of controlling virtual agent movement on a client device while listening to a user, the method comprising, at a server,

receiving user speech data from the client device over a network;

generating virtual agent movement data based on a prosodic analysis of the user speech data, the prosodic analysis further comprising selecting matching prosody patterns from a listening database; and

transmitting the virtual agent movement data to the client device over the network for controlling the virtual agent movement while listening to the user.

38. The method of claim 37, wherein the network is the Internet.

39. The method of claim 37, wherein the network is a packet network.

40. The method of claim 37, wherein the network is a wireless network.

41. The method of claim 37, wherein the network is an Internet Protocol network.

42. A computer-readable medium storing instructions for controlling a computer device to perform a method of controlling a virtual agent movement while listening to a user, the method comprising;

receiving speech data from the user;

performing a prosodic analysis of the speech data;

selecting matching prosody patterns from a listening database; and

controlling the virtual agent movement according to the selected prosody patterns.