System for low-latency animation of talking heads

• US 20040215460A1
• Filed: 04/25/2003
• Published: 10/28/2004
• Est. Priority Date: 04/25/2003
• Status: Active Grant

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Graf, Hans Peter, Cosatto, Eric, Ostermann, Joern
• Granted Patent: US 7,260,539 B2
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G06F 16/9577 Optimising the visualization of content, e.g. distillation of HTML documents

Abstract

Methods and apparatus for rendering a talking head on a client device are disclosed. The client device has a client cache capable of storing audio/visual data associated with rendering the talking head. The method comprises storing sentences in a client cache of a client device that relate to bridging delays in a dialog, storing sentence templates to be used in dialogs, generating a talking head response to a user inquiry from the client device, and determining whether sentences or stored templates stored in the client cache relate to the talking head response. If the stored sentences or stored templates relate to the talking head response, the method comprises instructing the client device to use the appropriate stored sentence or template from the client cache to render at least a part of the talking head response and transmitting a portion of the talking head response not stored in the client cache, if any, to the client device to render a complete talking head response. If the client cache has no stored data associated with the talking head response, the method comprises transmitting the talking head response to be rendered on the client device.

Claims

1. A method of rendering a talking head, the method comprising:

A. receiving a text-to-speech (TTS) response from a TTS module, the TTS response generated in response to a user request from a client;

B. determining whether data associated with the TTS response is stored in a client cache;

C. if the client cache has stored data associated with the TTS response;

1. instructing the client to play a first portion of the TTS response stored in the client cache; and

2. transmitting a second portion of the TTS response not stored in the client cache over the network, wherein the full TTS response comprises the first portion and the second portion; and

D. if the client cache has no stored data associated with the TTS response;

1. instructing the TTS module and a face animation server to provide the TTS response to the client.

2. The method of claim 1, wherein a dialog management module generates text in response to the user request and the generated text is transmitted to the TTS module to generate the TTS response.

3. The method of claim 2, wherein the dialog management module considers a state of the dialog when it generates the text in response to the user request.

4. The method of claim 3, wherein the dialog management module further considers user click-trail data when it generates the text in response to the user request.

5. The method of claim 1, wherein the response generated by the dialog management module comprises markup data indicating at least one emotional state for the talking head.

6. The method of claim 5, wherein the emotional state indicated by the markup data comprises happiness.

7. The method of claim 5, wherein the emotional state indicated by the markup data comprises disgust.

8. The method of claim 5, wherein the emotional state indicated by the markup data comprises anticipation.

9. The method of claim 1, wherein the TTS module generates a TTS response, a list of phonemes, durations, tags and a markup of the text for use by a talking head animation rendering module.

10. The method of claim 9, wherein the facial animation server computes animation parameters that enable the client to render the talking head.

11. The method of claim 10, wherein the computed animation parameters comprise head and eye motion, facial expressions, mouth videos, eye videos and rectangular or object-based full head videos.

12. The method of claim 1, wherein if the client cache has no stored data associated with the TTS response and the TTS module and face animation server are instructed to provide the TTS response to the client, the method further comprises sending a flag to the client instructing the client to store at least some data associated with the TTS response in the client cache.

13. The method of claim 1, further comprising storing a copy of the client cache on a server.

14. The method of claim 1, further comprising storing the dialog manager response as a representation of the data stored in the client cache on a server.

15. The method of claim 12, wherein the method further comprises sending a flag to the client instructing the client to store at least some data associated with the TTS response in the client cache if the TTS module and face animation server are instructed to provide the TTS response including visual TTS data.

26. The method of claim 1, wherein the method of rendering a talking head on a client device occurs over a network.

16. A method of rendering a talking head on a client device, wherein the client device has a client cache capable of storing audio/visual data associated with rendering the talking head, the method comprising:

A. generating a talking head response to a user inquiry from a client device;

B. determining whether data associated with the talking head response is stored in a client cache on the client device;

C. if the client cache has stored data associated with the talking head response;

1. instructing the client device to use the data associated with the talking head response from the client cache to render at least a part of the talking head response; and

2. transmitting a portion of the talking head response not stored in the client cache, if any, to the client device to render a complete talking head response; and

D. if the client cache has no stored data associated with the talking head response;

1. transmitting the talking head response to be rendered on the client device.

17. The method of claim 16, wherein if the client cache has no stored data associated with the talking head response, transmitting the talking head response to be rendered on the client device further comprises transmitting a flag instructing the client device to store at least a portion of the talking head response in the client cache.

18. The method of claim 16, further comprising transmitting http data to a browser operating on the client device.

19. The method of claim 16, wherein a server stores a copy of the data stored in the client cache.

20. The method of claim 16, wherein a server stores a dialog manager response to the user request as a representation of data stored in the client cache.

21. The method of claim 16, further comprising:

storing sentences associated with talking head responses prior to receiving a user inquiry.

27. The method of claim 16, wherein the method of rendering a talking head on a client device occurs over a network.

28. The method of claim 16, further comprising storing behavior templates.

29. The method of claim 28, wherein the behavior templates relate to listening behavior.

30. The method of claim 28, wherein the behavior templates relate to behavior associated with waiting for an answer to a question.

31. The method of claim 28, wherein the behavior templates relate to behavior associated with asking a question for the second time.

22. A method of rendering a talking head on a client device, wherein the client device has a client cache capable of storing audio/visual data associated with rendering the talking head, the method comprising:

A. storing sentences in a client cache of a client device that relate to bridging delays in a dialog;

B. storing sentence templates to be used in dialogs;

C. generating a talking head response to a user inquiry from a client device;

D. determining whether sentences or stored templates stored in the client cache relate to the talking head response;

E. if stored sentences or stored templates relate to the talking head response;

1. instructing the client device to use the appropriate stored sentence or template from the client cache to render at least a part of the talking head response; and

2. transmitting a portion of the talking head response not stored in the client cache, if any, to the client device to render a complete talking head response; and

F. if the client cache has no stored data associated with the talking head response;

1. transmitting the talking head response to be rendered on the client device.

23. The method of claim 22, further comprising storing idle behavior associated with the talking head in the client cache.

24. The method of claim 23, wherein the idle behavior relates to listening behavior and waiting behavior of the talking head.

25. The method of claim 22, wherein the method of rendering a talking head on a client device occurs over a network.