Conversational interface agent

• US 20050270293A1
• Filed: 08/04/2005
• Published: 12/08/2005
• Est. Priority Date: 12/28/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Bo, Shum, Heung-Yeung, Guo, Baining
• Granted Patent: US 7,391,421 B2
• Time in Patent Office: Days
• US Class Current: 345/473
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads

Abstract

A video rewrite technique for rendering a talking head or agent completely simulates a conversation by including a waiting or listening state. Smooth transitions are provided to and from a talking state.

Claims

1. A computer readable medium having instructions, which when executed on a computer provide a user interface, the instructions comprising:

a speech synthesizer receiving input for synthesis and providing an audio output signal; and

a video rendering module receiving information related to the audio output signal, the video rendering module rendering a representation of a talking head having a talking state with mouth movements in accordance with the audio output signal and a waiting state with movements in accordance with listening, and wherein the video rendering module renders a sequence of video frames having the talking head, and wherein the video rendering module also transforms affine parameters to physical movements of the talking head for each frame.

3. The computer readable medium of claim 1 wherein the video rendering module continuously renders the video frames having the talking head with non-talking mouth movements during the waiting state and adds a talking mouth position to each of the frames during the talking state.

5. The computer readable medium of claim 3 wherein the video rendering module tracks movements of the talking head in the sequence of video frames.

7. The computer readable medium of claim 5 wherein the physical movements include translations and rotations of the talking head.

8. The computer readable medium of claim 5 wherein the talking mouth positions are added based upon interpolated physical movements of the talking head.

9. The computer readable medium of claim 5 wherein for each of a plurality of frames, interpolated physical movements are calculated as a function of a corresponding preceding frame and a corresponding succeeding frame.

11. The computer readable medium of claim 5 wherein for each of a plurality of frames, interpolated physical movements are calculated as a function of a corresponding preceding frame and a corresponding succeeding frame.) A computer readable medium having instructions, which when executed on a computer provide a user interface, the instructions comprising:

a speech synthesizer receiving input for synthesis and providing an audio output signal; and

a video rendering module receiving information related to the audio output signal, the video rendering module rendering a representation of a talking head having a talking state with mouth movements in accordance with the audio output signal and a waiting state with mouth movements in accordance with listening, the video rendering module accessing a store having a sequence of frames of the talking head and continuously rendering at least a portion of each of the frames in the sequence of frames while selectively adding a corresponding mouth position for the talking state to each of the frames in accordance with the audio output signal and in accordance with tracking movements of the talking head during the sequence of frames, wherein the video rendering module transforms affine parameters to physical movements of the talking head for each frame.

13. The computer readable medium of claim 11 wherein the physical movements include translations and rotations of the talking head.

14. The computer readable medium of claim 13 wherein the mouth positions are added based upon interpolated physical movements of the talking head.

15. The computer readable medium of claim 14 wherein for each of a plurality of frames, interpolated physical movements are calculated as a function of a corresponding preceding frame and a corresponding succeeding frame.

2. (canceled)

4. (canceled)

6. (canceled)

10. (canceled)

12. (canceled)

16. (canceled)

17. A computer-implemented method for generating a talking head on a computer display to simulate a conversation, the method comprising:

continuously rendering a sequence of video frames of a talking head with each frame having mouth characteristics indicative of a non-talking state;

tracking movements of the talking head throughout the sequence of video frames;

outputting a voice audio;

selectively adding a corresponding mouth position to selected frames of the video sequence as a function of the voice audio and tracked movements of the talking head; and

wherein tracking movements includes transforming affine parameters to physical movements of the talking head for each frame.

18. (canceled)

20. The computer-implemented method of claim 18 wherein the physical movements include translations and rotations of the talking head.

21. The computer-implemented method of claim 20 and further comprising calculating interpolated physical movements of the talking head based on frames of the sequence.

22. The computer-implemented method of claim 21 wherein calculating interpolated physical movements includes calculating interpolated physical movements are calculated as a function of a corresponding preceding frame and a corresponding succeeding frame for each of a plurality of frames.

19. (canceled)

23. (canceled)