Method and system for converting text to lip-synchronized speech in real time

• US 20060129400A1
• Filed: 12/10/2004
• Published: 06/15/2006
• Est. Priority Date: 12/10/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Fields, Timothy V., Cotton, Brandon
• Granted Patent: US 7,613,613 B2
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04N 21/440236 by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
• CPC Class Codes: H04N 5/278 Subtitling

Abstract

A method and system for presenting lip-synchronized speech corresponding to the text received in real time is provided. A lip synchronization system provides an image of a character that is to be portrayed as speaking text received in real time. The lip synchronization system receives a sequence of text corresponding to the speech of the character. It may modify the received text in various ways before synchronizing the lips. It may generate phonemes for the modified text that are adapted to certain idioms. The lip synchronization system then generates the lip-synchronized images based on the phonemes generated from the modified texts and based on the identified expressions.

Claims

1. A method for presenting information in real time, the method comprising:

providing an image of a character;

receiving a sequence of text;

modifying the text of the received sequence;

generating speech corresponding to the modified text;

generating a sequence of images based on the provided image to represent the character speaking the generated speech; and

outputting the generated speech and sequence of images to portray the character speaking the text.

2. The method of claim 1 wherein the text is closed-captioned text of a television broadcast.

3. The method of claim 1 wherein the text is entered via a keyboard by a participant in a computer-based chat session.

4. The method of claim 1 wherein the text is modified by expanding acronyms.

5. The method of claim 1 wherein the text is modified to reflect an idiom.

6. The method of claim 5 wherein the idiom is associated with the character.

7. The method of claim 1 wherein the generating of speech includes identifying phonemes from the modified text.

8. The method of claim 7 wherein the phonemes are identified to reflect an idiom.

9. The method of claim 1 including identifying expressions from the sequence of text and wherein the generated sequence of images reflects the identified expressions.

10. The method of claim 9 wherein different images of the character are provided for different expressions.

11. The method of claim 1 wherein the generating of the sequence of images represents the character lip-syncing the generated speech.

12. The method of claim 1 including identifying expressions from the sequence of text and wherein the generated sequence of images reflects the identified expressions while the character lip-syncs the generated speech.

13. A system for presenting a lip-syncing character, comprising:

an image store containing an image of a character;

a modify text component that receives text in real time and modifies the text; and

a lip synchronization component that inputs the modified text and the image of the character and outputs speech corresponding to the modified text and images of the character speaking the output speech in real time as the text is received.

14. The system of claim 13 wherein the text is closed-captioned text of a television broadcast.

15. The system of claim 13 wherein the text is entered via a keyboard by a participant in a computer-based chat session.

16. The system of claim 13 wherein the text is modified by expanding acronyms.

17. The system of claim 13 wherein the text is modified to reflect an idiom.

18. The system of claim 13 wherein the generating of speech includes identifying phonemes from the modified text.

19. The system of claim 18 wherein the phonemes are identified to reflect an idiom.

20. The system of claim 13 including a component that identifies expressions from the sequence of text so that the images of the character speaking the output text can reflect the expressions.

21. A computer-readable medium containing instructions for controlling a computer to present images of a character speaking, by a method comprising:

providing an image of a character;

receiving a sequence of text in real time;

generating speech corresponding to the received text;

generating a sequence of images based on the provided image to represent the character speaking the generated speech; and

outputting the generated speech and sequence of images to portray the character speaking the text.

22. The computer-readable medium of claim 21 wherein the text is closed-captioned text.

23. The computer-readable medium of claim 21 wherein the text is entered by a participant in a computer-based chat session.

24. The computer-readable medium of claim 21 including modifying the text before generating the speech by expanding acronyms.

25. The computer-readable medium of claim 21 including modifying the text before generating the speech to reflect an idiom.

26. The computer-readable medium of claim 21 wherein the generating of speech includes identifying phonemes from the text.

27. The computer-readable medium of claim 26 wherein the phonemes are identified to reflect an idiom.

28. The computer-readable medium of claim 21 including identifying expressions from the sequence of text and wherein the generated sequence of images reflects the identified expressions.

29. The computer-readable medium of claim 28 wherein different images of the character are provided for different expressions.

30. The computer-readable medium of claim 21 wherein the generating of the sequence of images represents the character lip-syncing the generated speech.

31. The computer-readable medium of claim 21 including identifying expressions from the sequence of text and wherein the generated sequence of images reflects the identified expressions while the character lip-syncs the generated speech.