Method and apparatus for providing an animated display with translated speech

• US 20040068410A1
• Filed: 10/08/2002
• Published: 04/08/2004
• Est. Priority Date: 10/08/2002
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Mohamed, Magdi A., Tarlton, Mark A., Valliath, George T.
• Granted Patent: US 6,925,438 B2
• Time in Patent Office: Days
• US Class Current: 704/276
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: G10L 21/10 Transforming into visible information

Abstract

A first language to second language translator (11) converts text comprising a message in a first language into text representing that message in a second language. A text to speech transducer (12) then synthesizes an audiblized version of the text in the second language. An animated display driver (14) creates a corresponding display (15) of an image (16) that includes a mouth (17) that moves appropriately and synchronously with respect to the audiblized second language. In other embodiments, other facial features can be animated as well. Such animation can be random and unrelated to the appearance of the original speaker. Or, such animation can accurately represent at least some alterations to facial features of the original speaker while speaking. In some embodiments, the facial features can be altered by correlating specific detected alterations with specific corresponding substantive content in the original message and the translated text.

Claims

1. A method comprising:

providing text that corresponds to a first audiblizable language;

automatically translating the text into a second audiblizable language, which second audiblizable language is different than the first audiblizable language, to provide translated text;

rendering the translated text audible to provide audible sounds in the second language;

providing a visual representation of at least a mouth that moves in substantial synchronism with the audible spoken sounds in the second language.

2. The method of claim 1 wherein providing text that corresponds to a first audiblizable language includes:

providing an audiblized message;

automatically converting the audiblized message into the text.

3. The method of claim 2 wherein automatically converting the audiblized message into the text includes using an automatic speech recognition process.

4. The method of claim 1 wherein providing a visual representation of at least a mouth includes providing a visual representation of at least a face having a mouth.

5. The method of claim 4 wherein providing a visual representation of at least a face having a mouth includes providing a visual representation of at least a face that does not correspond to a person who sourced the text.

6. The method of claim 4 wherein providing a visual representation of at least a face having a mouth includes providing a visual representation of at least a face that corresponds to a person who sourced the text.

7. The method of claim 4 and further comprising altering at least one part of the face in addition to the mouth while rendering the translated text audible.

8. The method of claim 7 wherein altering at least one part of the face in addition to the mouth includes at least one of moving at least one part of the face and changing a color of at least one part of the face.

9. The method of claim 1 wherein providing text comprises recognizing speech as uttered by an individual having a face, and wherein providing a visual representation of at least a mouth includes providing a visual representation of at least a virtual face having a mouth, and further comprising:

detecting an alteration of at least one portion of the face while the individual utters the speech;

causing a part of the virtual face that corresponds to the at least one portion of the face to be altered while rendering the translated text audible.

10. The method of claim 9 wherein detecting an alteration of at least a portion of the face while the individual utters the speech includes correlating specific events of detected alterations with specific speech substantive content.

11. The method of claim 10 wherein causing a part of the virtual face that corresponds to the at least one portion of the face to be altered while rendering the translated text audible includes using the specific events of detected alteration to determine at least how and when to alter the at least one portion of the face.

12. The method of claim 11 wherein using the specific events of detected alteration to determine at least how and when to alter the at least one portion of the face includes causing a part of the virtual face to be altered substantially identically to one of the specific events of detected alteration when the audiblization of the translated text reaches audiblization of specific translated text that corresponds to specific speech substantive content for the speech that correlates to the one of the specific detected events.

13. The method of claim 12 wherein causing a part of the virtual face to be altered includes causing at least one of the following parts of the virtual face to be altered:

an eye, a part of an eye, an eyelid, an eyebrow, a cheek, a nose, a part of a nose, a forehead, an ear, and a chin.

20. The method of claim 13 wherein providing a display of at least an animated representation of the face wherein the face has a mouth that moves synchronously with the translated speech includes moving the mouth, at least from time to time, on a phoneme by phoneme basis with the audiblized translated speech.

14. A method comprising:

capturing speech in a first language being uttered by a person having a face;

detecting alteration of at least a part of the face, other than a movement of a mouth, and correlating such alteration to specific words in the speech to provide detected alterations;

translating the speech into a second language that is different from the first language to provide translated speech;

rendering the translated speech audible to provide audiblized translated speech;

while rendering the translated speech audible, substantially synchronously providing a display of at least an animated representation of the face wherein;

the face has a mouth that moves synchronously with the audiblized translated speech; and

the face has at least one other part that is altered synchronously with the detected alteration;

such that the at least animated representation of the face presents a mouth that moves in substantial accord with the audiblized translated speech and at least one part of the face that is altered in substantive accord with facial alterations as evinced by the person when uttering the speech.

15. The method of claim 14 wherein synchronously altering the at least one other part includes altering the at least one other part during a particular part of the audiblized translated speech in a way that is at least substantially similar to a detected alteration that correlates to the specific words having substantive content that substantially matches the particular part of the audiblized translated speech.

16. The method of claim 15 and further including identifying at least one category of facial alteration and wherein substantially synchronously providing a display of at least an animated representation of the face includes not altering the animated representation of the face to mimic the at least one category of facial alteration.

17. The method of claim 14 wherein detecting alteration of at least a part of the face includes detecting alteration of at least one of an eye, a part of an eye, an eyelid, an eyebrow, a cheek, a nose, a part of a nose, a forehead, an ear, and a chin.

18. The method of claim 14 wherein providing a display of at least an animated representation of the face includes providing a display of at least an animated three-dimensional representation of the face.

19. The method of claim 14 wherein rendering the translated speech audible includes selecting from amongst a plurality of synthesized speech voices.

21. An apparatus comprising:

a first-language to second-language translator having;

a first-language text input; and

a second-language translated text output;

a text-to-speech transducer having;

an input operably coupled to the second-language translated text output; and

an output comprising audiblized second-language translated text; and

an animated display driver having;

an input operably coupled to the second-language translated text output; and

an animated output that comprises data representing a mouth that is animated to synchronize with pronunciation of the audiblized second-language translated text.

22. The apparatus of claim 21 and further comprising:

a first-language speech recognizer having;

a first-language speech input; and

a recognized first-language speech text output;

and wherein the first-language text input of the first-language to second-language translator is operably coupled to the recognized first-language speech text output.

23. The apparatus of claim 22 wherein the animated output further comprises a face having the mouth.

24. The apparatus of claim 23 wherein the animated display driver further has a face animation input operably coupled to receive data representing facial alterations in addition to mouth movements of a person who voices the first-language speech input.

25. The apparatus of claim 24 wherein the animated output further represents the face as animated with facial alterations in addition to mouth movements, which facial alterations correspond to facial alterations of the person who voices the first-language speech input.

26. The apparatus of claim 21 wherein the animated display driver comprises animation means for animating a depiction of the mouth to appear as though the mouth is correctly speaking the audiblized second-language translated text.

27. The apparatus of claim 26 wherein the face comprises a likeness of a person who voices the first-language speech input.