Method and system for text-to-speech synthesis with personalized voice

• US 8,886,537 B2
• Filed: 03/20/2007
• Issued: 11/11/2014
• Est. Priority Date: 03/20/2007
• Status: Active Grant

• Current Assignee: Cerence Operating Company (Cerence Inc.)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Goldberg, Itzhack, Hoory, Ron, Mizrachi, Boaz, Kons, Zvi
• Primary Examiner(s): He, Jialong
• Application Number:
  US11/688,264
• Publication Number:
  US 20080235024A1
• Time in Patent Office: 2,793 Days
• Field of Search: 704258-269
• US Class Current: 704/258
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management

Abstract

A method and system are provided for text-to-speech synthesis with personalized voice. The method includes receiving an incidental audio input (403) of speech in the form of an audio communication from an input speaker (401) and generating a voice dataset (404) for the input speaker (401). The method includes receiving a text input (411) at the same device as the audio input (403) and synthesizing (312) the text from the text input (411) to synthesized speech including using the voice dataset (404) to personalize the synthesized speech to sound like the input speaker (401). In addition, the method includes analyzing (316) the text for expression and adding the expression (315) to the synthesized speech. The audio communication may be part of a video communication (453) and the audio input (403) may have an associated visual input (455) of an image of the input speaker. The synthesis from text may include providing a synthesized image personalized to look like the image of the input speaker with expressions added from the visual input (455).

Claims

1. A method for text-to-speech synthesis with personalized voice, comprising:

receiving, at a mobile communications device operated by a user, incidental audio speech data from a sending device operated by a remote input speaker, wherein the speech data of the remote input speaker is received over a first network communication link during a voice communication between the remote input speaker and the user of the mobile communications device;

generating, by the user'"'"'s mobile communications device, a voice dataset for the remote input speaker based, at least in part, on the incidental audio speech data;

receiving, over a second network communication link, text data at the user'"'"'s mobile communications device, wherein the text data is sent from the sending device subsequent to the voice communication; and

converting, by the user'"'"'s mobile communications device, the text data to synthesized speech, at least in part, using the voice dataset to personalize the synthesized speech to sound like the remote input speaker.

2. The method as claimed in claim 1, wherein personalizing the synthesized speech includes training a concatenative synthetic voice to sound like the input speaker by using a voice morphing transformation.

3. The method as claimed in claim 1, wherein the audio input of speech data has an associated visual input of an image of the input speaker and the method includes generating an image dataset, and wherein converting to synthesized speech includes synthesizing an associated synthesized image, including using the image dataset to personalize the synthesized image to look like the input speaker image.

4. The method as claimed in claim 3, including:

storing visual expressions from the visual input; and

adding the visual expressions to the personalized synthesized image.

5. The method as claimed in claim 1, including:

analyzing the text for expression;

adding the expression to the synthesized speech.

6. The method as claimed in claim 5, including:

storing paralinguistic expression elements from the audio input of speech;

adding the paralinguistic expression elements to the personalized synthesized speech.

7. The method as claimed in claim 5, wherein analyzing the text includes identifying one or more of the group of:

punctuation, letter case, paralinguistic elements, acronyms, emotion icons, and key words.

8. The method as claimed in claim 5, wherein metadata is provided in association with text elements to indicate the expression.

9. The method as claimed in claim 5, wherein the text is annotated to indicate the expression.

10. The method as claimed in claim 1, wherein the device is one of the group of:

an instant messaging client system, a mobile communication device, a broadcasting device, all with both audio and text capabilities.

11. The method as claimed in claim 1, wherein an identifier of the source of the audio speech data is stored in association with the voice dataset and the voice dataset is used in synthesis of text data from the same source.

12. A computer program product stored on a non-transitory computer readable storage medium for text-to-speech synthesis, comprising computer readable program code means for performing the steps of:

receiving, at a mobile communications device operated by a user, incidental audio speech data from a sending device operated by a remote input speaker, wherein the speech data of the remote input speaker is received over a first network communication link during a voice communication between the remote input speaker and the user of the mobile communications device;

generating, by the user'"'"'s mobile communications device, a voice dataset for the remote input speaker based, at least in part, on the incidental audio speech data;

receiving, over a second network communication link, text data at the user'"'"'s mobile communications device, wherein the text data is sent from the sending device subsequent to the voice communication; and

converting, by the user'"'"'s mobile communications device, the text data to synthesized speech, at least in part, using the voice dataset to personalize the synthesized speech to sound like the remote input speaker.

13. A mobile communications device capable of text-to-speech synthesis with personalized voice, comprising:

an audio communication input for receiving over a first network communication link incidental audio speech data from a sending device operated by a remote input speaker during a voice communication between the remote input speaker and a user of the mobile communications device;

a processor configured to generate, at the user'"'"'s mobile communications device, a voice dataset for the remote input speaker based, at least in part, on the incidental audio speech data;

at least one input for receiving over a second network communication link text data at the user'"'"'s mobile communication device, wherein the text data is sent from the sending device subsequent to the voice communication; and

a text-to-speech synthesizer for producing synthesized speech by converting the text data to synthesized speech to sound like the remote input speaker, at least in part, using the voice dataset.

14. The system as claimed in claim 13, wherein the text-to-speech synthesizer is configured to add expression to the synthesized speech.

15. The system as claimed in claim 13, including a video communication input including the audio communication input with an associated visual communication input for visual data of an image of the remote input speaker, wherein the processor is further configured to generate an image dataset for the remote input speaker, wherein the synthesizer provides a synthesized image which looks like the remote input speaker image.

16. The system as claimed in claim 15, wherein the synthesizer is configured to add expression to the synthesized image.

17. The system as claimed in claim 15, including:

at least one storage medium for storing expression elements from the speech data or visual data, wherein the processor is configured to add the expression elements to the synthesized speech or synthesized image.

18. The system as claimed in claim 13, including a training module for training a concatenative synthetic voice to sound like the input speaker, wherein the training module includes a voice morphing transformation.

19. The system as claimed in claim 13, wherein the text expression analyzer provides metadata in association with text elements to indicate the expression.

20. The system as claimed in claim 13, wherein the text expression analyzer provides text annotation to indicate the expression.