Local and remote feedback loop for speech synthesis

• US 8,380,508 B2
• Filed: 06/05/2009
• Issued: 02/19/2013
• Est. Priority Date: 06/05/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Plumpe, Michael D.
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US12/479,007
• Publication Number:
  US 20100312564A1
• Time in Patent Office: 1,355 Days
• Field of Search: 704/260
• US Class Current: 704/260
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: H04M 2201/39 using speech synthesis speech synthesis per se G10L13/00
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals

Abstract

A local text to speech feedback loop is utilized to modify algorithms used in speech synthesis to provide a user with an improved experience. A remote text to speech feedback loop is utilized to aggregate local feedback loop data and incorporate best solutions into new improved text to speech engine for deployment.

Claims

1. A method to be executed at least in part in a computing device for providing synthesized speech with feedback loops, the method comprising:

receiving text to be converted to speech from one of an input device, a removable storage device, a non-removable storage device, and a communication connection of the computing device;

determining the text to be converted to speech at a processor of the computing device;

synthesizing speech based on the text employing a predefined algorithm and at least one predefined parameter at the processor of the computing device;

presenting the synthesized speech to a user through an audio output device of the computing device;

receiving a user feedback through an input device of the computing device;

interpreting the user feedback at the processor of the computing device;

re-synthesizing the speech based on the user feedback employing at least one of;

a different predefined algorithm and at least one other parameter at the processor of the computing device; and

presenting the re-synthesized speech to the user through the audio output device of the computing device.

2. The method of claim 1, further comprising:

locally recording the re-synthesized speech based on the user feedback; and

utilizing the re-synthesized the speech subsequently, when the same text is to be converted to speech.

3. The method of claim 1, further comprising:

storing the user feedback locally;

providing the locally stored user feedback and information associated with the synthesized speech presented to the user to a developer for modification of a Text To Speech (TTS) system component; and

in response to receiving the modified TTS system component, synthesizing the speech employing the modified TTS system component.

4. The method of claim 3, wherein the information associated with the synthesized speech presented to the user includes at least one from a set of:

the text converted to speech, an identification of a TTS engine synthesizing the speech, a language of the speech, a dialect of the speech, a gender of a speaker, and the at least one parameter associated with the synthesized speech.

5. The method of claim 3, wherein the TTS system component is one of:

a TTS engine and training data for the TTS engine, and wherein modifying the TTS system component includes replacing at least one of the TTS engine and the training data for the TTS engine.

6. The method of claim 5, wherein modifying the TTS system component includes updating at least one of the TTS engine and the training data for the TTS engine.

7. The method of claim 3, wherein the stored user feedback and the information associated with the synthesized speech presented to the user are provided to the developer through one of:

a transmittal by email, a real time access through a hosted service, and an assistance application associated with the TTS system.

8. The method of claim 1, wherein the at least one parameter includes one from a set of:

a pitch of the synthesized speech, a dialect of the synthesized speech, a fundamental frequency of a waveform for generating the synthesized speech, a noise level, and a volume of the synthesized speech.

9. The method of claim 1, wherein the user feedback is an active feedback in form of a comment from the user.

10. The method of claim 1, wherein the user feedback is a passive feedback in form of a delay in responding to a voice prompt containing the synthesized speech from the user.

11. The method of claim 10, wherein the passive feedback is determined based on aggregating a plurality of delays in responding to voice prompts from the user employing a time-out trigger.

12. A communication system for providing synthesized speech with feedback loops, the system comprising:

a computing device executing a communication application, the computing device including;

a memory storing instructions; and

a processor configured to execute a Text To Speech (TTS) engine and a local feedback module, wherein the processor is configured to;

receive text to be converted to speech from one of an input device, a removable storage device, a non-removable storage device, and a communication connection of the computing device;

synthesize speech based on the text employing a predefined algorithm and at least one predefined parameter; and

present the synthesized speech to a user through an audio output device of the computing device ;

receive a user feedback through an input device of the computing device;

interpret the user feedback;

present a re-synthesized speech to the user employing a different algorithm through the audio output device of the computing device;

record the user feedback locally; and

utilize the re-synthesized the speech subsequently, when the same text is to be converted to speech.

13. The system of claim 12, wherein the processor is further configured to execute a remote feedback module and configured to:

locally record information associated with the synthesized speech presented to the user for modification of a TTS system component;

provide the locally recorded user feedback and the information associated with the synthesized speech to a developer;

receive a modified TTS system component; and

enable one of update and replacement of the TTS system component with the modified TTS system component.

14. The system of claim 12, wherein the communication system is part of an Interactive Voice Response (IVR) system.

15. The system of claim 14, wherein the local feedback module is further configured to interpret the user feedback by determining that a voice prompt containing the synthesized speech is not understood by the user based on a received user comment and instruct the TTS engine to re-synthesize the speech by modifying at least one from a set of:

a pitch of the synthesized speech, a dialect of the synthesized speech, a fundamental frequency of a waveform for generating the synthesized speech, a noise level, and a volume of the synthesized speech.

16. The system of claim 12, wherein the computing device includes a data store for storing at least one from a set of:

the user feedback, the information associated with the synthesized speech, and training data for the TTS engine.

17. A computer-readable memory device with instructions stored thereon for providing synthesized speech with feedback loops, the instructions comprising:

receiving text to be converted to speech from one of an input device, a removable storage device, a non-removable storage device, and a communication connection of the computing device;

synthesizing speech employing a predefined algorithm and a plurality of predefined parameters including at least one from a set of;

a language, a pitch, a dialect, a volume, and a fundamental frequency for a waveform of the synthesized speech at a processor of the computing device

presenting the synthesized speech to a user;

receiving one of an active and a passive user feedback through an input device of the computing device;

interpreting the user feedback at the processor of the computing device;

recording the user feedback and information associated with the synthesized speech including at least one from a set of;

the text converted to speech, an identification of the TTS engine, a language of the speech, a dialect of the speech, a gender of a speaker, and the plurality of parameters associated with the synthesized speech through the input device of the computing device;

providing instructions to a TTS engine such that the TSS engine re-synthesizes the speech based on the user feedback employing at least one of;

a different predefined algorithm and a modified parameter at the processor of the computing device;

providing the recorded user feedback and the information associated with the synthesized speech to a developer at the processor of the computing device;

receiving a modified TTS engine from the developer at the processor of the computing device; and

enabling one of update and replacement of the TTS engine with the modified TTS engine.

18. The computer-readable memory device of claim 17, wherein the active user feedback is a user utterance and the passive user feedback is a user hesitation determined based on aggregating a plurality of delays in responding to voice prompts by the user employing a time-out trigger.

19. The computer-readable memory device of claim 17, wherein the instructions further comprise:

providing the developer statistical information associated with presentation of original and re-synthesized speeches and respective user feedbacks.

20. The computer-readable memory device of claim 17, wherein the user feedback and the information associated with the synthesized speech are made available to the developer through a hosted service.