Adaptive text-to-speech outputs

• US 10,109,270 B2
• Filed: 07/19/2017
• Issued: 10/23/2018
• Est. Priority Date: 01/28/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Foerster, Jakob Nicolaus
• Primary Examiner(s): Pullias, Jesse
• Application Number:
  US15/653,872
• Publication Number:
  US 20170316774A1
• Time in Patent Office: 461 Days
• Field of Search: 704257-275
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination

Abstract

In some implementations, a language proficiency of a user of a client device is determined by one or more computers. The one or more computers then determines a text segment for output by a text-to-speech module based on the determined language proficiency of the user. After determining the text segment for output, the one or more computers generates audio data including a synthesized utterance of the text segment. The audio data including the synthesized utterance of the text segment is then provided to the client device for output.

Claims

1. A method performed by one or more computers, the method comprising:

receiving, by the one or more computers, context data from a client device of a user;

selecting, by the one or more computers, a user context corresponding to the context data from the client device, the user context being selected from among a plurality of user contexts, and the user context indicating a level of complexity of speech that the user is likely able to comprehend at a given time when the context data was received;

selecting, by the one or more computers and from among a plurality of candidate text segments that correspond to different levels of complexity of speech, the text segment for text-to-speech synthesis by a text-to-speech module that best matches the selected user context;

generating, by the one or more computers, audio data comprising a synthesized utterance of the selected text segment using the text-to-speech module; and

providing, by the one or more computers and to the client device, the audio data comprising the synthesized utterance of the selected text segment.

2. The method of claim 1, further comprising:

receiving, by the one or more computers, data indicating a voice query detected by the client device; and

generating, by the one or more computers, the plurality of candidate text segments, each candidate text segment responsive to the voice query,

wherein receiving the context data comprises receiving context data indicating a current context when the voice query was detected by the client device, and

wherein providing the audio data comprises providing the audio data to the client device for output as a response to the voice query.

3. The method of claim 1, further comprising generating each of the plurality of candidate text segments from one or more search results identified by a search engine in response to a query from the user.

4. The method of claim 1, wherein receiving the context data comprises receiving data indicating a location, speed, or movement pattern of the client device;

and

wherein selecting the user context comprises selecting the user context based on the location, speed, or movement pattern of the client device indicated by the context data.

5. The method of claim 1, wherein the context data indicates a GPS data indicating a current location associated with the user;

and

wherein selecting the user context comprises selecting the user context based on the GPS data indicating the current location associated with the user.

6. The method of claim 1, wherein the context data includes sensor data from a mobile device of the user;

and

wherein selecting the user context comprises selecting the user context based on the sensor data from the mobile device of the user.

7. The method of claim 1, further comprising:

generating, by the one or more computers, the plurality of candidate text segments in response to a query from the user;

ranking, by the one or more computers, the plurality of candidate text segments based on a likelihood that each candidate text segment is likely to be responsive to the query; and

re-ranking, by the one or more computers, the plurality of candidate text segments based on the selected user context.

8. The method of claim 7, wherein the selected text segment that best matches the selected user context corresponds to the candidate text segment having the highest rank after the re-ranking.

9. The method of claim 1, wherein the client device displays a mobile application that uses a text-to-speech interface.

10. The method of claim 1, wherein the user context is selected based on one or more queries that were previously submitted by the user, data indicating a current task of the user, or an indication that the user failed to complete a task.

11. A system comprising:

one or more computers; and

a non-transitory computer-readable medium coupled to the one or more computers having instructions stored thereon, which, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving context data from a client device of a user;

selecting a user context corresponding to the context data from the client device, the user context being selected from among a plurality of user contexts, and the user context indicating a level of complexity of speech that the user is likely able to comprehend at a given time when the context data was received;

selecting, from among a plurality of candidate text segments that correspond to different levels of complexity of speech, the text segment for text-to-speech synthesis by a text-to-speech module that best matches the selected user context;

generating audio data comprising a synthesized utterance of the selected text segment using the text-to-speech module; and

providing, to the client device, the audio data comprising the synthesized utterance of the selected text segment.

12. The system of claim 11, wherein the operations further comprise:

receiving data indicating a voice query detected by the client device; and

generating the plurality of candidate text segments, each candidate text segment responsive to the voice query,

wherein receiving the context data comprises receiving context data indicating a current context when the voice query was detected by the client device, and

wherein providing the audio data comprises providing the audio data to the client device for output as a response to the voice query.

13. The system of claim 11, wherein the operations further comprise generating each of the plurality of candidate text segments from one or more search results identified by a search engine in response to a query from the user.

14. The system of claim 11, wherein:

receiving the context data comprises receiving data indicating a location, speed, or movement pattern of the client device; and

selecting the user context comprises selecting the user context based on the location, speed, or movement pattern of the client device indicated by the context data.

15. A non-transitory computer-readable storage device encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving context data from a client device of a user;

selecting a user context corresponding to the context data from the client device, the user context being selected from among a plurality of user contexts, and the user context indicating a level of complexity of speech that the user is likely able to comprehend at a given time when the context data was received;

selecting from among a plurality of candidate text segments that correspond to different levels of complexity of speech, the text segment for text-to-speech synthesis by a text-to-speech module that best matches the selected user context;

generating, by the one or more computers, audio data comprising a synthesized utterance of the selected text segment using the text-to-speech module; and

providing, to the client device, the audio data comprising the synthesized utterance of the selected text segment.

16. The non-transitory computer-readable storage device of claim 15, wherein the operations further comprise:

receiving data indicating a voice query detected by the client device; and

generating the plurality of candidate text segments, each candidate text segment responsive to the voice query,

wherein receiving the context data comprises receiving context data indicating a current context when the voice query was detected by the client device, and

wherein providing the audio data comprises providing the audio data to the client device for output as a response to the voice query.

17. The non-transitory computer-readable storage device of claim 15, wherein the operations further comprise generating each of the plurality of candidate text segments from one or more search results identified by a search engine in response to a query from the user.

18. The non-transitory computer-readable storage device of claim 15, wherein:

receiving the context data comprises receiving data indicating a location, speed, or movement pattern of the client device; and

selecting the user context comprises selecting the user context based on the location, speed, or movement pattern of the client device indicated by the context data.

19. The non-transitory computer-readable storage device of claim 15, wherein the user context is selected based on one or more queries that were previously submitted by the user, data indicating a current task of the user, or an indication that the user failed to complete a task.