State-dependent query response

• US 10,496,718 B2
• Filed: 02/05/2016
• Issued: 12/03/2019
• Est. Priority Date: 08/06/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Jitkoff, John Nicholas, LeBeau, Michael J., Byrne, William J., Singleton, David P.
• Primary Examiner(s): Filipczyk, Marcin R
• Application Number:
  US15/016,707
• Publication Number:
  US 20160154881A1
• Time in Patent Office: 1,397 Days
• Field of Search: 707706, 707712
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/638 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/186 Templates
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: H04M 1/6041 Portable telephones adapted for handsfree use
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04R 29/004 for microphones H04R29/007 takes precedence

Abstract

In general, the subject matter described in this specification can be embodied in methods, systems, and program products for receiving user input that defines a search query, and providing the search query to a server system. Information that a search engine system determined was responsive to the search query is received at a computing device. The computing device is identified as in a first state, and a first output mode for audibly outputting at least a portion of the information is selected. The first output mode is selected from a collection of the first output mode and a second output mode. The second output mode is selected in response to the computing device being in a second state and is for visually outputting at least the portion of the information and not audibly outputting the at least portion of the information. At least the portion of information is audibly output.

Claims

1. A computer-implemented method, comprising:

transmitting, by one or more processors of a computing device, a query that was input at the computing device to a server system over a network;

receiving, by the one or more processors, a query response to the query from the server system over the network, the query response including data for presenting a textual representation of the query response;

determining, by the one or more processors, that the computing device is in a first mode of operation in which the query response is to be audibly provided; and

in response to receiving the query response from the server system and determining that the computing device is in the first mode of operation in which the query response is to be audibly provided;

applying, by the one or more processors, a speech template to automatically convert the data for presenting the textual representation of the query response into a set of text for generating synthesized speech without receiving any additional user input after transmitting the query to the server system, the set of text comprising all words present in the data for presenting the textual representation of the query response and additional pre-determined words specified by the speech template, the additional pre-determined words in the set of text are not present in the data for presenting the textual representation of the query response that would otherwise be output for display by the computing device if the computing device were not operating in the first mode of operation;

generating, by the one or more processors, a synthesized speech representation of the query response from the set of text comprising all of the words present in the data for presenting the textual representation of the query response and the additional pre-determined words specified by the speech template, the additional pre-determined words in the set of text for generating the synthesized speech representation of the query response configured to convey the query response into a form that improves sentence flow of the query response so that the query response is more understandable by a human listener; and

providing, by the one or more processors, the synthesized speech representation of the query response as an audible output.

2. The computer-implemented method of claim 1, wherein the first mode of operation is a hands free mode of operation in which the computing device is configured to output the synthesized speech representation of the query response using a speaker of the computing device.

3. The computer-implemented method of claim 1, further comprising selecting, by the one or more processors, the computing device to be in the first mode of operation based on the one or more processors having determined that the computing device is connected to a power source.

4. The computer-implemented method of claim 1, further comprising selecting, by the one or more processors, the computing device to be in the first mode of operation based on the one or more processors having determined that the computing device is in communication with a wireless headset.

5. The computer-implemented method of claim 1, further comprising selecting, by the one or more processors, the computing device to be in the first mode of operation based on the one or more processors having determined that the computing device is placed in a dock.

6. The computer-implemented method of claim 1, further comprising providing, by the one or more processors, the textual representation of the query response for presentation by a display of the computing device, in conjunction with providing the synthesized speech representation of the query response for output by a speaker of the computing device, based on the one or more processors having determined that the computing device is in the first mode of operation.

7. The computer-implemented method of claim 1, further comprising:

determining, by the one or more processors, the computing device is in a second mode of operation in which a display of the computing device is used to output responses generated by the computing device when the computing device is not operating in the first mode of operation; and

deactivating, by the one or more processors, a speaker of the computing device from providing the synthesized speech representation of the query response.

8. The computer-implemented method of claim 1, further comprising activating, by the one or more processors, a microphone of the computing device for receiving a voice input while the computing device is in the first mode of operation.

9. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

transmitting a query that was input at a computing device to a server system;

receiving a query response to the query from the server system, the query response including data for presenting a textual representation of the query response;

determining that the computing device is in a first mode of operation in which the query response is to be audibly provided; and

in response to receiving the query response from the server system and determining that the computing device is in the first mode of operation in which the query response is to be audibly provided;

applying a speech template to automatically convert the data for presenting the textual representation of the query response into a set of text for generating synthesized speech without receiving any additional user input after transmitting the query to the server system, the set of text comprising all words present in the data for presenting the textual representation of the query response and additional pre-determined words specified by the speech template, the additional pre-determined words in the set of text are not present in the data for presenting the textual representation of the query response that would otherwise be output for display by the computing device if the computing device were not operating in the first mode of operation;

generating a synthesized speech representation of the query response from the set of text comprising all of the words present in the data for presenting the textual representation of the query response and the additional pre-determined words specified by the speech template, the additional pre-determined words in the set of text for generating the synthesized speech representation of the query response configured to convey the query response into a form that improves sentence flow of the query response so that the query response is more understandable by a human listener; and

providing the synthesized speech representation of the query response as an audible output.

10. The system of claim 9, wherein the first mode of operation is a hands free mode of operation in which the computing device is configured to output the query response using a speaker of the computing device.

11. The system of claim 9, wherein the operations further comprise selecting the computing device to be in the first mode of operation when the computing device is connected to a power source.

12. The system of claim 9, wherein the operations further comprise selecting the computing device to be in the first mode of operation when the computing device is in communication with a wireless headset.

13. The system of claim 9, wherein the operations further comprise selecting the computing device to be in the first mode of operation when the computing device is placed in a dock.

14. The system of claim 9, wherein the operations further comprise providing the textual representation of the query response for presentation by a display of the computing device, in conjunction with providing the synthesized speech representation of the query response for output by a speaker of the computing device when the computing device is in the first mode of operation.

15. The system of claim 9, wherein the operations further comprise:

determining that the computing device is in a second mode of operation in which a display of the computing device is configured to output responses generated by the computing device when the computing device is not operating in the first mode of operation; and

deactivating a speaker of the computing device from providing the query response.

16. The system of claim 9, wherein the operations further comprise activating a microphone of the computing device for receiving a voice input while the computing device is in the first mode of operation.

17. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

transmitting a query that was input at a computing device to a server system;

receiving a query response to the query from the server system, the query response including data for presenting a textual representation of the query response;

determining that the computing device is in a first mode of operation in which the query response is to be audibly provided; and

in response to receiving the query response from the server system and determining that the computing device is in the first mode of operation in which the query response is to be audibly provided;

applying a speech template to automatically convert the data for presenting the textual representation of the query response into a set of text for generating synthesized speech without receiving any additional user input after transmitting the query to the server system, the set of text comprising all words present in the data for presenting the textual representation of the query response and additional pre-determined words specified by the speech template, the additional pre-determined words in the set of text are not present in the data for presenting the textual representation of the query response that would otherwise be output for display by the computing device if the computing device were not operating in the first mode of operation;

generating a synthesized speech representation of the query response from the set of text comprising the pre-determined words specified by the speech template, the pre-determined words in the set of text for generating the synthesized speech representation of the query response configured to convey the query response into a form that improves sentence flow of the query response so that the query response is more understandable by a human listener; and

providing the synthesized speech representation of the query response for audible output.

18. The medium of claim 17, wherein the first mode of operation is a hands free mode of operation in which the computing device is configured to output the query response using a speaker of the computing device.

19. The medium of claim 17, wherein the operations further comprise:

determining that the computing device is connected to a power source; and

selecting the computing device to be in the first mode of operation based on the determination that the computing device is connected to the power source.

20. The medium of claim 17, wherein the operations further comprise:

determining that the computing device is in communication with a wireless headset; and

selecting the computing device to be in the first mode of operation based on the determination that the computing device is in communication with the wireless headset.