Automatically monitoring for voice input based on context

• US 8,326,328 B2
• Filed: 09/29/2011
• Issued: 12/04/2012
• Est. Priority Date: 08/06/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: LeBeau, Michael J., Jitkoff, John Nicholas, Burke, Dave
• Primary Examiner(s): Gelin, Jean
• Assistant Examiner(s): Sarwar, Babar
• Application Number:
  US13/248,751
• Publication Number:
  US 20120035931A1
• Time in Patent Office: 432 Days
• Field of Search: 4554041-4042, 455/414.1, 455/418, 455/420, 455/550.1, 704/235, 704/233, 704/275, 704/E15.044, 704/E15.039, 704/270.1
• US Class Current: 455/456.4
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: H04M 1/04 Supports for telephone transmitters or receivers
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/72409 by interfacing with external accessories hands-free H04M1/60
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72445 for supporting Internet browser applications
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/046 involving vehicles, e.g. floating traffic data [FTD] or vehicle traffic prediction

Abstract

In one implementation, a computer-implemented method includes detecting a current context associated with a mobile computing device and determining, based on the current context, whether to switch the mobile computing device from a current mode of operation to a second mode of operation during which the mobile computing device monitors ambient sounds for voice input that indicates a request to perform an operation. The method can further include, in response to determining whether to switch to the second mode of operation, activating one or more microphones and a speech analysis subsystem associated with the mobile computing device so that the mobile computing device receives a stream of audio data. The method can also include providing output on the mobile computing device that is responsive to voice input that is detected in the stream of audio data and that indicates a request to perform an operation.

Claims

1. A computer-implemented method comprising:

detecting a current context associated with a mobile computing device, the current context indicating that the mobile computing device is coupled to a mobile computing device dock;

determining, based on the current context, whether to switch the mobile computing device from a current mode of operation, during which the mobile computing device does not monitor ambient sounds for voice input that indicates a request to perform an operation, to a second mode of operation during which the mobile computing device monitors ambient sounds selectively for voice input that indicates a request to perform an operation;

in response to determining to switch to the second mode of operation, activating one or more microphones and a speech analysis subsystem associated with the mobile computing device so that the mobile computing device receives a stream of audio data; and

providing output on the mobile computing device that is responsive to voice input that is detected in the stream of audio data and that indicates a request to perform an operation.

2. The computer-implemented method of claim 1, further comprising:

monitoring the stream of audio data for voice input using the speech analysis subsystem;

upon detecting voice input from the audio stream, determining whether the voice input indicates a request for the mobile computing device to perform an operation; and

based on the determination of whether the voice input indicates a request to perform an operation, causing the requested operation to be performed.

3. The computer-implemented method of claim 2, wherein the request to perform an operation comprises a search request;

and

wherein causing the requested operation to be performed comprises causing the search request to be performed.

4. The computer-implemented method of claim 2, wherein causing the requested operation to be performed comprises:

providing first information regarding the requested operation over a network to a remote computer system that is configured to perform the requested operation; and

receiving second information over the network from the remote computer system that indicates output generated from performance of the requested operation, wherein at least a portion of the second information is provided as the output on the mobile computing device.

5. The computer-implemented method of claim 2, further comprising:

identifying whether any of a predetermined group of keywords is present in the voice input; and

wherein the determination of whether the voice input indicates a request to perform an operation is based on the determination of whether any of the keywords are present in the voice input.

6. The computer-implemented method of claim 2, further comprising:

determining whether the voice input is a command or a question directed at the mobile computing device based on, at least, syntactical analysis of the voice input;

wherein the determination of whether the voice input indicates a request to perform an operation is based on, at least, the determination of whether the voice input is a command or a question directed at the mobile computing device.

7. The computer-implemented method of claim 2, further comprising:

identifying changes in a structure associated with the voice input; and

determining whether the voice input is directed at the mobile computing device based on the identified changes;

wherein the determination of whether the voice input indicates a request to perform an operation is based on, at least, the determination of whether the voice input is directed at the mobile computing device.

8. The computer-implemented method of claim 1, further comprising:

determining, based on the current context, whether a user associated with the mobile computing device has at least a threshold likelihood of providing voice input to the mobile computing device;

wherein determining whether to switch to the second mode of operation is additionally based on, at least, the determination of whether the user has at least the threshold likelihood of providing voice input.

9. The computer-implemented method of claim 1, further comprising:

determining, based on the current context, whether monitoring audio data for voice input will have at least a threshold level of convenience for a user associated with the mobile computing device and for the mobile computing device;

wherein determining whether to switch to the second mode of operation is based on, at least, the determination of whether the user has at least the threshold likelihood of providing voice input.

10. The computer-implemented method of claim 9, wherein the current context indicates at least the threshold level of convenience for the user when manual operation of the mobile computing device is not readily available.

11. The computer-implemented method of claim 9, wherein the current context indicates at least the threshold level of convenience for the mobile computing device when i) the mobile computing device is using an external power source or ii) the mobile computing device is using a portable power source that has at least a threshold charge remaining.

12. The computer-implemented method of claim 1, further comprising:

detecting that the current context for the mobile computing device has changed;

determining, based on the changed context, whether to switch the mobile computing device from the second mode of operation to a third mode of operation during which the mobile computing device does not monitor ambient sounds for voice input; and

in response to determining whether to switch to the third mode of operation, deactivating the microphones and the speech analysis subsystem.

13. The computer-implemented method of claim 1, wherein the current context further includes an indication of a type of device dock to which the mobile computing device is connected.

14. The computer-implemented method of claim 1, wherein the current context includes the mobile computing device receiving power from an external power source.

15. The computer-implemented method of claim 1, wherein detecting the current context, determining whether to switch to the second mode of operation, and activating the microphones and the speech analysis subsystem is performed without direction from a user.

16. A system for automatically monitoring for voice input, the system comprising:

a mobile computing device;

one or more microphones that are configured to receive ambient audio signals and to provide electronic audio data to the mobile computing device;

a context determination unit that is configured to detect a current context associated with the mobile computing device, the current context indicating that the mobile computing device is coupled to a mobile computing device dock;

a mode selection unit that is configured to determine, based on the current context determined by the context determination unit, whether to switch the mobile computing device from a current mode of operation, during which the mobile computing device does not monitor ambient sounds for voice input that indicates a request to perform an operation, to a second mode of operation during which the mobile computing device monitors ambient sounds selectively for voice input that indicates a request to perform an operation;

an input subsystem of the mobile computing device that is configured to activate the one or more microphones and a speech analysis subsystem associated with the mobile computing device in response to determining to switch to the second mode of operation so that the mobile computing device receives a stream of audio data;

an output subsystem of the mobile computing device that is configured to provide output on the mobile computing device that is responsive to voice input that is detected in the stream of audio data and that indicates a request to perform an operation.

17. A system for automatically monitoring for voice input, the system comprising:

a mobile computing device;

one or more microphones that are configured to receive ambient audio signals and to provide electronic audio data to the mobile computing device;

a context determination unit that is configured to detect a current context associated with the mobile computing device, the current context indicating that the mobile computing device is coupled to a mobile computing device dock;

means for determining, based on the current context, whether to switch the mobile computing device from a current mode of operation, during which the mobile computing device does not monitor ambient sounds for voice input that indicates a request to perform an operation, to a second mode of operation during which the mobile computing device monitors ambient sounds selectively for voice input that indicates a request to perform an operation;

an input subsystem of the mobile computing device that is configured to activate the one or more microphones and a speech analysis subsystem associated with the mobile computing device in response to determining to switch to the second mode of operation so that the mobile computing device receives a stream of audio data;

an output subsystem of the mobile computing device that is configured to provide output on the mobile computing device that is responsive to voice input that is detected in the stream of audio data and that indicates a request to perform an operation.