Selective aborting of online processing of voice inputs in a voice-enabled electronic device

• US 9,870,196 B2
• Filed: 05/27/2015
• Issued: 01/16/2018
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sung, Sangsoo, Gao, Yuli, Murugesan, Prathab
• Primary Examiner(s): THOMAS-HOMESCU, ANNE L
• Application Number:
  US14/723,324
• Publication Number:
  US 20160350070A1
• Time in Patent Office: 965 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

Online processing of a voice input directed to a voice-enabled electronic device is selectively aborted whenever it is determined that a voice input directed to the voice-enabled electronic device can be successfully processed locally by the device. Doing so may in some instances reduce the latency of responding to a voice input.

Claims

1. A method, comprising:

receiving a voice input with a voice-enabled electronic device; and

in the voice-enabled electronic device, and responsive to receiving at least a portion of the voice input;

initiating both local processing of the at least a portion of the voice input in the voice-enabled electronic device and online processing of the at least a portion of the voice input by the online voice-based query processor in response to receiving the at least a portion of the voice input;

performing local processing of the at least a portion of the voice input to dynamically build at least a portion of a voice action prior to completely receiving the voice input with the voice-enabled electronic device;

determining during the local processing and after initiating the online processing whether the voice action can be completed by the local processing without data received from the online voice-based query processor;

in response to a determination that the voice action can be completed by the local processing without data received from the online voice-based query processor, aborting online processing of the at least a portion of the voice input by the online voice-based query processor and completing building of the voice action with the local processing; and

after aborting online processing and completing building of the voice action with the local processing, receiving data from the online voice-based query processor, verifying the voice action based upon the data received from the online voice-based query processor, and if not verified, updating the voice action using at least a portion of the data received from the online voice-based query processor.

2. The method of claim 1, wherein the voice-enabled electronic device comprises a mobile device configured to communicate with the online voice-based query processor when in communication with a wireless network.

3. The method of claim 1, wherein performing the local processing of the portion of the voice input includes:

converting a digital audio signal of the voice input to text using a streaming voice to text module of the voice-enabled electronic device, wherein the streaming voice to text module dynamically generates a plurality of text tokens from the digital audio signal; and

dynamically building the portion of the voice action from at least a portion of the plurality of text tokens using a streaming semantic processor of the voice-enabled electronic device.

4. The method of claim 1, wherein initiating online processing includes sending data associated with the voice input to the online voice-based query processor.

5. The method of claim 4, wherein sending the data comprises sending at least a portion of a digital audio signal of the voice input, wherein the online voice-based query processor is configured to perform voice to text conversion and semantic processing of the portion of the digital audio signal to generate an online action for the voice input.

6. The method of claim 4, wherein performing the local processing of the portion of the voice input includes generating a plurality of text tokens from a digital audio signal of the voice input, wherein sending the data comprises sending at least a portion of the plurality of text tokens to the online voice-based query processor, and wherein the online voice-based query processor is configured to perform voice to text conversion and semantic processing of the portion of the digital audio signal to generate an online action for the voice input.

7. The method of claim 1, wherein aborting online processing of the portion of the voice input by the online voice-based query processor includes sending data to the online voice-based query processor to terminate online processing of the portion of the voice input by the online voice-based query processor.

8. The method of claim 1, wherein aborting online processing of the portion of the voice input by the online voice-based query processor includes disregarding a response received from the online voice-based query processor.

9. The method of claim 1, wherein aborting online processing of the portion of the voice input by the online voice-based query processor includes bypassing sending of data associated with the voice input to the online voice-based query processor.

10. The method of claim 1, wherein determining during the local processing whether the voice action can be completed by the local processing without data received from the online voice-based query processor includes determining whether an action requested by the voice input can be performed locally in the voice-enabled electronic device.

11. The method of claim 1, further comprising, in response to a determination that the voice action cannot be completed by the local processing without data received from the online voice-based query processor:

deferring completing building of the voice action with the local processing until reception of a response from the online voice-based query processor; and

completing building of the voice action using data from the response.

12. The method of claim 11, further comprising, in response to the determination that the voice action cannot be completed by the local processing without data received from the online voice-based query processor, querying the online voice-based query processor for the data.

13. The method of claim 1, wherein updating the voice action includes correcting the voice action in the voice-enabled electronic device in response to a failure to verify the voice action based upon the data received from the online voice-based query processor.

14. An apparatus including memory and one or more processors operable to execute instructions stored in the memory, comprising instructions to:

receive a voice input with a voice-enabled electronic device; and

in the voice-enabled electronic device, and responsive to receiving at least a portion of the voice input;

initiate both local processing of the at least a portion of the voice input in the voice-enabled electronic device and online processing of the at least a portion of the voice input by the online voice-based query processor in response to receiving the at least a portion of the voice input;

perform local processing of the at least a portion of the voice input to dynamically build at least a portion of a voice action prior to completely receiving the voice input with the voice-enabled electronic device;

determine during the local processing and after initiating the online processing whether the voice action can be completed by the local processing without data received from the online voice-based query processor;

in response to a determination that the voice action can be completed by the local processing without data received from the online voice-based query processor, abort online processing of the at least a portion of the voice input by the online voice-based query processor and complete building of the voice action with the local processing; and

after aborting online processing and completing building of the voice action with the local processing, receive data from the online voice-based query processor, verify the voice action based upon data received from the online voice-based query processor, and if not verified, update the voice action using at least a portion of the data received from the online voice-based query processor.

15. The apparatus of claim 14, wherein the instructions include:

first instructions implementing a streaming voice to text module that converts a digital audio signal of the voice input to text, wherein the first instructions dynamically generate a plurality of text tokens from the digital audio signal; and

second instructions implementing a streaming semantic processor that dynamically builds the portion of the voice action from at least a portion of the plurality of text tokens.

16. The apparatus of claim 14, wherein the instructions that abort online processing of the portion of the voice input by the online voice-based query processor send data to the online voice-based query processor to terminate online processing of the portion of the voice input by the online voice-based query processor.

17. The apparatus of claim 14, wherein the instructions that abort online processing of the portion of the voice input by the online voice-based query processor disregard a response received from the online voice-based query processor.

18. The apparatus of claim 14, wherein the instructions that abort online processing of the portion of the voice input by the online voice-based query processor bypass sending of data associated with the voice input to the online voice-based query processor.

19. A non-transitory computer readable storage medium storing computer instructions executable by one or more processors to perform a method comprising:

receiving a voice input with a voice-enabled electronic device; and

in the voice-enabled electronic device, and responsive to receiving at least a portion of the voice input;

initiating both local processing of the at least a portion of the voice input in the voice-enabled electronic device and online processing of the at least a portion of the voice input by the online voice-based query processor in response to receiving the at least a portion of the voice input;

performing local processing of the at least a portion of the voice input to dynamically build at least a portion of a voice action prior to completely receiving the voice input with the voice-enabled electronic device;

determining during the local processing and after initiating the online processing whether the voice action can be completed by the local processing without data received from the online voice-based query processor;

in response to a determination that the voice action can be completed by the local processing without data received from the online voice-based query processor, aborting online processing of the at least a portion of the voice input by the online voice-based query processor and completing building of the voice action with the local processing; and

after aborting online processing and completing building of the voice action with the local processing, receiving data from the online voice-based query processor, verifying the voice action based upon the data received from the online voice-based query processor, and if not verified, updating the voice action using at least a portion of the data received from the online voice-based query processor.