METHOD TO PROVIDE INCREMENTAL UI RESPONSE BASED ON MULTIPLE ASYNCHRONOUS EVIDENCE ABOUT USER INPUT

• US 20140058732A1
• Filed: 08/21/2012
• Published: 02/27/2014
• Est. Priority Date: 08/21/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Labsky, Martin, Macek, Tomas, Kleindienst, Jan, Kunc, Ladislav
• Granted Patent: US 9,384,736 B2
• Time in Patent Office: Days
• US Class Current: 704/254
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Techniques disclosed herein include systems and methods for managing user interface responses to user input including spoken queries and commands. This includes providing incremental user interface (UI) response based on multiple recognition results about user input that are received with different delays. Such techniques include providing an initial response to a user at an early time, before remote recognition results are available. Systems herein can respond incrementally by initiating an initial UI response based on first recognition results, and then modify the initial UI response after receiving secondary recognition results. Since an initial response begins immediately, instead of waiting for results from all recognizers, it reduces the perceived delay by the user before complete results get rendered to the user.

Claims

1. A computer-implemented method for managing speech recognition response, the computer-implemented method comprising:

receiving a spoken utterance at a client electronic device, the client electronic device having a local automated speech recognizer;

analyzing the spoken utterance using the local automated speech recognizer;

transmitting at least a portion of the spoken utterance over a communication network to a remote automated speech recognizer that analyzes spoken utterances and returns remote speech recognition results;

prior to receiving a remote speech recognition result from the remote automated speech recognizer, initiating a response via a user interface of the client electronic device, the response corresponding to the spoken utterance, at least an initial portion of the response is based on a local speech recognition result from the local automated speech recognizer; and

modifying the response after the response has been initiated and prior to completing delivery of the response via the user interface such that modifications to the response are delivered via the user interface as a portion of the response, the modifications being based on the remote speech recognition result.

2. The computer-implemented method of claim 1, wherein initiating the response includes using a text-to-speech system that begins producing audible speech.

3. The computer-implemented method of claim 2, wherein producing audible speech includes using one or more filler words that convey possession of the speech recognition results.

4. The computer-implemented method of claim 3, wherein modifying the response includes adding words to the response that are produced audibly by the text-to-speech system after the one or more filler words have been audibly produced.

5. The computer-implemented method of claim 1, wherein initiating the response includes using a graphical user interface that begins displaying graphics corresponding to the spoken utterance.

6. The computer-implemented method of claim 5, wherein displaying graphics includes using a sequence of graphics that appears as a commencement of search results.

7. The computer-implemented method of claim 1, wherein the remote speech recognition result is received during delivery of the response via the user interface.

8. The computer-implemented method of claim 1, further comprising:

identifying a predicted delay in receiving remote recognition results; and

basing the initiated response on the predicted delay.

9. The computer-implemented method of claim 8, wherein basing the initiated response based on the predicted delay includes selecting an initial response span based on the predicted delay.

10. The computer-implemented method of claim 9, wherein selecting an initial response span includes selecting a number of words to produce that provides sufficient time to receive the remote speech recognition results.

11. The computer-implemented method of claim 9, wherein basing the initial response based on the predicted delay includes adjusting a speed of speech production by the text-to-speech system based on the predicted delay.

12. The computer-implemented method of claim 8, wherein identifying a predicted delay includes recalculating predicted delays after producing each respective word of an initial response via the user interface.

13. The computer-implemented method of claim 1, further comprising:

classifying a local speech recognition result, from the local automated speech recognizer, into one of multiple reliability classes based on a confidence value representing accuracy of the local speech recognition result; and

wherein the initiated response is selected based on a particular reliability class assigned to the local speech recognition results.

14. The computer-implemented method of claim 1, further comprising identifying that the initiated response conveyed via text-to-speech is incorrect based on remote speech recognition results, and correcting the initiated response using an audible excuse transitions.

15. The computer-implemented method of claim 1, wherein the remote automated speech recognizer is a general-purpose open-domain recognizer.

16. The computer-implemented method of claim 1, wherein the remote automated speech recognizer is a content-specific recognizer.

17. The computer-implemented method of claim 1, wherein the spoken utterance is either a voice query or a voice command.

18. The computer-implemented method of claim 1, wherein the client electronic device is a mobile telephone.

19. A system for managing speech recognition response, the system comprising:

a processor; and

a memory coupled to the processor, the memory storing instructions that, when executed by the processor, causes the system to perform the operations of;

receiving a spoken utterance at a client electronic device, the client electronic device having a local automated speech recognizer;

analyzing the spoken utterance using the local automated speech recognizer;

transmitting at least a portion of the spoken utterance over a communication network to a remote automated speech recognizer that analyzes spoken utterances and returns remote speech recognition results;

prior to receiving a remote speech recognition result from the remote automated speech recognizer, initiating a response via a user interface of the client electronic device, the response corresponding to the spoken utterance, at least an initial portion of the response is based on a local speech recognition result from the local automated speech recognizer; and

modifying the response after the response has been initiated and prior to completing delivery of the response via the user interface such that modifications to the response are delivered via the user interface as a portion of the response, the modifications being based on the remote speech recognition result.

20. A computer program product including a non-transitory computer-storage medium having instructions stored thereon for processing data information, such that the instructions, when carried out by a processing device, cause the processing device to perform the operations of:

receiving a spoken utterance at a client electronic device, the client electronic device having a local automated speech recognizer;

analyzing the spoken utterance using the local automated speech recognizer;

transmitting at least a portion of the spoken utterance over a communication network to a remote automated speech recognizer that analyzes spoken utterances and returns remote speech recognition results;

prior to receiving a remote speech recognition result from the remote automated speech recognizer, initiating a response via a user interface of the client electronic device, the response corresponding to the spoken utterance, at least an initial portion of the response is based on a local speech recognition result from the local automated speech recognizer; and

modifying the response after the response has been initiated and prior to completing delivery of the response via the user interface such that modifications to the response are delivered via the user interface as a portion of the response, the modifications being based on the remote speech recognition result.