Controlling the listening horizon of an automatic speech recognition system for use in handsfree conversational dialogue

• US 7,716,057 B2
• Filed: 06/15/2007
• Issued: 05/11/2010
• Est. Priority Date: 05/17/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Horvitz, Eric J.
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): GODBOLD, DOUGLAS
• Application Number:
  US11/764,082
• Publication Number:
  US 20070239459A1
• Time in Patent Office: 1,061 Days
• Field of Search: 704/231, 704/270, 704/275
• US Class Current: 704/275
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Conversational dialog with a computer or other processor-based device without requiring push-to-talk functionality. In one embodiment, a computer-implemented method first determines that a user desires to engage in a dialog. Based thereon the method turns on a speech recognition functionality for a period of time referred to as a listening horizon. Upon the listening horizon expiring, the method turns off the speech recognition functionality.

Claims

1. A hand held mobile system comprising:

at least one processor;

a plurality of components executable on the at least one processor, the plurality of components comprising;

a dialog component to engage in dialog with a user, the dialog component determining whether an utterance of the user recognized with a confidence level greater than a threshold is associated with a pattern of dysfluency or reflection; and

an application component that determines that a user desires to engage in a dialog, the application turning on the dialog component for a listening horizon, the length of the listening horizon is a function of the subject matter of the dialog, wherein the listening horizon is extended when the utterance is determined to be associated with a pattern of dysfluency or reflection.

2. The system of claim 1, wherein the application component determines that a user desires to engage in a dialog comprises performing a cost-benefit analysis to determine whether engaging in a dialog comprises a highest expected utility action.

3. The system of claim 2, wherein determining that a user desires to engage in a dialog further comprises initially inferring a probability that the user desires an automated service.

4. The system of claim 1, further comprising prior to turning on a speech recognition functionality, the dialog component engaging the user with a question.

5. The system of claim 4, wherein engaging the user with a question comprises engaging the user with a question regarding a desire for an automated service.

6. The system of claim 4, wherein engaging the user with a question comprises the dialog component displaying an automated assistant to ask the question.

7. The system of claim 1, wherein the dialog component determines that the user desires an automated service.

8. The system of claim 7, wherein the length of the listening horizon is further determined based upon the complexity of the automated service.

9. The system of claim 7, wherein the dialog component determines that there is hearing difficulty.

10. The system of claim 9, wherein the dialog component determines that there is hearing difficulty based upon a predetermined number of times that the user makes an utterance that the dialog component rates as having a confidence level lower than a predetermined threshold.

11. The system of claim 1, further comprising the application component turning off the dialog component, the dialog component detecting an utterance from the user during the listening horizon and determining a confidence level of the utterance prior to being turned off.

12. The system of claim 11, further comprising prior to turning off the dialog component, upon determining that the confidence level of the utterance is greater than a predetermined threshold, the dialog component displaying an automated assistant indicating understanding.

13. The system of claim 11, further comprising prior to turning off the dialog component, upon determining that the confidence level of the utterance is greater than a predetermined threshold and the utterance indicates deliberation, the dialog component displaying an automated assistant indicating increased attentiveness and continuing to detect an utterance from the user during the listening horizon.

14. The system of claim 11, further comprising prior to turning off the dialog component, upon determining that the confidence level of the utterance is less than a predetermined threshold, the dialog component displaying an automated assistant indicating hearing difficulty and continuing to detect an utterance from the user during the listening horizon.

15. The system of claim 11, further comprising prior to turning off the dialog component, upon determining that the confidence level of the utterance is less than a predetermined threshold, and based on continued hearing difficulty, the dialog component displaying an automated assistant indicating failure to hear.

16. The system of claim 11, further comprising prior to turning off the dialog component, upon failure to detect an utterance from the user and upon expiration of the listening horizon, the dialog component displaying an automated assistant indicating recognition that the user is busy.

17. A computer implemented method comprising:

with at least one processor, performing steps comprising;

determining that a user desires to engage in a dialog;

turning on a speech recognition function with a listening horizon, the length of the listening horizon is initially set as a function of the subject matter of the dialog;

listening for an utterance of the user until an utterance is detected or the listening horizon expires;

determining whether the detected utterance is recognized with a confidence above a threshold;

when the utterance is recognized with a confidence above the threshold;

determining whether the utterance represents a deliberation; and

when the utterance does not represent a deliberation, turning off the speech recognition function; and

when the utterance represents a deliberation, resuming listening for an utterance until the listening horizon expires; and

when the utterance is not recognized with a confidence above the threshold;

when a number of utterances have been received since the speech recognition function was turned on exceeds a second threshold, outputting an indication of a hearing difficulty and turning off the speech recognition function;

when the number of utterances that have been received since the speech recognition function was turned on is below the second threshold, resuming listening for an utterance until the listening horizon expires.

18. The method of claim 17, wherein the initial listening horizon is set based on prior difficulty recognizing utterances from the user.

19. The method of claim 17, wherein the listening horizon is extended when it is determined that the detected utterance represents a deliberation.

20. A mobile device comprising:

at least one processor;

means for determining that a user desires to engage in a dialog; and

means, executable on the at least one processor, the means comprising;

means for determining whether an utterance of the user recognized with a confidence level greater than a threshold is associated with a pattern representative of user deliberation;

means for turning on a speech recognition functionality for a listening horizon upon determining that the user desires to engage in a dialog, the length of the listening horizon is a function of the subject matter of the dialog;

means for extending the listening horizon when the recognized utterance is associated with a pattern representative of user deliberation; and

,

means for turning off the speech recognition functionality after the listening horizon has expired.