Signalling and controlling the status of an automatic speech recognition system for use in handsfree conversational dialogue

• US 6,434,527 B1
• Filed: 05/17/1999
• Issued: 08/13/2002
• Est. Priority Date: 05/17/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Horvitz, Eric
• Primary Examiner(s): Banks-Harold, Marsha D.
• Assistant Examiner(s): Storm, Donald L.
• Application Number:
  US09/312,679
• Time in Patent Office: 1,184 Days
• Field of Search: 704/200, 704/270, 704/275, 704/251
• US Class Current: 704/275
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Conversational dialog with a computer or other processor-based device without requiring push-to-talk functionality. In one embodiment, a computer-implemented method first determines that a user desires to engage in a dialog. Based thereon the method turns on a speech recognition functionality for a period of time referred to as a listening horizon. Upon the listening horizon expiring, the method turns off the speech recognition functionality.

Claims

1. A computer-implemented method comprising:

determining that a user desires to engage in a dialog;

upon determining that the user desires to engage in a dialog, turning on a speech recognition functionality for a listening horizon, wherein turning on a speech recognition functionality for a listening horizon comprises determining a length of the listening horizon; and

,

turning off the speech recognition functionality after the listening horizon has expired.

2. The method of claim 1, wherein determining that a user desires to engage in a dialog comprises performing a cost-benefit analysis to determine whether engaging in a dialog comprises a highest expected utility action.

3. The method of claim 2, wherein determining that a user desires to engage in a dialog further comprises initially inferring a probability that the user desires an automated service.

4. The method of claim 1, further comprising prior to turning on a speech recognition functionality, engaging the user with a question.

5. The method of claim 4, wherein engaging the user with a question comprises engaging the user with a question regarding a desire for an automated service.

6. The method of claim 4, wherein engaging the user with a question comprises displaying an automated assistant asking the question.

7. The method of claim 1, wherein determining a length of the listening horizon comprises determining the length of the listening horizon as a function of at least an inferred probability that the user desires automated service.

8. The method of claim 1, wherein determining a length of the listening horizon comprises determining the length of the listening horizon as a function of at least an inferred probability that the user desires automated service and an acute listening history.

9. The method of claim 1, wherein turning on a speech recognition functionality comprises displaying an automated assistant having listening-for-user-utterances indications.

10. The method of claim 1, further comprising prior to turning off the speech recognition functionality,

detecting an utterance from the user during the listening horizon; and,

determining a confidence level of the utterance.

11. The method of claim 10, further comprising prior to turning off the speech recognition functionality, upon determining that the confidence level of the utterance is greater than a predetermined threshold, displaying an automated assistant indicating understanding and proceeding to turning off the speech recognition functionality.

12. The method of claim 10, further comprising prior to turning off the speech recognition functionality, upon determining that the confidence level of the utterance is greater than a predetermined threshold and the utterance indicates deliberation, displaying an automated assistant indicating increased attentiveness and continuing to detect an utterance from the user during the listening horizon.

13. The method of claim 10, further comprising prior to turning off the speech recognition functionality, upon determining that the confidence level of the utterance is less than a predetermined threshold, displaying an automated assistant indicating hearing difficulty and continuing to detect an utterance from the user during the listening horizon.

14. The method of claim 10, further comprising prior to turning off the speech recognition functionality, upon determining that the confidence level of the utterance is less than a predetermined threshold, and based on continued hearing difficulty, displaying an automated assistant indicating failure to hear and proceeding to turning off the speech recognition functionality.

15. The method of claim 10, further comprising prior to turning off the speech recognition functionality, upon failure to detect an utterance from the user and upon expiration of the listening horizon, displaying an automated assistant indicating sensitivity that the user is busy and proceeding to turning off the speech recognition functionality.

16. The method of claim 1, further comprising modifying the length of the initially determined listening horizon.

17. A computer-implemented method comprising:

determining that a user desires to engage in a dialog;

upon determining that the user desires to engage in a dialog,

engaging the user with a question;

displaying an automated assistant asking the question;

determining a length of a listening horizon;

turning on a speech recognition functionality for the listening horizon;

during the listening horizon,

detecting an utterance from the user;

determining a confidence level of the utterance; and

,

no later than after expiration of the listening horizon,

removing the automated assistant;

turning off the speech recognition functionality.

18. The method of claim 17, wherein determining that a user desires to engage in a dialog comprises:

inferring a probability that the user desires an automated service; and

,

performing a cost-benefit analysis to determine whether engaging in a dialog comprises a highest expected utility action.

19. The method of claim 17, wherein engaging the user with a question comprises engaging the user with a question regarding a desire for an automated service.

20. The method of claim 17, wherein determining a length of the listening horizon is a function of at least an inferred probability that the user desires automated service and an acute listening history.

21. The method of claim 17, further comprising subsequent to turning on the speech recognition functionality, rendering the automated assistant as having listen-for-user-utterances indications.

22. The method of claim 17, further comprising after determining a confidence level of the utterance, upon determining that the confidence level is greater than a predetermined threshold, rendering the automated assistant as indicating understanding and proceeding to turning off the speech recognition functionality.

23. The method of claim 17, further comprising after determining a confidence level of the utterance, upon determining that the confidence level of the utterance is greater than a predetermined threshold and the utterance indicates deliberation, rendering the automated assistant as indicating increased attentiveness and continuing to detecting an utterance from the user during the listening horizon.

24. The method of claim 17, further comprising after determining a confidence level of the utterance, upon determining that the confidence level of the utterance is less than a predetermined threshold, rendering the automated assistant as indicating hearing difficulty and continuing to detecting an utterance from the user during the listening horizon.

25. The method of claim 17, further comprising after determining a confidence level of the utterance, upon determining that the confidence level of the utterance is less than a predetermined threshold, and based on continued hearing difficulty, rendering the automated assistant as indicating failure to hear and proceeding to turning off the speech recognition functionality.

26. The method of claim 17, prior to turning off the speech recognition functionality upon failure to detect an utterance from the user and upon expiration of the listening horizon, rendering the automated assistant as indicating sensitivity that the user is busy.

27. The method of claim 17, further comprising modifying the length of the initially determined listening horizon.

28. A machine-readable medium having instructions stored thereon for execution by a processor to cause performance of a method comprising:

determining that a user desires to engage in a dialog;

upon determining that the user desires to engage in a dialog, turning on a speech recognition functionality for a listening horizon, wherein turning on a speech recognition functionality for a listening horizon comprises determining a length of the listening horizon; and

,

turning off the speech recognition functionality after the listening horizon has expired.

29. The medium of claim 28, wherein determining that a user desires to engage in a dialog comprises:

inferring a probability that the user desires an automated service; and

,

performing a cost-benefit analysis to determine whether engaging in a dialog comprises a highest expected utility action.

30. The medium of claim 28, further comprising prior to turning on a speech recognition functionality, engaging the user with a question.

31. The medium of claim 28, further comprising prior to turning off the speech recognition functionality,

detecting an utterance from the user during the listening horizon;

and,

determining a confidence level of the utterance.

32. The method of claim 28, wherein the method further comprises modifying the length of the initially determined listening horizon.

33. A machine-readable medium having instructions stored thereon for execution by a processor to cause performance of a method comprising:

determining that a user desires to engage in a dialog;

upon determining that the user desires to engage in a dialog,

engaging the user with a question;

displaying an automated assistant asking the question;

determining an initial length of a listening horizon;

turning on a speech recognition functionality for a listening horizon;

during the listening horizon,

detecting an utterance from the user;

determining a confidence level of the utterance; and

,

no later than after expiration of the listening horizon,

removing the automated assistant;

turning off the speech recognition functionality.

34. The medium of claim 33, further comprising subsequent to turning on the speech recognition functionality, rendering the automated assistant as having listen-for-user-utterances indications.

35. The medium of claim 33, further comprising after determining a confidence level of the utterance, upon determining that the confidence level is greater than a predetermined threshold, rendering the automated assistant as indicating understanding and proceeding to turning off the speech recognition functionality.

36. The medium of claim 33, further comprising after determining a confidence level of the utterance, upon determining that the confidence level of the utterance is greater than a predetermined threshold and the utterance indicates deliberation, rendering the automated assistant as indicating increased attentiveness and continuing to detecting an utterance from the user during the listening horizon.

37. The medium of claim 33, further comprising after determining a confidence level of the utterance, upon determining that the confidence level of the utterance is less than a predetermined threshold, rendering the automated assistant as indicating hearing difficulty and continuing to detecting an utterance from the user during the listening horizon.

38. The medium of claim 33, further comprising after determining a confidence level of the utterance, upon determining that the confidence level of the utterance is less than a predetermined threshold, and based on continued hearing difficulty, rendering the automated assistant as indicating failure to hear and proceeding to turning off the speech recognition functionality.

39. The medium of claim 33, prior to turning off the speech recognition functionality upon failure to detect an utterance from the user and upon expiration of the listening horizon, rendering the automated assistant as indicating sensitivity that the user is busy.

40. The medium of claim 33, wherein the method further comprises modifying the length of the initially determined listening horizon.