System and method of supporting adaptive misrecognition in conversational speech

• US 8,620,659 B2
• Filed: 02/07/2011
• Issued: 12/31/2013
• Est. Priority Date: 08/10/2005
• Status: Active Grant

• Current Assignee: Dialect, LLC
• Original Assignee: VoiceBox Technologies, Inc. (Microsoft Corporation)
• Inventors: Di Cristo, Philippe, Weider, Chris, Kennewick, Robert A.
• Primary Examiner(s): He, Jialong
• Application Number:
  US13/022,370
• Publication Number:
  US 20110131036A1
• Time in Patent Office: 1,058 Days
• Field of Search: 704200-278
• US Class Current: 704/257
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A system and method are provided for receiving speech and/or non-speech communications of natural language questions and/or commands and executing the questions and/or commands. The invention provides a conversational human-machine interface that includes a conversational speech analyzer, a general cognitive model, an environmental model, and a personalized cognitive model to determine context, domain knowledge, and invoke prior information to interpret a spoken utterance or a received non-spoken message. The system and method creates, stores, and uses extensive personal profile information for each user, thereby improving the reliability of determining the context of the speech or non-speech communication and presenting the expected results for a particular question or command.

Claims

1. A system for processing natural language utterances, comprising:

one or more processors configured to;

receive a first input of a user that comprises a natural language utterance;

generate an interpretation of the natural language utterance based on one or more recognized words of the natural language utterance;

generate a request based on the interpretation of the natural language utterance;

invoke a domain agent to process the request;

determine whether the interpretation of the natural language utterance is correct or incorrect based on whether a second input is received from the user within an amount of time shorter than an expected execution time associated with the request;

update a personalized cognitive model associated with the user based on the determination of whether the interpretation is correct or incorrect, wherein the personalized cognitive model is based on a tracking of a pattern of interactions between the user and the system; and

predict one or more actions associated with the user based on the updated personalized cognitive model.

2. The system of claim 1, wherein the one or more processors are configured to generate an event indicative of misrecognition in response to a determination that the interpretation of the natural language utterance is incorrect.

3. The system of claim 2, wherein the one or more processors are configured to:

analyze the event to determine how the natural language utterance was incorrectly interpreted; and

determine one or more tuning parameters based on how the natural language utterance was incorrectly interpreted, wherein the one or more tuning parameters are used to improve interpretations of subsequent natural language utterances relating to the request.

4. The system of claim 1, wherein the predicted actions associated with the user are used to improve interpretations of one or more subsequent natural language utterances.

5. The system of claim 1, wherein the one or more processors are configured to track interaction patterns with the system over time for a plurality of users.

6. The system of claim 5, wherein the one or more processors are configured to generate a generalized cognitive model associated with the plurality of users based on the interaction patterns tracked for the plurality of users, and wherein the generalized cognitive model includes a statistical abstract that corresponds to the interaction patterns.

7. The system of claim 6, wherein one or more processors are configured to predict one or more other actions associated with the user based on the generalized cognitive model in response to receiving one or more subsequent natural language utterances from the user, wherein the predicted other actions associated with the user are used to improve interpretations of the one or more subsequent natural language utterances.

8. The system of claim 1, wherein the one or more processors are configured to generate an environmental model that includes information associated with at least one of an environmental condition or surrounding associated with the user that provided the natural language utterance.

9. The system of claim 8, wherein the environmental condition or surrounding identifies a level of noise associated with an environment or the surrounding of the user.

10. The system of claim 8, wherein the environmental model provides one or more of context, domain knowledge, preferences, or cognitive qualities to enhance the interpretation of the natural language utterance.

11. The system of claim 1, wherein the one or more processors are configured to:

determine a most likely context for the natural language utterance;

compare one or more text combinations against one or more grammar expression entries in a context description grammar to identify one or more contexts that completely or partially match the one or more text combinations;

provide a relevance score for each of the identified matching contexts;

select the matching context having a highest score as the most likely context for the natural language utterance, wherein the domain agent is associated with the selected context;

communicate the request to the domain agent associated with the selected context; and

generate a response to the request using content gathered as a result of the domain agent processing the request, wherein the response arranges the content in an order based on the relevance scores for the identified matching contexts.

12. The system of claim 11, wherein the response includes an aggregation of the content gathered as a result of the domain agent processing the request.

13. The system of claim 11, wherein the one or more processors are configured to:

determine a personality based on the identified matching contexts, the domain agent processing the request, or a user profile associated with the user; and

format the response based on the personality.

14. The system of claim 11,

wherein the one or more processors are configured to compare the text combinations against a context stack that stores one or more expected contexts to identify the one or more contexts.

15. The system of claim 11, wherein the one or more processors are configured to apply prior probabilities or fuzzy possibilities to at least one of keyword matching, user profiles, or a dialog history to identify the one or more contexts.

16. The system of claim 11, wherein the domain agent is configured to direct a query to at least one of a local information source or a network information source to process the request.

17. The system of claim 16, wherein the domain agent is configured to evaluate a plurality of responses to the query to process the request.

18. The system of claim 11, wherein the domain agent is configured to direct a command to at least one of a local device or a remote device to process the request.

19. The system of claim 1, wherein the second input includes a follow-up request associated with a same context as the request being processed by the domain agent.

20. The system of claim 1, wherein the one or more processors are configured to determine that the interpretation of the natural language utterance was incorrect in response to a determination that the second input includes a request to stop the request being processed by the domain agent.

21. The system of claim 1, wherein the one or more processors are configured to determine that the interpretation of the natural language utterance was incorrect in response to the user repeating the natural language utterance.

22. The system of claim 1, wherein the one or more processors are configured to:

receive a non-speech input relating to the natural language utterance;

transcribe the non-speech input to create a non-speech-based transcription; and

merge the recognized words and the non-speech-based transcription to create a merged transcription, wherein the interpretation is generated further based on the merged transcription.

23. A method of processing natural language utterances, the method being implemented on a computer system that includes one or more processors, the method comprising:

receiving a first input of a user that comprises a natural language utterance;

generating an interpretation of the natural language utterance based on one or more recognized words of the natural language utterances;

generating a request based on the interpretation of the natural language utterance;

invoking a domain agent to process the request;

monitoring one or more actions associated with the domain agent processing the request; and

determining whether the interpretation of the natural language utterance is correct or incorrect based on whether a second input is received from the user within an amount of time shorter than an expected execution time associated with the request;

updating a personalized cognitive model associated with the user based on the determination of whether the interpretation is correct or incorrect, wherein the personalized cognitive model is based on a tracking of a pattern of interactions between the user and the system; and

predicting one or more actions associated with the user based on the updated personalized cognitive model.

24. The method of claim 23, further comprising:

generating an event indicative of misrecognition in response to a determination that the interpretation of the natural language utterance is incorrect.

25. The method of claim 24, further comprising:

analyzing the event to determine how the natural language utterance was incorrectly interpreted; and

determining one or more tuning parameters based on how the natural language utterance was incorrectly interpreted, wherein the one or more tuning parameters are used to improve interpretations of subsequent natural language utterances relating to the request.

26. The method of claim 23, further comprising:

wherein the predicted actions associated with the user are used to improve interpretations of one or more subsequent natural language utterances.

27. The method of claim 23, further comprising tracking interaction patterns over time for a plurality of users.

28. The method of claim 27, further comprising generating a generalized cognitive model associated with the plurality of users based on the interaction patterns tracked for the plurality of users, wherein the generalized cognitive model includes a statistical abstract that corresponds to the interaction patterns.

29. The method of claim 28, further comprising predicting one or more other actions associated with the user based on the generalized cognitive model in response to receiving one or more subsequent natural language utterances from the user, wherein the predicted other actions associated with the user are used to improve interpretations of the one or more subsequent natural language utterances.

30. The method of claim 23, further comprising generating an environmental model that includes information associated with at least one of an environmental condition or surrounding associated with the user that provided the natural language utterance.

31. The method of claim 30, wherein the environmental condition or surrounding identifies a level of noise associated with an environment or the surrounding of the user.

32. The method of claim 30, wherein the environmental model provides one or more of context, domain knowledge, preferences, or cognitive qualities to enhance the interpretation of the natural language utterance.

33. The method of claim 23, further comprising:

determining a most likely context for the natural language utterances;

comparing one or more text combinations against one or more grammar expression entries in a context description grammar to identify one or more contexts that completely or partially match the one or more text combinations;

providing a relevance score for each of identified matching contexts;

selecting the matching context having a highest score as the most likely context for the natural language utterance, wherein the domain agent is associated with the selected context;

communicating the request to the domain agent associated with the selected context; and

generating the response to the request using content gathered as a result of the domain agent processing the request, wherein the response arranges the content in an order based on the relevance scores for the identified matching contexts.

34. The method of claim 33, wherein the response includes an aggregation of the content gathered as a result of the domain agent processing the request.

35. The method of claim 33, further comprising:

determining a personality based on the identified matching contexts, the domain agent processing the request, or a user profile associated with the user; and

formatting the response based on the personality using a personality module.

36. The method of claim 33, further comprising:

comparing the text combinations against a context stack that stores one or more expected contexts to identify the one or more contexts.

37. The method of claim 33, further comprising:

applying prior probabilities or fuzzy possibilities to at least one of keyword matching, user profiles, or a dialog history to identify the one or more contexts.

38. The method of claim 23, further comprising:

determining that the interpretation of the natural language utterance was incorrect in response to a determination that the second input includes a request to stop the request being processed by the domain agent.

39. The method of claim 23, further comprising:

determining that the interpretation of the natural language utterance was incorrect in response to the user repeating the natural language utterance.

40. The method of claim 23, further comprising:

receiving a non-speech input relating to the natural language utterance at the device;

transcribing the non-speech input to create a non-speech-based transcription; and

merging the recognized words and the non-speech-based transcription to create a merged transcription, wherein the interpretation is generated further based on the merged transcription.

41. A method of processing natural language utterances, the method being implemented by a computer system that includes one or more processors executing one or more computer program instructions which, when executed, perform the method, the method comprising:

receiving a first input of a user that comprises a natural language utterance;

generating an interpretation of the natural language utterance based on one or more recognized words of the natural language utterance;

generating a request based on the interpretation;

transmitting the request to a domain agent for processing;

determining whether the interpretation is correct or incorrect based on whether a second input is received from the user within an amount of time shorter than an expected execution time associated with the request;

updating a personalized cognitive model associated with the user based on the determination of whether the interpretation is correct or incorrect, wherein the personalized cognitive model is based on a tracking of a pattern of interactions between the user and the system; and

predicting one or more actions associated with the user based on the updated personalized cognitive model.

42. A method of processing natural language utterances, the method being implemented by a computer system that includes one or more processors executing one or more computer program instructions which, when executed, perform the method, the method comprising:

receiving a first input of a user that comprises a natural language utterance;

generating an interpretation of the natural language utterance based on one or more recognized words of the natural language utterance;

generating a request based on the interpretation;

transmitting the request to a domain agent for processing;

determining whether a personalized cognitive model associated with the user includes sufficient information for predicting one or more subsequent actions associated with the user, wherein the personalized cognitive model is generated based on a tracking of a pattern of interactions between the user and the system, and wherein the one or more subsequent actions include one or more actions predicted to occur after receiving the first input; and

predicting the one or more subsequent actions based on a generalized cognitive model in response to a determination that the personalized cognitive model does not include the sufficient information, wherein the generalized cognitive model is generated based on a tracking of patterns of interactions between a plurality of users and the system.

43. The method of claim 42, further comprising:

determining whether the interpretation is correct or incorrect based on whether a second input is received from the user within an amount of time shorter than an expected execution time associated with the request; and

updating the personalized cognitive model based on the determination of whether the interpretation is correct or incorrect.