System and method of supporting adaptive misrecognition conversational speech

• US 8,332,224 B2
• Filed: 10/01/2009
• Issued: 12/11/2012
• Est. Priority Date: 08/10/2005
• Status: Active Grant

• Current Assignee: Dialect, LLC
• Original Assignee: VoiceBox Technologies, Inc. (Microsoft Corporation)
• Inventors: Di Cristo, Philippe, Weider, Chris, Kennewick, Robert A.
• Primary Examiner(s): SKED, MATTHEW J
• Application Number:
  US12/571,795
• Publication Number:
  US 20100023320A1
• Time in Patent Office: 1,167 Days
• Field of Search: None
• US Class Current: 704/257
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A system and method are provided for receiving speech and/or non-speech communications of natural language questions and/or commands and executing the questions and/or commands. The invention provides a conversational human-machine interface that includes a conversational speech analyzer, a general cognitive model, an environmental model, and a personalized cognitive model to determine context, domain knowledge, and invoke prior information to interpret a spoken utterance or a received non-spoken message. The system and method creates, stores and uses extensive personal profile information for each user, thereby improving the reliability of determining the context of the speech or non-speech communication and presenting the expected results for a particular question or command.

Claims

1. A system for processing natural language utterances, comprising:

a multimodal device configured to receive a natural language utterance;

a speech recognition engine configured to recognize one or more words from the natural language utterance;

a parser configured to generate an interpretation of the natural language utterance from the one or more recognized words and generate a request based on the interpretation of the natural language utterance;

a domain agent configured to process the generated request;

an adaptive misrecognition engine configured to monitor one or more actions associated with the domain agent processing the request, determine whether the interpretation of the natural language utterance is correct or incorrect based on the one or more monitored actions, and generate an unrecognized event in response to determining that the interpretation of the natural language utterance is incorrect; and

an analyzer configured to;

track an interaction pattern with the system over time for a user that provided the natural language utterance;

generate a personalized cognitive model for the user based on the interaction pattern tracked for the user;

use the personalized cognitive model to predict the one or more actions associated with the domain agent processing the request; and

update the personalized cognitive model based on a frequency of incorrect interpretations for the request.

2. The system of claim 1, wherein the analyzer is further configured to track interaction patterns with the system over time for a plurality users.

3. The system of claim 2, wherein the analyzer is further configured to generate a generalized cognitive model for the plurality of users based on the interaction patterns tracked for the plurality of users, wherein the generalized cognitive model includes a statistical abstract that corresponds to the tracked interaction patterns.

4. The system of claim 3, wherein the analyzer is further configured to use the generalized cognitive model to predict the one or more actions associated with the domain agent processing the request.

5. The system of claim 1, wherein the analyzer is further configured to generate an environmental model that includes information associated with at least one of environmental conditions or surroundings associated with the user.

6. The system of claim 5, wherein the environmental conditions or surroundings include one or more of a global position of the user, movement information associated with the user, quiet or noisy conditions associated with an environment of the user, or a vicinity to one or more voice-enabled devices.

7. The system of claim 5, wherein the environmental model provides one or more of context, domain knowledge, preferences, or cognitive qualities to enhance the interpretation of the natural language utterance.

8. The system of claim 1, wherein the analyzer is further configured to:

analyze the unrecognized event to determine how the natural language utterance was incorrectly interpreted; and

determine one or more tuning parameters for at least one of the speech recognition engine or the parser based on how the natural language utterance was incorrectly interpreted, wherein the tuning parameters are used to improve interpretations of subsequent natural language utterances relating to the request.

9. The system of claim 1, further comprising:

a knowledge-enhanced speech recognition engine configured to determine a most likely context for the natural language utterance, wherein the knowledge-enhanced speech recognition engine is further configured to;

compare one or more text combinations against one or more grammar expression entries in a context description grammar to identify one or more contexts that completely or partially match the one or more text combinations;

provide a relevance score for each of the identified matching contexts; and

select the matching context having a highest score as the most likely context for the natural language utterance, wherein the domain agent configured to process the generated request is associated with the selected context; and

a response generating module configured to;

communicate the request to the domain agent associated with the selected context; and

generate a response to the natural language utterance using content gathered as a result of the domain agent processing the request, wherein the response arranges the content in an order based on the relevance scores for the identified matching contexts.

10. The system of claim 9, wherein the response generated by the response generating module includes an aggregation of the content gathered as a result of the domain agent processing the request.

11. The system of claim 9, further comprising a personality module configured to format the response.

12. The system of claim 9, wherein the knowledge-enhanced speech recognition engine is further configured to compare the text combinations against a context stack that stores one or more expected contexts to identify the one or more contexts.

13. The system of claim 9, wherein the knowledge-enhanced speech recognition engine is further configured to apply prior probabilities or fuzzy possibilities to at least one of keyword matching, user profiles, or a dialog history to identify the one or more contexts.

14. The system of claim 9, wherein the domain agent is further configured to direct a query to at least one of a local information source or a network information source to process the request.

15. The system of claim 14, wherein the domain agent is further configured to evaluate a plurality of responses to the query to process the request.

16. The system of claim 9, wherein the domain agent is further configured to direct a command to at least one of a local device or a remote device to process the request.

17. The system of claim 1, wherein the multimodal device includes at least one of a personal digital assistant, a cellular telephone, a portable computer, or a desktop computer.

18. The system of claim 1, wherein the multimodal device is further configured to subsequently receive one or more follow-up multimodal inputs.

19. The system of claim 18, wherein the speech recognition engine is further configured to recognize one or more words from a natural language utterance provided in the follow-up multimodal input, and wherein the parser is further configured to generate an interpretation of the follow-up multimodal input from the one or more words recognized from the natural language utterance provided in the follow-up multimodal input.

20. The system of claim 19, wherein the follow-up multimodal input includes a follow-up request associated with a same context as the request being processed by the domain agent.

21. The system of claim 1, wherein the adaptive misrecognition engine determines that the interpretation of the natural language utterance was incorrect in response to the user providing a subsequent request to stop the request being processed by the domain agent.

22. The system of claim 1, wherein the adaptive misrecognition engine determines that the interpretation of the natural language utterance was incorrect in response to the user repeating the natural language utterance.

23. The system of claim 1, wherein the multimodal device is further configured to receive a non-speech input relating to the natural language utterance, and wherein the system further comprises:

a transcription module configured to transcribe the non-speech input to create a non-speech-based transcription; and

a merging module configured to merge the recognized words and the non-speech-based transcription to create a merged transcription, wherein the parser is further configured to generate the interpretation of the natural language utterance from the merged transcription.

24. A system for processing natural language utterances, comprising:

a multimodal device configured to receive a natural language utterance;

a speech recognition engine configured to recognize one or more words from the natural language utterance;

a parser configured to generate a plurality of interpretations of the natural language utterance and generate a request based on a best interpretation selected from the plurality of interpretations of the natural language utterance;

a domain agent configured to process the generated request;

an adaptive misrecognition engine configured to monitor one or more actions associated with the domain agent processing the request, determine whether the best interpretation of the natural language utterance is correct or incorrect based on the one or more monitored actions, and generate an unrecognized event in response to determining that the best interpretation of the natural language utterance is incorrect; and

an analyzer configured to;

track an interaction pattern with the system over time for a user that provided the natural language utterance;

generate a personalized cognitive model for the user based on the interaction pattern tracked for the user; and

use the personalized cognitive model to predict the one or more actions associated with the domain agent processing the request and select a next best interpretation from the plurality of interpretations in response to the adaptive misrecognition engine determining that the best interpretation selected by the parser is incorrect.

25. A system for processing natural language utterances, comprising:

a multimodal device configured to receive a natural language utterance;

a speech recognition engine configured to recognize one or more words from the natural language utterance;

a parser configured to generate an interpretation of the natural language utterance from the one or more recognized words and generate a request based on the interpretation of the natural language utterance;

a domain agent configured to process the generated request;

an adaptive misrecognition engine configured to monitor one or more actions associated with the domain agent processing the request, determine whether the interpretation of the natural language utterance is correct or incorrect based on the one or more monitored actions, and generate an unrecognized event in response to determining that the interpretation of the natural language utterance is incorrect; and

an analyzer configured to;

track interaction patterns with the system over time for a plurality of users, including a user that provided the natural language utterance;

generate a generalized cognitive model for the plurality of users, wherein the generalized cognitive model includes a statistical abstract that corresponds to the interaction patterns tracked for the plurality of users; and

update the generalized cognitive model based on a frequency of incorrect interpretations for the request.

26. A method for processing natural language utterances, comprising:

receiving a natural language utterance at a multimodal device;

recognizing one or more words from the natural language utterance using a speech recognition engine coupled to the multimodal device;

generating an interpretation of the natural language utterance from the one or more recognized words using a parser coupled to the multimodal device, wherein the parser generates a request based on the interpretation of the natural language utterance;

invoking a domain agent configured to process the generated request;

monitoring one or more actions associated with the domain agent processing the request using an adaptive misrecognition engine;

determining, at the adaptive misrecognition engine, whether the interpretation of the natural language utterance is correct or incorrect based on the one or more monitored actions, wherein the adaptive misrecognition engine generates an unrecognized event in response determining that the interpretation of the natural language utterance is incorrect;

tracking an interaction pattern over time for a user that provided the natural language utterance using an analyzer associated with the adaptive misrecognition engine;

generating, at the analyzer, a personalized cognitive model for the user based on the interaction pattern tracked for the user;

using the personalized cognitive model to predict the one or more actions associated with the domain agent processing the request; and

updating the personalized cognitive model using the analyzer based on a frequency of incorrect interpretations for the request.

27. The method of claim 26, further comprising tracking interaction patterns over time for a plurality of users.

28. The method of claim 27, further comprising generating a generalized cognitive model for the plurality of users based on the interaction patterns tracked for the plurality of users, wherein the generalized cognitive model includes a statistical abstract that corresponds to the tracked interaction patterns.

29. The method of claim 28, further comprising using the generalized cognitive model to predict the one or more actions associated with the domain agent processing the request.

30. The method of claim 26, further comprising generating an environmental model that includes information associated with at least one of environmental conditions or surroundings associated with the user.

31. The method of claim 30, wherein the environmental conditions or surroundings include one or more of a global position of the user, movement information associated with the user, quiet or noisy conditions associated with an environment of the user, or a vicinity to one or more voice-enabled devices.

32. The method of claim 30, wherein the environmental model provides one or more of context, domain knowledge, preferences, or cognitive qualities to enhance the interpretation of the natural language utterance.

33. The method of claim 26, further comprising:

analyzing the unrecognized event to determine how the natural language utterance was incorrectly interpreted; and

determining one or more tuning parameters for at least one of the speech recognition engine or the parser based on how the natural language utterance was incorrectly interpreted, wherein the tuning parameters are used to improve interpretations of subsequent natural language utterances relating to the request.

34. The method of claim 26, further comprising determining a most likely context for the natural language utterance using a knowledge-enhanced speech recognition engine, wherein determining the most likely context further includes:

comparing one or more text combinations against one or more grammar expression entries in a context description grammar to identify one or more contexts that completely or partially match the one or more text combinations;

providing a relevance score for each of identified matching contexts;

selecting the matching context having a highest score as the most likely context for the natural language utterance, wherein the domain agent configured to process the generated request is associated with the selected context;

communicating the request to the domain agent associated with the selected context; and

generating a response to the natural language utterance using content gathered as a result of the domain agent processing the request, wherein the response arranges the content in an order based on the relevance scores for the identified matching contexts.

35. The method of claim 34, wherein the response includes an aggregation of the content gathered as a result of the domain agent processing the request.

36. The method of claim 34, further comprising formatting the response using a personality module.

37. The method of claim 34, wherein the knowledge-enhanced speech recognition engine further compares the text combinations against a context stack that stores one or more expected contexts to identify the one or more contexts.

38. The method of claim 37, wherein the knowledge-enhanced speech recognition engine further applies prior probabilities or fuzzy possibilities to at least one of keyword matching, user profiles, or a dialog history to identify the one or more contexts.

39. The method of claim 26, further comprising receiving one or more follow-up multimodal inputs at the multimodal device.

40. The method of claim 26, wherein the adaptive misrecognition engine determines that the interpretation of the natural language utterance was incorrect in response to the user providing a subsequent request to stop the request being processed by the domain agent.

41. The method of claim 26, wherein the adaptive misrecognition engine determines that the interpretation of the natural language utterance was incorrect in response to the user repeating the natural language utterance.

42. The method of claim 26, further comprising:

receiving a non-speech input relating to the natural language utterance at the multimodal device;

transcribing the non-speech input to create a non-speech-based transcription; and

merging the recognized words and the non-speech-based transcription to create a merged transcription, wherein the parser is further configured to generate the interpretation of the natural language utterance from the merged transcription.

43. A method for processing natural language utterances, comprising:

receiving a natural language utterance at a multimodal device;

recognizing one or more words from the natural language utterance using a speech recognition engine coupled to the multimodal device;

generating a plurality of interpretations of the natural language utterance from the one or more recognized words using a parser coupled to the multimodal device, wherein the parser generates a request based on a best interpretation selected from the plurality of interpretations of the natural language utterance;

invoking a domain agent configured to process the generated request;

monitoring one or more actions associated with the domain agent processing the request using an adaptive misrecognition engine;

determining, at the adaptive misrecognition engine, whether the interpretation of the natural language utterance is correct or incorrect based on the one or more monitored actions, wherein the adaptive misrecognition engine generates an unrecognized event in response determining that the interpretation of the natural language utterance is incorrect;

tracking an interaction pattern over time for a user that provided the natural language utterance using an analyzer associated with the adaptive misrecognition engine;

generating, at the analyzer, a personalized cognitive model for the user based on the interaction pattern tracked for the user; and

using the personalized cognitive model to predict the one or more actions associated with the domain agent processing the request and select a next best interpretation from the plurality of interpretations in response to the adaptive misrecognition engine determining that the best interpretation selected by the parser is incorrect.

44. A method for processing natural language utterances, comprising:

receiving a natural language utterance at a multimodal device;

recognizing one or more words from the natural language utterance using a speech recognition engine coupled to the multimodal device;

generating an interpretation of the natural language utterance from the one or more recognized words using a parser coupled to the multimodal device, wherein generating the interpretation of the natural language utterance includes the parser generating a request based on the interpretation of the natural language utterance;

invoking a domain agent configured to process the generated request;

monitoring one or more actions associated with the domain agent processing the request using an adaptive misrecognition engine;

determining, at the adaptive misrecognition engine, whether the interpretation of the natural language utterance is correct or incorrect based on the one or more monitored actions, wherein the adaptive misrecognition engine generates an unrecognized event in response determining that the interpretation of the natural language utterance is incorrect;

tracking interaction patterns over time for a plurality of users, including a user that provided the natural language utterance, using an analyzer associated with the adaptive misrecognition engine;

generating, at the analyzer, a generalized cognitive model for the plurality of users, wherein the generalized cognitive model includes a statistical abstract that corresponds to the interaction patterns tracked for the plurality of users; and

updating the generalized cognitive model using the analyzer based on a frequency of incorrect interpretations for the request.