Method and system for asynchronously processing natural language utterances

• US 8,155,962 B2
• Filed: 07/19/2010
• Issued: 04/10/2012
• Est. Priority Date: 06/03/2002
• Status: Expired due to Term

• Current Assignee: Dialect, LLC
• Original Assignee: VoiceBox Technologies, Inc. (Microsoft Corporation)
• Inventors: Kennewick, Robert A., Locke, David, Kennewick, Michael R. Sr., Kennewick, Michael R. Jr., Kennewick, Richard, Freeman, Tom
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US12/838,982
• Publication Number:
  US 20100286985A1
• Time in Patent Office: 631 Days
• Field of Search: 704/9, 704/257, 704/270, 704/270.1, 704/275, 704/236, 707/709, 707/710, 707/722, 707/723, 707/771
• US Class Current: 704/257
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

The methods and systems described herein may asynchronously process natural language utterances to provide real-time response performance and natural interaction with users. In particular, the methods and systems described herein may use various natural language speech recognition and interpretation components to identify a request (e.g., a query or command) in an utterance. The request identified in the utterance may then be processed with one or more domain agents, which may submit duplicate queries to multiple different data sources to process the request. The domain agents may then asynchronously evaluate responses to the duplicate queries to return results to users in a timely and natural manner, and further to account the fact that the different data sources may respond to the queries at different speeds, provide unsatisfactory responses to the queries, or fail to respond to the queries at all.

Claims

1. A method for asynchronously processing natural language utterances, comprising:

receiving a natural language utterance at a speech unit connected to a computer device, wherein the speech unit converts the received natural language utterance into an electronic signal;

recognizing one or more words in the electronic signal with a speech recognition engine that operates on the computer device;

identifying a request contained in the natural language utterance with a parser that further operates on the computer device, wherein the parser identifies the request contained in the natural language utterance from the one or more recognized words; and

asynchronously processing the request contained in the natural language utterance in a multi-threaded environment with a domain agent associated with a context relating to the identified request, wherein asynchronously processing the request with the domain agent includes;

submitting a plurality of asynchronous queries created with the domain agent to a plurality of information sources, wherein the plurality of information sources include one or more local information sources and one or more network information sources;

asynchronously evaluating results that the plurality of information sources return to the domain agent in response to the plurality of asynchronous queries;

scoring, at the domain agent, the asynchronously evaluated results returned from the plurality of information sources until one or more of the asynchronously evaluated results have a score that satisfies a predetermined confidence level; and

presenting a single best response to the request on the computer device, wherein the domain agent extracts the single best response from the one or more of the asynchronously evaluated results having the score that satisfies the predetermined confidence level.

2. The method of claim 1, wherein asynchronously processing the request with the domain agent further includes formatting, by the domain agent, the single best response using a format template associated with the context relating to the identified request, criteria identified from the natural language utterance, or a profile associated with a user that provided the natural language utterance.

3. The method of claim 2, wherein asynchronously processing the request with the domain agent further includes performing, by the domain agent, or more variable substitutions and transformations using the format template to format the single best response.

4. The method of claim 3, wherein asynchronously processing the request with the domain agent further includes using, by the domain agent, probabilistic or fuzzy set decision methods to vary one or more of a personality to present the single best response, an order to present tokens contained in the single best response, or a terminology to present the tokens contained in the single best response.

5. The method of claim 1, wherein scoring the asynchronously evaluated results includes applying, at the domain agent, probabilistic or fuzzy scoring techniques to previous results asynchronously evaluated by the domain agent, the context relating to the identified request, criteria identified from the natural language utterance, tokens associated with a desired response to the request, a dialog history relating to one or more prior natural language utterances, a profile associated with a user that provided the natural language utterance, or information specific to the domain agent.

6. The method of claim 1, wherein asynchronously processing the request with the domain agent further includes:

requesting additional information relating to the identified request via the speech unit if none of the asynchronously evaluated results returned from the plurality of information sources has the score that satisfies the predetermined confidence level or an irresolvable ambiguity arises in formulating the single best response to the request; and

submitting one or more new asynchronous queries to one or more of the plurality of information sources in response to receiving the requested additional information relating to the identified request, wherein the domain agent creates the one or more new asynchronous queries based on the received additional information.

7. The method of claim 1, wherein asynchronously processing the request with the domain agent further includes:

formulating one or more new asynchronous queries if none of the asynchronously evaluated results returned from the plurality of information sources has the score that satisfies the predetermined confidence level or do not contain all information required to extract the single best response; and

submitting the one or more new asynchronous queries to one or more of the plurality of information sources, wherein the domain agent infers the one or more of the plurality of information sources to query in the one or more new asynchronous queries based on the asynchronously evaluated results returned from the plurality of information sources.

8. The method of claim 1, wherein asynchronously processing the request with the domain agent further includes waiting to receive additional results to plurality of asynchronous queries from the plurality of information sources if none of the asynchronously evaluated results has the score that satisfies the predetermined confidence level.

9. A system for asynchronously processing natural language utterances, comprising:

a speech unit connected to a computer device, wherein the speech unit is configured to receive a natural language utterance and convert the received natural language speech into an electronic signal;

a speech recognition engine that operates on the computer device, wherein the speech recognition engine is configured to recognize one or more words in the electronic signal converted from the natural language utterance;

a parser that further operates on the computer device, wherein the parser is configured to identify a request contained in the natural language utterance from the one or more recognized words and determine a context relating to the identified request; and

a domain agent associated with the context relating to the identified request, wherein the domain agent further operates on the computer device and is configured to;

submit a plurality of asynchronous queries created with the domain agent to a plurality of information sources, wherein the plurality of information sources include one or more local information sources and one or more network information sources;

asynchronously evaluate results that the plurality of information sources return to the domain agent in response to the plurality of asynchronous queries;

score the asynchronously evaluated results returned from the plurality of information sources until one or more of the asynchronously evaluated results have a score that satisfies a predetermined confidence level;

extract a single best response to the request from the one or more of the asynchronously evaluated results having the score that satisfies the predetermined confidence level; and

present the single best response to the request on the computer device.

10. The system of claim 9, wherein the domain agent is further configured to format the single best response using a format template associated with the context relating to the identified request, criteria identified from the natural language utterance, or a profile associated with a user that provided the natural language utterance.

11. The system of claim 10, wherein the domain agent is further configured to perform one or more variable substitutions and transformations using the format template to format the single best response.

12. The system of claim 11, wherein the domain agent is further configured to use probabilistic or fuzzy set decision methods to vary one or more of a personality to present the single best response, an order to present tokens contained in the single best response, or a terminology to present the tokens contained in the single best response.

13. The system of claim 9, wherein to score the asynchronously evaluated results, the domain agent is further configured to apply probabilistic or fuzzy scoring techniques to previous results asynchronously evaluated by the domain agent, the context relating to the identified request, criteria identified from the natural language utterance, tokens associated with a desired response to the request, a dialog history relating to one or more prior natural language utterances, a profile associated with a user that provided the natural language utterance, or information specific to the domain agent.

14. The system of claim 9, wherein the domain agent is further configured to:

request additional information relating to the identified request via the speech unit if none of the asynchronously evaluated results returned from the plurality of information sources has the score that satisfies the predetermined confidence level or an irresolvable ambiguity arises in formulating the single best response to the request; and

submit one or more new asynchronous queries to one or more of the plurality of information sources in response to receiving the requested additional information relating to the identified request, wherein the domain agent is configured to create the one or more new asynchronous queries based on the received additional information.

15. The system of claim 9, wherein the domain agent is further configured to:

formulate one or more new asynchronous queries if none of the asynchronously evaluated results returned from the plurality of information sources has the score that satisfies the predetermined confidence level or do not contain all information required to extract the single best response;

infer one or more of the plurality of information sources to query in the one or more new asynchronous queries based on the asynchronously evaluated results returned from the plurality of information sources; and

submit the one or more new asynchronous queries to the one or more of the plurality of information sources inferred based on the asynchronously evaluated results returned from the plurality of information sources.

16. The system of claim 9, wherein the domain agent is further configured to wait to receive additional results to plurality of asynchronous queries from the plurality of information sources if none of the asynchronously evaluated results has the score that satisfies the predetermined confidence level.