Mobile systems and methods of supporting natural language human-machine interactions

• US 8,849,652 B2
• Filed: 05/20/2013
• Issued: 09/30/2014
• Est. Priority Date: 08/29/2005
• Status: Active Grant

• Current Assignee: Dialect, LLC
• Original Assignee: VoiceBox Technologies Corporation (Microsoft Corporation)
• Inventors: Weider, Chris, Kennewick, Richard, Kennewick, Mike, Di Cristo, Philippe, Kennewick, Robert A., Menaker, Samuel, Armstrong, Lynn Elise
• Primary Examiner(s): YEN, ERIC L
• Application Number:
  US13/898,045
• Publication Number:
  US 20130253929A1
• Time in Patent Office: 498 Days
• Field of Search: 704/9, 704270-275
• US Class Current: 704/9
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

A mobile system is provided that includes speech-based and non-speech-based interfaces for telematics applications. The mobile system identifies and uses context, prior information, domain knowledge, and user specific profile data to achieve a natural environment for users that submit requests and/or commands in multiple domains. The invention creates, stores and uses extensive personal profile information for each user, thereby improving the reliability of determining the context and presenting the expected results for a particular question or command. The invention may organize domain specific behavior and information into agents, that are distributable or updateable over a wide area network.

Claims

1. A system for processing natural language utterances where recognized words of the natural language utterances alone are insufficient to completely determine one or more commands or requests, the system comprising:

one or more physical processors programmed with one or more computer program instructions which, when executed, cause the one or more physical processors to;

generate a first context stack associated with a first device, the first context stack comprising context information that corresponds to a plurality of prior utterances;

synchronize the first context stack with a second context stack associated with a second device such that the context information of the first context stack is updated based on related context information of the second context stack;

receive a natural language utterance associated with a command or request;

determine one or more words of the natural language utterance by performing speech recognition on the natural language utterance; and

determine the command or request based on the one or more words and the updated context information.

2. The system of claim 1, wherein the one or more physical processors are further caused to:

prompt a user associated with the natural language utterance for one or more of (i) additional information regarding the command or request or (ii) confirmation regarding the command or request; and

receive a non-speech input regarding one or more of the additional information or the confirmation in response to the prompt, wherein the command or request is determined further based on the non-speech input.

3. The system of claim 1, wherein the first context stack includes a plurality of context entries, and wherein the one or more physical processors are further caused to:

identify, from among the plurality of context entries, one or more context entries that correspond to the one or more words, wherein the updated context information includes the one or more context entries.

4. The system of claim 3, wherein identifying the one or more context entries comprises:

comparing the plurality of context entries with the one or more words;

generating one or more rank scores for individual context entries of the plurality of context entries based on the comparison; and

identifying, from among the plurality of context entries, the one or more context entries based on the one or more rank scores.

5. The system of claim 4, wherein the plurality of context entries are ordered in the first context stack, and wherein the one or more physical processors are further caused to update the ordering of the plurality of context entries in the first context stack based on the one or more rank scores.

6. The system of claim 5, wherein the natural language utterance is associated with a user, and wherein the one or more physical processors are further caused to:

receive a subsequent natural language utterance associated with the user; and

determine one or more subsequent words of the subsequent natural language utterance by performing speech recognition on the subsequent natural language utterance;

identify, from among the plurality of context entries, one or more other context entries that correspond to the one or more subsequent words based on the updated ordering; and

determine a subsequent command or request based on the one or more subsequent words and the one or more other context entries.

7. The system of claim 3, wherein identifying the one or more context entries comprises identifying, from among the plurality of context entries, the one or more context entries that most closely correspond to the one or more words.

8. The system of claim 3, wherein the one or more physical processors are further caused to:

identify one or more domain agents associated with the one or more context entries, wherein the one or more domain agents are configured to process the command or request; and

generate a response to the command or request using the one or more domain agents.

9. The system of claim 8, wherein each of the one or more domain agents comprises domain knowledge associated with a particular domain, the domain knowledge comprising:

one or more of (i) a keyword;

(ii) a link to an information source;

(iii) a list of responses associated with a plurality commands or requests;

(iv) a substitution list used to format the plurality of commands or requests;

or (v) content including dictionaries, encyclopedias, or almanacs.

10. The system of claim 8, wherein the response comprises an aggregation of one or more responses generated by the one or more domain agents based on the command or request.

11. The system of claim 8, wherein the one or more physical processors are further caused to:

obtain information relating to a license agreement that is associated with at least one of the one or more domain agents;

determine, based on the information related to the license agreement, that use of the at least one of the one or more domain agents to process the command or request is permitted; and

use, based on the determination that the use is permitted, the at least one of the one or more domain agents to generate the response.

12. The system of claim 1, wherein the natural language utterance is associated with a user, and wherein the one or more physical processors are further caused to:

obtain one or more of a cognitive model or an environmental model associated with the user, wherein the cognitive model comprises information relating to one or more interactions between the user and the system, and the environmental model comprises information indicative of how noisy an environment surrounding the user is,

wherein the one or more words are determined further based on one or more of the cognitive model or the environmental model.

13. The system of claim 1, wherein the natural language utterance is associated with a user, and wherein the one or more physical processors are further caused to:

obtain a first cognitive model that comprises information relating to one or more interactions between the user and the system; and

obtain a second cognitive model that comprises information relating to one or more interactions between the system and a plurality of users of the system,

wherein the one or more words are determined further based on the first cognitive model and the second cognitive model.

14. A computer-implemented method of processing natural language utterances where recognized words of the natural language utterances alone are insufficient to completely determine one or more commands or requests, the method being implemented by a computer system that includes one or more physical processors executing one or more computer program instructions which, when executed, perform the method, the method comprising:

generating, by the one or more physical processors, a first context stack associated with a first device, the first context stack comprising context information that corresponds to a plurality of prior utterances;

synchronizing the first context stack with a second context stack associated with a second device such that the context information of the first context stack is updated based on related context information of the second context stack;

receiving, at the one or more physical processors, a natural language utterance associated with a command or request;

determining, by the one or more physical processors, one or more words of the natural language utterance by performing speech recognition on the natural language utterance; and

determining, by the one or more physical processors, the command or request based on the one or more words and the updated context information.

15. The method of claim 14, further comprising:

prompting, by the one or more physical processors, a user associated with the natural language utterance for one or more of (i) additional information regarding the command or request or (ii) confirmation regarding the command or request; and

receiving, at the one or more physical processors, a non-speech input regarding one or more of the additional information or the confirmation in response to the prompt, wherein the command or request is determined further based on the non-speech input.

16. The method of claim 14, wherein the first context stack includes a plurality of context entries, the method further comprising:

identifying, by the one or more physical processors, from among the plurality of context entries, one or more context entries that correspond to the one or more words, wherein the updated context information includes the one or more context entries.

17. The method of claim 16, wherein identifying the one or more context entries comprises:

comparing the plurality of context entries with the one or more words;

generating one or more rank scores for individual context entries of the plurality of context entries based on the comparison; and

identifying, from among the plurality of context entries, the one or more context entries based on the one or more rank scores.

18. The method of claim 16, further comprising:

identifying, by the one or more physical processors, one or more domain agents associated with the one or more context entries, wherein the one or more domain agents are configured to process the command or request; and

generating, by the one or more physical processors, a response to the command or request using the one or more domain agents.

19. The method of claim 14, wherein the natural language utterance is associated with a user, the method further comprising:

obtaining, by the one or more physical processors, a first cognitive model that comprises information relating to one or more interactions between the user and the system; and

obtaining, by the one or more physical processors, a second cognitive model that comprises information relating to one or more interactions between the system and a plurality of users of the system,

wherein the one or more words are determined further based on the first cognitive model and the second cognitive model.

20. A computer-implemented method of processing natural language utterances where recognized words of the natural language utterances alone are insufficient to completely determine one or more commands or requests, the method being implemented by a computer system that includes one or more physical processors executing one or more computer program instructions which, when executed, perform the method, the method comprising:

generating, by the one or more physical processors, a first context set associated with a first device, the first context set comprising context information that corresponds to a plurality of prior utterances;

synchronizing the first context set with a second context set associated with a second device such that the context information of the first context set is updated based on related context information of the second context set;

receiving, at the one or more physical processors, a natural language utterance associated with a command or request;

determining, by the one or more physical processors, one or more words of the natural language utterance by performing speech recognition on the natural language utterance; and

determining, by the one or more physical processors, the command or request based on the one or more words and the updated context information.

21. The method of claim 20, further comprising:

prompting, by the one or more physical processors, a user associated with the natural language utterance for one or more of (i) additional information regarding the command or request or (ii) confirmation regarding the command or request; and

receiving, at the one or more physical processors, a non-speech input regarding one or more of the additional information or the confirmation in response to the prompt, wherein the command or request is determined further based on the non-speech input.

22. The method of claim 20, wherein the first context set includes a plurality of context entries, the method further comprising:

identifying, by the one or more physical processors, from among the plurality of context entries, one or more context entries that correspond to the one or more words, wherein the updated context information includes the one or more context entries.

23. The method of claim 22, wherein identifying the one or more context entries comprises identifying, from among the plurality of context entries, the one or more context entries that most closely correspond to the one or more words.

24. The method of claim 22, further comprising:

identifying, by the one or more physical processors, one or more domain agents associated with the one or more context entries, wherein the one or more domain agents are configured to process the command or request; and

generating, by the one or more physical processors, a response to the command or request using the one or more domain agents.

25. The method of claim 24, wherein each of the one or more domain agents comprises domain knowledge associated with a particular domain, the domain knowledge comprising:

one or more of (i) a keyword;

(ii) a link to an information source;

(iii) a list of responses associated with a plurality commands or requests;

(iv) a substitution list used to format the plurality of commands or requests;

or (v) content including dictionaries, encyclopedias, or almanacs.

26. The method of claim 24, wherein the response comprises an aggregation of one or more responses generated by the one or more domain agents based on the command or request.

27. The method of claim 24, further comprising:

obtaining, by the one or more physical processors, information relating to a license agreement that is associated with at least one of the one or more domain agents;

determining, by the one or more physical processors, based on the information related to the license agreement, that use of the at least one of the one or more domain agents to process the command or request is permitted; and

using, by the one or more physical processors, based on the determination that the use is permitted, the at least one of the one or more domain agents to generate the response.