Mobile systems and methods of supporting natural language human-machine interactions

• US 8,447,607 B2
• Filed: 06/04/2012
• Issued: 05/21/2013
• Est. Priority Date: 08/29/2005
• Status: Active Grant

• Current Assignee: Dialect, LLC
• Original Assignee: VoiceBox Technologies, Inc. (Microsoft Corporation)
• Inventors: Weider, Chris, Kennewick, Richard, Kennewick, Mike, Di Cristo, Philippe, Kennewick, Robert A., Menaker, Samuel, Armstrong, Lynn Elise
• Primary Examiner(s): YEN, ERIC L
• Application Number:
  US13/488,299
• Publication Number:
  US 20120278073A1
• Time in Patent Office: 351 Days
• Field of Search: 704270-275, 704/9, 704/250
• US Class Current: 704/250
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

A mobile system is provided that includes speech-based and non-speech-based interfaces for telematics applications. The mobile system identifies and uses context, prior information, domain knowledge, and user specific profile data to achieve a natural environment for users that submit requests and/or commands in multiple domains. The invention creates, stores and uses extensive personal profile information for each user, thereby improving the reliability of determining the context and presenting the expected results for a particular question or command. The invention may organize domain specific behavior and information into agents, that are distributable or updateable over a wide area network.

Claims

1. A device for processing natural language inputs, comprising one or more processors configured to:

receive a multi-modal natural language input from a user, the multi-modal natural language input including a natural language utterance and a non-speech input;

generate a non-speech transcription from the non-speech input;

identify the user who provided the multi-modal natural language input;

generate a speech-based transcription based on a cognitive model associated with the user, wherein the cognitive model includes information on one or more prior interactions between the user and the device;

generate a merged transcription from the speech-based transcription and the non-speech transcription;

identify, from among a plurality of entries that are in a context stack and that are each indicative of context, an entry in the context stack that matches information in the merged transcription;

identify a domain agent associated with the entry in the context stack;

determine a request based on the merged transcription; and

communicate the request to the domain agent, wherein the domain agent is configured to generate a response to the user.

2. The device of claim 1, wherein the response includes an aggregation of content gathered when the identified domain agent processes the request.

3. The device of claim 1, wherein the multi-modal natural language input is a first multi-modal natural language input and is received during a first session, the one or more processors further configured to receive a second multi-modal natural language input from another user during the first session.

4. The device of claim 1, wherein the one or more processors are configured to support interactions with a plurality of users during an interleaved session.

5. The device of claim 1, wherein the cognitive model is a first cognitive model, wherein the one or more processors are configured to generate the speech-based transcription based on a second cognitive model that includes information on one or more interactions between the device and a plurality of users of the device.

6. The device of claim 5, wherein the one or more processors are configured to generate the speech-based transcription further based on an environmental model that has information indicative of an environment of the user.

7. The device of claim 6, wherein the environmental model has information indicative of how noisy the environment of the user is.

8. The device of claim 1, wherein the one or more processors are configured to receive one or more subsequent multi-modal inputs, the subsequent multi-modal inputs including at least one of a subsequent natural language utterance or a subsequent non-speech input.

9. The device of claim 1, wherein the identified domain agent is configured to update the context stack in response to processing the request.

10. The device of claim 1, wherein the one or more processors are configured to synchronize the context stack with a context stack of another device.

11. The device of claim 10, wherein the one or more processors are configured to synchronize the cognitive model with a cognitive model of another device.

12. A method for processing natural language inputs, comprising:

receiving, by one or more processors, a multi-modal natural language input from a user, the multi-modal natural language input including a natural language utterance and a non-speech input;

generating, by the one or more processors, a non-speech transcription from the non-speech input;

identifying, by the one or more processors, the user who provided the multi-modal natural language input;

generating, by the one or more processors, a speech-based transcription based on a cognitive model associated with the user, wherein the cognitive model includes information on one or more prior interactions between the user and the device;

generating, by the one or more processors, a merged transcription from the speech-based transcription and the non-speech transcription;

identifying, by the one or more processors, an entry in a context stack, from among a plurality of entries that are in the context stack and that are each indicative of context, wherein the identified entry matches information in the merged transcription;

identifying, by the one or more processors, a domain agent associated with the entry in the context stack;

determining, by the one or more processors, a request based on the merged transcription; and

communicating, by the one or more processors, the request to the domain agent, wherein the domain agent is configured to generate a response to the user.

13. The method of claim 12, wherein the cognitive model is a first cognitive model, and wherein the generating the speech-based transcription is based on a second cognitive model that includes information on one or more interactions between the device and a plurality of users of the device.

14. A device for processing natural language inputs, comprising one or more processors configured to:

receive a natural language utterance from a user;

identify the user who provided the natural language utterance;

generate a speech-based transcription based on a personal cognitive model associated with the user and a general cognitive model, wherein the personal cognitive model includes information on one or more prior interactions between the device and the user, and wherein the general cognitive model includes information on one or more prior interactions between the device and a plurality of users;

identify, from among a plurality of entries that are in a context stack and that are each indicative of context, an entry that matches information in the speech-based transcription;

identify a domain agent associated with the entry in the context stack;

determine a request based on the speech-based transcription; and

communicate the request to the domain agent, wherein the domain agent is configured to generate a response to the user.

15. The device of claim 14, wherein the one or more processors are configured to generate the speech-based transcription further based on an environmental model, wherein the environmental model includes information indicative of how noisy the environment of the user is.