SYSTEMS AND METHODS FOR RESPONDING TO NATURAL LANGUAGE SPEECH UTTERANCE

• US 20100057443A1
• Filed: 11/12/2009
• Published: 03/04/2010
• Est. Priority Date: 08/05/2005
• Status: Active Grant

• Current Assignee: Dialect, LLC
• Original Assignee: VoiceBox Technologies, Inc. (Microsoft Corporation)
• Inventors: Armstrong, Lynn Elise, Ke, Min, Di Cristo, Philippe, Kennewick, Robert A.
• Granted Patent: US 7,917,367 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Systems and methods are provided for receiving speech and non-speech communications of natural language questions and/or commands, transcribing the speech and non-speech communications to textual messages, and executing the questions and/or commands. The invention applies context, prior information, domain knowledge, and user specific profile data to achieve a natural environment for one or more users presenting questions or commands across multiple domains. The systems and methods creates, stores and uses extensive personal profile information for each user, thereby improving the reliability of determining the context of the speech and non-speech communications and presenting the expected results for a particular question or command.

Claims

1. A system for processing multi-modal natural language inputs, comprising:

a context manager communicatively coupled to a plurality of mobile devices, wherein the context manager is configured to;

register the plurality of mobile devices with the context manager in response to a registration module associated with the context manager receiving a communication from the plurality of mobile devices;

subscribe the plurality of mobile devices registered with the context manager to one or more context events;

receive a context input from one or more of the plurality of mobile devices registered with the context manager, wherein the context input includes a context change event; and

inform the plurality of mobile devices registered with the context manager of the context change event, thereby synchronizing a context across the plurality of mobile devices.

2. The system of claim 1, wherein the context manager is configured to inform the plurality of mobile devices registered with the context manager of the context change event through a context tracking module associated with the context manager.

3. The system of claim 2, wherein the context tracking module is configured to track the context synchronized across the plurality of mobile devices.

4. The system of claim 1, wherein the context manager receives the context input in a Context XML format.

5. The system of claim 1, wherein the context manager is further configured to remove one or more of the plurality of mobile devices from the plurality of mobile devices registered with the context manager.

6. The system of claim 1, wherein the context manager includes a plurality of dynamic link libraries that are respectively specific to the plurality of mobile devices registered with the context manager.

7. The system of claim 1, wherein at least one of the plurality of mobile devices registered with the context manager include:

a multi-modal voice user interface configured to receive a natural language utterance; and

a knowledge-enhanced speech recognition engine configured to determine a most likely context for the natural language utterance, wherein the knowledge-enhanced speech recognition engine is further configured to;

identify one or more contexts that completely or partially match one or more text combinations contained in text transcribed from the natural language utterance, wherein identifying the matching contexts includes comparing the text combinations against one or more grammar expression entries in a context description grammar and one or more expected contexts stored in a context stack synchronized through the context manager;

score each of the identified matching contexts; and

select the matching context having a highest score as the most likely context for the natural language utterance.

8. The system of claim 7, wherein the context change event received at the context manager includes a communication from the at least one mobile device that indicates a change in context to the most likely context.

9. The system of claim 7, wherein the at least one mobile device further includes a multi-pass speech recognition module configured to:

use at least one of a dictation grammar or a virtual dictation grammar to transcribe the natural language utterance into the text, wherein the dictation grammar includes an unconstrained large vocabulary of words, and wherein the virtual dictation grammar includes a constrained vocabulary of words and a plurality of decoy words for out-of-vocabulary words;

dynamically generate the constrained vocabulary of words based on one or more previously successful transcriptions; and

transcribe the utterance using the multi-pass speech recognition module.

10. The system of claim 9, wherein the decoy words include utility words, nonsense words, isolated syllables, and isolated distinct sounds associated with a particular spoken language.

11. A method for processing multi-modal natural language inputs, comprising:

registering a plurality of mobile devices with a context manager in response to a registration module associated with the context manager receiving a communication from the plurality of mobile devices;

subscribing the plurality of mobile devices registered with the context manager to one or more context events;

receiving, at the context manager, a context input from one or more of the plurality of mobile devices registered with the context manager, wherein the context input includes a context change event; and

informing the plurality of mobile devices registered with the context manager of the context change event, wherein informing the plurality of mobile devices registered with the context manager of the context change event synchronizes a context across the plurality of mobile devices.

12. The method of claim 11, wherein a context tracking module associated with the context manager informs the plurality of mobile devices registered with the context manager of the context change event.

13. The method of claim 12, further comprising tracking, by the context tracking module, the context synchronized across the plurality of mobile devices.

14. The method of claim 11, wherein the context manager receives the context input in a Context XML format.

15. The method of claim 11, further comprising removing, by the context manager, one or more of the plurality of mobile devices from the plurality of mobile devices registered with the context manager.

16. The method of claim 11, wherein the context manager includes a plurality of dynamic link libraries that are respectively specific to the plurality of mobile devices registered with the context manager.

17. The method of claim 11, further comprising:

receiving a natural language utterance at a multi-modal voice user interface associated with at least one of the plurality of mobile devices registered with the context manager;

identifying, using a knowledge-enhanced speech recognition engine associated with the at least one mobile device, one or more contexts that completely or partially match one or more text combinations contained in text transcribed from the natural language utterance, wherein identifying the matching contexts includes comparing the text combinations against one or more grammar expression entries in a context description grammar and one or more expected contexts stored in a context stack synchronized through the context manager;

scoring each of the identified matching contexts; and

selecting the matching context having a highest score as a most likely context for the natural language utterance.

18. The method of claim 17, wherein the context change event received at the context manager includes a communication from the at least one mobile device that indicates a change in context to the most likely context.

19. The method of claim 17, further comprising:

using, by a multi-pass speech recognition module associated with the at least one mobile device, at least one of a dictation grammar or a virtual dictation grammar to transcribe the natural language utterance into the text, wherein the dictation grammar includes an unconstrained large vocabulary of words, and wherein the virtual dictation grammar includes a constrained vocabulary of words and a plurality of decoy words for out-of-vocabulary words;

dynamically generating the constrained vocabulary of words based on one or more previously successful transcriptions; and

transcribing the utterance using the multi-pass speech recognition module.

20. The method of claim 19, wherein the decoy words include utility words, nonsense words, isolated syllables, and isolated distinct sounds associated with a particular spoken language.