Dynamic semantic control of a speech recognition system

• US 20040006465A1
• Filed: 02/10/2003
• Published: 01/08/2004
• Est. Priority Date: 02/25/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: SpeechWorks International, Inc. (Microsoft Corporation)
• Inventors: Phillips, Michael S., Dahan, Jean-Guy, Barnard, Etienne, Metzger, Michael J.
• Granted Patent: US 7,127,393 B2
• Time in Patent Office: Days
• US Class Current: 704/240
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning

Abstract

A method and apparatus are provided for automatically recognizing words of spoken speech using a computer-based speech recognition system according to a dynamic semantic model. In an embodiment, the speech recognition system recognizes speech and generates one or more word strings, each of which is a hypothesis of the speech, and creates and stores a probability value or score for each of the word strings. The word strings are ordered by probability value. The speech recognition system also creates and stores, for each of the word strings, one or more keyword-value pairs that represent semantic elements and semantic values of the semantic elements for the speech that was spoken. One or more dynamic semantic rules are defined that specify how a probability value of a word string should be modified based on information about external conditions, facts, or the environment of the application in relation to the semantic values of that word string. A speech recognition application, upon receiving the keyword-value pairs, instructs the speech recognizer to modify one or more of the probability values, according to the dynamic semantic rules. The dynamic semantic rules are applied to the word strings and the keyword-value pairs. The speech recognizer modifies one or more of the probability values, re-orders the word strings, and returns control to the application. As a result, the speech recognizer may adjust dynamically to a changing likelihood that a speaker uttered a particular utterance, as the application executes, depending on the context of the application and the external factors.

Claims

1. A method of dynamically modifying a probability value associated with one or more word strings that are recognized by a speech recognizer, based on semantic values associated with the word strings, comprising the steps of:

creating and storing one or more rules that define a change in one or more of the probability values when one of the semantic values matches a pre-determined semantic tag, in which the rules are based on one or more external conditions about the context in which the speech recognizer is used;

determining whether one of the conditions currently is true, and if so, modifying one or more of the probability values of word strings associated with semantic values that match the tag that is associated with the condition that is true.

2. The method as recited in claim 1, wherein the speech recognizer delivers the word strings to an application program, and wherein the step of determining comprises the steps of determining, in the application program, whether one of the conditions currently is true, and if so, instructing the speech recognizer to modify one or more of the probability values of word strings associated with semantic values that match the tag that is associated with the condition that is true.

3. The method as recited in claim 1, further comprising the steps of:

storing the semantic values as one or more keyword-value pairs that are associated with the word strings recognized by the speech recognizer;

delivering the keyword-value pairs to an application program; and

determining, in the application program, whether one of the conditions currently is true, and if so, instructing the speech recognizer to modify, in one or more keyword-value pairs, one or more probability values of word strings associated with semantic values that match the tag that is associated with the condition that is true.

4. The method as recited in claim 1, further comprising the steps of:

delivering the words and semantic values to an application program that is logically coupled to the speech recognizer;

creating and storing, in association with the speech recognizer;

a function callable by the application program that can modify one or more of the probability values of word strings having semantic values that match the tag that is associated with the condition that is true;

determining, in the application program, whether one of the conditions currently is true, and if so, calling the function with parameter values that identify how to modify one or more of the probability values.

5. The method as recited in claim 4, further comprising the step of re-ordering the word strings after modifying one or more of the probability values.

6. The method as recited in claim 3, further comprising the step of re-ordering the word strings by probability value after modifying one or more of the probability values.

7. The method as recited in claim 1, in which the modifying step further comprises the step of modifying the probability values by multiplying one or more of the probability values by a scaling factor that is associated with the condition that is true.

8. The method as recited in claim 1, further comprising the steps of:

delivering one or more word-value pairs that include the semantic values to an application program that is logically coupled to the speech recognizer;

creating and storing, in association with the speech recognizer, a function callable by the application program that can modify one or more of the probability values of word strings associated with word-value pairs that match the tag word that is associated with the condition that is true;

determining, in the application program, whether one of the conditions currently is true, and if so, calling the function with parameter values that identify how to modify one or more of the probability values, including a scaling factor that is associated with the condition that is true;

modifying one of the probability values of the word strings associated with one of the word-value pairs that matches the tag word that is associated with the condition that is true by multiplying its probability value by the scaling factor.

12. The method as recited in claim 1, wherein the creating and storing step comprises the steps of:

creating and storing a table of predetermined semantic tags, wherein each of the semantic tags is associated with a substitute probability value;

creating and storing a function call that changes one or more of the values to the substitute probability value when one or more of the hypothesized words matches a pre-determined semantic tag, according to rules in the function call that are based on one or more external conditions about the context in which the speech recognizer is used.

13. The method as recited in claim 1, wherein the creating and storing step comprises the steps of:

creating and storing a table of pre-determined semantic tags, wherein each of the semantic tags is associated with a substitute probability value, a weight value, and an offset value;

creating and storing a function call that changes one or more of the values to the substitute probability value or applies the weight value or the offset value to the probability value when one or more of the hypothesized words matches a pre-determined semantic tag, according to rules in the function call that are based on one or more external conditions about the context in which the speech recognizer is used.

9. A method of recognizing utterances received at a speech recognizer, comprising the steps of:

converting the utterances into one or more word strings, each associated with one or more keyword-value pairs, in which each of the pairs comprises a keyword that represents a semantic element of one of the utterances and a semantic value that represents a portion of the utterance that corresponds to the semantic element;

storing a probability value in association with each of the word strings;

creating and storing one or more rules that define a change in one or more of the probability values when one or more of the semantic values matches a pre-determined tag word, in which the rules are based on one or more external conditions about the context in which the speech recognizer is used;

determining whether one of the conditions currently is true, and if so, modifying one or more of the probability values of one of the word strings associated with semantic values that match the tag word that is associated with the condition that is true;

delivering the word-value pairs to an application program that is logically coupled to the speech recognizer;

creating and storing, in association with the speech recognizer, a function callable by the application program that can modify one or more of the probability values that are associated with words that match the tag word that is associated with the condition that is true;

determining, in the application program, whether one of the conditions currently is true, and if so, calling the function with parameter values that identify how to modify one or more of the probability values;

modifying one or more of the probability values using the function; and

re-ordering the word strings according to the probability values.

10. The method as recited in claim 9, in which the modifying step further comprises the step of modifying the word strings by multiplying one or more of the probability values by a scaling factor that is associated with the condition that is true.

11. The method as recited in claim 9, further comprising the steps of:

delivering the word-value pairs to an application program that is logically coupled to the speech recognizer;

creating and storing, in association with the speech recognizer, a function callable by the application program that can modify one or more of the probability values that are associated with words that match the tag word that is associated with the condition that is true;

determining, in the application program, whether one of the conditions currently is true, and if so, calling the function with parameter values that identify how to modify one or more of the probability values, including a scaling factor that is associated with the condition that is true;

modifying one of the probability values that is associated with one of the words that matches the tag word that is associated with the condition that is true by multiplying its probability value by the scaling factor.

14. A computer-readable medium carrying one or more sequences of instructions for dynamically modifying a probability value associated with one or more word strings that are recognized by a speech recognizer, based on semantic values associated with the word strings, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform the steps of:

creating and storing one or more rules that define a change in one or more of the probability values when one of the semantic values matches a pre-determined semantic tag, in which the rules are based on one or more external conditions about the context in which the speech recognizer is used;

determining whether one of the conditions currently is true, and if so, modifying one or more of the probability values of word strings associated with semantic values that match the tag that is associated with the condition that is true.

15. The computer-readable medium as recited in claim 14, wherein the speech recognizer delivers the word strings to an application program, and wherein the step of determining comprises the steps of determining, in the application program, whether one of the conditions currently is true, and if so, instructing the speech recognizer to modify one or more of the probability values of word strings associated with semantic values that match the tag that is associated with the condition that is true.

16. The computer-readable medium as recited in claim 14, further comprising the steps of:

storing the semantic values as one or more keyword-value pairs that are associated with the word strings recognized by the speech recognizer;

delivering the keyword-value pairs to an application program; and

determining, in the application program, whether one of the conditions currently is true, and if so, instructing the speech recognizer to modify, in one or more keyword-value pairs, one or more probability values of word strings associated with semantic values that match the tag that is associated with the condition that is true.

17. The computer-readable medium as recited in claim 14, further comprising the steps of:

delivering the words and semantic values to an application program that is logically coupled to the speech recognizer;

creating and storing, in association with the speech recognizer, a function callable by the application program that can modify one or more of the probability values of word strings having semantic values that match the tag that is associated with the condition that is true;

determining, in the application program, whether one of the conditions currently is true, and if so, calling the function with parameter values that identify how to modify one or more of the probability values.

18. The computer-readable medium as recited in claim 14, further comprising the step of re-ordering the word strings after modifying one or more of the probability values.

19. The computer-readable medium as recited in claim 16, further comprising the step of re-ordering the word strings by probability value after modifying one or more of the probability values.

20. The computer-readable medium as recited in claim 14, in which the modifying step further comprises the step of modifying the probability values by multiplying one or more of the probability values by a scaling factor that is associated with the condition that is true.

21. The computer-readable medium as recited in claim 14, further comprising the steps of:

delivering one or more word-value pairs that include the semantic values to an application program that is logically coupled to the speech recognizer;

creating and storing, in association with the speech recognizer, a function callable by the application program that can modify one or more of the probability values of word strings associated with word-value pairs that match the tag word that is associated with the condition that is true;

determining, in the application program, whether one of the conditions currently is true, and if so, calling the function with parameter values that identify how to modify one or more of the probability values, including a scaling factor that is associated with the condition that is true;

modifying one of the probability values of the word strings associated with one of the word-value pairs that matches the tag word that is associated with the condition that is true by multiplying its probability value by the scaling factor.

22. A computer-readable medium carrying one or more sequences of instructions for recognizing utterances that are received by a speech recognizer, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform the steps of:

converting the utterances into one or more word strings, each associated with one or more keyword-value pairs, in which each of the pairs comprises a keyword that represents a semantic element of one of the utterances and a semantic value that represents a portion of the utterance that corresponds to the semantic element;

storing a probability value in association with each of the word strings;

creating and storing one or more rules that define a change in one or more of the probability values when one or more of the semantic values matches a pre-determined tag word, in which the rules are based on one or more external conditions about the context in which the speech recognizer is used;

determining whether one of the conditions currently is true, and if so, modifying one or more of the probability values of one of the word strings associated with semantic values that match the tag word that is associated with the condition that is true;

delivering the word-value pairs to an application program that is logically coupled to the speech recognizer;

creating and storing, in association with the speech recognizer, a function callable by the application program that can modify one or more of the probability values that are associated with words that match the tag word that is associated with the condition that is true;

determining, in the application program, whether one of the conditions currently is true, and if so, calling the function with parameter values that identify how to modify one or more of the probability values;

modifying one or more of the probability values using the function; and

re-ordering the word strings according to the probability values.

23. The computer-readable medium as recited in claim 22, in which the modifying step further comprises the step of modifying the word strings by multiplying one or more of the probability values by a scaling factor that is associated with the condition that is true.

24. The computer-readable medium as recited in claim 23, further comprising the steps of:

delivering the word-value pairs to an application program that is logically coupled to the speech recognizer;

creating and storing, in association with the speech recognizer, a function callable by the application program that can modify one or more of the probability values that are associated with words that match the tag word that is associated with the condition that is true;

determining, in the application program, whether one of the conditions currently is true, and if so, calling the function with parameter values that identify how to modify one or more of the probability values, including a scaling factor that is associated with the condition that is true;

modifying one of the probability values that is associated with one of the words that matches the tag word that is associated with the condition that is true by multiplying its probability value by the scaling factor.

25. The computer-readable medium as recited in claim 22, wherein the creating and storing step comprises the steps of:

creating and storing a table of pre-determined semantic tags, wherein each of the semantic tags is associated with a substitute probability value;

creating and storing a function call that changes one or more of the values to the substitute probability value when one or more of the hypothesized words matches a pre-determined semantic tag, according to rules in the function call that are based on one or more external conditions about the context in which the speech recognizer is used.

26. The computer-readable medium as recited in claim 22, wherein the creating and storing step comprises the steps of:

creating and storing a table of pre-determined semantic tags, wherein each of the semantic tags is associated with a substitute probability value, a weight value, and an offset value;

creating and storing a function call that changes one or more of the values to the substitute probability value or applies the weight value or the offset value to the probability value when one or more of the hypothesized words matches a pre-determined semantic tag, according to rules in the function call that are based on one or more external conditions about the context in which the speech recognizer is used.