Method for building a natural language understanding model for a spoken dialog system

• US 7,620,550 B1
• Filed: 10/03/2007
• Issued: 11/17/2009
• Est. Priority Date: 01/09/2004
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Van der Mude, Antony, Gupta, Narendra K., Rahim, Mazin G., Tur, Gokhan
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US11/866,685
• Time in Patent Office: 776 Days
• Field of Search: 704/257, 704/270, 704/275, 704/9, 704/246, 704/249, 704/254, 704/242, 709/228
• US Class Current: 704/257
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks

Abstract

A method of generating a natural language model for use in a spoken dialog system is disclosed. The method comprises using sample utterances and creating a number of hand crafted rules for each call-type defined in a labeling guide. A first NLU model is generated and tested using the hand crafted rules and sample utterances. A second NLU model is built using the sample utterances as new training data and using the hand crafted rules. The second NLU model is tested for performance using a first batch of labeled data. A series of NLU models are built by adding a previous batch of labeled data to training data and using a new batch of labeling data as test data to generate the series of NLU models with training data that increases constantly. If not all the labeling data is received, the method comprises repeating the step of building a series of NLU models until all labeling data is received. After all the training data is received, at least once, the method comprises building a third NLU model using all the labeling data, wherein the third NLU model is used in generating the spoken dialog service.

Claims

1. A computing device for generating a natural language understanding (NLU) model for use in a spoken dialog service, the computing device comprising a processor and further comprising:

(a) a module configured to control the processor to use sample utterances to create at least one hand crafted rule for each call-type defined in a labeling guide, wherein each call-type comprises a call category with at least one defined attribute;

(b) a module configured to control the processor to generate and test a first NLU model using the at least one hand crafted rule and sample utterances;

(c) a module configured to control the processor to build a second NLU model using the sample utterances as new training data and using the at least one hand crafted rule;

(d) a module configured to control the processor to test the performance of the second NLU model using a first batch of labeled data;

(e) a module configured to control the processor to (1) build a series of NLU models by adding a previous batch of labeled data to training data and (2) use a new batch of labeling data as test data to generate the series of NLU models with training data that increases constantly;

(f) a module configured, if not all the labeling data is received, to control the processor to repeat steps performed by module (e) until all labeling data is received; and

(g) a module configured, after all the training data is received, at least once, to control the processor to build a third NLU model using all the labeling data, wherein the third NLU model is used in generating the spoken dialog service.

2. The computing device of claim 1, further comprising:

a module configured to control the processor to identify at least one utterance with a known classification;

a module configured to control the processor to apply the at least one utterance to the NLU model; and

a module configured to control the processor to check whether the at least one utterance was correctly classified by the NLU model.

3. The computing device of claim 1, further comprising:

(h) a module configured to control the processor to perform an improvement performance step on the third NLU model;

(i) a module configured to control the processor to add the test data to the training data and build a fourth NLU model; and

(j) a module configured to control the processor to augment the fourth NLU model using utterances obtained from a customer acceptance test.

4. The computing device of claim 3, further comprising a module configured to control the processor to ignore the utterances that are longer than a specific threshold.

5. A system for generating a natural language understanding (NLU) model for use in a spoken dialog service, the system having a processor and comprising:

(a) a module configured to control the processor to build a first NLU model using sample utterances from a labeling guide, hand crafted rules and labeled utterances of available human/human dialogs or human/machine dialogs, if available;

(b) a module configured to control the processor to test the performance of the first NLU model using sample utterances in the labeling guide; and

(c) a module configured to control the processor to build a series of NLU models and evaluate the performance of the series of NLU models as labeled data becomes available by;

(i) adding a previous batch of labeled data to training data; and

(ii) using a new batch of labeling data as test data to generate the series of NLU models with training data that increases constantly.

6. The system of claim 5, wherein the module configured to control the processor to build a series of NLU models further mixes all the data and divide the mixed data into training data and test data.

7. The system of claim 5, wherein the module configured to control the processor to build a series of NLU models further keeps a fixed text set and uses all other data for training.

8. The system of claim 5, wherein the module configured to control the processor to build a series of NLU models further mixes all the data and divides it into training and text data.

9. The system of claim 5, wherein one batch containing all the data is received after completion of all the data labeling.

10. A tangible computer readable medium storing instructions for controlling a computing device to generate a natural language understanding (NLU) model for use in a spoken dialog system, the instructions comprising:

(a) controlling a processor to build a first NLU model using sample utterances from a labeling guide, hand crafted rules and labeled utterances of available human/human dialogs or human/machine dialogs, if available;

(b) controlling the processor to test the performance of the first NLU model using sample utterances in the labeling guide; and

(c) controlling the processor to build a series of NLU models and evaluate the performance of the series of NLU models as labeled data becomes available by;

(i) adding a previous batch of labeled data to training data; and

(ii) using a new batch of labeling data as test data to generate the series of NLU models with training data that increases constantly.

11. The tangible computer readable medium of claim 10, wherein building a series of NLU models further comprises mixing all the data and dividing the mixed data into training data and test data.

12. The tangible computer readable medium of claim 10, wherein building a series of NLU models further comprises keeping a fixed text set and using all other data for training.

13. The tangible computer readable medium of claim 10, wherein building a series of NLU models further comprises mixing all the data and dividing it into training and text data.

14. The tangible computer readable medium of claim 10, wherein one batch containing all the data is received after completion of all the data labeling.

15. The tangible computer readable medium of claim 10, wherein the NLU model is generated by a method further comprising:

augmenting the series of NLU models using utterances obtained from a customer acceptance test.

16. The tangible computer readable medium of claim 15, wherein the NLU model is generated by a method further comprising ignoring the utterances that are longer than a specific threshold.

17. The tangible computer readable medium of claim 15, wherein if the sample utterances are human-human utterances, the instructions further comprise not labeling utterances longer than a threshold amount.