Method for building a natural language understanding model for a spoken dialog system

• US 7,933,766 B2
• Filed: 10/20/2009
• Issued: 04/26/2011
• Est. Priority Date: 01/09/2004
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Gupta, Narendra K., Rahim, Mazin G., Tur, Gokhan, Van der Mude, Antony
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US12/582,062
• Publication Number:
  US 20100042404A1
• Time in Patent Office: 553 Days
• Field of Search: 704/9, 704/257, 704/270.1, 704/275, 704/8, 704/10
• US Class Current: 704/9
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks

Abstract

A method of generating a natural language model for use in a spoken dialog system is disclosed. The method comprises using sample utterances and creating a number of hand crafted rules for each call-type defined in a labeling guide. A first NLU model is generated and tested using the hand crafted rules and sample utterances. A second NLU model is built using the sample utterances as new training data and using the hand crafted rules. The second NLU model is tested for performance using a first batch of labeled data. A series of NLU models are built by adding a previous batch of labeled data to training data and using a new batch of labeling data as test data to generate the series of NLU models with training data that increases constantly. If not all the labeling data is received, the method comprises repeating the step of building a series of NLU models until all labeling data is received. After all the training data is received, at least once, the method comprises building a third NLU model using all the labeling data, wherein the third NLU model is used in generating the spoken dialog service.

Claims

1. A computing device for generating a natural language understanding (NLU) model for use in a spoken dialog service, the computing device comprising a processor and further comprising:

(a) a module that controls the processor to generate and test a first NLU model using a hand crafted rule and sample utterances;

(b) a module that controls the processor to build a second NLU model based on the sample utterances as new training data and based on the hand crafted rule;

(c) a module that controls the processor to (1) build a series of NLU models by adding a previous batch of labeled data to training data and (2) use a new batch of labeling data as test data to generate the series of NLU models with dynamically increasing training data; and

(d) a module that controls the processor, after all the training data is received, at least once, to build a third NLU model based on all the labeling data, wherein the third NLU model is used in a spoken dialog service.

2. The computing device of claim 1, further comprising:

a module that controls the processor to identify at least one utterance with a known classification;

a module that controls the processor to apply the at least one utterance to the third NLU model; and

a module that controls the processor to check whether the at least one utterance was correctly classified by the third NLU model.

3. The computing device of claim 1, further comprising:

a module that controls the processor to perform an improvement performance step on the third NLU model;

a module that controls the processor to add the test data to the training data and build a fourth NLU model; and

a module that controls the processor to augment the fourth NLU model using utterances obtained from a customer acceptance test.

4. The computing device of claim 3, further comprising a module that controls the processor to ignore the utterances that are longer than a specific threshold.

5. A system for generating a natural language understanding (NLU) model for use in a spoken dialog service, the system having a processor and comprising:

(a) a module that controls the processor to build a first NLU model using sample utterances from a labeling guide, hand crafted rules and labeled utterances of available dialogs;

(b) a module that controls the processor to test the performance of the first NLU model using sample utterances in the labeling guide; and

(c) a module that controls the processor to build a series of NLU models and evaluate the performance of the series of NLU models as labeled data becomes available by;

(i) adding a previous batch of labeled data to training data; and

(ii) using a new batch of labeling data as test data to generate the series of NLU models with training data that increases constantly.

6. The system of claim 5, wherein the module that controls the processor to build a series of NLU models further mixes all the data and divide the mixed data into training data and test data.

7. The system of claim 5, wherein the module that controls the processor to build a series of NLU models further keeps a fixed text set and uses all other data for training.

8. The system of claim 5, wherein the module that controls the processor to build a series of NLU models further mixes all the data and divides it into training and text data.

9. The system of claim 5, wherein one batch containing all the data is received after completion of all the data labeling.

10. A method for generating a natural language understanding (NLU) model for use in a spoken dialog service, the method causing a computing device to perform steps comprising:

(a) generating and testing a first NLU model using a hand crafted rule and sample utterances;

b) building a second NLU model based on the sample utterances as new training data and based on the hand crafted rule;

(c) building a series of NLU models by adding a previous batch of labeled data to training data;

(d) using a new batch of labeling data as test data to generate the series of NLU models with dynamically increasing training data; and

(e) after all the training data is received, at least once, building a third NLU model based at least in part on all the labeling data, wherein the third NLU model is used in a spoken dialog service.

11. The method of claim 10, further comprising:

identifying at least one utterance with a known classification;

applying the at least one utterance to the third NLU model; and

checking whether the at least one utterance was correctly classified by the third NLU model.

12. The method of claim 10, further comprising:

performing an improvement performance step on the third NLU model;

adding the test data to the training data and build a fourth NLU model; and

augmenting the fourth NLU model using utterances obtained from a customer acceptance test.

13. The method of claim 12, further comprising, ignoring the utterances that are longer than a specific threshold.