Method and system for automatically building natural language understanding models

• US 7,835,911 B2
• Filed: 12/30/2005
• Issued: 11/16/2010
• Est. Priority Date: 12/30/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Boyer, Linda M., Balchandran, Rajesh
• Primary Examiner(s): Wozniak; James S
• Application Number:
  US11/324,057
• Publication Number:
  US 20070156392A1
• Time in Patent Office: 1,782 Days
• Field of Search: 704/1, 704/9, 704/10, 704/257, 704/270, 704/270.1, 704243-244
• US Class Current: 704/257
• CPC Class Codes: G06F 40/237 Lexical tools

Abstract

The invention disclosed herein concerns a system (100) and method (600) for building a language model representation of an NLU application. The method 500 can include categorizing an NLU application domain (602), classifying a corpus in view of the categorization (604), and training at least one language model in view of the classification (606). The categorization produces a hierarchical tree of categories, sub-categories and end targets across one or more features for interpreting one or more natural language input requests. During development of an NLU application, a developer assigns sentences of the NLU application to categories, sub-categories or end targets across one or more features for associating each sentence with desire interpretations. A language model builder (140) iteratively builds multiple language models for this sentence data, and iteratively evaluating them against a test corpus, partitioning the data based on the categorization and rebuilding models, so as to produce an optimal configuration of language models to interpret and respond to language input requests for the NLU application.

Claims

1. A method for building a language model configuration comprising the steps of:

categorizing a natural language understanding (NLU) application to produce an application categorization having a plurality of categories;

classifying a corpus of example expressions to produce a classified corpus by identifying at least one of the categories for each example expression of the example expressions; and

operating at least one computer configured with a plurality of instructions that, when executed, cause the at least one computer to train at least one statistical language model using said classified corpus by;

building from the classified corpus a first language model configuration comprising a first statistical language model;

evaluating an interpretation accuracy of the first language model configuration using test data;

determining whether the evaluated interpretation accuracy of the first language model configuration is less than a desired accuracy;

when it is determined that the evaluated interpretation accuracy of the first language model configuration is at least the desired accuracy, then adopting the first language model configuration; and

when it is determined that the evaluated interpretation accuracy of the first language model configuration is less than the desired accuracy, then;

sub-dividing the application categorization into a plurality of sub-categories; and

building a second language model configuration comprising a plurality of statistical language models corresponding to the plurality of sub-categories.

2. The method of claim 1, wherein categorizing the natural language understanding (NLU) application to produce an application categorization having a plurality of categories comprises representing the application as a hierarchical tree of categories, sub-categories and end targets in the application categorization for one or more features or types of interpretation.

3. The method of claim 2, wherein operating the at least one computer to train the statistical language model comprises causing the statistical language model to learn associations between, categories, sub-categories and end targets across multiple features within said application categorization, such that a language input request is identified, using a statistical language model, with at least one action that corresponds to a target.

4. The method of claim 2, further comprising producing a visual representation of said application categorization for visually categorizing and visually classifying said natural language understanding application.

5. The method of claim 4, wherein visually classifying includes dragging and dropping a sentence of said corpus into at least one target of said visual representation.

6. The method of claim 4, further comprising automatically classifying an example sentence in category targets above a node in the visual representation when a developer classifies said example sentence in said node.

7. The method of claim 1, wherein identifying at least one of the categories for each example expression of the example expressions comprises identifying at least one target in said application categorization to provide a correct interpretation of the example expression.

8. The method of claim 1, wherein:

said classifying further comprises associating an example expression with multiple targets corresponding to one or more features; and

the method further comprises providing to a user of said natural language understanding application who has entered a language input request multiple pieces of information from said language model representation.

9. The method of claim 1, wherein the method further comprises interpreting a language input request at runtime using the at least one statistical language model in a defined sequence that provides the highest trained accuracy of natural language interpretation.

10. The method of claim 1, wherein:

sub-dividing the application categorization into a plurality of sub-categories comprises sub-dividing the application categorization into a plurality of sub-categories at least one branch within said application categorization;

building a second language model configuration comprising a plurality of statistical language models comprises;

building a statistical language model for each of said at least one branch corresponding to the plurality of sub-categories; and

saving a configuration file describing a sequential interconnection of each statistical language model of the plurality of statistical language models; and

the method further comprises evaluating the interpretation accuracy of the second language model configuration by passing sentences of test data through said statistical language models of the second language model configuration in a sequence described by said configuration file.

11. The method of claim 10, further comprising logging a history of a performance accuracy for each new configuration, comparing a historic performance accuracy of a previous configuration to a new performance accuracy of said new configuration, and reverting to said historic configuration if said new performance accuracy is less than said historic performance accuracy,

wherein if no further partitioning is possible, then of the previous partitions, the model configuration that yielded the best performance accuracy on the test data is considered as the optimum model configuration.

12. The method of claim 1, wherein during a runtime, a user request is first submitted to a high language model for which the high language model produces a response, and if the response is an end target then the high language model has sufficiently interpreted the user request and no lower language models are employed to interpret the user request, whereas if the response of the high level language model is a category having its own set of end targets then a lower level language model is further accessed for interpreting the user request.

13. The method of claim 1, wherein the plurality of sub-categories is a first plurality of sub-categories and the plurality of statistical language models is a first plurality of statistical language models, and training at least one statistical language model further comprises, when it is determined that the evaluated interpretation accuracy of the first language model configuration is less than the desired accuracy:

evaluating an interpretation accuracy of the second language model configuration using the test data;

determining whether the evaluated interpretation accuracy of the second language model configuration is less than the desired accuracy;

when it is determined that the evaluated interpretation accuracy of the second language model configuration is at least the desired accuracy, then adopting the second language model configuration; and

when it is determined that the evaluated interpretation accuracy of the second language model configuration is less than the desired accuracy, then;

further sub-dividing the first plurality of sub-categories of the application categorization into a second plurality of sub-categories; and

building a third language model configuration comprising a second plurality of statistical language models corresponding to the second plurality of sub-categories.

14. A natural language understanding (NLU) system comprising:

a computer comprising at least one processor configured to perform acts of;

categorizing an NLU application to produce an application categorization having a plurality of categories;

classifying an NLU database corpus of example expressions to produce a classified corpus by identifying at least one of the categories for each example expression of the example expressions; and

building from the classified corpus a first language model configuration comprising a first statistical language model;

evaluating an interpretation accuracy of the first language model configuration using test data;

determining whether the evaluated interpretation accuracy is less than a desired accuracy;

when it is determined that the evaluated interpretation accuracy is at least the desired accuracy, then adopting the first language model configuration; and

when it is determined that the evaluated interpretation accuracy is less than the desired accuracy, then;

sub-dividing the application categorization into a plurality of sub-categories; and

building a second language model configuration comprising a plurality of statistical language models corresponding to the plurality of sub-categories.

15. The NLU system of claim 14, wherein said application categorization includes at least one topic of said natural language understanding application that is presented as one of a category, sub-category and end target for one or more features.

16. The NLU system of claim 14, wherein said classifying partitions said NLU database corpus based on said application categorization.

17. The NLU system of claim 14, further comprising:

a visual toolkit having a graphical user interface (GUI) for visually presenting said application categorization and classifying said NLU database corpus.

18. The NLU system of claim 14, wherein a language input request is processed through at least one language model configuration for yielding a classification result, wherein a target with the highest classification result is selected to provide information in response to the language input request.

19. The NLU system of claim 14, wherein the plurality of sub-categories is a first plurality of sub-categories and the plurality of statistical language models is a first plurality of statistical language models, and when it is determined that the evaluated interpretation accuracy of the first language model configuration is less than the desired accuracy, the computer comprising the at least one processor is further configured to perform acts of:

evaluating an interpretation accuracy of the second language model configuration using the test data;

determining whether the evaluated interpretation accuracy of the second language model configuration is less than the desired accuracy;

when it is determined that the evaluated interpretation accuracy of the second language model configuration is at least the desired accuracy, then adopting the second language model configuration; and

when it is determined that the evaluated interpretation accuracy of the second language model configuration is less than the desired accuracy, then;

further sub-dividing the first plurality of sub-categories of the application categorization into a second plurality of sub-categories; and

building a third language model configuration comprising a second plurality of statistical language models corresponding to the second plurality of sub-categories.

20. A natural language understanding (NLU) system comprising:

a computer comprising at least one processor configured to perform acts of;

categorizing an NLU application to produce an application categorization having a plurality of categories;

classifying an NLU database corpus of example expressions to produce a classified corpus by identifying at least one of the categories for each example expression of the example expressions;

building from the classified corpus a first language model configuration comprising a first statistical language model;

evaluating an interpretation accuracy of the first language model configuration using test data;

determining whether the evaluated interpretation accuracy of the first language model configuration is less than a desired accuracy;

when it is determined that the evaluated interpretation accuracy of the first language model configuration is at least the desired accuracy, then adopting the first language model configuration; and

when it is determined that the evaluated interpretation accuracy of the first language model configuration is less than the desired accuracy, then;

sub-dividing the application categorization into a first plurality of sub-categories;

building a second language model configuration comprising a first plurality of statistical language models corresponding to the first plurality of sub-categories;

evaluating an interpretation accuracy of the second language model configuration using the test data;

determining whether the evaluated interpretation accuracy of the second language model configuration is less than the desired accuracy;

when it is determined that the evaluated interpretation accuracy of the second language model configuration is at least the desired accuracy, then adopting the second language model configuration; and

when it is determined that the evaluated interpretation accuracy of the second language model configuration is less than the desired accuracy, then;

determining whether the first plurality of sub-categories can be further sub-divided;

when it is determined that the first plurality of sub-categories can be further sub-divided, then;

 

further sub-dividing the first plurality of sub-categories of the application categorization into a second plurality of sub-categories; and

 

building a third language model configuration comprising a second plurality of statistical language models corresponding to the second plurality of sub-categories; and

when it is determined that the first plurality of sub-categories cannot be further sub-divided, then adopting a language model configuration selected from the group consisting of the first language model configuration and the second language model configuration, the selected language model configuration having the greatest evaluated interpretation accuracy.