METHOD AND APPARATUS FOR SELECTING AMONG COMPETING MODELS IN A TOOL FOR BUILDING NATURAL LANGUAGE UNDERSTANDING MODELS

• US 20150032442A1
• Filed: 07/26/2013
• Published: 01/29/2015
• Est. Priority Date: 07/26/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Marcus, Jeffrey N.
• Granted Patent: US 10,339,216 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars

Abstract

Selecting a grammar for use in a machine question-answering system, such as a Natural Language Understanding System, can be difficult for non-experts in such grammars. A tool, according to an example embodiment, can compare annotations of sample sentences, performed correctly by a human, the annotations having intents and mentions, against annotations performed by multiple grammars. Each grammar can be scored, and the system can select the best scored grammar for the user. In one embodiment, a method of selecting a grammar includes comparing manually-generated annotations against machine-generated annotations as a function of a given grammar among multiple grammars. The method can further include applying scores to the machine-generated annotations that are a function of weightings of the intents and mentions. The method can additionally include recommending whether to employ the given grammar based on the scores.

Claims

1. A method of selecting a grammar from among multiple grammars, the method comprising:

comparing manually-generated annotations, including intents and mentions, corresponding a set of sentences, against machine-generated annotations, the comparing being as a function of a given grammar among multiple grammars, the machine-generated annotations also including intents and mentions corresponding to the set of sentences;

applying scores to the machine-generated annotations, the scores being a function of weightings of the intents and mentions; and

recommending whether to employ the given grammar based on the scores.

2. The method of claim 1, wherein applying the scores to the machine-generated annotations includes applying a respective intra-sentence score to each of the intents and mentions of the machine-generated annotations based on the comparison of the manually-generated annotations and the machine-generated annotations.

3. The method of claim 2, wherein applying the scores to the machine-generated annotations further includes applying an inter-sentence score to the grammar based on the respective intra-sentence scores of each of the intents and mentions of the machine-generated annotations of the grammar.

4. The method of claim 1, further comprising parsing, by the machine, each of the set of sentences to generate the machine-generated annotations.

5. The method of claim 1, further comprising adjusting weightings of intents and mentions by performing a statistical regression of the weightings and metrics related to an end user.

6. The method of claim 1, wherein applying the scores to the machine-generated annotations further includes applying a plurality of scores, each of the plurality of scores based on the grammar employing a respective confidence threshold.

7. The method of claim 6, wherein applying the scores to the machine-generated annotations further includes averaging the plurality of scores over a range of respective confidence thresholds.

8. A system for selecting a grammar from among multiple grammars, the method comprising:

a comparison module configured to compare manually-generated annotations including a set of sentences, against machine-generated annotations, the comparing being a function of a given grammar among multiple grammars, the machine-generated annotations also including intents and mentions corresponding to the set of sentences;

a scoring module configured to apply scores to the machine-generated annotations, the scores being a function of weightings of the intents and mentions; and

a recommendation module configured to recommend whether to employ the grammar based on the scores.

9. The system of claim 8, wherein the scoring module is further configured to apply a respective intra-sentence score to each of the intents and mentions of the machine-generated annotations based on the comparison of the annotations of the manually-generated annotations and the machine-generated annotations.

10. The system of claim 9, wherein the scoring module is further configured to apply an inter-sentence score to the grammar based on the respective intra-sentence scores of each of the intents and mentions of the machine-generated annotations of the grammar.

11. The system of claim 8, further comprising a parsing module configured to parse, by the machine, each of the set of sentences to generate the machine-generated annotations.

12. The system of claim 8, further comprising a weighting module configured to adjust weightings of intents and mentions by performing a statistical regression of the weightings and metrics related to an end user.

13. The method of claim 8, wherein the scoring module is further configured to apply a plurality of scores, each of the plurality of scores based on the grammar employing a respective confidence threshold.

14. The method of claim 13, wherein the scoring module is further configured to average the plurality of scores over a range of the respective confidence thresholds.

15. A non-transitory computer-readable medium configured to store instructions for selecting a grammar from among multiple grammars, the instructions, when loaded and executed by a processor, cause the processor to:

compare manually-generated annotations including intents and mentions corresponding to a set of sentences, against machine-generated annotations, the comparing being a function of a given grammar among multiple grammars, the machine-generated annotations including intents and mentions corresponding to the set of sentences;

apply scores to the machine-generated annotations, the scores being a function of weightings of the intents and mentions; and

recommend whether to employ the given grammar based on the scores.

16. The non-transitory computer-readable of claim 15, wherein the instructions further cause the processor to apply the scores to the machine-generated annotations by applying a respective intra-sentence score to each of the intents and mentions of the machine-generated annotations based on the comparison of the manually-generated annotations and the machine-generated annotations.

17. The non-transitory computer-readable of claim 16, wherein the instructions further cause the processor to apply the scores to the machine-generated annotations further by applying an inter-sentence score to the grammar based on the respective intra-sentence scores of each of the intents and mentions of the machine-generated annotations of the grammar.

18. The non-transitory computer-readable of claim 15, wherein the instructions further cause the processor to parse, by the machine, each of the set of sentences to generate the machine-generated annotations.

19. The non-transitory computer-readable of claim 15, wherein the instructions further cause the processor to adjust weightings of intents and mentions by performing a statistical regression of the weightings and metrics related to an end user.

20. The non-transitory computer-readable of claim 15, wherein the instructions further cause the processor to apply the scores to the machine-generated annotations by applying a plurality of scores, each of the plurality of scores based on the grammar employing a respective confidence threshold.