Method and apparatus for selecting among competing models in a tool for building natural language understanding models
First Claim
1. A computer-implemented method:
- building, by a processor, an ontology of grammar models for performing natural language understanding (NLU) processing by an NLU system by;
receiving, from an annotating user via a user interface, electronically formatted manually-generated annotations of intents and mentions of a set of sentences that are displayed via the user interface to the annotating user;
comparing the manually-generated annotations of the intents and the mentions of the set of sentences against multiple machine-generated annotations, each of the machine-generated annotations being generated using a grammar model from among multiple grammar models, the comparing being as a function of each grammar model among the multiple grammar models stored in a memory, the machine-generated annotations corresponding to intents and mentions of the set of sentences, each of the multiple grammar models being a grammar model having a plurality of stub grammar models, and wherein the grammar model correlates an input of a given user to a response to the given user;
computing a respective inter-sentence score for each respective grammar model based on respective intra-sentence scores of intents and mentions of each machine-generated annotation for the respective grammar model compared to the intents and mentions of the manually-generated annotations, the respective score being a match of each machine-generated annotation for the respective grammar model to each manually-generated annotation; and
providing, via the user interface, a list of recommended grammar models of the multiple grammar models to employ based on the scores corresponding to the machine-generated annotations corresponding to the respective grammar model, the list including grammar models scoring above a particular semantic confidence threshold;
receiving selection, from the annotating user via the user interface, of a grammar model of the recommended grammar models; and
adding the user-selected grammar model of the list of recommended grammar models to the ontology of the NLU system.
2 Assignments
0 Petitions
Accused Products
Abstract
Selecting a grammar for use in a machine question-answering system, such as a Natural Language Understanding System, can be difficult for non-experts in such grammars. A tool, according to an example embodiment, can compare annotations of sample sentences, performed correctly by a human, the annotations having intents and mentions, against annotations performed by multiple grammars. Each grammar can be scored, and the system can select the best scored grammar for the user. In one embodiment, a method of selecting a grammar includes comparing manually-generated annotations against machine-generated annotations as a function of a given grammar among multiple grammars. The method can further include applying scores to the machine-generated annotations that are a function of weightings of the intents and mentions. The method can additionally include recommending whether to employ the given grammar based on the scores.
30 Citations
14 Claims
-
1. A computer-implemented method:
-
building, by a processor, an ontology of grammar models for performing natural language understanding (NLU) processing by an NLU system by; receiving, from an annotating user via a user interface, electronically formatted manually-generated annotations of intents and mentions of a set of sentences that are displayed via the user interface to the annotating user; comparing the manually-generated annotations of the intents and the mentions of the set of sentences against multiple machine-generated annotations, each of the machine-generated annotations being generated using a grammar model from among multiple grammar models, the comparing being as a function of each grammar model among the multiple grammar models stored in a memory, the machine-generated annotations corresponding to intents and mentions of the set of sentences, each of the multiple grammar models being a grammar model having a plurality of stub grammar models, and wherein the grammar model correlates an input of a given user to a response to the given user; computing a respective inter-sentence score for each respective grammar model based on respective intra-sentence scores of intents and mentions of each machine-generated annotation for the respective grammar model compared to the intents and mentions of the manually-generated annotations, the respective score being a match of each machine-generated annotation for the respective grammar model to each manually-generated annotation; and providing, via the user interface, a list of recommended grammar models of the multiple grammar models to employ based on the scores corresponding to the machine-generated annotations corresponding to the respective grammar model, the list including grammar models scoring above a particular semantic confidence threshold; receiving selection, from the annotating user via the user interface, of a grammar model of the recommended grammar models; and adding the user-selected grammar model of the list of recommended grammar models to the ontology of the NLU system. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A system comprising:
-
a processor configured to implement building an ontology for performing natural language understanding (NLU) processing by an NLU system by; a user input module configured to receive input, via a user interface, from an annotating user of electronically formatted manually-generated annotations of intents and mentions of a set of sentences that are displayed via the user interface to the annotating user; a comparison module configured to compare the manually-generated annotations of the intents and the mentions of the set of sentences against multiple machine-generated annotations, each of the machine-generated annotations being generated using a grammar model from among multiple grammar models, the comparing being a function of each grammar model among the multiple grammar models stored in a memory, the machine-generated annotations corresponding to intents and mentions of the set of sentences, each of the multiple grammar models being a grammar model having a plurality of stub grammar models, and wherein the grammar model correlates an input of a given user to a response to the given user; a scoring module configured to compute a respective intra-sentence score for each respective grammar model based on respective intra-sentence scores of intents and mentions of each machine-generated annotation for the respective grammar model compared to the intents and mentions of the manually-generated annotations, the respective score being a match of each machine-generated annotation for the respective grammar model to each manually-generated annotation; and a recommendation module configured to; provide, via a user interface, a list of recommended grammar models of the multiple grammar models to employ based on the scores corresponding to the machine-generated annotations corresponding to the respective grammar model, the list including grammar models scoring above a particular semantic confidence threshold, receive a selection, from the annotating user via the user interface, of a grammar model of the recommended grammar model, and adding the user-selected grammar model of the list of recommended grammar models to the ontology of the NLU system. - View Dependent Claims (7, 8, 9, 10)
-
-
11. A non-transitory computer-readable medium configured to store instructions, the instructions, when loaded and executed by a processor, cause the processor to:
-
build an ontology for performing natural language understanding (NLU) processing by an NLU system by; receiving inputs, via a user interface, from an annotating user of electronically formatted manually-generated annotations of intents and mentions of a set of sentences that are displayed via a user interface to the annotating user; comparing the manually-generated annotations of the intents and the mentions of the set of sentences against multiple machine-generated annotations, each of the machine-generated annotations being generated using a grammar model from among multiple grammar models, the comparing being a function of each grammar model among the multiple grammar models stored in a memory, the machine-generated annotations corresponding to intents and mentions of the set of sentences, each of the multiple grammar models being a grammar model having a plurality of stub grammar models, and wherein the grammar model correlates an input of a given user to a response to the given user; computing a respective intra-sentence score for each respective grammar model based on respective intra-sentence scores of intents and mentions of each machine-generated annotation for the respective grammar model compared to the intents and mentions of the manually-generated annotations, the respective score being a match of each machine-generated annotation for the respective grammar model to each manually-generated annotation; and providing, via the user interface, a list of recommended grammar models of the multiple grammar models to employ based on the scores corresponding to the machine-generated annotations corresponding to the respective grammar model, the list including grammar models scoring above a particular semantic confidence threshold; receiving a selection, from the annotating user via the user interface, of a grammar model of the recommended grammar models; and adding the user-selected grammar model of the list of recommended grammar models the ontology of the NLU system. - View Dependent Claims (12, 13, 14)
-
Specification