Automated quality assurance checks for improving the construction of natural language understanding systems
First Claim
1. A method for improving development of natural language understanding systems, the method comprising:
- storing a plurality of annotations at a data store of a computing device wherein each of the plurality of annotations corresponds to one of a plurality of text samples;
processing, by the computing device, one or more annotations of the plurality of annotations wherein the processing selects an annotation from the plurality of annotations to present to a user for review, wherein processing the one or more annotations comprises;
selecting an annotation candidate and a comparison annotation candidate from the plurality of annotations; and
filtering the annotation candidate from the plurality of annotations to present to the user for review in response to determining that the concept of the annotation candidate is a hyponym of the concept of the comparison annotation candidate and determining that the annotation candidate covers a smaller portion of the text sample compared to the comparison annotation candidate;
presenting the annotation to the user at a display of the computing device;
receiving input from the user via an input interface of the computing device;
modifying the annotation based on the input to obtain a modified annotation; and
modifying, by the computing device, at least one of a grammar or an ontology based on the modified annotation.
2 Assignments
0 Petitions
Accused Products
Abstract
Aspects described herein provide quality assurance checks for improving the construction of natural language understanding grammars. An annotation module may obtain a set of annotations for a set of text samples based, at least in part, on an ontology and a grammar. A quality assurance module may automatically perform one or more quality assurance checks on the set of annotations, the ontology, the grammar, or combinations thereof. The quality assurance module may generate a list of flagged annotations during performance of a quality assurance check. The list of flagged annotations may be presented at an annotation review interface displayed at a display device. One of the flagged annotations may be selected and presented at an annotation interface displayed at the display device. Responsive to presentation of the flagged annotation, the ontology, the grammar, the flagged annotation selected, or combinations thereof may be updated based on user input received.
24 Citations
20 Claims
-
1. A method for improving development of natural language understanding systems, the method comprising:
-
storing a plurality of annotations at a data store of a computing device wherein each of the plurality of annotations corresponds to one of a plurality of text samples; processing, by the computing device, one or more annotations of the plurality of annotations wherein the processing selects an annotation from the plurality of annotations to present to a user for review, wherein processing the one or more annotations comprises; selecting an annotation candidate and a comparison annotation candidate from the plurality of annotations; and filtering the annotation candidate from the plurality of annotations to present to the user for review in response to determining that the concept of the annotation candidate is a hyponym of the concept of the comparison annotation candidate and determining that the annotation candidate covers a smaller portion of the text sample compared to the comparison annotation candidate; presenting the annotation to the user at a display of the computing device; receiving input from the user via an input interface of the computing device; modifying the annotation based on the input to obtain a modified annotation; and modifying, by the computing device, at least one of a grammar or an ontology based on the modified annotation. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A system for improving development of natural language understanding systems, the system comprising:
-
one or more processors; a data store storing a plurality of annotations wherein each of the plurality of annotations corresponds to one of a plurality of text samples; a display; an input interface; and memory storing instructions that, when executed by the one or more processors, cause the system to; process one or more annotations of the plurality of annotations wherein the processing selects an annotation from the plurality of annotations to present to a user for review, wherein the system processes the one or more annotations at least by; selecting an annotation candidate and a comparison annotation candidate from the plurality of annotations, and filtering the annotation candidate from the plurality of annotations to present to the user for review in response to determining that the concept of the annotation candidate is a hyponym of the concept of the comparison annotation candidate and determining that the annotation candidate covers a smaller portion of the text sample compared to the comparison annotation candidate, present the annotation to the user at the display, receive input from the user via the input interface, modify the annotation based on the input to obtain a modified annotation, and modify at least one of a grammar or an ontology based on the modified annotation. - View Dependent Claims (9, 10, 11, 12, 13, 14)
-
-
15. Non-transitory computer-readable storage media for improving the development of natural language understanding systems, the non-transitory computer-readable storage media having computer-executable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to:
-
process one or more annotations of a plurality of annotations stored at a data store of the computing device wherein the processing selects an annotation from the plurality of annotations to present to a user for review, wherein the instructions, when executed, cause the computing device to process the one or more annotations at least by; selecting an annotation candidate and a comparison annotation candidate from the plurality of annotations, and filtering the annotation candidate from the plurality of annotations to present to the user for review in response to determining that the concept of the annotation candidate is a hyponym of the concept of the comparison annotation candidate and determining that the annotation candidate covers a smaller portion of the text sample compared to the comparison annotation candidate, present the annotation to the user at a display of the computing device, receive input from the user via an input interface of the computing device, modify the annotation based on the input to obtain a modified annotation, and modify at least one of a grammar or an ontology based on the modified annotation. - View Dependent Claims (16, 17, 18, 19)
-
-
20. The non-transitory computer-readable storage media of claim wherein:
the instructions, when executed, cause the computing device to process the one or more annotations at least by selecting, as the annotation to present to the user for review, at least one annotation of the one or more annotations based on determining that the at least one annotation is associated with a cycle of the grammar.
Specification