Automated quality assurance checks for improving the construction of natural language understanding systems

• US 10,339,217 B2
• Filed: 06/26/2017
• Issued: 07/02/2019
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Tremblay, Real, Tremblay, Jerome, Robillard, Serge, Liscombe, Jackson, Andreevskaia, Alina, Chung, Tagyoung
• Primary Examiner(s): Spooner, Lamont M
• Application Number:
  US15/633,030
• Publication Number:
  US 20180143962A1
• Time in Patent Office: 736 Days
• Field of Search: 704 2- 8
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

Aspects described herein provide quality assurance checks for improving the construction of natural language understanding grammars. An annotation module may obtain a set of annotations for a set of text samples based, at least in part, on an ontology and a grammar. A quality assurance module may automatically perform one or more quality assurance checks on the set of annotations, the ontology, the grammar, or combinations thereof. The quality assurance module may generate a list of flagged annotations during performance of a quality assurance check. The list of flagged annotations may be presented at an annotation review interface displayed at a display device. One of the flagged annotations may be selected and presented at an annotation interface displayed at the display device. Responsive to presentation of the flagged annotation, the ontology, the grammar, the flagged annotation selected, or combinations thereof may be updated based on user input received.

Claims

1. A method for improving development of natural language understanding systems, the method comprising:

storing a plurality of annotations at a data store of a computing device wherein each of the plurality of annotations corresponds to one of a plurality of text samples;

processing, by the computing device, one or more annotations of the plurality of annotations wherein the processing selects an annotation from the plurality of annotations to present to a user for review, wherein processing the one or more annotations comprises;

selecting an annotation candidate and a comparison annotation candidate from the plurality of annotations; and

filtering the annotation candidate from the plurality of annotations to present to the user for review in response to determining that the concept of the annotation candidate is a hyponym of the concept of the comparison annotation candidate and determining that the annotation candidate covers a smaller portion of the text sample compared to the comparison annotation candidate;

presenting the annotation to the user at a display of the computing device;

receiving input from the user via an input interface of the computing device;

modifying the annotation based on the input to obtain a modified annotation; and

modifying, by the computing device, at least one of a grammar or an ontology based on the modified annotation.

2. The method of claim 1, wherein:

the one or more annotations comprise a first annotation and a second annotation that each comprise a common set of hypernyms; and

the processing comprises comparing a first arrangement of the common set of hypernyms in the first annotation to a second arrangement of the common set of hypernyms in the second annotation and selecting, as the annotation to present to the user for review, at least one of the first annotation or the second annotation based on determining that the first arrangement does not match the second arrangement.

3. The method of claim 1, wherein:

the one or more annotations comprise a first annotation comprising at least one first hypernym and a second annotation comprising at least one second hypernym; and

the processing comprises comparing the at least one first hypernym to the at least one second hypernym and selecting, as the annotation to present to the user for review, at least one of the first annotation or the second annotation based on determining that none of the at least one first hypernym is the same as the at least one second hypernym.

4. The method of claim 3, wherein:

the input received comprises a selection of a hypernym; and

modifying the first annotation and the second annotation comprises including, in the first annotation and the second annotation, the hypernym selected.

5. The method of claim 1, wherein:

the processing comprises selecting, as the annotation to present to the user for review, at least one annotation of the one or more annotations based on determining that the at least one annotation includes an n-gram that can be parsed by more than one grammar rule of the grammar.

6. The method of claim 1, wherein:

the processing comprises selecting, as the annotation to present to the user for review, at least one annotation of the one or more annotations based on determining that the at least one annotation is associated with a cycle of the grammar.

7. The method of claim 1, wherein:

the annotation selected comprises a hypernym that is not defined in the ontology.

8. A system for improving development of natural language understanding systems, the system comprising:

one or more processors;

a data store storing a plurality of annotations wherein each of the plurality of annotations corresponds to one of a plurality of text samples;

a display;

an input interface; and

memory storing instructions that, when executed by the one or more processors, cause the system to;

process one or more annotations of the plurality of annotations wherein the processing selects an annotation from the plurality of annotations to present to a user for review, wherein the system processes the one or more annotations at least by;

selecting an annotation candidate and a comparison annotation candidate from the plurality of annotations, and

filtering the annotation candidate from the plurality of annotations to present to the user for review in response to determining that the concept of the annotation candidate is a hyponym of the concept of the comparison annotation candidate and determining that the annotation candidate covers a smaller portion of the text sample compared to the comparison annotation candidate,

present the annotation to the user at the display,

receive input from the user via the input interface,

modify the annotation based on the input to obtain a modified annotation, and

modify at least one of a grammar or an ontology based on the modified annotation.

9. The system of claim 8, wherein:

the one or more annotations comprise a first annotation and a second annotation that each comprise a common set of hypernyms; and

the system processes the one or more annotations at least by comparing a first arrangement of the common set of hypernyms in the first annotation to a second arrangement of the common set of hypernyms in the second annotation and selecting, as the annotation to present to the user for review, at least one of the first annotation or the second annotation based on determining that the first arrangement does not match the second arrangement.

10. The system of claim 8, wherein:

the one or more annotations comprise a first annotation comprising at least one first hypernym and a second annotation comprising at least one second hypernym; and

the system processes the one or more annotations at least by comparing the at least one first hypernym to the at least one second hypernym and selecting, as the annotation to present to the user for review, at least one of the first annotation or the second annotation based on determining that none of the at least one first hypernym is the same as the at least one second hypernym.

11. The system of claim 10, wherein:

the input received comprises a selection of a hypernym; and

the system modifies the first annotation and the second annotation at least by including, in the first annotation and the second annotation, the hypernym selected.

12. The system of claim 8, wherein:

the system processes the one or more annotations at least by selecting, as the annotation to present to the user for review, at least one annotation of the one or more annotations based on determining that the at least one annotation includes an n-gram that can be parsed by more than one grammar rule of the grammar.

13. The system of claim 8, wherein:

the system processes the one or more annotations at least by selecting, as the annotation to present to the user for review, at least one annotation of the one or more annotations based on determining that the at least one annotation is associated with a cycle of the grammar.

14. The system of claim 8, wherein:

the annotation selected comprises a hypernym that is not defined in the ontology.

15. Non-transitory computer-readable storage media for improving the development of natural language understanding systems, the non-transitory computer-readable storage media having computer-executable instructions stored thereon that, when executed by one or more processors of a computing device, cause the computing device to:

process one or more annotations of a plurality of annotations stored at a data store of the computing device wherein the processing selects an annotation from the plurality of annotations to present to a user for review, wherein the instructions, when executed, cause the computing device to process the one or more annotations at least by;

selecting an annotation candidate and a comparison annotation candidate from the plurality of annotations, and

filtering the annotation candidate from the plurality of annotations to present to the user for review in response to determining that the concept of the annotation candidate is a hyponym of the concept of the comparison annotation candidate and determining that the annotation candidate covers a smaller portion of the text sample compared to the comparison annotation candidate,

present the annotation to the user at a display of the computing device,

receive input from the user via an input interface of the computing device,

modify the annotation based on the input to obtain a modified annotation, and

modify at least one of a grammar or an ontology based on the modified annotation.

16. The non-transitory computer-readable storage media of claim 15, wherein:

the one or more annotations comprise a first annotation and a second annotation that each comprise a common set of hypernyms; and

the instructions, when executed, cause the computing device to process the one or more annotations at least by comparing a first arrangement of the common set of hypernyms in the first annotation to a second arrangement of the common set of hypernyms in the second annotation and selecting, as the annotation to present to the user for review, at least one of the first annotation or the second annotation based on determining that the first arrangement does not match the second arrangement.

17. The non-transitory computer-readable storage media of claim 15, wherein:

the one or more annotations comprise a first annotation comprising at least one first hypernym and a second annotation comprising at least one second hypernym; and

the instructions, when executed, cause the computing device to process the one or more annotations at least by comparing the at least one first hypernym to the at least one second hypernym and selecting, as the annotation to present to the user for review, at least one of the first annotation or the second annotation based on determining that none of the at least one first hypernym is the same as the at least one second hypernym.

18. The non-transitory computer-readable storage media of claim 17, wherein:

the input received comprises a selection of a hypernym; and

the instructions, when executed, cause the computing device to modify the first annotation and the second annotation at least by including, in the first annotation and the second annotation, the hypernym selected.

19. The non-transitory computer-readable storage media of claim 15, wherein:

the instructions, when executed, cause the computing device to process the one or more annotations at least by selecting, as the annotation to present to the user for review, at least one annotation of the one or more annotations based on determining that the at least one annotation includes an n-gram that can be parsed by more than one grammar rule of the grammar.

20. The non-transitory computer-readable storage media of claim wherein:

the instructions, when executed, cause the computing device to process the one or more annotations at least by selecting, as the annotation to present to the user for review, at least one annotation of the one or more annotations based on determining that the at least one annotation is associated with a cycle of the grammar.