System for automatically annotating training data for a natural language understanding system

• US 20030212544A1
• Filed: 05/10/2002
• Published: 11/13/2003
• Est. Priority Date: 05/10/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Acero, Alejandro, Wang, Ye-Yi, Wong, Leon
• Granted Patent: US 7,548,847 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

The present invention uses a natural language understanding system that is currently being trained to assist in annotating traomomg data for training that natural language understanding system. Unannotated training data is provided to the system and the system proposes annotations to the training data. The user is offered an opportunity to confirm or correct the proposed annotations, and the system is trained with the corrected or verified annotations.

Claims

1. A method of generating annotated training data to train a natural language understanding (NLU) system having one or models, comprising:

generating a proposed annotation with the NLU system for each unit of unannotated training data;

displaying the proposed annotations for user verification or correction to obtain a user-confirmed annotation; and

training the NLU system with the user-confirmed annotation.

2. The method of claim 1 and further comprising:

initializing one or more models in the NLU system.

3. The method of claim 1 wherein displaying the proposed annotation for user verification or correction comprises:

receiving a user input indicative of a user-identified portion of the proposed annotation; and

displaying a plurality of alternative proposed annotations for the user-identified portion.

4. The method of claim 3 wherein the one or more models impose model constraints and wherein displaying a plurality of proposed annotations comprises:

displaying an alternative proposed annotation for the user-identified portion only if the alternative proposed annotation can lead to an overall annotation for the unit that is consistent with the model constraints.

5. The method of claim 4 wherein the proposed annotation includes parent and child nodes and wherein displaying a plurality of alternative proposed annotations includes displaying a user actuable delete node input which, when actuated deletes a child node.

6. The method of claim 5 wherein displaying a plurality of alternative proposed annotations comprises:

displaying a portion of the unit not covered by the proposed annotation; and

displaying alternative proposed annotations for the portion not covered.

7. The method of claim 6 wherein the user can select a segment of the portion not covered and wherein displaying alternative proposed annotations comprises:

displaying alternative proposed annotations for the selected segment.

8. The method of claim 7 wherein displaying alternative proposed annotations for the selected segment comprises:

displaying a proposed annotation only if it is consistent with the model constraints.

9. The method of claim 1 wherein each unit of the training data comprises a sentence.

10. The method of claim 1 wherein each unit of the training data comprises a phrase.

11. The method of claim 1 wherein each unit of the training data comprises multiple sentences.

12. The method of claim 1 wherein each unit of the training data comprises a single word.

13. The method of claim 1 and further comprising:

displaying an indication of a volume of training data used to train a plurality of different portions of the one or more models.

14. The method of claim 13 wherein displaying an indication of a volume of training data comprises:

displaying a representation of the one or more models; and

visually contrasting portions of the one or more models that have been trained with a threshold volume of training data.

15. The method of claim 14 wherein the threshold volume of training data is dynamic, based on one or more performance criteria for the one or more models.

16. The method of claim 1 wherein displaying the proposed annotation for verification or correction comprises:

generating a confidence metric for the proposed annotation; and

visually contrasting a portion of the displayed proposed annotation based on the confidence metric.

17. The method of claim 16 wherein visually contrasting comprises:

visually contrasting the portion of the displayed annotation that has an associated confidence metric that is below a threshold level.

18. The method of claim 1 and further comprising:

prior to generating a proposed annotation, receiving a limiting user indication; and

limiting natural language understanding processing used to generate the proposed annotation based on the limiting user indication.

19. The method of claim 18 wherein limiting natural language understanding processing comprises:

limiting the natural language understanding processing to utilizing only user-identified portions of the one or more models.

20. The method of claim 1 and further comprising:

identifying inconsistencies between the user-confirmed annotation and prior annotations.

21. The method of claim 20 and further comprising:

if an inconsistency is identified, displaying the user-confirmed annotation visually contrasting inconsistent portions of the user-confirmed annotation.

22. The method of claim 1 wherein displaying the proposed annotations comprises:

generating a confidence metric for each proposed annotation; and

displaying the proposed annotations in an order based on the confidence metrics.

23. The method of claim 22 wherein displaying the proposed annotations based on the confidence metrics comprises:

displaying the proposed annotations in order of ascending confidence metrics.

24. The method of claim 22 wherein displaying the proposed annotations based on the confidence metrics comprises:

displaying the proposed annotations in order of descending confidence metrics.

25. The method of claim 1 wherein displaying the proposed annotations comprises:

sorting the proposed annotations based on type of annotation.

26. The method of claim 25 wherein displaying the proposed annotations comprises:

displaying similar types of annotations closely proximate one another.

27. The method of claim 25 wherein displaying the proposed annotations comprises:

providing a user actuable input which, when actuated, allows the user to correct or verify similar types of annotations sequentially.

28. The method of claim 1 wherein generating proposed annotations with the NLU system, comprises:

generating a plurality of proposed annotations for each unit using a plurality of different NLU subsystems.

29. The method of claim 28 wherein generating proposed annotations further comprises:

choosing one of the proposed annotations for each unit to be displayed.

30. A user interface for training a natural language understanding (NLU) system that has one or more models, the user interface comprising:

a first portion displaying a model display representative of the one or more models;

a second portion displaying unannotated training inputs; and

a third portion displaying a proposed annotation for a selected one of the unannotated training inputs.

31. The user interface of claim 30 and further comprising:

a fourth portion displaying, in response to a user input identifying a portion of the displayed proposed annotation, alternative proposed annotations for the identified portion.

32. The user interface of claim 31 wherein the NLU system includes one or more model constraints and wherein the fourth portion displays alternative proposed annotations for the identified portion only of those alternative proposed annotations are part of an overall annotation for the selected unannotated training input that is consistent with the model constraints.

33. The user interface of claim 30 wherein the NLU system calculates a confidence measure for each proposed annotation and wherein the second portion displays the unannotated training inputs in an order based on the confidence measures associated with proposed annotations corresponding to the training inputs.

34. The user interface of claim 33 wherein the second portion displays the unannotated training inputs in ascending order of the confidence measures.

35. The user interface of claim 30 and further comprising:

a fourth portion displaying the selected one of the unannotated training inputs.

36. The user interface of claim 30 wherein the first portion displays the model display in the form of a dependency structure having parent and child nodes.

37. A method of generating annotated training data for training a natural language understanding (NLU) system having at least one model, comprising:

generating a proposed annotation for a unit of unannotated training data;

calculating a confidence measure for a plurality of different portions of the proposed annotation; and

displaying the proposed annotation by visually contrasting portions that have a corresponding confidence measure that falls below a threshold level.

38. The method of claim 37 wherein displaying comprises:

visually contrasting only a portion having the lowest confidence measure.

39. The method of claim 37 wherein displaying comprises:

displaying user actuable inputs, actuable by the user to confirm or correct the proposed annotation to obtain an user-confirmed annotation.

40. The method of claim 39 and further comprising:

training the model with the user-confirmed annotation.

41. A method of generating annotated training data for training a natural language understanding (NLU) system having at least one model, comprising:

generating, with the NLU system, a proposed annotation for a unit of unannotated training data;

displaying the proposed annotation with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation;

training the model with the user-confirmed annotation; and

checking for inconsistencies among user-confirmed annotation and data already used to train the model by determining whether the model accurately predicts the prior user-confirmed annotations.

42. The method of claim 41 and further comprising:

if an inconsistency is detected, re-displaying the user-confirmed annotations spawning the inconsistency.

43. The method of claim 42 wherein re-displaying comprises:

displaying the user-confirmed annotations by visually contrasting portions in which the NLU system has a calculated confidence measure below a threshold.

44. The method of claim 41 wherein checking comprises:

re-generating annotations with the NLU system for the unit of unannotated training data and comparing the re-generated annotations with the user-confirmed annotations.

45. A method of generating annotated training data for training a natural language understanding (NLU) system, comprising:

generating, with the NLU system, a proposed annotation for each of a plurality of units of unannotated training data;

calculating a confidence measure for each of the proposed annotations;

displaying the proposed annotations in an order based on the corresponding confidence measures, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation.

46. The method of claim 45 wherein displaying comprises:

displaying the proposed annotations in order based on ascending value of the corresponding confidence measures.

47. The method of claim 45 wherein displaying comprises:

displaying the proposed annotations in order based on descending value of the corresponding confidence measures.

48. A method of generating annotated training data for training a natural language understanding (NLU) system, comprising:

generating, with the NLU system, a proposed annotation for each of a plurality of units of unannotated training data, each proposed annotation having a type;

displaying the proposed annotations in an order based on the type, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation.

49. The method of claim 48 wherein displaying comprises:

sorting the proposed annotations based on type; and

displaying the proposed annotations with those of a similar type grouped together.

50. A method of generating annotated training data for training a natural language understanding (NLU) system employing a plurality of different natural language understanding techniques, comprising:

generating, with each natural language training technique, a proposed annotation for a unit of unannotated training data, to obtain a plurality of proposed annotations;

selecting one of the plurality of proposed annotations; and

displaying the selected proposed annotation, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation.

51. The method of claim 50 and further comprising:

training the NLU system with the user-confirmed annotation.