System for automatically annotating training data for a natural language understanding system
First Claim
Patent Images
1. A method of generating annotated training data to train a natural language understanding (NLU) system having one or models, comprising:
- generating a proposed annotation with the NLU system for each unit of unannotated training data;
displaying the proposed annotations for user verification or correction to obtain a user-confirmed annotation; and
training the NLU system with the user-confirmed annotation.
2 Assignments
0 Petitions
Accused Products
Abstract
The present invention uses a natural language understanding system that is currently being trained to assist in annotating traomomg data for training that natural language understanding system. Unannotated training data is provided to the system and the system proposes annotations to the training data. The user is offered an opportunity to confirm or correct the proposed annotations, and the system is trained with the corrected or verified annotations.
-
Citations
51 Claims
-
1. A method of generating annotated training data to train a natural language understanding (NLU) system having one or models, comprising:
-
generating a proposed annotation with the NLU system for each unit of unannotated training data;
displaying the proposed annotations for user verification or correction to obtain a user-confirmed annotation; and
training the NLU system with the user-confirmed annotation. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29)
-
-
30. A user interface for training a natural language understanding (NLU) system that has one or more models, the user interface comprising:
-
a first portion displaying a model display representative of the one or more models;
a second portion displaying unannotated training inputs; and
a third portion displaying a proposed annotation for a selected one of the unannotated training inputs. - View Dependent Claims (31, 32, 33, 34, 35, 36)
-
-
37. A method of generating annotated training data for training a natural language understanding (NLU) system having at least one model, comprising:
-
generating a proposed annotation for a unit of unannotated training data;
calculating a confidence measure for a plurality of different portions of the proposed annotation; and
displaying the proposed annotation by visually contrasting portions that have a corresponding confidence measure that falls below a threshold level. - View Dependent Claims (38, 39, 40)
-
-
41. A method of generating annotated training data for training a natural language understanding (NLU) system having at least one model, comprising:
-
generating, with the NLU system, a proposed annotation for a unit of unannotated training data;
displaying the proposed annotation with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation;
training the model with the user-confirmed annotation; and
checking for inconsistencies among user-confirmed annotation and data already used to train the model by determining whether the model accurately predicts the prior user-confirmed annotations. - View Dependent Claims (42, 43, 44)
-
-
45. A method of generating annotated training data for training a natural language understanding (NLU) system, comprising:
-
generating, with the NLU system, a proposed annotation for each of a plurality of units of unannotated training data;
calculating a confidence measure for each of the proposed annotations;
displaying the proposed annotations in an order based on the corresponding confidence measures, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation. - View Dependent Claims (46, 47)
-
-
48. A method of generating annotated training data for training a natural language understanding (NLU) system, comprising:
-
generating, with the NLU system, a proposed annotation for each of a plurality of units of unannotated training data, each proposed annotation having a type;
displaying the proposed annotations in an order based on the type, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation. - View Dependent Claims (49)
-
-
50. A method of generating annotated training data for training a natural language understanding (NLU) system employing a plurality of different natural language understanding techniques, comprising:
-
generating, with each natural language training technique, a proposed annotation for a unit of unannotated training data, to obtain a plurality of proposed annotations;
selecting one of the plurality of proposed annotations; and
displaying the selected proposed annotation, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation. - View Dependent Claims (51)
-
Specification