TECHNIQUES FOR COMBINING HUMAN AND MACHINE LEARNING IN NATURAL LANGUAGE PROCESSING

• US 20190311024A1
• Filed: 11/09/2018
• Published: 10/10/2019
• Est. Priority Date: 12/09/2014
• Status: Abandoned Application

• Current Assignee: AI IP Investments Limited
• Original Assignee: AIPARC holdings Pte. Ltd.
• Inventors: Munro, Robert J., Walker, Christopher, Luger, Sarah K., Callahan, Brendan D., King, Gary C., Tepper, Paul A., Thompson, Jana N., Schnoebelen, Tyler J., Brenier, Jason, Long, Jessica D.
• Application Number:
  US16/185,843
• Publication Number:
  US 20190311024A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/367 Ontology
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/137 Hierarchical processing, e.g. outlines
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/221 Parsing markup language streams streaming G06F40/149
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/42 Data-driven translation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

Methods, apparatuses and computer readable medium are presented for generating a natural language model. A method for generating a natural language model comprises: receiving more than one annotation of a document; calculating a level of agreement among the received annotations; determining that a criterion among a first criterion, a second criterion, and a third criterion is satisfied based at least in part on the level of agreement; determining an aggregated annotation representing an aggregation of information in the received annotations and training a natural language model using the aggregated annotation, when the first criterion is satisfied; generating at least one human readable prompt configured to receive additional annotations of the document, when the second criterion is satisfied; and discarding the received annotations from use in training the natural language model, when the third criterion is satisfied.

Claims

1. A method for generating a natural language model, the method comprising:

receiving more than one annotation of a document;

calculating a level of agreement among the received annotations;

determining that a criterion among a first criterion, a second criterion, and a third criterion is satisfied based at least in part on the level of agreement;

determining an aggregated annotation representing an aggregation of information in the received annotations and training a natural language model using the aggregated annotation, when the first criterion is satisfied;

generating at least one human readable prompt configured to receive additional annotations of the document, when the second criterion is satisfied; and

discarding the received annotations from use in training the natural language model, when the third criterion is satisfied.

2. The method of claim 1, wherein the second criterion is satisfied when the number of annotations received is less than a minimum number.

3. The method of claim 1, wherein the annotations of the document comprise selection of one or more portions of the document relevant to one or more topics.

4. The method of claim 1, wherein the annotations of the document comprise selection of one or more categories among a plurality of categories.

5. The method of claim 4, wherein the level of agreement is determined for each category based on a percentage of annotations that select said category.

6. The method of claim 5, wherein:

the first criterion is satisfied when the number of annotations received is at least a minimum number and the level of agreement for a category is at least a threshold level; and

the aggregated annotation is determined as selecting or not selecting said category.

7. The method of claim 5, wherein the second criterion is satisfied when the number of annotations received is less than a maximum number and the level of agreement is less than a threshold level.

8. The method of claim 5, wherein the third criterion is satisfied when the number of annotations received is at least a maximum number and the level of agreement is less than a threshold level.

9. The method of claim 4, wherein a numerical value is assigned to each of the plurality of categories.

10. The method of claim 9, wherein:

the level of agreement comprises a difference between the highest numerical value and the lowest numerical value among the selected categories;

the first criterion is satisfied when the difference is no more than a threshold value; and

the third criterion is satisfied when the difference is more than the threshold value.

11. The method of claim 10, wherein the aggregated annotation is determined as selection of a category with the numerical value closest to a mean of the numerical values of all received annotations.

12. The method of claim 10, wherein the aggregated annotation is determined as selection of a category with the numerical value closest to a median of the numerical values of all received annotations.

13. The method of claim 1, wherein determining that the criterion among the first criterion, the second criterion, and the third criterion is satisfied is further based on a result of an analysis of the document by one or more pre-existing natural language models.

14. The method of claim 1, wherein determining that the criterion among the first criterion, the second criterion, and the third criterion is satisfied is further based on known performance levels of annotators.

15. The method of claim 1, wherein at least one of the annotations received comprises prediction by a pre-existing natural language model.

16. An apparatus for generating a natural language model, the apparatus comprising one or more processors configured to:

receive more than one annotation of a document;

calculate a level of agreement among the received annotations;

determine that a criterion among a first criterion, a second criterion, and a third criterion is satisfied based at least in part on the level of agreement;

determine an aggregated annotation representing an aggregation of information in the received annotations and train a natural language model using the aggregated annotation, when the first criterion is satisfied;

generate at least one human readable prompt configured to receive additional annotations of the document, when a second criterion is satisfied; and

discard the received annotations from use in training the natural language model, when the third criterion is satisfied.

17. The apparatus of claim 16, wherein the annotations of the document comprise selection of one or more categories among a plurality of categories.

18. The apparatus of claim 17, wherein the level of agreement is determined for each category based on a percentage of annotations that select said category.

19. The apparatus of claim 17, wherein

a numerical value is assigned to each of the plurality of categories;

the level of agreement comprises a difference between the highest numerical value and the lowest numerical value among the selected categories;

the first criterion is satisfied when the difference is no more than a threshold value; and

the third criterion is satisfied when the difference is more than a threshold value.

20. A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to:

receive more than one annotation of a document;

calculate a level of agreement among the received annotations;

determine that a criterion among a first criterion, a second criterion, and a third criterion is satisfied based at least in part on the level of agreement;

determine an aggregated annotation representing an aggregation of information in the received annotations and train a natural language model using the aggregated annotation, when the first criterion is satisfied;

generate at least one human readable prompt configured to receive additional annotations of the document, when a second criterion is satisfied; and

discard the received annotations from use in training the natural language model, when the third criterion is satisfied.