Ingesting documents using multiple ingestion pipelines

• US 10,318,591 B2
• Filed: 06/02/2015
• Issued: 06/11/2019
• Est. Priority Date: 06/02/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Andrejko, Pamela D., Freed, Andrew R., Murch, Cynthia M., Nordland, Jan M., Rivero, Humberto R.
• Primary Examiner(s): Ries, Laurie A
• Application Number:
  US14/728,050
• Publication Number:
  US 20160359894A1
• Time in Patent Office: 1,470 Days
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes

Abstract

A primary ingestion pipeline configured for use in natural language processing includes annotators configured for annotating documents. The annotators and documents to be annotated are evaluated. Based on the evaluations, an ingestion risk score is generated for each document. Each ingestion risk score represents a likelihood that an associated document will not successfully be annotated by the annotators. Each ingestion risk score is compared to a set of risk criteria. Based on the comparisons, a determination is made that each document of a first set of documents satisfies the set of risk criteria. A further determination is made, based on the comparisons, that each document of a second set of documents does not satisfy the set of risk criteria. In response to these determinations, the first set of documents is entered into the primary ingestion pipeline and the second set of documents is provided special handling.

Claims

1. A method for analyzing a primary ingestion pipeline, the primary ingestion pipeline configured for using natural language processing (NLP) to populate a corpus with documents annotated with metadata tags such that the corpus becomes usable by a computer system for generating answers to questions as they are posed by users, the primary ingestion pipeline including a plurality of annotators configured for annotating documents passing through the primary ingestion pipeline, wherein each annotator of the plurality of annotators is configured to annotate the documents with a different defined subset of the metadata tags, the method comprising:

evaluating the plurality of annotators;

evaluating a plurality of documents to be annotated by the plurality of annotators;

generating, based on the evaluating the plurality of annotators and further based on the evaluating the plurality of documents, an ingestion risk score for each document of the plurality of documents, wherein each ingestion risk score represents a likelihood that an associated document will not successfully be annotated by all of the plurality of annotators while passing sequentially through each annotator in the primary ingestion pipeline;

comparing each ingestion risk score to a set of risk criteria;

determining, based on the comparing, that each document of a first set of documents of the plurality of documents satisfies the set of risk criteria and that each document of a second set of documents of the plurality of documents does not satisfy the set of risk criteria;

entering, in response to the determining, the first set of documents into the primary ingestion pipeline; and

providing, in response to the determining, special handling to the second set of documents.

2. The method of claim 1, wherein the providing special handling includes entering the second set of documents in a secondary ingestion pipeline configured for use in NLP, the secondary ingestion pipeline including the plurality of annotators configured for annotating documents passing through the secondary ingestion pipeline.

3. The method of claim 2, wherein the primary ingestion pipeline has a first log level setting, and wherein the secondary ingestion pipeline has a second log level setting that is higher than the first log level setting, the method further comprising:

logging, based on the first log level setting and while the first set of documents are passing through the primary ingestion pipeline, execution of program code of the plurality of annotators of the primary ingestion pipeline;

logging, based on the second log level setting and while the second set of documents are passing through the secondary ingestion pipeline, execution of program code of the plurality of annotators of the secondary ingestion pipeline.

4. The method of claim 2 further comprising:

annotating the first set of documents with the plurality of annotators in the primary ingestion pipeline; and

annotating the second set of documents with the plurality of annotators in the secondary ingestion pipeline.

5. The method of claim 2 further comprising:

generating, based on the evaluating the plurality of annotators and further based on the evaluating the plurality of documents, a document-annotator pair risk score for each document of the plurality of documents for each annotator of the plurality of annotators, wherein each document-annotator pair risk score represents a likelihood that an associated document will not successfully be annotated by an associated annotator while passing through the primary ingestion pipeline;

comparing each document-annotator pair risk score to a secondary set of risk criteria;

determining, based on the comparing each document-annotator pair risk score to the secondary set of risk criteria, that, for a first annotator of the plurality of annotators, a first document of the first set of documents does not satisfy the secondary set of risk criteria;

transferring, after entering the first document into the primary ingestion pipeline and in response to the secondary set of risk criteria failure determination, the first document from the primary ingestion pipeline to the secondary ingestion pipeline;

annotating the first document with the first annotator in the secondary ingestion pipeline; and

transferring, after the annotating with the first annotator, the first document back to the primary ingestion pipeline.

6. The method of claim 5 further comprising:

annotating the first document in the primary ingestion pipeline with the plurality of annotators except for the first annotator.

7. The method of claim 1, wherein the set of risk criteria is a risk threshold.

8. A computer program for analyzing a primary ingestion pipeline configured for use in natural language processing (NLP), the primary ingestion pipeline including a plurality of annotators configured for annotating documents passing through the primary ingestion pipeline, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

evaluating the plurality of annotators;

evaluating a plurality of documents to be annotated by the plurality of annotators;

generating, based on the evaluating the plurality of annotators and further based on the evaluating the plurality of documents, an ingestion risk score for each document of the plurality of documents, wherein each ingestion risk score represents a likelihood that an associated document will not successfully be annotated by the plurality of annotators while passing through the primary ingestion pipeline;

comparing each ingestion risk score to a set of risk criteria;

determining, based on the comparing, that each document of a first set of documents of the plurality of documents satisfies the set of risk criteria and that each document of a second set of documents of the plurality of documents does not satisfy the set of risk criteria;

entering, in response to the determining, the first set of documents into the primary ingestion pipeline; and

providing, in response to the determining, special handling to the second set of documents.

9. The computer program product of claim 8, wherein the providing special handling includes entering the second set of documents in a secondary ingestion pipeline configured for use in NLP, the secondary ingestion pipeline including the plurality of annotators configured for annotating documents passing through the secondary ingestion pipeline.

10. The computer program product of claim 9, wherein the primary ingestion pipeline has a first log level setting, wherein the secondary ingestion pipeline has a second log level setting that is higher than the first log level setting, and wherein the method further comprises:

logging, based on the first log level setting and while the first set of documents are passing through the primary ingestion pipeline, execution of program code of the plurality of annotators of the primary ingestion pipeline;

logging, based on the second log level setting and while the second set of documents are passing through the secondary ingestion pipeline, execution of program code of the plurality of annotators of the secondary ingestion pipeline.

11. The computer program product of claim 9, wherein the method further comprises:

annotating the first set of documents with the plurality of annotators in the primary ingestion pipeline; and

annotating the second set of documents with the plurality of annotators in the secondary ingestion pipeline.

12. The computer program product of claim 9, wherein the method further comprises:

generating, based on the evaluating the plurality of annotators and further based on the evaluating the plurality of documents, a document-annotator pair risk score for each document of the plurality of documents for each annotator of the plurality of annotators, wherein each document-annotator pair risk score represents a likelihood that an associated document will not successfully be annotated by an associated annotator while passing through the primary ingestion pipeline;

comparing each document-annotator pair risk score to a secondary set of risk criteria;

determining, based on the comparing each document-annotator pair risk score to the secondary set of risk criteria, that, for a first annotator of the plurality of annotators, a first document of the first set of documents does not satisfy the secondary set of risk criteria;

transferring, after entering the first document into the primary ingestion pipeline and in response to the secondary set of risk criteria failure determination, the first document from the primary ingestion pipeline to the secondary ingestion pipeline;

annotating the first document with the first annotator in the secondary ingestion pipeline; and

transferring, after the annotating with the first annotator, the first document back to the primary ingestion pipeline.

13. The computer program product of claim 12, wherein the method further comprises:

annotating the first document in the primary ingestion pipeline with the plurality of annotators except for the first annotator.

14. The computer program product of claim 8, wherein the set of risk criteria is a risk threshold.

15. A system for analyzing a primary ingestion pipeline configured for use in natural language processing (NLP), the primary ingestion pipeline including a plurality of annotators configured for annotating documents passing through the primary ingestion pipeline, the system comprising:

a memory;

at least one processor in communication with the memory, wherein the at least one processor is configured to perform a method comprising;

evaluating the plurality of annotators;

evaluating a plurality of documents to be annotated by the plurality of annotators;

generating, based on the evaluating the plurality of annotators and further based on the evaluating the plurality of documents, an ingestion risk score for each document of the plurality of documents, wherein each ingestion risk score represents a likelihood that an associated document will not successfully be annotated by the plurality of annotators while passing through the primary ingestion pipeline;

comparing each ingestion risk score to a set of risk criteria;

determining, based on the comparing, that each document of a first set of documents of the plurality of documents satisfies the set of risk criteria and that each document of a second set of documents of the plurality of documents does not satisfy the set of risk criteria;

entering, in response to the determining, the first set of documents into the primary ingestion pipeline; and

providing, in response to the determining, special handling to the second set of documents.

16. The system of claim 15, wherein the providing special handling includes entering the second set of documents in a secondary ingestion pipeline configured for use in NLP, the secondary ingestion pipeline including the plurality of annotators configured for annotating documents passing through the secondary ingestion pipeline.

17. The system of claim 16, wherein the primary ingestion pipeline has a first log level setting, wherein the secondary ingestion pipeline has a second log level setting that is higher than the first log level setting, and wherein the method further comprises:

logging, based on the first log level setting and while the first set of documents are passing through the primary ingestion pipeline, execution of program code of the plurality of annotators of the primary ingestion pipeline;

logging, based on the second log level setting and while the second set of documents are passing through the secondary ingestion pipeline, execution of program code of the plurality of annotators of the secondary ingestion pipeline.

18. The system of claim 16, wherein the method further comprises:

annotating the first set of documents with the plurality of annotators in the primary ingestion pipeline; and

annotating the second set of documents with the plurality of annotators in the secondary ingestion pipeline.

19. The system of claim 16, wherein the method further comprises:

generating, based on the evaluating the plurality of annotators and further based on the evaluating the plurality of documents, a document-annotator pair risk score for each document of the plurality of documents for each annotator of the plurality of annotators, wherein each document-annotator pair risk score represents a likelihood that an associated document will not successfully be annotated by an associated annotator while passing through the primary ingestion pipeline;

comparing each document-annotator pair risk score to a secondary set of risk criteria;

determining, based on the comparing each document-annotator pair risk score to the secondary set of risk criteria, that, for a first annotator of the plurality of annotators, a first document of the first set of documents does not satisfy the secondary set of risk criteria;

transferring, after entering the first document into the primary ingestion pipeline and in response to the secondary set of risk criteria failure determination, the first document from the primary ingestion pipeline to the secondary ingestion pipeline;

annotating the first document with the first annotator in the secondary ingestion pipeline; and

transferring, after the annotating with the first annotator, the first document back to the primary ingestion pipeline.

20. The system of claim 19, wherein the method further comprises:

annotating the first document in the primary ingestion pipeline with the plurality of annotators except for the first annotator.