Real-time or frequent ingestion by running pipeline in order of effectiveness

• US 9,697,099 B2
• Filed: 06/04/2014
• Issued: 07/04/2017
• Est. Priority Date: 06/04/2014
• Status: Expired due to Fees

• Current Assignee: AirBNB Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Petri, John E., Freed, Andrew R., Schroeder, Michael W., Dubbels, Joel C.
• Primary Examiner(s): Holmes, Michael B
• Assistant Examiner(s): Bharadwaj, Kalpana
• Application Number:
  US14/295,913
• Publication Number:
  US 20150356456A1
• Time in Patent Office: 1,126 Days
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A mechanism is provided in a data processing system for partial ingestion of content. The mechanism receives new content to be ingested into a corpus of information. The mechanism applies a plurality of sub-pipelines of annotation engines against the new content in order of effectiveness. The plurality of sub-pipelines include all annotation engines of an ingestion pipeline. Each sub-pipeline within the plurality of sub-pipelines generates one or more intermediate output objects. The mechanism provides access to the one or more intermediate output objects.

Claims

1. A method, in a data processing system, for partial ingestion of content, the method comprising:

identifying a set of features that contribute to generating candidate answers for input questions;

identifying a set of annotation engines in the ingestion pipeline that contribute to each of the set of features and at least one annotation engine on which the one or more annotation engines depend;

generating a sub-pipeline for each set of annotation engines to form a plurality of sub-pipelines of annotation engines;

receiving new content to be ingested into a corpus of information;

applying the plurality of sub-pipelines of annotation engines against the new content in order of effectiveness, wherein the plurality of sub-pipelines include all annotation engines of an ingestion pipeline and wherein each sub-pipeline within the plurality of sub-pipelines generates one or more intermediate output objects; and

providing access to the one or more intermediate output objects, wherein the one or more intermediate output objects represent the partially ingested new content.

2. The method of claim 1, further comprising responsive to applying all of the plurality of sub-pipelines, storing fully ingested new content in the corpus of information.

3. The method of claim 1, wherein identifying the set of features comprises examining feature score weighting from a trained model.

4. The method of claim 1, wherein generating a sub-pipeline for each set of annotation engines comprises:

determining performance or runtime cost of each annotation engine in the ingestion pipeline;

determining an efficiency score for each set of annotation engines based on the performance or runtime costs of the set of annotation engines and a feature score weighting of a corresponding feature; and

ranking the sub-pipelines by efficiency score.

5. The method of claim 4, wherein generating a sub-pipeline for each set of annotation engines comprises for each sub-pipeline, removing annotation engines that are present in a next higher ranked sub-pipeline.

6. The method of claim 5, wherein generating a sub-pipeline for each set of annotation engines comprises combining a given sub-pipeline with a next higher or lower ranked sub-pipeline responsive to the given sub-pipeline having fewer than a predetermined number of annotation engines.

7. The method of claim 4, wherein applying the plurality of sub-pipelines of annotation engines against the new content comprises applying the plurality of sub-pipelines according to the ranking by efficiency score.

8. The method of claim 1, wherein each sub-pipeline within the plurality of sub-pipelines deletes one or more intermediate output objects generated by a previous sub-pipeline.

9. The method of claim 1, wherein providing access to the one or more intermediate output objects comprises mapping the new content being ingested to the one or more intermediate output objects.

10. The method of claim 9, further comprising:

responsive to receiving an input question in a question answering system, running a question answering pipeline of software engines against available partially and fully ingested content according to the mapping;

generating one or more candidate answers for the input question;

ranking the one or more candidate answers; and

presenting the ranked one or more candidate answers.

11. The method of claim 10, further comprising marking the given candidate answer as being based on partially ingested content responsive to determining evidence for the given candidate answer is partially ingested.

12. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

identify a set of features that contribute to generating candidate answers for input questions;

identify a set of annotation engines in the ingestion pipeline that contribute to each of the set of features and at least one annotation engine on which the one or more annotation engines depend;

generate a sub-pipeline for each set of annotation engines to form a plurality of sub-pipelines of annotation engines;

receive new content to be ingested into a corpus of information;

apply a plurality of sub-pipelines of annotation engines against the new content in order of effectiveness, wherein the plurality of sub-pipelines include all annotation engines of an ingestion pipeline and wherein each sub-pipeline within the plurality of sub-pipelines generates one or more intermediate output objects; and

provide access to the one or more intermediate output objects, wherein the one or more intermediate output objects represent the partially ingested new content.

13. The computer program product of claim 12, wherein generating a sub-pipeline for each set of annotation engines comprises:

determining performance or runtime cost of each annotation engine in the ingestion pipeline;

determining an efficiency score for each set of annotation engines based on the performance or runtime costs of the set of annotation engines and a feature score weighting of a corresponding feature; and

ranking the sub-pipelines by efficiency score.

14. The computer program product of claim 13, wherein applying the plurality of sub-pipelines of annotation engines against the new content comprises applying the plurality of sub-pipelines according to the ranking by efficiency score.

15. The computer program product of claim 12, wherein providing access to the one or more intermediate output objects comprises mapping the new content being ingested to the one or more intermediate output objects, wherein the computer readable program further causes the computing device to:

responsive to receiving an input question in a question answering system, run a question answering pipeline of software engines against available partially and fully ingested content according to the mapping;

generate one or more candidate answers for the input question;

rank the one or more candidate answers; and

present the ranked one or more candidate answers.

16. The computer program product of claim 15, wherein the computer readable program further causes the computing device to mark the given candidate answer as being based on partially ingested content responsive to determining evidence for the given candidate answer is partially ingested.

17. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

identify a set of features that contribute to generating candidate answers for input questions;

identify a set of annotation engines in the ingestion pipeline that contribute to each of the set of features and at least one annotation engine on which the one or more annotation engines depend;

generate a sub-pipeline for each set of annotation engines to form a plurality of sub-pipelines of annotation engines;

receive new content to be ingested into a corpus of information;

apply a plurality of sub-pipelines of annotation engines against the new content in order of effectiveness, wherein the plurality of sub-pipelines include all annotation engines of an ingestion pipeline and wherein each sub-pipeline within the plurality of sub-pipelines generates one or more intermediate output objects; and

provide access to the one or more intermediate output objects, wherein the one or more intermediate output objects represent the partially ingested new content.

18. The apparatus of claim 17, wherein generating a sub-pipeline for each set of annotation engines comprises:

determining performance or runtime cost of each annotation engine in the ingestion pipeline;

determining an efficiency score for each set of annotation engines based on the performance or runtime costs of the set of annotation engines and a feature score weighting of a corresponding feature; and

ranking the sub-pipelines by efficiency score.

19. The apparatus of claim 17, wherein providing access to the one or more intermediate output objects comprises mapping the new content being ingested to the one or more intermediate output objects, wherein the instructions further cause the processor to:

responsive to receiving an input question in a question answering system, run a question answering pipeline of software engines against available partially and fully ingested content according to the mapping;

generate one or more candidate answers for the input question;

rank the one or more candidate answers; and

present the ranked one or more candidate answers.

20. The apparatus of claim 19, wherein the instructions further cause the processor to mark the given candidate answer as being based on partially ingested content responsive to determining evidence for the given candidate answer is partially ingested.