Generating an answer from multiple pipelines using clustering

• US 9,348,900 B2
• Filed: 12/11/2013
• Issued: 05/24/2016
• Est. Priority Date: 12/11/2013
• Status: Active Grant

• Current Assignee: Hyundai Motor Company (Hyundai Motor Group), Kia Corp.
• Original Assignee: International Business Machines Corporation
• Inventors: Alkov, Christopher S., Estrada, Suzanne L., Haggar, Peter F., Haverlock, Kevin B.
• Primary Examiner(s): Moser, Bruce
• Application Number:
  US14/102,671
• Publication Number:
  US 20150161230A1
• Time in Patent Office: 895 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/353 into predefined classes

Abstract

Mechanisms for generating an answer to an input question are provided. The mechanisms receive an input question and route the input question to a plurality of Question and Answer (QA) system pipelines. The mechanisms receive results of processing the input question from each QA system pipeline in the plurality of QA system pipelines. The results comprise one or more candidate answers to the input question and associated confidence scores. The mechanisms adjust, for each QA system pipeline, the confidence scores of the one or more candidate answers associated with the QA system pipeline based on one or more weighting factors. In addition, the mechanisms generate the answer to the input question based on the adjusted confidence scores of the one or more candidate answers for each QA system pipeline.

Claims

1. A method, in a data processing system comprising a processor and a memory, for generating an answer to an input question, the method comprising:

receiving, in the data processing system, an input question;

routing, by the data processing system, the input question to a plurality of Question and Answer (QA) system pipelines;

receiving, by the data processing system, results of processing the input question from each QA system pipeline in the plurality of QA system pipelines, wherein the results comprise one or more candidate answers to the input question and associated confidence scores;

adjusting, by the data processing system, for each QA system pipeline, the confidence scores of the one or more candidate answers associated with the QA system pipeline based on one or more weighting factors; and

generating, by the data processing system, the answer to the input question based on the adjusted confidence scores of the one or more candidate answers for each QA system pipeline.

2. The method of claim 1, wherein adjusting the confidence scores of the one or more candidate answers associated with the QA system pipeline comprises:

determining a weighting factor for the confidence scores based on determined relevance of the QA system pipeline to the input question.

3. The method of claim 2, wherein determining a weighting factor for the confidence scores based on a determined relevance of the QA system pipeline to the input question comprises calculating the weighting factor as a function of a distance between the input question and a center of a cluster of training data used to train the QA system pipeline.

4. The method of claim 3, wherein the distance is calculated as a Euclidean distance as part of a clustering operation to cluster the input question with one or more clusters of training data sets associated with the plurality of QA system pipelines.

5. The method of claim 2, wherein determining a weighting factor for the confidence scores based on a determined relevance of the QA system pipeline to the input question comprises calculating the weighting factor as a function of a degree of matching of the input question to a corpora of documents used to train the QA system pipeline.

6. The method of claim 5, wherein the degree of matching of the input question to the corpora of documents used to train the QA system pipeline is determined based on a matching of features extracted from the input question to metadata associated with the corpora of documents used to train the QA system pipeline.

7. The method of claim 6, wherein the extracted features comprise terms in the input question, and wherein the metadata associated with the corpora of documents used to train the QA system pipeline comprises at least one of a domain description, one or more keywords, or a name of the corpora of documents used to train the QA system pipeline.

8. The method of claim 2, wherein determining a weighting factor for the confidence scores based on a determined relevance of the QA system pipeline to the input question comprises calculating the weighting factor as a function of a ranking of a corpora upon which the QA system pipeline operates to answer input questions.

9. The method of claim 8, wherein the ranking of the corpora is determined based on user feedback identifying one or more ratings of the corpora with regard to correctness of answers generated based on the corpora.

10. The method of claim 1, wherein routing the input question to the plurality of QA system pipelines comprises:

analyzing, by the data processing system, the input question to identify at least one feature of the input question;

performing, by the data processing system, clustering of the input question with previously generated clusters of questions based on the at least one feature of the input question;

identifying, by the data processing system, based on results of the clustering, a plurality of matching clusters, of the previously generated clusters, with which the input question is associated; and

identifying, by the data processing system, a QA system pipeline associated with each matching cluster in the plurality of matching clusters.

11. A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an input question;

route the input question to a plurality of Question and Answer (QA) system pipelines;

receive results of processing the input question from each QA system pipeline in the plurality of QA system pipelines, wherein the results comprise one or more candidate answers to the input question and associated confidence scores;

adjust, for each QA system pipeline, the confidence scores of the one or more candidate answers associated with the QA system pipeline based on one or more weighting factors; and

generate the answer to the input question based on the adjusted confidence scores of the one or more candidate answers for each QA system pipeline.

12. The computer program product of claim 11, wherein the computer readable program further causes the computing device to adjust the confidence scores of the one or more candidate answers associated with the QA system pipeline at least by:

determining a weighting factor for the confidence scores based on determined relevance of the QA system pipeline to the input question.

13. The computer program product of claim 12, wherein the computer readable program further causes the computing device to determine a weighting factor for the confidence scores based on a determined relevance of the QA system pipeline to the input question at least by calculating the weighting factor as a function of a distance between the input question and a center of a cluster of training data used to train the QA system pipeline.

14. The computer program product of claim 13, wherein the distance is calculated as a Euclidean distance as part of a clustering operation to cluster the input question with one or more clusters of training data sets associated with the plurality of QA system pipelines.

15. The computer program product of claim 12, wherein the computer readable program further causes the computing device to determine a weighting factor for the confidence scores based on a determined relevance of the QA system pipeline to the input question at least by calculating the weighting factor as a function of a degree of matching of the input question to a corpora of documents used to train the QA system pipeline.

16. The computer program product of claim 15, wherein the degree of matching of the input question to the corpora of documents used to train the QA system pipeline is determined based on a matching of features extracted from the input question to metadata associated with the corpora of documents used to train the QA system pipeline.

17. The computer program product of claim 16, wherein the extracted features comprise terms in the input question, and wherein the metadata associated with the corpora of documents used to train the QA system pipeline comprises at least one of a domain description, one or more keywords, or a name of the corpora of documents used to train the QA system pipeline.

18. The computer program product of claim 12, wherein the computer readable program further causes the computing device to determine a weighting factor for the confidence scores based on a determined relevance of the QA system pipeline to the input question at least by calculating the weighting factor as a function of a ranking of a corpora upon which the QA system pipeline operates to answer input questions.

19. The computer program product of claim 18, wherein the ranking of the corpora is determined based on user feedback identifying one or more ratings of the corpora with regard to correctness of answers generated based on the corpora.

20. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question;

route the input question to a plurality of Question and Answer (QA) system pipelines;

receive results of processing the input question from each QA system pipeline in the plurality of QA system pipelines, wherein the results comprise one or more candidate answers to the input question and associated confidence scores;

adjust, for each QA system pipeline, the confidence scores of the one or more candidate answers associated with the QA system pipeline based on one or more weighting factors; and

generate the answer to the input question based on the adjusted confidence scores of the one or more candidate answers for each QA system pipeline.