Returning related previously answered questions based on question affinity

• US 9,703,860 B2
• Filed: 10/06/2014
• Issued: 07/11/2017
• Est. Priority Date: 10/06/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: O'Keeffe, William G., Taieb, David D., Haverlock, Kevin B., Berajawala, Suneil H.
• Primary Examiner(s): Rostami, Mohammad S
• Application Number:
  US14/506,728
• Publication Number:
  US 20160098477A1
• Time in Patent Office: 1,009 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

Mechanisms are provided, in a Question and Answer (QA) system, for identifying a related previously processed question. A first question is processed by the QA system to generate an answer to the first question and a supporting evidence passage that supports the answer to the first question being a correct answer to the first question. The QA system performs a lookup operation in a question repository based on an identifier of the supporting evidence passage to identify an entry corresponding to a previously processed question, different than the first question, that previously had an answer generated based on the supporting evidence passage. The QA system returns the previously processed question and the answer to the first question to a submitter of the first question.

Claims

1. A method, in a Question and Answer (QA) system comprising a processor and a memory, for identifying a related previously processed question, the method comprising:

processing, by the QA system, a first question to generate an answer to the first question and a supporting evidence passage that supports the answer to the first question being a correct answer to the first question;

performing, by the QA system, a lookup operation in a question repository based on an identifier of the supporting evidence passage to identify an entry corresponding to a previously processed question, different than the first question, that previously had an answer generated based on the supporting evidence passage;

returning, by the QA system, the previously processed question and the answer to the first question to a submitter of the first question;

calculating, by the QA system, an affinity score of the first question, wherein the affinity score represents an affinity between the first question and the supporting evidence passage; and

storing the first question and the affinity score in the question repository in association with the supporting evidence passage, wherein calculating the affinity score of the first question comprises;

clustering one or more topics in the first question;

determining a correspondence of the supporting evidence passage to one or more clusters of the one or more topics in the first question; and

calculating an affinity score based on the correspondence, wherein the affinity score is calculated to be relatively higher in response supporting evidence passage being clustered in the one or more clusters than in response to the supporting evidence passage not being clustered in the one or more clusters.

2. The method of claim 1, further comprising:

generating a ranked listing of previously processed questions that previously had an answer generated based on the supporting evidence passage, wherein returning the answer to the first question and the previously processed question to the submitter of the first question further comprises returning the ranked listing of previously processed questions.

3. The method of claim 2, wherein the ranked listing is ranked according to affinity scores of previously processed questions, and wherein the affinity score of a previously processed question identifies an affinity between the previously processed question and the supporting evidence passage.

4. The method of claim 2, further comprising:

receiving a user selection of a previously processed question in the ranked listing of previously processed questions; and

in response to receiving the user selection, presenting an answer for the previously processed question to the user.

5. The method of claim 4, wherein presenting the answer for the previously processed question comprises at least one of presenting a stored previously generated answer for the previously processed question or resubmitting the previously processed question to the QA system for processing and generation of the answer that is presented to the user.

6. The method of claim 1, wherein calculating the affinity score of the first question comprises calculating the affinity score of the first question based on one or more of a confidence score associated with the answer to the first question, term co-reference scoring, cosine similarity scoring, and clustering of documents.

7. The method of claim 1, wherein calculating the affinity score of the first question comprises calculating the affinity score of the first question based on at least one of a number of times the first question is processed by the QA system, or a number of times the first question is processed by the QA system and a number of times a follow up question is subsequently processed by the QA system.

8. A method, in a Question and Answer (QA) system comprising a processor and a memory, for identifying a related previously processed question, the method comprising:

processing, by the QA system, a first question to generate an answer to the first question and a supporting evidence passage that supports the answer to the first question being a correct answer to the first question;

performing, by the QA system, a lookup operation in a question repository based on an identifier of the supporting evidence passage to identify an entry corresponding to a previously processed question, different than the first question, that previously had an answer generated based on the supporting evidence passage;

returning, by the QA system, the previously processed question and the answer to the first question to a submitter of the first question;

generating a ranked listing of previously processed questions that previously had an answer generated based on the supporting evidence passage, wherein;

returning the answer to the first question and the previously processed question to the submitter of the first question further comprises returning the ranked listing of previously processed questions,

the ranked listing is ranked according to affinity scores of previously processed questions, and

the affinity score of a previously processed question identifies an affinity between the previously processed question and the supporting evidence passage; and

performing a corpus management operation on a corpus of evidence passages in which the supporting evidence passage is a part based on the affinity scores associated with the evidence passages in the corpus, wherein the corpus management operation comprises at least one of modifying a relevance weight value associated with one or more evidence passages in the corpus, removing an evidence passage from the corpus, or adding additional evidence passages, from a same source as an already existing evidence passage in the corpus, to the corpus.

9. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system implementing a Question and Answer (QA) system, causes the data processing system to:

process, by the QA system, a first question to generate an answer to the first question and a supporting evidence passage that supports the answer to the first question being a correct answer to the first question;

perform, by the QA system, a lookup operation in a question repository based on an identifier of the supporting evidence passage to identify an entry corresponding to a previously processed question, different than the first question, that previously had an answer generated based on the supporting evidence passage;

return, by the QA system, the previously processed question and the answer to the first question to a submitter of the first question;

calculate, by the QA system, an affinity score of the first question, wherein the affinity score represents an affinity between the first question and the supporting evidence passage; and

store the first question and the affinity score in the question repository in association with the supporting evidence passage, wherein the computer readable program further causes the data processing system to calculate the affinity score of the first question at least by;

clustering one or more topics in the first question;

determining a correspondence of the supporting evidence passage to one or more clusters of the one or more topics in the first question; and

calculating an affinity score based on the correspondence, wherein the affinity score is calculated to be relatively higher in response to the supporting evidence passage being clustered in the one or more clusters than in response to the supporting evidence passage not being clustered in the one or more clusters.

10. The computer program product of claim 9, wherein the computer readable program further causes the data processing system to:

generate a ranked listing of previously processed questions that previously had an answer generated based on the supporting evidence passage, and wherein the computer readable program further causes the data processing system to return the answer to the first question and the previously processed question to the submitter of the first question further at least by returning the ranked listing of previously processed questions.

11. The computer program product of claim 10, wherein the ranked listing is ranked according to affinity scores of previously processed questions, and wherein the affinity score of a previously processed question identifies an affinity between the previously processed question and the supporting evidence passage.

12. The computer program product of claim 10, wherein the computer readable program further causes the data processing system to:

receive a user selection of a previously processed question in the ranked listing of previously processed questions; and

in response to receiving the user selection, present an answer for the previously processed question to the user.

13. The computer program product of claim 12, wherein the computer readable program further causes the data processing system to present the answer for the previously processed question at least by one of presenting a stored previously generated answer for the previously processed question or resubmitting the previously processed question to the QA system for processing and generation of the answer that is presented to the user.

14. The computer program product of claim 9, wherein the computer readable program further causes the data processing system to calculate the affinity score of the first question at least by calculating the affinity score of the first question based on one or more of a confidence score associated with the answer to the first question, term co-reference scoring, cosine similarity scoring, and clustering of documents.

15. The computer program product of claim 9, wherein the computer readable program further causes the data processing system to calculate the affinity score of the first question at least by calculating the affinity score of the first question based on at least one of a number of times the first question is processed by the QA system, or a number of times the first question is processed by the QA system and a number of times a follow up question is subsequently processed by the QA system.

16. A data processing system comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

process a first question to generate an answer to the first question and a supporting evidence passage that supports the answer to the first question being a correct answer to the first question;

perform a lookup operation in a question repository based on an identifier of the supporting evidence passage to identify an entry corresponding to a previously processed question, different than the first question, that previously had an answer generated based on the supporting evidence passage;

return the previously processed question and the answer to the first question to a submitter of the first question;

calculate an affinity score of the first question, wherein the affinity score represents an affinity between the first question and the supporting evidence passage; and

store the first question and the affinity score in the question repository in association with the supporting evidence passage, wherein the instructions further cause the processor to calculate the affinity score of the first question at least by;

clustering one or more topics in the first question;

determining a correspondence of the supporting evidence passage to one or more clusters of the one or more topics in the first question; and

calculating an affinity score based on the correspondence, wherein the affinity score is calculated to be relatively higher in response to the supporting evidence passage being clustered in the one or more clusters than in response to the supporting evidence passage not being clustered in the one or more clusters.

17. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system implementing a Question and Answer (QA) system, causes the data processing system to:

process, by the QA system, a first question to generate an answer to the first question and a supporting evidence passage that supports the answer to the first question being a correct answer to the first question;

perform, by the QA system, a lookup operation in a question repository based on an identifier of the supporting evidence passage to identify an entry corresponding to a previously processed question, different than the first question, that previously had an answer generated based on the supporting evidence passage;

return, by the QA system, the previously processed question and the answer to the first question to a submitter of the first question;

generate a ranked listing of previously processed questions that previously had an answer generated based on the supporting evidence passage, wherein;

returning the answer to the first question and the previously processed question to the submitter of the first question further comprises returning the ranked listing of previously processed questions,

the ranked listing is ranked according to affinity scores of previously processed questions, and

the affinity score of a previously processed question identifies an affinity between the previously processed question and the supporting evidence passage; and

perform a corpus management operation on a corpus of evidence passages in which the supporting evidence passage is a part based on the affinity scores associated with the evidence passages in the corpus, wherein the corpus management operation comprises at least one of modifying a relevance weight value associated with one or more evidence passages in the corpus, removing an evidence passage from the corpus, or adding additional evidence passages, from a same source as an already existing evidence passage in the corpus, to the corpus.

18. A data processing system comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

process a first question to generate an answer to the first question and a supporting evidence passage that supports the answer to the first question being a correct answer to the first question;

perform a lookup operation in a question repository based on an identifier of the supporting evidence passage to identify an entry corresponding to a previously processed question, different than the first question, that previously had an answer generated based on the supporting evidence passage;

return the previously processed question and the answer to the first question to a submitter of the first question;

generate a ranked listing of previously processed questions that previously had an answer generated based on the supporting evidence passage, wherein;

returning the answer to the first question and the previously processed question to the submitter of the first question further comprises returning the ranked listing of previously processed questions,

the ranked listing is ranked according to affinity scores of previously processed questions, and

the affinity score of a previously processed question identifies an affinity between the previously processed question and the supporting evidence passage; and

perform a corpus management operation on a corpus of evidence passages in which the supporting evidence passage is a part based on the affinity scores associated with the evidence passages in the corpus, wherein the corpus management operation comprises at least one of modifying a relevance weight value associated with one or more evidence passages in the corpus, removing an evidence passage from the corpus, or adding additional evidence passages, from a same source as an already existing evidence passage in the corpus, to the corpus.