Generating usage report in a question answering system based on question categorization

• US 9,996,604 B2
• Filed: 02/09/2015
• Issued: 06/12/2018
• Est. Priority Date: 02/09/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: McCloskey, Daniel J., O'Keeffe, William G., Paciolla, Timothy V., Vardy, Cale R.
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Tran, Loc
• Application Number:
  US14/616,961
• Publication Number:
  US 20160232222A1
• Time in Patent Office: 1,219 Days
• Field of Search: 707737, 707748, 707723
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

A mechanism is provided in a question answering system for generating a usage report based on question categorization. The mechanism clusters documents from a corpus of documents to form a nested structure of clusters of documents. The mechanism record questions answered by question answering system in a database in association with answers returned and answer confidence values. The mechanism maps the questions to the nested structure of clusters of documents to form a nested structure of clusters of questions. The mechanism generates a usage report based on the nested structure of clusters of questions and presenting the usage report responsive to a requesting user.

Claims

1. A method, in a question answering system, for generating a usage report based on question categorization, the method comprising:

clustering documents from a corpus of documents to form a nested structure of clusters of documents;

recording questions answered by a question answering system in a database in association with answers returned and answer confidence values;

mapping the questions to the nested structure of clusters of documents to form a nested structure of clusters of questions;

ranking the nested structure of clusters of questions according to number of questions in each cluster;

associating each cluster of questions with a topic;

generating a usage report based on the nested structure of clusters of questions, wherein generating the usage report comprises;

identifying one or more topics having a number of questions that is below a predetermined threshold, and recommending removing or replacing documents from the corpus in the one or more clusters corresponding to the one or more topics;

presenting the usage report responsive to a requesting user; and

subsequent to the user removing or replacing documents from the corpus, thereby creating a modified corpus of documents, utilizing the modified corpus of documents to generate an answer to a question submitted to the question answering system.

2. The method of claim 1, wherein clustering the documents comprises performing k-means clustering on the corpus of documents.

3. The method of claim 1, wherein clustering the documents comprises assigning a cluster identifier to each cluster of documents.

4. The method of claim 1, wherein mapping the questions comprises assigning a descriptive label to each cluster of questions.

5. The method of claim 1, wherein generating the usage report further comprises:

identifying one or more additional topics having a number of questions that is greater than a predetermined threshold.

6. The method of claim 1, wherein generating the usage report further comprises:

identifying one or more additional topics having at least a predetermined percentage of questions with answer confidence values below a predetermined threshold; and

marking the one or more additional topics as requiring improved documentation, more training of machine learning models, improved ground truth, or new reasoning algorithms.

7. The method of claim 1, wherein generating the usage report further comprises:

identifying one or more additional topics having an average or median answer confidence value below a predetermined threshold; and

presenting the one or more additional topics as requiring improved documentation, more training of machine learning models, improved ground truth, or new reasoning algorithms.

8. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

cluster documents from a corpus of documents to form a nested structure of clusters of documents;

record questions answered by a question answering system in a database in association with answers returned and answer confidence values;

map the questions to the nested structure of clusters of documents to form a nested structure of clusters of questions;

rank the nested structure of clusters of questions according to number of questions in each cluster;

associate each cluster of questions with a topic;

generate a usage report based on the nested structure of clusters of questions, wherein generating the usage report comprises;

identifying one or more topics having a number of questions that is below a predetermined threshold, and recommending removing or replacing documents from the corpus in the one or more clusters corresponding to the one or more topics;

present the usage report responsive to a requesting user; and

subsequent to the user removing or replacing documents from the corpus, thereby creating a modified corpus of documents, utilize the modified corpus of documents to generate an answer to a question submitted to the question answering system.

9. The computer program product of claim 8, wherein generating the usage report further comprises:

identifying one or more additional topics having a number of questions that is greater than a predetermined threshold.

10. The computer program product of claim 8, wherein generating the usage report further comprises:

identifying one or more additional topics having at least a predetermined percentage of questions with answer confidence values below a predetermined threshold; and

marking the one or more additional topics as requiring improved documentation, more training of machine learning models, improved ground truth, or new reasoning algorithms.

11. The computer program product of claim 8, wherein generating the usage report comprises:

identifying one or more additional topics having an average or median answer confidence value below a predetermined threshold; and

presenting the one or more additional topics as requiring improved documentation, more training of machine learning models, improved ground truth, or new reasoning algorithms.

12. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

cluster documents from a corpus of documents to form a nested structure of clusters of documents;

record questions answered by a question answering system in a database in association with answers returned and answer confidence values;

map the questions to the nested structure of clusters of documents to form a nested structure of clusters of questions;

rank the nested structure of clusters of questions according to number of questions in each cluster;

associate each cluster of questions with a topic;

generate a usage report based on the nested structure of clusters of questions, wherein generating the usage report comprises;

identifying one or more topics having a number of questions that is below a predetermined threshold, and recommending removing or replacing documents from the corpus in the one or more clusters corresponding to the one or more topics;

present the usage report responsive to a requesting user; and

subsequent to the user removing or replacing documents from the corpus, thereby creating a modified corpus of documents, utilize the modified corpus of documents to generate an answer to a question submitted to the question answering system.

13. The apparatus of claim 12, wherein generating the usage report further comprises:

identifying one or more additional topics having at least a predetermined percentage of questions with answer confidence values below a predetermined threshold; and

marking the one or more additional topics as requiring improved documentation, more training of machine learning models, improved ground truth, or new reasoning algorithms.

14. The apparatus of claim 12, wherein generating the usage report further comprises:

identifying one or more additional topics having an average or median answer confidence value below a predetermined threshold; and

presenting the one or more additional topics as requiring improved documentation, more training of machine learning models, improved ground truth, or new reasoning algorithms.