Categorizing questions in a question answering system

• US 10,366,107 B2
• Filed: 02/06/2015
• Issued: 07/30/2019
• Est. Priority Date: 02/06/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: McCloskey, Daniel J., O'Keeffe, William G., Paciolla, Timothy V., Vardy, Cale R.
• Primary Examiner(s): Tran, Anhtai V
• Application Number:
  US14/615,731
• Publication Number:
  US 20160232221A1
• Time in Patent Office: 1,635 Days
• Field of Search: None
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A mechanism is provided in a question answering system for categorizing questions. The mechanism clusters documents from a corpus of documents to form a nested structure of clusters of documents. The mechanism records questions answered by question answering system in a database in association with answers returned and answer confidence values. The mechanism maps the questions to the nested structure of clusters of documents to form a nested structure of clusters of questions. The mechanism presents recommended questions based on the nested structure of clusters of questions in a user interface of the question answering system.

Claims

1. A method, in a question answering system, for categorizing questions, the method comprising:

storing a corpus of documents in a first database;

clustering documents from the corpus of documents to form a nested structure of clusters of documents;

recording questions answered by the question answering system in a second database that is different from the first database, wherein the questions are answered using supporting evidence from the corpus of documents, and wherein the questions are recorded in association with answers returned and answer confidence values;

mapping the questions to the nested structure of clusters of documents to form a nested structure of clusters of questions, wherein the mapping includes, for a given question, determining a document containing supporting evidence for the answer to the question, and mapping the question to a cluster containing the document;

mapping a question history of a user and context information of the user to the nested structure of clusters of questions;

ranking the nested structure of clusters of questions based on the mapping of the question history and context information, and on the number of questions in each cluster;

presenting, in a user interface, one or more topics corresponding to one or more of the highest ranking clusters of questions; and

responsive to receiving user selection of a topic of the presented one or more topics, presenting recommended questions in the user interface, wherein the recommended questions are mapped to a cluster corresponding to the selected topic.

2. The method of claim 1, wherein clustering the documents comprises performing k-means clustering on the corpus of documents.

3. The method of claim 1, wherein clustering the documents comprises assigning a unique cluster identifier to each cluster of documents.

4. The method of claim 1, wherein mapping the questions comprises assigning a descriptive label to each cluster of questions.

5. The method of claim 1, further comprising:

responsive to receiving a user input, mapping the user input to a given cluster in the nested structure of clusters of questions; and

presenting a set of questions mapped to the given cluster in the user interface.

6. The method of claim 1, further comprising:

responsive to receiving a user input, mapping the user input to a given cluster in the nested structure of clusters of questions;

generating a type-ahead suggestion output based on a set of questions mapped to the given cluster; and

presenting the type-ahead suggestion output in the user interface.

8. The method of claim 1, further comprising:

responsive to receiving an input question in the user interface, mapping the input question to a given cluster in the nested structure of clusters of questions;

generating a set of candidate answers for the input question; and

providing a higher ranking to candidate answers in the set of candidate answers supported by documents in the given cluster.

7. The method 6, wherein generating the type-ahead suggestion output comprises generating a set of suggested questions based on the user input in a range of topics;

wherein presenting the commonly asked questions further comprises narrowing the range of topics and modifying the set of suggested questions responsive to receiving subsequent user input.

9. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

store a corpus of documents in a first database;

cluster documents from the corpus of documents to form a nested structure of clusters of documents;

record questions answered by the question answering system in a second database that is different from the first database, wherein the questions are answered using supporting evidence from the corpus of documents, and wherein the questions are recorded in association with answers returned and answer confidence values;

map the questions to the nested structure of clusters of documents to form a nested structure of clusters of questions, wherein the mapping includes, for a given question, determining a document containing supporting evidence for the answer to the question, and mapping the question to a cluster containing the document;

map a question history of a user and context information of the user to the nested structure of clusters of questions;

rank the nested structure of clusters of questions based on the mapping of the question history and context information, and on the number of questions in each cluster;

present, in a user interface, one or more topics corresponding to one or more of the highest ranking clusters of questions; and

responsive to receiving user selection of a topic of the presented one or more topics, present recommended questions in the user interface, wherein the recommended questions are mapped to a cluster corresponding to the selected topic.

10. The computer program product of claim 9, further comprising:

responsive to receiving a user input, mapping the user input to a given cluster in the nested structure of clusters of questions; and

presenting a set of questions mapped to the given cluster in the user interface.

11. The computer program product of claim 9, further comprising:

responsive to receiving a user input, mapping the user input to a given cluster in the nested structure of clusters of questions;

generating a type-ahead suggestion output based on a set of questions mapped to the given cluster; and

presenting the type-ahead suggestion output in the user interface.

12. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

store a corpus of documents in a first database;

cluster documents from the corpus of documents to form a nested structure of clusters of documents;

record questions answered by the question answering system in a second database that is different from the first database, wherein the questions are answered using supporting evidence from the corpus of documents, and wherein the questions are recorded in association with answers returned and answer confidence values;

map the questions to the nested structure of clusters of documents to form a nested structure of clusters of questions, wherein the mapping includes, for a given question, determining a document containing supporting evidence for the answer to the question, and mapping the question to a cluster containing the document;

map a question history of a user and context information of the user to the nested structure of clusters of questions;

rank the nested structure of clusters of questions based on the mapping of the question history and context information, and on the number of questions in each cluster;

present, in a user interface, one or more topics corresponding to one or more of the highest ranking clusters of questions; and

responsive to receiving user selection of a topic of the presented one or more topics, present recommended questions in the user interface, wherein the recommended questions are mapped to a cluster corresponding to the selected topic.

13. The apparatus of claim 12, further comprising:

responsive to receiving a user input, mapping the user input to a given cluster in the nested structure of clusters of questions; and

presenting a set of questions mapped to the given cluster in the user interface.

14. The apparatus of claim 12, further comprising:

responsive to receiving a user input, mapping the user input to a given cluster in the nested structure of clusters of questions;

generating a type-ahead suggestion output based on a set of questions mapped to the given cluster; and

presenting the type-ahead suggestion output in the user interface.