Answer Confidence Output Mechanism for Question and Answer Systems

• US 20150261859A1
• Filed: 03/11/2014
• Published: 09/17/2015
• Est. Priority Date: 03/11/2014
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Isensee, Scott H., O'Keeffe, William G., Schwartz, David R., Vardy, Cale R.
• Application Number:
  US14/204,068
• Publication Number:
  US 20150261859A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/287 Visualization; Browsing
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials

Abstract

Mechanisms are provided for generating an output of confidence score for candidate answers of an input question. The mechanisms, implemented in a data processing system configured to answer an input question receive candidate answer information comprising confidence scores associated with candidate answers for the input question. The mechanisms categorize, for each candidate answer in the candidate answer information, the corresponding confidence score into one of a plurality of confidence score categories. For each candidate answer in the candidate answer information, a graphical representation of the confidence score category of the candidate answer is generated. A graphical user interface output comprising an entry for each candidate answer in the candidate answer information is generated where each entry comprises the corresponding graphical representation of the confidence score category of the candidate answer corresponding to the entry.

Claims

1. A method, in a data processing system comprising a processor and a memory, and being configured to answer an input question, comprising:

receiving, by the data processing system, candidate answer information comprising confidence scores associated with candidate answers for the input question;

categorizing, by the data processing system, for each candidate answer in the candidate answer information, the corresponding confidence score into one of a plurality of confidence score categories;

generating, by the data processing system, for each candidate answer in the candidate answer information, a graphical representation of the confidence score category of the candidate answer; and

generating, by the data processing system, a graphical user interface output comprising an entry for each candidate answer in the candidate answer information, wherein each entry comprises the corresponding graphical representation of the confidence score category of the candidate answer corresponding to the entry.

2. The method of claim 1, wherein the graphical representation of confidence score category of the candidate answer is a segmented bar graph in which segments of the segmented bar graph are configured to represent the confidence score category of the candidate answer.

3. The method of claim 2, wherein the segments of the segmented bar graph are configured at least by outputting a number of segments of the segmented bar graph in accordance with the confidence score category of the candidate answer.

4. The method of claim 2, wherein the segments of the segmented bar graph are configured at least by modifying a color of the segments of the segmented bar graph in accordance with the confidence score category of the candidate answer.

5. The method of claim 2, wherein the segments of the segmented bar graph are configured at least by outputting a number of segments of the segmented bar graph in accordance with the confidence score category of the candidate answer and outputting a color of the segments of the segmented bar graph in accordance with a relative position of the confidence score within a range of confidence scores corresponding to the confidence score category of the candidate answer.

6. The method of claim 1, wherein generating the graphical user interface output comprising an entry for each candidate answer in the candidate answer information, further comprises outputting, for each entry, a textual label in association with the corresponding graphical representation of the confidence score category, wherein the textual label comprises a description of the confidence score category.

7. The method of claim 1, wherein generating the graphical user interface output comprises, for each entry, outputting a user selectable graphical user interface element for receiving an input from a user to drill down into evidence passage information supporting the corresponding confidence score of the entry.

8. The method of claim 7, further comprising:

receiving a user input selecting the graphical user interface element of an entry in the graphical user interface; and

in response to receiving the user input selecting the graphical user interface element, outputting a summary evidence passage information graphical user interface summarizing evidence passage information organized into a first set of evidence passage information that is in support of a candidate answer corresponding to the entry being a correct answer for the input question and a second set of evidence passage information that is not in support of the candidate answer being a correct answer for the input question.

9. The method of claim 8, wherein the summary evidence passage information graphical user interface comprises, for the first set of evidence passage information, a first drill down graphical user interface element that is selectable by a user to drill down into individual evidence passages corresponding to the first set of evidence passage information, and further comprises, for the second set of evidence passage information, a second drill down graphical user interface element that is selectable by the user to drill down into individual evidence passages corresponding to the second set of evidence passage information.

10. The method of claim 1, wherein generating the graphical user interface output comprises, for each entry, outputting a user selectable graphical user interface element for receiving an input from a user indicating a subjective evaluation of correctness of a corresponding confidence score of the entry.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein for outputting an indication of confidence score in candidate answers for an input question, when executed on a computing device, causes the computing device to:

receive candidate answer information comprising confidence scores associated with candidate answers for the input question;

categorize, for each candidate answer in the candidate answer information, the corresponding confidence score into one of a plurality of confidence score categories;

generate, for each candidate answer in the candidate answer information, a graphical representation of the confidence score category of the candidate answer; and

generate a graphical user interface output comprising an entry for each candidate answer in the candidate answer information, wherein each entry comprises the corresponding graphical representation of the confidence score category of the candidate answer corresponding to the entry.

12. The computer program product of claim 11, wherein the graphical representation of confidence score category of the candidate answer is a segmented bar graph in which segments of the segmented bar graph are configured to represent the confidence score category of the candidate answer.

13. The computer program product of claim 12, wherein the segments of the segmented bar graph are configured at least by outputting a number of segments of the segmented bar graph in accordance with the confidence score category of the candidate answer.

14. The computer program product of claim 12, wherein the segments of the segmented bar graph are configured at least by modifying a color of the segments of the segmented bar graph in accordance with the confidence score category of the candidate answer.

15. The computer program product of claim 12, wherein the segments of the segmented bar graph are configured at least by outputting a number of segments of the segmented bar graph in accordance with the confidence score category of the candidate answer and outputting a color of the segments of the segmented bar graph in accordance with a relative position of the confidence score within a range of confidence scores corresponding to the confidence score category of the candidate answer.

16. The computer program product of claim 11, wherein the computer readable program further causes the computing device to generate the graphical user interface output comprising an entry for each candidate answer in the candidate answer information, at least by outputting, for each entry, a textual label in association with the corresponding graphical representation of the confidence score category, wherein the textual label comprises a description of the confidence score category.

17. The computer program product of claim 11, wherein the computer readable program further causes the computing device to generate the graphical user interface output at least by, for each entry, outputting a user selectable graphical user interface element for receiving an input from a user to drill down into evidence passage information supporting a corresponding confidence score of the entry.

18. The computer program product of claim 17, wherein the computer readable program further causes the computing device to:

receive a user input selecting the graphical user interface element of an entry in the graphical user interface; and

in response to receiving the user input selecting the graphical user interface element, output a summary evidence passage information graphical user interface summarizing evidence passage information organized into a first set of evidence passage information that is in support of a candidate answer corresponding to the entry being a correct answer for the input question and a second set of evidence passage information that is not in support of the candidate answer being a correct answer for the input question.

19. The computer program product of claim 18, wherein the summary evidence passage information graphical user interface comprises, for the first set of evidence passage information, a first drill down graphical user interface element that is selectable by a user to drill down into individual evidence passages corresponding to the first set of evidence passage information, and further comprises, for the second set of evidence passage information, a second drill down graphical user interface element that is selectable by the user to drill down into individual evidence passages corresponding to the second set of evidence passage information.

20. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions for outputting an indication of confidence score in candidate answers for an input question which, when executed by the processor, cause the processor to;

receive candidate answer information comprising confidence scores associated with candidate answers for the input question;

categorize, for each candidate answer in the candidate answer information, the corresponding confidence score into one of a plurality of confidence score categories;

generate, for each candidate answer in the candidate answer information, a graphical representation of the confidence score category of the candidate answer; and

generate a graphical user interface output comprising an entry for each candidate answer in the candidate answer information, wherein each entry comprises the corresponding graphical representation of the confidence score category of the candidate answer corresponding to the entry.