Displaying answers in accordance with answer classifications

• US 10,061,842 B2
• Filed: 12/09/2014
• Issued: 08/28/2018
• Est. Priority Date: 12/09/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gordon, John B., Isensee, Scott H., Marrara, Mark, O'Keeffe, William G., Vardy, Cale
• Primary Examiner(s): Vincent, David R
• Application Number:
  US14/564,912
• Publication Number:
  US 20160162790A1
• Time in Patent Office: 1,358 Days
• Field of Search: 706 12, 706 15, 706 45
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A question is received in a computerized answering system via an input device. A plurality of classes corresponding to the question are retrieved from a memory device. One or more answers to the question are generated by a processor. Each answer is associated with a confidence value. The confidence values associated with the answers are compared with thresholds associated with the plurality of classes. The answers are classified by the processor into the plurality of classes based on the comparing. The answers are presented via a display device in accordance with the classification of the answers. A user response to at least one of the answers is detected. At least one of the plurality of classes is adjusted based on the user response.

Claims

1. A method for providing answers in a computerized answering system to client devices of a network, comprising:

receiving, by a network interface of the computerized answering system via the network, a question for the computerized answering system from a client device of the network;

retrieving, from a memory device of the computerized answering system, a plurality of classes corresponding to the question;

generating, by a processor of the computerized answering system, answers to the question, wherein each answer is associated with a confidence value;

comparing, by the processor, the confidence values associated with the answers with thresholds associated with the plurality of classes;

classifying, by the processor, the answers into the plurality of classes based on the comparing;

providing, by the network interface via the network, the answers for presentation in a display of the client device in accordance with the classification of the answers;

receiving, by the network interface via the network, one or more user responses to one of the answers that was generated by the computerized answering system and classified in a first class of the plurality of classes; and

adjusting, by the processor and based on the one or more user responses, at least a threshold associated with a second class of the plurality of classes to reclassify the one of the answers that was generated by the computerized answering system from the first class to the second class in the computerized answering system.

2. The method of claim 1, wherein said adjusting, based on the one or more user responses, at least the threshold associated with the second class of the plurality of classes comprises:

determining the one or more user responses to the one of the answers classified in the first class of the plurality of classes to be a negative response; and

adjusting a threshold of the first class of the plurality of classes to exclude answers having the same confidence value as the one of the answers from being classified into the first class; and

adjusting the threshold of the second class of the plurality of classes to include answers having the same confidence value as the one of the answers in order to be classified into the second class.

3. The method of claim 1, wherein each of the plurality of classes comprises:

a threshold value operable to indicate which ones of a set of answers belong in a class based on comparing associated answer confidence values with the threshold value; and

an indication of how the answers in the class are to be presented via the display of the client device.

4. The method of claim 3, wherein the indication of how the answers are to be presented comprises at least one of:

an indication that the answers be displayed;

an indication that the answers be available to view;

or

an indication that the answers not be displayed.

5. The method of claim 1, wherein the confidence values fall within a range, and wherein higher confidence values indicate increased confidence that the answers are correct.

6. The method of claim 1, further comprising:

monitoring user interaction with a user interface of the display of the client device;

selecting a user profile corresponding to the user interaction; and

presenting answers to a second question in accordance with the selected user profile.

7. The method of claim 6, wherein the user profile comprises at least one of:

an indication that only a specified number of answers be displayed;

an indication that only answers with a certain confidence value be displayed;

or

an indication that evidence for answers be displayed.

20. The method of claim 1, wherein said receiving the one or more user responses and adjusting, based on the one or more user responses, the threshold associated with the second class of the plurality of classes comprises:

adjusting, based on the one or more user responses, at least a threshold associated with the first class of the plurality of classes and at least the threshold associated with the second class of the plurality of classes to reclassify the one of the answers from the first class to the second class in the computerized answering system.

8. A computer program product for providing answers in a computerized answering system to client devices of a network, comprising:

a computer readable storage medium having program instructions embodied therewith, the program instructions comprising program instructions to,

receive, by a network interface of the computerized answering system via the network, a question for the computerized answering system from a client device of the network;

retrieve, from a memory device of the computerized answering system, a plurality of classes corresponding to the question;

generate, by a processor, answers to the question of the computerized answering system, wherein each answer is associated with a confidence value;

compare, by the processor, the confidence values associated with the answers with thresholds associated with the plurality of classes;

classify, by the processor, the answers into the plurality of classes based on the comparing;

provide, by the network interface via the network, the answers for presentation in a display of the client device in accordance with the classification of the answers;

receive, by the network interface via the network, one or more user responses to one of the answers that was generated by the computerized answering system and classified in a first class of the plurality of classes; and

adjust, by the processor and based on the one or more user responses, at least a threshold associated with a second class of the plurality of classes to reclassify the one of the answers that was generated by the computerized answering system from the first class to the second class in the computerized answering system.

9. The computer program product of claim 8, wherein the program instructions to adjust, based on the one or more user responses, at least the threshold associated with the second class of the plurality of classes comprises the program instructions to:

determine the one or more user responses to the one of the answers classified in the first class of the plurality of classes to be a negative response;

adjust a threshold of the first class of the plurality of classes to exclude answers having the same confidence value as the one of the answers from being classified into the first class; and

adjust the threshold of the second class of the plurality of classes to include answers having the same confidence value as the one of the answers in order to be classified into the second class.

10. The computer program product of claim 8, wherein each of the plurality of classes comprises:

a threshold value operable to indicate which ones of a set of answers belong in a class based on comparing associated answer confidence values with the threshold value; and

an indication of how the answers in the class are to be presented via the display of the client device.

11. The computer program product of claim 10, wherein the indication of how the answers are to be presented comprises at least one of:

an indication that the answers be displayed;

an indication that the answers be available to view;

or

an indication that the answers not be displayed.

12. The computer program product of claim 8, further having program instructions to:

monitor user interaction with a user interface of the display of the client device;

select a user profile corresponding to the user interaction; and

present answers to a second question in accordance with the selected user profile.

13. The computer program product of claim 12, wherein the user profile comprises at least one of:

an indication that only a specified number of answers be displayed;

an indication that only answers with a certain confidence value be displayed;

or

an indication that evidence for answers be displayed.

14. A server for implementing a computerized answering system for providing answers to client devices of a network, comprising:

a processor;

a network interface; and

a computer readable storage medium having program instructions embodied therewith, the program instructions executable by the processor to cause the server to,

receive, by the network interface via the network, a question for the computerized answering system from a client device of the network;

retrieve, from the computer readable storage medium, a plurality of classes corresponding to the question;

generate, by the processor, answers to the question, wherein each answer is associated with a confidence value;

compare, by the processor, the confidence values associated with the answers with thresholds associated with the plurality of classes;

classify, by the processor, the answers into the plurality of classes based on the comparing;

provide, by the network interface via the network, the answers for presentation in a display of the client device in accordance with the classification of the answers;

receive, by the network interface via the network, one or more user responses to one of the answers that was generated by the computerized answering system and classified in a first class of the plurality of classes; and

adjust, by the processor and based on the one or more user responses, at least a threshold associated with a second class of the plurality of classes to reclassify the one of the answers that was generated by the computerized answering system from the first class to the second class in the computerized answering system.

15. The server of claim 14, wherein the program instructions executable by the processor to cause the server to adjust, based on the one or more user responses, at least the threshold associated with the second class of the plurality of classes comprises the program instructions executable by the processor to cause the server to:

determine the one or more user responses to the one of the answers classified in the first class of the plurality of classes to be a negative response;

adjust a threshold of the first class of the plurality of classes to exclude answers having the same confidence value as the one of the answers from being classified into the first class; and

adjust the threshold of the second class of the plurality of classes to include answers having the same confidence value as the one of the answers in order to be classified into the second class.

16. The server of claim 14, wherein each of the plurality of classes comprises:

a threshold value operable to indicate which ones of a set of answers belong in a class based on comparing associated answer confidence values with the threshold value; and

an indication of how the answers in the class are to be presented via the display of the client device.

17. The server of claim 16, wherein the indication of how the answers are to be presented comprises one of:

an indication that the answers be displayed;

an indication that the answers be available to view;

or

an indication that the answers not be displayed.

18. The server of claim 14, further having program instructions executable by the processor to cause the server to:

monitor user interaction with a user interface of the display of the client device;

select a user profile corresponding to the user interaction; and

present answers to a second question in accordance with the selected user profile.

19. The server of claim 18, wherein the user profile comprises one of:

an indication that only a specified number of answers be displayed;

an indication that only answers with a certain confidence value be displayed;

or

an indication that evidence for answers be displayed.