Reward Based Ranker Array for Question Answer System

• US 20140272884A1
• Filed: 03/13/2013
• Published: 09/18/2014
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Croutwater, Kyle L., Mansjur, Dwi Sianto
• Granted Patent: US 9,251,474 B2
• Time in Patent Office: Days
• US Class Current: 434/322
• CPC Class Codes: G06F 16/903 Querying for retrieval from the web G06F16/953
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G09B 7/04 characterised by modifying the teaching programme in response to a wrong answer, e.g. repeating the question, supplying a further explanation

Abstract

Mechanisms are provided for configuring a ranker array of a question and answer (QA) system. An initial configuration of a ranker array of the QA system is generated which comprises a plurality of rankers with each ranker applying a statistical classification function to candidate answers generated by the QA system. A training question is input to the QA system to generate a training candidate answer set and the ranker array is applied to the training candidate answer set to generate, for each ranker in the ranker array, a ranked listing of candidate answers. For each ranker in the ranker array, a correctness of a corresponding ranked listing of candidate answers is determined and the ranker array is reconfigured based on results of the determined correctness of the corresponding ranked listing of candidate answers for each ranker in the ranker array.

Claims

1. A method, in a data processing system comprising a processor and a memory, for configuring a ranker array of a question and answer (QA) system, comprising:

generating, by the data processing system, an initial configuration of a ranker array of the QA system, wherein the ranker array comprises a plurality of rankers, and wherein each ranker in the ranker array applies a statistical classification function to candidate answers generated by the QA system in response to an input question to generate a ranked listing of candidate answers;

inputting, by the data processing system, a training question to the QA system to generate a training candidate answer set;

applying, by the data processing system, the ranker array to the training candidate answer set to generate, for each ranker in the ranker array, a ranked listing of candidate answers in the training candidate answer set;

determining, by the data processing system, for each ranker in the ranker array, correctness of a corresponding ranked listing of candidate answers; and

reconfiguring, by the data processing system, the ranker array based on results of the determined correctness of the corresponding ranked listing of candidate answers for each ranker in the ranker array.

2. The method of claim 1, wherein reconfiguring the ranker array based on results of the determined correctness comprises:

generating a ranker array reward value for the ranker array based on the correctness of the ranked listing of candidate answers for each of the rankers in the ranker array;

comparing the ranker array reward value to a reward value threshold value; and

reconfiguring the ranker array in response to the ranker array reward value not satisfying a predetermined relationship with regard to the reward value threshold value.

3. The method of claim 2, wherein the ranker array reward value is generated based on a combination of the correctness of the ranked listing of candidate answers for each of the rankers in the ranker array and a quality value associated with each of the rankers in the ranker array.

4. The method of claim 2, wherein the quality value associated with a ranker in the ranker array is a quality value accumulated over a plurality of iterations of training of the ranker in the ranker array.

5. The method of claim 2, wherein generating a ranker array reward value comprises generating a ranker reward value for each ranker in the ranker array and generating the ranker array reward value based on a combination of the ranker reward values for each of the rankers in the ranker array.

6. The method of claim 5, wherein reconfiguring the ranker array comprises:

selecting one or more rankers in the ranker array to be reconfigured based on ranker reward values corresponding to the one or more rankers in the ranker array; and

reconfiguring the selected one or more rankers to generate a reconfigured ranker array.

7. The method of claim 2, wherein reconfiguring the ranker array comprises at least one of modifying a statistical classification function used by one or more of the rankers in the ranker array, or modifying a sampling of a corpus of information upon which one or more of the rankers in the ranker array operate.

8. The method of claim 2, further comprising:

in response to the ranker array reward value satisfying the predetermined relationship with regard to the reward value threshold value, storing ranker array configuration data specifying a configuration of the ranker array, in a ranker array configuration data structure.

9. The method of claim 8, wherein the ranker array configuration data is stored in the ranker array configuration data structure in association with an identifier of a domain of the training question.

10. The method of claim 1, wherein the ranker array comprises at least two rankers of different types, wherein each ranker of a particular type is configured for ranking candidate answers to input questions in a domain different from another ranker of a different type.

11. The method of claim 1, wherein the ranker array comprises a plurality of rankers of different types, and wherein each set of rankers of each type has a different number of rankers than a set of rankers of another type.

12. The method of claim 1, wherein the method is performed iteratively over a plurality of training iterations, and wherein, for each training iteration in the plurality of training iterations, at least one of the training question or a sampling of a corpus of data upon which the training question or ranker array operates is different from other training iterations in the plurality of training iterations.

13. The method of claim 1, wherein determining, for each ranker in the ranker array, correctness of a corresponding ranked listing of candidate answers comprises:

receiving a golden answer set specifying one or more correct answers for the training question;

comparing the golden answer set to the corresponding ranked listing of candidate answers; and

generating a value indicative of correctness of the corresponding ranked listing of candidate answers based on results of the comparison of the golden answer set to the corresponding ranked listing of candidate answers.

14. The method of claim 2, wherein generating a ranker array reward value comprises utilizing the following relationship:

15. The method of claim 1, further comprising:

receiving, after training of the ranker array, a runtime input question from a client computing device;

processing the runtime input question to generate one or more queries to be applied to a corpus of data;

applying the one or more queries to the corpus of data to generate a runtime listing of candidate answers;

processing, by the ranker array, the runtime listing of candidate answers to generate, for each ranker in the ranker array, a corresponding runtime ranked list of candidate answers, thereby constituting a plurality of runtime ranked lists of candidate answers; and

selecting a final answer for the runtime input question from the plurality of runtime ranked lists of candidate answers.

16. The method of claim 15, wherein selecting a final answer for the runtime input question from the plurality of runtime ranked lists of candidate answers comprises using a majority vote algorithm to select a candidate answer that a majority of the rankers in the ranker array indicate to be a most likely candidate answer for the runtime input question.

17. The method of claim 15, wherein the processing of the runtime input question further comprises identifying a domain of the runtime input question, and wherein the method further comprises:

performing a lookup operation in a ranker array configuration data structure based on the identified domain of the runtime input question to retrieve ranker array configuration data corresponding to the identified domain; and

configuring the ranker array according to the retrieved ranker array configuration data.

18. The method of claim 17, wherein the ranker array configuration data specifies at least one of one or more types of rankers to include in the ranker array or a number of each type of ranker to include in the ranker array.

19. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

generate an initial configuration of a ranker array of the QA system, wherein the ranker array comprises a plurality of rankers, and wherein each ranker in the ranker array applies a statistical classification function to candidate answers generated by the QA system in response to an input question to generate a ranked listing of candidate answers;

input a training question to the QA system to generate a training candidate answer set;

apply the ranker array to the training candidate answer set to generate, for each ranker in the ranker array, a ranked listing of candidate answers in the training candidate answer set;

determine, for each ranker in the ranker array, correctness of a corresponding ranked listing of candidate answers; and

reconfigure the ranker array based on results of the determined correctness of the corresponding ranked listing of candidate answers for each ranker in the ranker array.

20. An apparatus, comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

generate an initial configuration of a ranker array of the QA system, wherein the ranker array comprises a plurality of rankers, and wherein each ranker in the ranker array applies a statistical classification function to candidate answers generated by the QA system in response to an input question to generate a ranked listing of candidate answers;

input a training question to the QA system to generate a training candidate answer set;

apply the ranker array to the training candidate answer set to generate, for each ranker in the ranker array, a ranked listing of candidate answers in the training candidate answer set;

determine, for each ranker in the ranker array, correctness of a corresponding ranked listing of candidate answers; and

reconfigure the ranker array based on results of the determined correctness of the corresponding ranked listing of candidate answers for each ranker in the ranker array.