Combining different type coercion components for deferred type evaluation

• US 10,108,904 B2
• Filed: 12/04/2015
• Issued: 10/23/2018
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bagchi, Sugato, Fan, James J., Ferrucci, David A., Kalyanpur, Aditya A., Murdock, IV, James W., Welty, Christopher A.
• Primary Examiner(s): Gonzales, Vincent
• Application Number:
  US14/959,069
• Publication Number:
  US 20160086503A1
• Time in Patent Office: 1,054 Days
• CPC Class Codes: G06N 5/00 Computing arrangements using knowledge-based models
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

In a method of answering questions, a question is received, a question LAT is determined, and a candidate answer to the question is identified. Preliminary types for the candidate answer are determined using first components to produce the preliminary types. Each of the first components produces a preliminary type using different methods. A first type-score representing a degree of match between the preliminary type and the question LAT is produced. Each preliminary type and each first type-score is evaluated using second components. Each of the second components produces a second score based on a combination of the first type-score and a measure of degree that the preliminary type matches the question LAT. The second components use different methods to produce the second score. A final score representing a degree of confidence that the candidate answer matches the question LAT is calculated based on the second score.

Claims

1. A method comprising:

receiving a question into a computerized device;

determining a lexical answer type associated with said question, using said computerized device;

identifying candidate answers to said question, using said computerized device;

using a first plurality of first type coercion components of said computerized device, determining preliminary types for each of said candidate answers, each first type coercion component of said first plurality of first type coercion components using different resources and different processes to determine a preliminary type for each of said candidate answers;

comparing said preliminary types for each of said candidate answers to said lexical answer type associated with said question and producing a plurality of first type-scores for each of said candidate answers, using said first plurality of first type coercion components of said computerized device, each first type coercion component of said first plurality of first type coercion components producing a first type-score for each of said candidate answers, each first type-score of said plurality of first type-scores representing the degree of match between said preliminary types for each of said candidate answers and said lexical answer type associated with said question based on different methods particular to each first type coercion component of said first plurality of first type coercion components, each said first type-score of said plurality of first type-scores being differentiated based on which first type coercion component of said first plurality of first type coercion components produced said first type-score;

inputting said preliminary types for each of said candidate answers from each first type coercion component of said first plurality of first type coercion components and each first type-score of said plurality of first type-scores from each first type coercion component of said first plurality of first type coercion components into second type coercion components of a second plurality of second type coercion components of said computerized device;

evaluating said preliminary types for each of said candidate answers from each first type coercion component of said first plurality of first type coercion components and each first type-score of said plurality of first type-scores from each first type coercion component of said first plurality of first type coercion components using each second type coercion component of said second plurality of second type coercion components and producing a plurality of second scores using said second plurality of second type coercion components of said computerized device, each second type coercion component of said second plurality of second type coercion components separately determining a type for each of said candidate answers and producing a separate measure of the degree of match between said type for each of said candidate answers and said lexical answer type associated with said question, each second type coercion component of said second plurality of second type coercion components evaluating each said preliminary type, each said first type-score from each first type coercion component of said first plurality of first type coercion components, and the separate measure of the degree of match between said type for each of said candidate answers and said lexical answer type associated with said question, each second score of said plurality of second scores representing a combination of said first type-scores from each first type coercion component of said first plurality of first type coercion components and the separate measure of the degree of match between said type for each of said candidate answers and said lexical answer type associated with said question determined by each second type coercion component of said second plurality of second type coercion components based on different methods particular to each second type coercion component of said second plurality of second type coercion components;

inputting said second scores for each of said candidate answers from each second type coercion component of said second plurality of second type coercion components of said computerized device into a classifier of said computerized device;

calculating a final score based on said second scores for each of said candidate answers by aggregating the second score from each second type coercion component of said second plurality of second type coercion components, using said classifier of said computerized device; and

outputting said final score representing a degree of confidence that each of said candidate answers is a type that matches said lexical answer type associated with said question, using said computerized device.

2. The method according to claim 1, further comprising:

performing automated query analysis to determine said lexical answer type associated with said question, using said computerized device.

3. The method according to claim 1, further comprising:

matching said candidate answers against instances in a corpus of data, using said computerized device;

retrieving preliminary types from said corpus of data, using said computerized device, said preliminary types being associated with said instances;

matching said lexical answer type associated with said question with said preliminary types retrieved from said corpus of data, using said computerized device; and

producing scores representing a degree of match between said lexical answer type associated with said question and said preliminary types for said candidate answers, using said computerized device.

4. The method according to claim 1, said producing first type-scores comprising using said first type coercion components of said computerized device, each of said first type coercion components having different resource-specific type classification methodologies determining preliminary types for each of said candidate answers and scoring a measure of degree that said preliminary types match said lexical answer type associated with said question, each of said first type coercion components producing a preliminary type and a first type-score.

5. The method according to claim 1, said producing second scores comprising using said second type coercion components of said computerized device, each of said second type coercion components having different resource-specific type classification methodologies determining types for each of said candidate answers and scoring a measure of degree that said types match said lexical answer type associated with said question, each of said second type coercion components producing a second score.

6. The method according to claim 1, said calculating a final score based on said second scores comprising using an aggregation function resolving said second scores from each second type coercion component of said second plurality of second type coercion components to a single final score, using said computerized device.

7. The method according to claim 1, further comprising determining whether any of said preliminary types are subtypes of said lexical answer type associated with said question, using said computerized device.

8. A method comprising:

identifying a question lexical answer type (LAT) for a question in a question-answering system, using a computerized device;

generating candidate answers to said question, using said computerized device;

determining preliminary types for each of said candidate answers using different resources of said computerized device to produce said preliminary types, each of said different resources using different processes to determine a preliminary type for each of said candidate answers;

comparing said preliminary types for each of said candidate answers to said question LAT and scoring a match between said preliminary types and said question LAT using a first plurality of first type coercion components of said computerized device, each first type coercion component of said first plurality of first type coercion components producing a first type-score representing the degree of match between said preliminary types and said question LAT, each first type coercion component of said first plurality of first type coercion components using different resources to produce said first type-score, and said scoring being differentiated based on which first type coercion component of said first plurality of first type coercion components produced a given preliminary type;

evaluating each said preliminary type and each said first type-score from each first type coercion component of said first plurality of first type coercion components, using a second plurality of second type coercion components of said computerized device, each second type coercion component of said second plurality of second type coercion components producing a second score for said preliminary type for said candidate answer based on a combination of said first type-score and a separate determination of type for each of said candidate answers and measure of degree that said type matches said question LAT, each second type coercion component of said second plurality of second type coercion components using different resources to produce said second score, and said scoring being differentiated based on which second type coercion component of said second plurality of second type coercion components produced a given type; and

outputting a final score by aggregating said second score from each second type coercion component of said second plurality of second type coercion components, said final score representing a degree of confidence that said candidate answer is a type that matches said question LAT, using said computerized device.

9. The method according to claim 8, further comprising:

receiving a question into said computerized device; and

performing automated query analysis to determine said question LAT, using said computerized device.

10. The method according to claim 8, said question-answering system further comprising a corpus of data, said method further comprising:

generating said candidate answers for said question from said corpus of data, using said computerized device.

11. The method according to claim 10, said scoring further comprising:

matching said candidate answers against instances in said corpus of data, using said computerized device;

retrieving preliminary types from said corpus of data, using said computerized device, said preliminary types being associated with said instances;

matching said question LAT with said preliminary types retrieved from said corpus of data, using said computerized device; and

producing scores representing a degree of match between said question LAT and said preliminary types for said candidate answer, using said computerized device.

12. The method according to claim 8, said first type coercion components having different resource-specific type classification methodologies determining preliminary types for each of said candidate answers and scoring said measure of degree that said preliminary types match said question LAT.

13. The method of claim 8, said second type coercion components having different resource-specific type classification methodologies determining types for each of said candidate answers and scoring said measure of degree that said types match said question LAT.

14. A computer program product for determining a confidence score for candidate answers to questions in a question-answering system, said computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, said program code being readable and executable by a computer to perform a method comprising:

receiving a question into a computerized device;

determining a question lexical answer type (LAT) associated with said question;

identifying candidate answers to said question;

using a first plurality of first type coercion components, determining preliminary types for each of said candidate answers, each first type coercion component of said first plurality of first type coercion components using different resources and different processes to determine a preliminary type for each of said candidate answers;

producing first type-scores representing the degree of match between said preliminary types and said question LAT, using said first plurality of first type coercion components, each first type coercion component of said first plurality of first type coercion components producing a first type-score for each of said candidate answers;

using a second plurality of second type coercion components, separately determining types for each of said candidate answers, each second type coercion component of said second plurality of second type coercion components determining a type for each of said candidate answers based on different methods particular to each second type coercion component of said second plurality of second type coercion components;

producing second scores representing a combination of said first type-scores from each first type coercion component of said first plurality of first type coercion components and a separate measure of the degree of match between said types and said question LAT, using said second plurality of second type coercion components, said second scores being based on different methods particular to each second type coercion component of said second plurality of second type coercion components;

calculating a final score based on said second scores for each of said candidate answers by aggregating the second score from each second type coercion component of said second plurality of second type coercion components; and

outputting said final score representing a degree of confidence that said candidate answer is a type that matches said question LAT.

15. The computer program product according to claim 14, said method further comprising:

performing automated query analysis to determine said question LAT.

16. The computer program product according to claim 14, said method further comprising:

matching said candidate answer against instances in a corpus of data;

retrieving preliminary types from said corpus of data, said preliminary types being associated with said instances;

matching said question LAT with said preliminary types retrieved from said corpus of data; and

producing scores representing a degree of match between said question LAT and said preliminary types for said candidate answers.

17. The computer program product according to claim 14, said producing first type-scores comprising using said first type coercion components, each of said first type coercion components having different resource-specific type classification methodologies determining preliminary types for each of said candidate answers and scoring a measure of degree that said preliminary types match said question LAT, each of said first type coercion components producing a preliminary type and a first type-score.

18. The computer program product according to claim 14, said producing second scores comprising using said second type coercion components, each of said second type coercion components having different resource-specific type classification methodologies determining types for each of said candidate answers and scoring a measure of degree that said types match said question LAT, each of said second type coercion components producing a second score.

19. The computer program product according to claim 14, said calculating a final score based on said second scores comprising using an aggregation function resolving said second scores from each second type coercion component of said second plurality of second type coercion components to a single final score.

20. The computer program product according to claim 14, further comprising determining whether any of said preliminary types are subtypes of said question LAT.