Combining different type coercion components for deferred type evaluation

• US 9,262,938 B2
• Filed: 03/15/2013
• Issued: 02/16/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bagchi, Sugato, Fan, James J., Ferrucci, David A., Kalyanpur, Aditya A., Murdock, James W. IV, Welty, Christopher A.
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Gonzales, Vincent
• Application Number:
  US13/835,411
• Publication Number:
  US 20140272904A1
• Time in Patent Office: 1,068 Days
• US Class Current: 1/1
• CPC Class Codes: G06N 5/00 Computing arrangements using knowledge-based models
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

In a method of answering questions, a question is received, a question LAT is determined, and a candidate answer to the question is identified. Preliminary types for the candidate answer are determined using first components to produce the preliminary types. Each of the first components produces a preliminary type using different methods. A first type-score representing a degree of match between the preliminary type and the question LAT is produced. Each preliminary type and each first type-score is evaluated using second components. Each of the second components produces a second score based on a combination of the first type-score and a measure of degree that the preliminary type matches the question LAT. The second components use different methods to produce the second score. A final score representing a degree of confidence that the candidate answer matches the question LAT is calculated based on the second score.

Claims

1. A method comprising:

receiving a question into a computerized device;

determining a question LAT, using said computerized device, said question LAT being a lexical answer type associated with said question;

identifying a candidate answer to said question, using said computerized device;

automatically determining preliminary types for said candidate answer using first components of said computerized device, said first components using different methods to produce said preliminary types, and each of said first components producing a preliminary type;

automatically scoring a match between said preliminary type and said question LAT, using said computerized device, each of said first components producing a first type-score, said first type-score representing a degree of match between said preliminary type and said question LAT, said scoring being differentiated based on which of said first components produced said preliminary type;

automatically evaluating each said preliminary type and each said first type-score using second components of said computerized device, each of said second components producing a second score based on a combination of said first type-score and a measure of degree that said preliminary type matches said question LAT, said second components using different methods to produce said second score;

automatically calculating a final score based on said second score from each of said second components, using said computerized device; and

automatically outputting said final score representing a degree of confidence that said candidate answer is a type that matches said question LAT, using said computerized device.

2. The method according to claim 1, further comprising:

performing automated query analysis to determine said question LAT, using said computerized device.

3. The method according to claim 1, said scoring further comprising:

matching said candidate answer against instances in a corpus of data, using said computerized device;

retrieving preliminary types from said corpus of data, using said computerized device, said preliminary types being associated with said instances;

matching said question LAT with said preliminary types retrieved from said corpus of data, using said computerized device; and

producing scores representing a degree of match between said question LAT and said preliminary types for said candidate answer, using said computerized device.

4. The method according to claim 1, said first components comprising type coercion components having different resource-specific type classification methodologies to score said measure of degree that said preliminary type matches said question LAT,

said scoring further comprising comparing said preliminary type to said question LAT to produce said first type-score for each said type coercion component and an automated scoring function of said computerized device.

5. The method according to claim 1, further comprising automatically resolving said second score from each of said second components to a single final score using an aggregation function, using said computerized device.

6. The method according to claim 1, further comprising said second components automatically determining whether any of said preliminary types are subtypes of said question LAT based on how well said preliminary types match said question LAT, using said computerized device.

7. A method comprising:

automatically identifying a question lexical answer type (LAT) for a question in a question-answering system, using a computerized device;

automatically generating a candidate answer to said question, using said computerized device;

automatically determining preliminary types for said candidate answer using first components of said computerized device, said first components using different resources to produce said preliminary types, and each of said first components producing a preliminary type for said candidate answer;

automatically scoring a match between said preliminary type and said question LAT using said first components of said computerized device and producing a first type-score for each preliminary type based on an amount that said preliminary type corresponds to said question LAT, said first components using different resources to produce said first type-score for said preliminary type, and said scoring being differentiated based on which of said first components produced said preliminary type;

each of second components automatically evaluating each said preliminary type and said first type-score from each of said first components, using said computerized device, each of said second components producing a second score for said preliminary type for said candidate answer based on a combination of said first type-score and a measure of degree that said preliminary type matches said question LAT; and

automatically outputting a final score based on said second score from each of said second components, said final score representing a degree of confidence that said candidate answer is a type that matches said question LAT, using said computerized device.

8. The method according to claim 7, further comprising:

receiving a question into said computerized device; and

performing automated query analysis to determine said question LAT, using said computerized device.

9. The method according to claim 7, said question-answering system further comprising a corpus of data, said method further comprising:

automatically generating said candidate answer for said question using said corpus of data, using said computerized device.

10. The method according to claim 9, said scoring further comprising:

matching said candidate answer against instances in said corpus of data, using said computerized device;

retrieving preliminary types from said corpus of data, using said computerized device, said preliminary types being associated with said instances;

matching said question LAT with said preliminary types retrieved from said corpus of data, using said computerized device; and

producing scores representing a degree of match between said question LAT and said preliminary types for said candidate answer, using said computerized device.

11. The method according to claim 7, said first components comprising type coercion components having different resource-specific type classification methodologies to score said measure of degree that said preliminary type matches said question LAT,

said scoring further comprising comparing said preliminary type to said question LAT to produce said first type-score for each said preliminary type using said type coercion components and an automated scoring function of said computerized device.

12. The method according to claim 7, further comprising automatically resolving said second score from each of said second components to a single final score using an aggregation function, using said computerized device.

13. The method of claim 7, further comprising:

said second components determining whether any of said preliminary types are a subtype of said question LAT based on how well said preliminary types match said question LAT, using said computerized device.

14. A computer system for determining a confidence score for candidate answers to questions in a question-answering system, comprising:

an automated question answering (QA) system comprising;

a query analysis module;

a candidate answer generator operatively connected to said query analysis module; and

a processor comprising;

a plurality of first components;

a plurality of second components operatively connected to said plurality of first components; and

a classifier operatively connected to said plurality of second components;

and

a user interface operatively connected to said query analysis module, said user interface receiving a question into said automated QA system, said query analysis module determining a question lexical answer type (LAT) for said question;

said candidate answer generator automatically computing a candidate answer to said question;

said processor automatically computing a preliminary type for said candidate answer from each of said plurality of first components, said first components using different methods to produce said preliminary type;

said processor automatically scoring a match between said preliminary type and said question LAT to produce a first type-score from each of said plurality of first components based on an amount that said preliminary type corresponds to said question LAT, said scoring being differentiated based on which component of said plurality of first components produced said preliminary type, and a same preliminary type receiving a different type-score based on differences between said components that produced said same preliminary type;

said processor automatically evaluating each said preliminary type and said first type-score from each of said plurality of first components using each of said plurality of second components and producing a second score for said preliminary type based on a measure of degree that said preliminary type matches said question LAT for each of said plurality of second components; and

said processor automatically outputting a final score for said preliminary type from said classifier based on said second score for said preliminary type from each of said plurality of second components.

15. The system according to claim 14, said automated QA system further comprising a corpus of data,

said candidate answer generator automatically computing a candidate answer to said question using said corpus of data.

16. The system according to claim 15, further comprising:

said processor automatically matching said candidate answer against instances in said corpus of data;

said processor automatically retrieving preliminary types from said corpus of data, said preliminary types being associated with said instances;

said processor automatically matching said question LAT with said preliminary types retrieved from said corpus of data; and

said processor automatically producing scores representing a degree of match between said question LAT and said preliminary types for said candidate answer.

17. The system according to claim 14, said plurality of first components comprising type coercion components having different resource-specific type classification methodologies to score said measure of degree that said preliminary type matches said question LAT,

said processor automatically comparing said preliminary type to said question LAT to produce said score for each said preliminary type using said type coercion components and an automated scoring function of said processor.

18. The system according to claim 14, further comprising:

said processor using said classifier to automatically resolve said second score from each of said plurality of second components to a single final score using an aggregation function.

19. The system according to claim 14, further comprising:

said processor using said plurality of second components to determine whether any of said preliminary types are a subtype of said question LAT based on how well said preliminary types match said question LAT.

20. A computer program product for determining a confidence score for candidate answers to questions in a question-answering system, said computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, said program code being readable and executable by a computer to perform a method comprising:

receiving a question into a computerized device;

determining a question LAT, said question LAT being a lexical answer type associated with said question;

identifying a candidate answer to said question;

automatically determining preliminary types for said candidate answer using first components of said computerized device, said first components using different methods to produce said preliminary types, and each of said first components producing a preliminary type;

automatically scoring a match between said preliminary type and said question LAT, each of said first components producing a first type-score, said first type-score representing a degree of match between said preliminary type and said question LAT, said scoring being differentiated based on which of said first components produced said preliminary type;

automatically evaluating each said preliminary type and each said first type-score using second components of said computerized device, each of said second components producing a second score based on a combination of said first type-score and a measure of degree that said preliminary type matches said question LAT, said second components using different methods to produce said second score;

automatically calculating a final score based on said second score from each of said second components; and

automatically outputting said final score representing a degree of confidence that said candidate answer is a type that matches said question LAT.

21. The computer program product according to claim 20, said method further comprising:

performing automated query analysis to determine said question LAT.

22. The computer program product according to claim 20, said scoring further comprising:

matching said candidate answer against instances in a corpus of data;

retrieving preliminary types from said corpus of data, said preliminary types being associated with said instances;

matching said question LAT with said preliminary types retrieved from said corpus of data; and

producing scores representing a degree of match between said question LAT and said preliminary types for said candidate answer.

23. The computer program product according to claim 20, said first components comprising type coercion components having different resource-specific type classification methodologies to score said measure of degree that said preliminary types match said question LAT,

said scoring further comprising comparing said preliminary type to said question LAT to produce said first type-score for each said preliminary type using said type coercion components and an automated scoring function.

24. The computer program product according to claim 20, said method further comprising:

automatically resolving said second score from each of said second components to a single final score using an aggregation function.

25. The computer program product according to claim 20, said method further comprising:

said second components automatically determining whether said preliminary types are subtypes of said question LAT based on how well said preliminary types match said question LAT.