Providing question and answers with deferred type evaluation using text with limited structure

• US 10,482,115 B2
• Filed: 10/23/2017
• Issued: 11/19/2019
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Murdock, IV, James W., Fan, James J., Welty, Christopher A., Duboue, Pablo A., Ferrucci, David A., Zadrozny, Wlodek W.
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): Morris, John J
• Application Number:
  US15/790,839
• Publication Number:
  US 20180046705A1
• Time in Patent Office: 757 Days
• Field of Search: None
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence

Abstract

A system, method and computer program product for conducting questions and answers with deferred type evaluation based on any corpus of data. The method includes processing a query including waiting until a “Type” (i.e. a descriptor) is determined AND a candidate answer is provided. Then, a search is conducted to look (search) for evidence that the candidate answer has the required Lexical Answer Type (e.g., as determined by a matching function that can leverage a parser, a semantic interpreter and/or a simple pattern matcher). Prior to or during candidate answer evaluation, a process is provided for extracting and storing collections of entity-type pairs from semi-structured text documents. During QA processing and candidate answer scoring, a process is implemented to match the query LAT against the lexical type of each provided candidate answer and generate a score judging a degree of match.

Claims

1. A system for automatically generating answers to questions, said system comprising:

a memory storage device;

one or more hardware processors, each operatively connected to said memory device and configured to;

analyze a corpus of documents to identify a document containing a list, wherein said list contains item-delimiting markup;

parse said list to identify type information and entities in said list, wherein to identify said type information and entities in said list, said one or more hardware processors are further configured to;

extract a type from a title indicating said list;

determine a presence of item-delimeter mark-up associated with said list, each mark-up delimeter including or not including one or more associated hyperlinks, and if a hyperlink is included;

obtain an instance of a hyperlink in closest proximity to the mark-up item-delimeter, and extracting an entity from a target of said hyperlink instance; and

if a hyperlink is not included;

use an annotator to identify phrases included in text associated with the item-delimeter mark-up, and extract a most salient phrase as said entity;

create entity-type pairs, wherein said entity-type pairs comprise said entities and the identified type from said list;

receive a lexical answer type associated with an input query;

receive a candidate answer to said query;

determine whether said candidate answer is associated with an entity in said entity-type pairs;

for any associated entity-type pairs, comparing said extracted type in said associated entity-type pair with said lexical answer type;

generate a type-matching score, wherein said type-matching score is indicative of a quality of said obtained candidate answer based on matching types; and

use said type-matching score to evaluate said candidate answer as an answer to said query.

2. The system of claim 1, wherein said one or more hardware processors are further configured to:

store, in a further memory storage device, said created entity-type pairs, each entity-pair structure representing said one or more entities and associated type,

wherein to determine whether said candidate answer is associated with an entity in said entity-type pairs, said one or more hardware processor is further configured to;

access said stored entity-type pairs to identify a type from an entity-type pair.

3. The system of claim 1, wherein said corpus analyzing, said list identifying, said list parsing and said creating entity-type pairs are performed off-line as a pre-processing step.

4. The system of claim 2, wherein said corpus analyzing, said list identifying, said list parsing and said creating entity-type pairs are performed on-demand by a Question Answer system at a time of receiving said input query.

5. The system of claim 2, wherein said one or more hardware processors are further configured to:

parse each respective said lexical answer type associated with an input query and said identified type for any associated entity-type pair of a candidate answer, to obtain respective terms or phrases for each.

6. The system of claim 1, wherein to compare said type in said associated entity-type pair with said lexical answer type, said one or more hardware processors are further configured to:

match individual terms of respective lexical answer type associated with an input query and said type for any associated entity-type pair, or

match entire phrases of each respective lexical answer type associated with an input query and said type for any associated entity-type pair.

7. The system of claim 5, wherein said one or more hardware processors are further configured to:

determine which terms of respective said lexical answer type associated with an input query and said type for any associated entity-type pair are to be used for said term matching;

obtain individual scores based on degree of match between terms of respective said lexical answer type associated with an input query and said identified type for any associated entity-type pair; and

combine obtained scores of said matches determined for said phrase matching.

8. The system as claimed in claim 1, wherein to analyze a corpus of documents to identify a document containing a list, said one or more hardware processors are further configured to:

parse content of said corpus to identify said item-delimiting markup, said item delimited mark-up specifying an associated type and entities forming an entity-type pair.

9. The system as claimed in claim 1, wherein said item-delimiting markup includes a title, a header, a recitation of the word “

list”

of entities of a specified type, bullet markers, parentheses, a hypertext link, a Uniform Resource Locator, a table in said data source.

10. The system as claimed in claim 1, wherein said item-delimiting markup includes a tag representing a category or sub-category including entities of a specified type.

11. The system of claim 6, wherein term matching is one of:

aggregate term matching or primitive term matching.

12. A computer program product for automatically generating answers to questions, the computer program device comprising a non-transitory storage medium, said medium readable by a processing circuit and storing instructions run by the processing circuit for performing a method, the method comprising:

analyzing a corpus of documents to identify a document containing a list, wherein said list contains item-delimiting markup;

parsing said list to identify type information and entities in said list, wherein to identify said type information and entities in said list, said processing circuit further performs;

extracting a type from a title indicating said list;

determining a presence of item-delimeter mark-up associated with said list, each mark-up delimeter including or not including one or more associated hyperlinks, and if a hyperlink is included;

obtaining an instance of a hyperlink in closest proximity to the mark-up item-delimeter, and extracting an entity from a target of said hyperlink instance; and

if a hyperlink is not included;

using an annotator to identify phrases included in text associated with the item-delimeter mark-up, and extracting a most salient phrase as said entity;

creating entity-type pairs, wherein said entity-type pairs comprise said entities and the identified type from said list;

receiving a lexical answer type associated with an input query;

receiving a candidate answer to said query;

determining whether said candidate answer is associated with an entity in said entity-type pairs;

for any associated entity-type pairs, comparing said extracted type in said associated entity-type pair with said lexical answer type;

generating a type-matching score, wherein said type-matching score is indicative of a quality of said obtained candidate answer based on matching types; and

using said type-matching score to evaluate said candidate answer as an answer to said query.

13. The computer program product of claim 12, further comprising:

storing, in a further memory storage device, said created entity-type pairs, each entity-pair structure representing said one or more entities and associated type,

wherein said determining whether said candidate answer is associated with an entity in said entity-type pairs comprises;

accessing said stored entity-type pairs to identify a type from an entity-type pair.

14. The computer program product of claim 13, wherein said corpus analyzing, said list identifying, said list parsing and said creating entity-type pairs are performed off-line as a pre-processing step.

15. The computer program product of claim 13, wherein said corpus analyzing, said list identifying, said list parsing and said creating entity-type pairs are performed on-demand by a Question Answer system at a time of receiving said input query.

16. The computer program product of claim 13, further comprising:

parsing each respective said lexical answer type associated with an input query and said identified type for any associated entity-type pair of a candidate answer, to obtain respective terms or phrases for each.

17. The computer program product of claim 12, wherein said comparing said type in said associated entity-type pair with said lexical answer type comprises:

matching individual terms of respective lexical answer type associated with an input query and said type for any associated entity-type pair, or

matching entire phrases of each respective lexical answer type associated with an input query and said type for any associated entity-type pair.

18. The computer program product of claim 16, further comprising:

determining which terms of respective said lexical answer type associated with an input query and said type for any associated entity-type pair are to be used for said term matching;

obtaining individual scores based on degree of match between terms of respective said lexical answer type associated with an input query and said identified type for any associated entity-type pair; and

combining obtained scores of said matches determined for said phrase matching.

19. The computer program product as claimed in claim 12, wherein said analyzing a corpus of documents to identify a document containing a list comprises:

parsing content of said corpus to identify said item-delimiting markup, said item delimited mark-up specifying an associated type and entities forming an entity-type pair.

20. The computer program product as claimed in claim 12, wherein said item-delimiting markup includes a title, a header, a recitation of the word “

list”

of entities of a specified type, bullet markers, parentheses, a hypertext link, a Uniform Resource Locator, a table in said data source, or a tag representing a category or sub-category including entities of a specified type.