Using paraphrase metrics for answering questions

• US 9,684,714 B2
• Filed: 12/22/2014
• Issued: 06/20/2017
• Est. Priority Date: 12/22/2014
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Winkler, Timothy P., Murdock, IV, James W., Segal, Benjamin P., Levas, Anthony T., Prager, John M.
• Primary Examiner(s): NGUYEN, CAM LINH T
• Application Number:
  US14/579,612
• Publication Number:
  US 20160179939A1
• Time in Patent Office: 911 Days
• Field of Search: 707728, 707774, 707749, 707725, 707769
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri

Abstract

A mechanism is provided in a data processing system for using paraphrase metrics for answering questions. The mechanism receives an input question and generating a candidate answer from a corpus of information. The candidate answer has a supporting passage from the corpus of information. The mechanism divides the input question into a first sequence of tokens and divides the supporting passage into a second sequence of tokens. The mechanism identifies a plurality of subsequences of tokens within the second sequence of tokens and applies a paraphrase metric to compare the first sequence of tokens to each of the plurality of subsequences of tokens to generate a plurality of paraphrase metric scores. The mechanism then determines a confidence score for the candidate answer based on a highest paraphrase metric score within the plurality of paraphrase metric scores.

Claims

1. A method, in a data processing system, for using paraphrase metrics for answering questions, the method comprising:

receiving an input question;

generating a candidate answer from a corpus of information, wherein the candidate answer has a supporting passage from the corpus of information;

dividing the input question into a first sequence of tokens;

dividing the supporting passage into a second sequence of tokens;

matching question tokens from the first set of tokens to passage tokens from the second set of tokens, wherein matching the question tokens to the passage tokens comprises identifying a focus of the input question and matching the focus of the input question to an occurrence of the candidate answer in the supporting passage, wherein matching the focus of the input question to an occurrence of the candidate answer in the supporting passage comprises;

treating the focus of the question as a match for the candidate answer;

modifying the input question to replace text of the focus of the question with text of the candidate answer;

modifying the passage to replace text of the candidate answer with text of the focus of the question;

or

modify the input question and the passage to replace text of the focus of the question and text of the candidate answer with a common reserved constant string;

identifying a plurality of subsequences of tokens within the second sequence of tokens;

applying a paraphrase metric to compare the first sequence of tokens to each of the plurality of subsequences of tokens to generate a plurality of paraphrase metric scores; and

determining a confidence score for the candidate answer based on a highest paraphrase metric score within the plurality of paraphrase metric scores.

2. The method of claim 1, wherein the paraphrase metric is a machine translation metric.

3. The method of claim 1, further comprising applying a plurality of paraphrase metrics to produce a plurality of results.

4. The method of claim 3, further comprising eliminating a given paraphrase metric that is determined to be ineffective based on the plurality of results from future application.

5. The method of claim 1, wherein dividing the input question into the first sequence of tokens comprises:

dividing the input question into a plurality of sequences of question tokens;

identifying a given sequence of question tokens within the plurality of sequences of question tokens having important content of the input question as the first sequence of tokens.

6. The method of claim 1, wherein identifying the plurality of subsequences of tokens within the second sequence of tokens comprises restricting the plurality of subsequences of tokens such that:

a number of tokens in each subsequence of tokens must be within a first predetermined number of a number of tokens in the first sequence of tokens;

each subsequence must contain at least a second predetermined number of tokens that match tokens in the first sequence of tokens;

each subsequence must contain at least a third predetermined number of non-stop-word tokens that match tokens in the first sequence of tokens;

each subsequence must contain the candidate answer,

each subsequence may not contain a sentence boundary;

each subsequence must begin at a sentence boundary;

each subsequence must end at a sentence boundary;

each subsequence must begin at a clause boundary;

each subsequence must end at a clause boundary;

each subsequence must be contiguous;

or

each subsequence must include no more than a fourth predetermined number of contiguous sub-subsequences.

7. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an input question;

generate a candidate answer from a corpus of information, wherein the candidate answer has a supporting passage from the corpus of information;

divide the input question into a first sequence of tokens;

divide the supporting passage into a second sequence of tokens;

match question tokens from the first set of tokens to passage tokens from the second set of tokens, wherein matching the question tokens to the passage tokens comprises identifying a focus of the input question and matching the focus of the input question to an occurrence of the candidate answer in the supporting passage, wherein matching the focus of the input question to an occurrence of the candidate answer in the supporting passage comprises;

treating the focus of the question as a match for the candidate answer;

modifying the input question to replace text of the focus of the question with text of the candidate answer;

modifying the passage to replace text of the candidate answer with text of the focus of the question;

or

modify the input question and the passage to replace text of the focus of the question and text of the candidate answer with a common reserved constant string;

identify a plurality of subsequences of tokens within the second sequence of tokens;

apply a paraphrase metric to compare the first sequence of tokens to each of the plurality of subsequences of tokens to generate a plurality of paraphrase metric scores; and

determine a confidence score for the candidate answer based on a highest paraphrase metric score within the plurality of paraphrase metric scores.

8. The computer program product of claim 7, wherein dividing the input question into the first sequence of tokens comprises:

dividing the input question into a plurality of sequences of question tokens;

identifying a given sequence of question tokens within the plurality of sequences of question tokens having important content of the input question as the first sequence of tokens.

9. The computer program product of claim 7, wherein identifying the plurality of subsequences of tokens within the second sequence of tokens comprises restricting the plurality of subsequences of tokens such that:

a number of tokens in each subsequence of tokens must be within a first predetermined number of a number of tokens in the first sequence of tokens;

each subsequence must contain at least a second predetermined number of tokens that match tokens in the first sequence of tokens;

each subsequence must contain at least a third predetermined number of non-stop-word tokens that match tokens in the first sequence of tokens;

each subsequence must contain the candidate answer;

each subsequence may not contain a sentence boundary;

each subsequence must begin at a sentence boundary;

each subsequence must end at a sentence boundary;

each subsequence must begin at a clause boundary;

each subsequence must end at a clause boundary;

each subsequence must be contiguous;

or

each subsequence must include no more than a fourth predetermined number of contiguous sub-subsequences.

10. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question;

generate a candidate answer from a corpus of information, wherein the candidate answer has a supporting passage from the corpus of information;

divide the input question into a first sequence of tokens;

divide the supporting passage into a second sequence of tokens;

match question tokens from the first set of tokens to passage tokens from the second set of tokens, wherein matching the question tokens to the passage tokens comprises identifying a focus of the input question and matching the focus of the input question to an occurrence of the candidate answer in the supporting passage, wherein matching the focus of the input question to an occurrence of the candidate answer in the supporting passage comprises;

treating the focus of the question as a match for the candidate answer;

modifying the input question to replace text of the focus of the question with text of the candidate answer;

modifying the passage to replace text of the candidate answer with text of the focus of the question;

or

modify the input question and the passage to replace text of the focus of the question and text of the candidate answer with a common reserved constant string;

identify a plurality of subsequences of tokens within the second sequence of tokens;

apply a paraphrase metric to compare the first sequence of tokens to each of the plurality of subsequences of tokens to generate a plurality of paraphrase metric scores; and

determine a confidence score for the candidate answer based on a highest paraphrase metric score within the plurality of paraphrase metric scores.

11. The apparatus of claim 10, wherein dividing the input question into the first sequence of tokens comprises:

dividing the input question into a plurality of sequences of question tokens;

identifying a given sequence of question tokens within the plurality of sequences of question tokens having important content of the input question as the first sequence of tokens.