Using paraphrase metrics for answering questions

• US 10,642,874 B2
• Filed: 06/08/2017
• Issued: 05/05/2020
• Est. Priority Date: 12/22/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Levas, Anthony T., Murdock, IV, James W., Prager, John M., Segal, Benjamin P., Winkler, Timothy P.
• Primary Examiner(s): Nguyen, Cam Linh T
• Application Number:
  US15/617,617
• Publication Number:
  US 20170270191A1
• Time in Patent Office: 1,062 Days
• Field of Search: 707728
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri

Abstract

A mechanism is provided in a data processing system for using paraphrase metrics for answering questions. The mechanism receives an input question and generating a candidate answer from a corpus of information. The candidate answer has a supporting passage from the corpus of information. The mechanism divides the input question into a first sequence of tokens and divides the supporting passage into a second sequence of tokens. The mechanism identifies a plurality of subsequences of tokens within the second sequence of tokens and applies a paraphrase metric to compare the first sequence of tokens to each of the plurality of subsequences of tokens to generate a plurality of paraphrase metric scores. The mechanism then determines a confidence score for the candidate answer based on a highest paraphrase metric score within the plurality of paraphrase metric scores.

Claims

1. A method, in a data processing system, for using paraphrase metrics for answering questions, the method comprising:

receiving an input question;

generating a candidate answer from a corpus of information, wherein the candidate answer has a supporting passage from the corpus of information;

dividing the input question into a first sequence of tokens;

dividing the supporting passage into a second sequence of tokens;

identifying a plurality of subsequences of tokens within the second sequence of tokens;

applying a paraphrase metric to compare the first sequence of tokens to each of the plurality of subsequences of tokens to generate a plurality of paraphrase metric scores; and

determining a confidence score for the candidate answer based on a highest paraphrase metric score within the plurality of paraphrase metric scores.

2. The method of claim 1, wherein the paraphrase metric is a machine translation metric.

3. The method of claim 1, further comprising:

applying a plurality of paraphrase metrics to produce a plurality of results; and

eliminating a given paraphrase metric that is determined to be ineffective based on the plurality of results from future application.

4. The method of claim 1, wherein dividing the input question into the first sequence of tokens comprises:

dividing the input question into a plurality of sequences of question tokens;

identifying a given sequence of question tokens within the plurality of sequences of question tokens having important content of the input question as the first sequence of tokens.

5. The method of claim 1, further comprising:

matching question tokens from the first set of tokens to passage tokens from the second set of tokens.

6. The method of claim 5, wherein matching the question tokens to the passage tokens comprises:

identifying a focus of the input question; and

matching the focus of the input question to an occurrence of the candidate answer in the supporting passage.

7. The method of claim 1, wherein identifying the plurality of subsequences of tokens within the second sequence of tokens comprises restricting the plurality of subsequences of tokens such that:

a number of tokens in each subsequence of tokens must be within a first predetermined number of a number of tokens in the first sequence of tokens;

each subsequence must contain at least a second predetermined number of tokens that match tokens in the first sequence of tokens;

each subsequence must contain at least a third predetermined number of non-stop-word tokens that match tokens in the first sequence of tokens;

each subsequence must contain the candidate answer;

each subsequence may not contain a sentence boundary;

each subsequence must begin at a sentence boundary;

each subsequence must end at a sentence boundary;

each subsequence must begin at a clause boundary;

each subsequence must end at a clause boundary;

each subsequence must be contiguous;

or

each subsequence must include no more than a fourth predetermined number of contiguous sub-subsequences.

8. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive an input question;

generate a candidate answer from a corpus of information, wherein the candidate answer has a supporting passage from the corpus of information;

divide the input question into a first sequence of tokens;

divide the supporting passage into a second sequence of tokens;

identify a plurality of subsequences of tokens within the second sequence of tokens;

apply a paraphrase metric to compare the first sequence of tokens to each of the plurality of subsequences of tokens to generate a plurality of paraphrase metric scores; and

determine a confidence score for the candidate answer based on a highest paraphrase metric score within the plurality of paraphrase metric scores.

9. The computer program product of claim 8, wherein dividing the input question into the first sequence of tokens comprises:

dividing the input question into a plurality of sequences of question tokens;

identifying a given sequence of question tokens within the plurality of sequences of question tokens having important content of the input question as the first sequence of tokens.

10. The computer program product of claim 8, wherein the computer readable program further causes the computing device to match question tokens from the first set of tokens to passage tokens from the second set of tokens.

11. The computer program product of claim 10, wherein matching the question tokens to the passage tokens comprises:

identifying a focus of the input question; and

matching the focus of the input question to an occurrence of the candidate answer in the supporting passage.

12. The computer program product of claim 8, wherein identifying the plurality of subsequences of tokens within the second sequence of tokens comprises restricting the plurality of subsequences of tokens such that:

a number of tokens in each subsequence of tokens must be within a first predetermined number of a number of tokens in the first sequence of tokens;

each subsequence must contain at least a second predetermined number of tokens that match tokens in the first sequence of tokens;

each subsequence must contain at least a third predetermined number of non-stop-word tokens that match tokens in the first sequence of tokens;

each subsequence must contain the candidate answer;

each subsequence may not contain a sentence boundary;

each subsequence must begin at a sentence boundary;

each subsequence must end at a sentence boundary;

each subsequence must begin at a clause boundary;

each subsequence must end at a clause boundary;

each subsequence must be contiguous;

or

each subsequence must include no more than a fourth predetermined number of contiguous sub-subsequences.

13. The computer program product of claim 8, wherein the paraphrase metric is a machine translation metric.

14. The computer program product of claim 8, wherein the computer readable program further causes the computing device to:

apply a plurality of paraphrase metrics to produce a plurality of results; and

eliminate a given paraphrase metric that is determined to be ineffective based on the plurality of results from future application.

15. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive an input question;

generate a candidate answer from a corpus of information, wherein the candidate answer has a supporting passage from the corpus of information;

divide the input question into a first sequence of tokens;

divide the supporting passage into a second sequence of tokens;

identify a plurality of subsequences of tokens within the second sequence of tokens;

apply a paraphrase metric to compare the first sequence of tokens to each of the plurality of subsequences of tokens to generate a plurality of paraphrase metric scores; and

determine a confidence score for the candidate answer based on a highest paraphrase metric score within the plurality of paraphrase metric scores.

16. The apparatus of claim 15, wherein dividing the input question into the first sequence of tokens comprises:

dividing the input question into a plurality of sequences of question tokens;

identifying a given sequence of question tokens within the plurality of sequences of question tokens having important content of the input question as the first sequence of tokens.

17. The apparatus of claim 15, wherein the instructions further cause the processor to match question tokens from the first set of tokens to passage tokens from the second set of tokens.

18. The apparatus of claim 17, wherein matching the question tokens to the passage tokens comprises:

identifying a focus of the input question; and

matching the focus of the input question to an occurrence of the candidate answer in the supporting passage.

19. The apparatus of claim 15, wherein the paraphrase metric is a machine translation metric.

20. The apparatus of claim 15, wherein the instructions further cause the processor to:

applying a plurality of paraphrase metrics to produce a plurality of results; and

eliminating a given paraphrase metric that is determined to be ineffective based on the plurality of results from future application.