Providing computable guidance to relevant evidence in question-answering systems

• US 9,704,099 B2
• Filed: 07/25/2016
• Issued: 07/11/2017
• Est. Priority Date: 02/10/2010
• Status: Active Grant

• Current Assignee: Solventum Intellectual Properties Company (Solventum Corp.)
• Original Assignee: MModal IP LLC (3M Company)
• Inventors: Koll, Detlef, Polzin, Thomas
• Primary Examiner(s): Tran, Anh Tai
• Application Number:
  US15/218,510
• Publication Number:
  US 20160335554A1
• Time in Patent Office: 351 Days
• Field of Search: None
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06N 5/048 Fuzzy inferencing
• CPC Class Codes: G06Q 30/04 Billing or invoicing
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A computer-based system includes a computer-processable definition of a region in a data set. The system identifies a region of the data set based on the definition of the region. The system provides output to a user representing a question and the identified region of the data set. The system may also automatically generate an answer to the question based on the question and the data set, and provide output to the user representing the answer. The system may generate the answer based on a subset of the data set, and provide output to the user representing the subset of the data set. The user may provide feedback on the first answer to the system, which the system may use to improve subsequent answers to the same and other questions, and to disable the system'"'"'s automatic question-answering function in response to disagreement between the user and the system.

Claims

1. A method performed by at least one computer processor, the method comprising:

(a) selecting, by the at least one computer processor, a first question instance including first text and a first region definition designed to identify a first region of a data set likely to contain information that may be used to provide an answer to the question represented by the first text, the first region being aligned with a tagged element in the data set;

(b) automatically identifying, by the at least one computer processor, before (c) and (e), the first region of the data set, based on the first region definition;

(c) providing output, by the at least one computer processor, to a user representing the first question instance;

(d) generating, by the at least one computer processor, an answer to the first question instance based on the first region of the data set before (e);

(e) receiving, by the at least one computer processor, from the user, feedback before providing to the user output representing the generated answer;

(f) determining whether the received user feedback indicates the user relied upon the first region of the data set identified in the first region definition;

(g) determining that a frequency of reliance upon the first region of the data set falls below a predetermined threshold; and

(h) modifying the first region definition to exclude the first region of the data set from the first region definition.

2. The method of claim 1, wherein (d) further comprises:

determining that an automatic answering function is disabled; and

determining not to provide to the user the generated answer to the first question instance, based on the determination that the automatic answering function is disabled.

3. The method of claim 1, further comprising:

identifying a degree of agreement between the answer feedback received from the user and the generated answer;

determining that the degree of agreement exceeds a predetermined threshold; and

determining to enable an automatic answering function based upon the determination that the degree of agreement exceeds the predetermined threshold.

4. The method of claim 1, wherein (e) further comprises receiving input providing a verification status of the generated answer.

5. The method of claim 1, further comprising:

(i) accessing a second question instance including second text and a second region definition designed to identify a second region of the data set likely to contain information that may be used to provide an answer to a second question represented by the second text, the second region being aligned with a second tagged element in the data set;

(j) providing output to the user representing the second question instance; and

(k) providing output to the user, the output representing the second region of the data set before receiving, from the user, an answer to the second question instance.

6. A non-transitory computer-readable medium comprising computer-readable instructions tangibly stored on the non-transitory computer-readable medium, wherein the instructions are executable by at least one computer processor to execute a method for use with a system, the non-transitory computer-readable medium comprising:

instructions to select, by the at least one computer processor, a first question instance including first text and a first region definition designed to identify a first region of a data set likely to contain information that may be used to provide an answer to the question represented by the first text, the first region being aligned with a tagged element in the data set;

instructions to automatically identify, by the at least one computer processor, before providing output to a user representing the first question instance and before receiving user feedback on the first question instance, the first region of the data set, based on the first region definition;

instructions to provide output, by the at least one computer processor, to a user representing the first question instance;

instructions to generate, by the at least one computer processor, an answer to the first question instance based on the first region of the data set before receiving user feedback on the first question instance;

instructions to receive, by the at least one computer processor, from the user, feedback before providing to the user output representing the generated answer;

instructions to determine whether the received user feedback indicates the user relied upon the first region of the data set identified in the first region definition;

instructions to determine that a frequency of reliance upon the first region of the data set falls below a predetermined threshold; and

instructions to modify the first region definition to exclude the first region of the data set from the first region definition.

7. The non-transitory computer-readable medium of claim 6, wherein (d) further comprises:

determining that an automatic answering function is disabled; and

determining not to provide to the user the generated answer to the first question instance, based on the determination that the automatic answering function is disabled.

8. The non-transitory computer-readable medium of claim 6, wherein the method further comprises:

identifying a degree of agreement between the answer feedback received from the user and the generated answer;

determining that the degree of agreement exceeds a predetermined threshold; and

determining to enable an automatic answering function based upon the determination that the degree of agreement exceeds the predetermined threshold.

9. The non-transitory computer-readable medium of claim 6, wherein (e) further comprises receiving input providing a verification status of the generated answer.

10. The non-transitory computer-readable medium of claim 6, wherein the method further comprises:

(i) accessing a second question instance including second text and a second region definition designed to identify a second region of the data set likely to contain information that may be used to provide an answer to a second question represented by the second text, the second region being aligned with a second tagged element in the data set;

(j) providing output to the user representing the second question instance; and

(k) providing output to the user, the output representing the second region of the data set before receiving, from the user, an answer to the second question instance.