Providing computable guidance to relevant evidence in question-answering systems

• US 9,082,310 B2
• Filed: 02/10/2011
• Issued: 07/14/2015
• Est. Priority Date: 02/10/2010
• Status: Active Grant

• Current Assignee: Solventum Intellectual Properties Company (Solventum Corp.)
• Original Assignee: MModal IP LLC (3M Company)
• Inventors: Koll, Detlef, Polzin, Thomas
• Primary Examiner(s): TRAN, ANHTAI V
• Application Number:
  US13/025,051
• Publication Number:
  US 20120041950A1
• Time in Patent Office: 1,615 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06N 5/048 Fuzzy inferencing
• CPC Class Codes: G06Q 30/04 Billing or invoicing
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

A computer-based system includes a computer-processable definition of a region in a data set. The system identifies a region of the data set based on the definition of the region. The system provides output to a user representing a question and the identified region of the data set. The system may also automatically generate an answer to the question based on the question and the data set, and provide output to the user representing the answer. The system may generate the answer based on a subset of the data set, and provide output to the user representing the subset of the data set. The user may provide feedback on the first answer to the system, which the system may use to improve subsequent answers to the same and other questions, and to disable the system'"'"'s automatic question-answering function in response to disagreement between the user and the system.

Claims

1. A method performed by at least one computer processor, the method comprising:

(a) selecting, by the at least one computer processor, a first question instance including first text and a first region definition designed to identify a region of a data set likely to contain information that may be used to provide an answer to the question represented by the first text, the region definition identifying a region aligned with a tagged element in the data set;

(b) automatically identifying, by the at least one computer processor, before (c) and (d), the first region of the data set, based on the first region definition;

(c) providing output, by the at least one computer processor, to a user representing the first question instance; and

(d) providing output, by the at least one computer processor, to the user, the output representing the first region of the data set before receiving an answer to the first question instance.

2. The method of claim 1, wherein (b) further comprises:

automatically identifying, by the at least one computer processor, a first document type specified by the first region definition; and

automatically identifying, by the at least one computer processor, a document of the first document type in the data set.

3. The method of claim 1, wherein (b) further comprises:

identifying a section type specified by the first region definition; and

identifying a section of the section type in the data set.

4. The method of claim 1, wherein (b) further comprises:

identifying a keyword specified by the first region definition; and

identifying text in the data set matching the keyword.

5. The method of claim 1, wherein (b) further comprises:

identifying a keyword specified by the first region definition; and

identifying text near the keyword in the data set.

6. The method of claim 1, wherein (b) further comprises:

identifying a relationship to a concept, the relationship specified by the first region definition; and

identifying first coded data in the data set having the specified relationship to second coded data representing the concept in the data set.

7. The method of claim 6, wherein the relationship comprises an “

is a”

relationship.

8. The method of claim 6, wherein the relationship comprises a “

has a”

relationship.

9. The method of claim 1, wherein (b) comprises identifying a plurality of portions of the data set based on the first region definition, wherein the first region includes the plurality of portions.

10. The method of claim 9, wherein (d) comprises providing output to the user representing the plurality of portions of the data set.

11. The method of claim 10, wherein (d) comprises providing output to the user simultaneously representing the plurality of portions of the data set.

12. The method of claim 10, wherein (d) comprises:

(d)(1) providing output to the user representing a first one of the plurality of portions of the data set;

(d)(2) receiving input from the user; and

(d)(3) in response to the input from the user, providing output to the user representing a second one of the plurality of portions of the data set.

13. The method of claim 9:

wherein the data set includes a plurality of documents;

wherein the plurality of portions includes a first portion of a first one of the plurality of documents; and

wherein the plurality of portions includes a second portion of a second one of the plurality of documents.

14. The method of claim 1, wherein (d) comprises providing output to the user representing the entirety of the first region of the data set.

15. The method of claim 1, wherein (d) comprises providing output to the user representing a subset of the first region of the data set.

16. The method of claim 1, wherein (d) comprises providing output to the user representing a summary of the first region of the data set.

17. The method of claim 1, further comprising:

(e) automatically generating a first answer to the first question instance based on the first region definition in the first question instance and the data set.

18. The method of claim 17, further comprising:

(f) providing output to the user representing the first answer.

19. The method of claim 17:

wherein (e) comprises automatically generating the first answer to the first question instance based on a particular subset of the data set; and

wherein the method further comprises;

(f) providing output to the user representing the particular subset of the data set.

20. The method of claim 17, wherein the first answer includes answer content and a confidence score indicating a likelihood that the answer content accurately answers the first question instance.

21. The method of claim 20, wherein (e) comprises automatically generating the confidence score based on a degree of agreement between the answer content and at least one previous answer to the first question instance received from a human.

22. The method of claim 17, wherein the first answer includes at least one pointer to at least one portion of the data set relied upon in (e) to generate the first answer.

23. The method of claim 17, further comprising:

(f) receiving input from the user representing feedback on the first answer; and

(g) storing, in a computer-readable medium, a record of the input.

24. The method of claim 23, wherein the input from the user indicates a verification status of the first answer.

25. The method of claim 24 further comprising:

(h) accessing a second question instance including second text and a second region definition; and

(i) determining whether to generate an answer to the second question based on the verification status of the first answer.

26. The method of claim 23, wherein the input from the user represents a second answer to the first question instance.

27. The method of claim 26 further comprising:

(h) accessing a second question instance including second text and a second region definition; and

(i) generating an answer to the second question instance only if the first answer is in agreement with the second answer.

28. The method of claim 17, further comprising:

(f) generating a confidence score indicating a likelihood that the answer content accurately answers the first question instance;

(g) determining, based on the confidence score, whether to prompt the user for input representing feedback on the first answer; and

(h) prompting the user for input representing feedback on the first answer only if it is determined that the user should be prompted for the input.

29. The method of claim 17, wherein the first question instance further comprises a first question definition, and wherein (e) comprises processing the first question definition to automatically generate the first answer.

30. The method of claim 29, wherein (e) comprises applying the first question definition as a query to the data set to select a subset of the data set as the first answer.

31. The method of claim 29, wherein the first question definition is independent of the first text.

32. The method of claim 1, further comprising:

(e) attempting to generate a first answer to the first question instance based on the first question instance and the data set;

(f) if the attempt to generate the first answer failed, then providing output to the user indicating that the attempt to generate the first answer failed.

33. The method of claim 1, wherein the first region definition is independent of the first text.

81. The method of claim 17, wherein the first region definition specifies a document type.

82. The method of claim 17, wherein the first region definition specifies a keyword.

83. The method of claim 17, wherein the first region definition specifies a relationship to a concept.

84. The method of claim 17, wherein the first region definition specifies a section type.

34. A non-transitory computer-readable medium comprising computer-readable instructions tangibly stored on the non-transitory computer-readable medium, wherein the instructions are executable by at least one computer processor to execute a method for use with a system, the non-transitory computer-readable medium comprising:

instructions to select, by the at least one computer processor, a first question instance including first text and a first region definition designed to identify a region of a data set likely to contain information that may be used to provide an answer to the question represented by the first text, the region definition identifying a region aligned with a tagged element in the data set;

instructions to automatically identify, by the at least one computer processor, before providing output to a user representing the first question instance and before providing output to the user representing the first region of the data set, the first region of the data set, based on the first region definition;

instructions to provide output, by the at least one computer processor, to a user representing the first question instance; and

instructions to provide output, by the at least one computer processor, to the user, the output representing the first region of the data set before receiving an answer to the first question instance.

35. The non-transitory computer-readable medium of claim 34, wherein the instructions to identify further comprise:

instructions to automatically identify, by the at least one computer processor, a first document type specified by the first region definition; and

instructions to automatically identify, by the at least one computer processor, a document of the first document type in the data set.

36. The non-transitory computer-readable medium of claim 34, wherein the instructions to identify further comprise:

instructions to identify a section type specified by the first region definition; and

instructions to identify a section of the section type in the data set.

37. The non-transitory computer-readable medium of claim 34, wherein the instructions to identify further comprise:

instructions to identify a keyword specified by the first region definition; and

instructions to identify text in the data set matching the keyword.

38. The non-transitory computer-readable medium of claim 34, wherein the instructions to identify further comprise:

instructions to identify a keyword specified by the first region definition; and

instructions to identify text near the keyword in the data set.

39. The non-transitory computer-readable medium of claim 34, wherein the instructions to identify further comprise:

instructions to identify a relationship to a concept, the relationship specified by the first region definition; and

instructions to identify first coded data in the data set having the specified relationship to second coded data representing the concept in the data set.

40. The non-transitory computer-readable medium of claim 39, wherein the relationship comprises an “

is a”

relationship.

41. The non-transitory computer-readable medium of claim 39, wherein the relationship comprises a “

has a”

relationship.

42. The non-transitory computer-readable medium of claim 34, wherein the instructions to identify) comprise instructions to identify a plurality of portions of the data set based on the first region definition, wherein the first region includes the plurality of portions.

43. The non-transitory computer-readable medium of claim 42, wherein the instructions to provide output representing the first region comprises providing output to the user representing the plurality of portions of the data set.

44. The non-transitory computer-readable medium of claim 43, wherein the instructions to provide output representing the first region comprises providing output to the user simultaneously representing the plurality of portions of the data set.

45. The non-transitory computer-readable medium of claim 43, wherein the instructions to provide output representing the first region) comprises:

instructions to provide output to the user representing a first one of the plurality of portions of the data set;

instructions to receive input from the user; and

instructions to provide, in response to the input from the user, output to the user representing a second one of the plurality of portions of the data set.

46. The non-transitory computer-readable medium of claim 42:

wherein the data set includes a plurality of documents;

wherein the plurality of portions includes a first portion of a first one of the plurality of documents; and

wherein the plurality of portions includes a second portion of a second one of the plurality of documents.

47. The non-transitory computer-readable medium of claim 34, wherein the instructions to provide output representing the first region comprises instructions to provide output the user representing the entirety of the first region of the data set.

48. The non-transitory computer-readable medium of claim 34, wherein the instructions to provide output representing the first region comprises instructions to provide output to the user representing a subset of the first region of the data set.

49. The non-transitory computer-readable medium of claim 34, wherein the instructions to provide output representing the first region comprises instructions to provide output to the user representing a summary of the first region of the data set.

50. The non-transitory computer-readable medium of claim 34, wherein the method further comprises:

instructions to automatically generate a first answer to the first question instance based on the first question instance and the data set.

51. The non-transitory computer-readable medium of claim 50, wherein the method further comprises:

instructions to provide output to the user representing the first answer.

52. The non-transitory computer-readable medium of claim 50:

wherein the instructions to automatically generate a first answer comprise instructions to automatically generate the first answer to the first question instance based on a particular subset of the data set; and

wherein the computer-readable medium further comprises;

instructions to provide output to the user representing the particular subset of the data set.

53. The non-transitory computer-readable medium of claim 50, wherein the first answer includes answer content and a confidence score indicating a likelihood that the answer content accurately answers the first question instance.

54. The non-transitory computer-readable medium of claim 53, wherein the instructions to automatically generate a first answer comprises instructions to automatically generate the confidence score based on a degree of agreement between the answer content and at least one previous answer to the first question instance received from a human.

55. The non-transitory computer-readable medium of claim 50, wherein the first answer includes at least one pointer to at least one portion of the data set relied upon in the instructions to automatically generate a first answer to generate the first answer.

56. The non-transitory computer-readable medium of claim 50, further comprising:

instructions to receive input from the user representing feedback on the first answer; and

instructions to store, in a computer-readable medium, a record of the input.

57. The non-transitory computer-readable medium of claim 56, wherein the input from the user indicates a verification status of the first answer.

58. The non-transitory computer-readable medium of claim 57:

wherein the system further comprises a second question instance including second text and a second region definition; and

wherein the computer-readable medium further comprises;

instructions to determine whether to generate an answer to the second question based on the verification status of the first answer.

59. The non-transitory computer-readable medium of claim 56, wherein the input from the user represents a second answer to the first question instance.

60. The non-transitory computer-readable medium of claim 59:

instructions to generate an answer to a second question instance only if the first answer is in agreement with the second answer, the second question instance including second text and a second region definition.

61. The non-transitory computer-readable medium of claim 50, further comprising:

instructions to generate a confidence score indicating a likelihood that the answer content accurately answers the first question instance;

instructions to determine, based on the confidence score, whether to prompt the user for input representing feedback on the first answer; and

instructions to prompt the user for input representing feedback on the first answer only if it is determined that the user should be prompted for the input.

62. The non-transitory computer-readable medium of claim 50, wherein the first question instance further comprises a first question definition, and wherein the instructions to automatically generate a first answer comprises instructions to process the first question definition to automatically generate the first answer.

63. The non-transitory computer-readable medium of claim 62, wherein the instructions to automatically generate a first answer comprises instructions to apply the first question definition as a query to the data set to select a subset of the data set as the first answer.

64. The non-transitory computer-readable medium of claim 62, wherein the first question definition is independent of the first text.

65. The non-transitory computer-readable medium of claim 34 further comprising:

instructions to attempt to generate a first answer to the first question instance based on the first question instance and the data set;

instructions to provide, if the attempt to generate the first answer failed, output to the user indicating that the attempt to generate the first answer failed.

66. The non-transitory computer-readable medium of claim 34, wherein the first region definition is independent of the first text.

67. A computer-implemented method for use with a system, the method comprising:

(a) selecting a first question instance including first text and a first region definition designed to identify a region of a data set likely to contain information that may be used to provide an answer to the question represented by the first text, the region definition identifying a region aligned with a tagged element in the data set;

(b) before (c), for each of a plurality of instances Q of the first question;

(b-1) identifying a region R_Q of the data set based on the first region definition;

(b-2) receiving a computer-generated answer C_Q in response to instance Q of the first question;

(b-3) providing output to a user representing instance Q of the first question;

(b-4) providing output to the user representing the region R_Q of the data set, before (b-5);

(b-5) receiving input H_Q from the user representing human-generated answer to instance Q of the first question; and

(c) identifying a degree of agreement between the computer-generated answers C_Q and the human-generated answers H_Q.

68. The method of claim 67, further comprising:

(d) determining, based on the degree of agreement, whether to automatically generate an answer to an additional instance of the first question; and

(e) automatically generating an answer to the additional question only if it is determined in (d) that such an answer should be automatically generated.

69. The method of claim 67, further comprising:

(d) receiving an additional instance of the first question;

(e) determining whether the degree of agreement satisfies a predetermined threshold; and

(f) if the degree of agreement does not satisfy the predetermined threshold, then providing output in response to the additional instance of the first question indicating that no automated answer to the additional instance of the first question has been generated.

70. The method of claim 67, further comprising:

(d) automatically generating an additional answer to an additional instance of the first question; and

(e) identifying a degree of agreement between the additional answer and the human-generated answers H_Q.

71. The method of claim 67, wherein the human-generated answers H_Q includes a plurality of human-generated answers provided by a first set of human answerers and a plurality of human-generated answers provided by a second set of at least one human answerer, and wherein the method further comprises:

(d) calculating a first degree of agreement between the plurality of human-generated answers provided by the first set of human answerers and the computer-generated answers C_Q;

(e) calculating a second degree of agreement between the plurality of human-generated answers provided by the second set of at least one human answerer and the plurality of computer-generated answers C_Q; and

(f) calculating a difference between the first degree of agreement and the second degree of agreement.

72. The method of claim 67, further comprising:

(d) determining that the human-generated answers H_Q include an indication of an additional region R_H in the data set relied upon to generate at least one of the human-generated answers H_Q, wherein the region R_H is not one of the regions R_Q; and

(e) modifying the first region definition to include a definition of the additional region R_H.

73. The method of claim 67, wherein (b) further comprises:

(b-6) providing output to the user representing the computer-generated answer C_Q.

74. A non-transitory computer-readable medium comprising computer-readable instructions tangibly stored on the non-transitory computer-readable medium, wherein the instructions are executable by at least one computer processor to execute a method for use with a system, the non-transitory computer-readable medium comprising:

(a) instructions to select a first question instance including first text and a first region definition designed to identify a region of a data set likely to contain information that may be used to provide an answer to the question represented by the first text, the region definition identifying a region aligned with a tagged element in the data set;

(b) before (c), for each of a plurality of instances Q of the first question;

(b-1) instructions to identify a region R_Q of the data set based on the first region definition;

(b-2) instructions to receive a computer-generated answer C_Q in response to instance Q of the first question;

(b-3) instructions to provide output to a user representing instance Q of the first question;

(b-4) instructions to provide output to the user representing the region R_Q of the data set;

(b-5) instructions to receive input H_Q from the user representing human-generated answer to instance Q of the first question; and

(c) instructions to identify a degree of agreement between the computer-generated answers C_Q and the human-generated answers H_Q.

75. The non-transitory computer-readable medium of claim 74, wherein the non-transitory computer-readable medium further comprises:

(d) instructions to determine, based on the degree of agreement, whether to automatically generate an answer to an additional instance of the first question; and

(e) instructions to automatically generate an answer to the additional question only if it is determined in (d) that such an answer should be automatically generated.

76. The non-transitory computer-readable medium of claim 74, wherein the non-transitory computer-readable medium further comprises:

(d) instructions to receive an additional instance of the first question;

(e) instructions to determine whether the degree of agreement satisfies a predetermined threshold; and

(f) instructions to provide, if the degree of agreement does not satisfy the predetermined threshold, output in response to the additional instance of the first question indicating that no automated answer to the additional instance of the first question has been generated.

77. The non-transitory computer-readable medium of claim 74, wherein the non-transitory computer-readable medium further comprises:

(d) instructions to automatically generate an additional answer to an additional instance of the first question; and

(e) instructions to identify a degree of agreement between the additional answer and the human-generated answers H_Q.

78. The non-transitory computer-readable medium of claim 74, wherein the human-generated answers H_Q includes a plurality of human-generated answers provided by a first set of human answerers and a plurality of human-generated answers provided by a second set of at least one human answerer, and wherein the non-transitory computer-readable medium further comprises:

(d) instructions to calculate a first degree of agreement between the plurality of human-generated answers provided by the first set of human answerers and the computer-generated answers C_Q;

(e) instructions to calculate a second degree of agreement between the plurality of human-generated answers provided by the second set of at least one human answerer and the plurality of computer-generated answers C_Q; and

(f) instructions to calculate a difference between the first degree of agreement and the second degree of agreement.

79. The non-transitory computer-readable medium of claim 74, wherein the non-transitory computer-readable medium further comprises:

(d) instructions to determine that the human-generated answers H_Q include an indication of an additional region R_H in the data set relied upon to generate at least one of the human-generated answers H_Q, wherein the region R_H is not one of the regions R_Q; and

(e) instructions to modify the first region definition to include a definition of the additional region R_H.

80. The non-transitory computer-readable medium of claim 74, wherein (b) further comprises:

(b-6) instructions to provide output to the user representing the computer-generated answer C_Q.