Providing answers to questions including assembling answers from multiple document segments

• US 10,318,529 B2
• Filed: 01/03/2018
• Issued: 06/11/2019
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Brown, Eric W., Chu-Carroll, Jennifer, Ferrucci, David A., Murdock, IV, James W.
• Primary Examiner(s): Nguyen, Thu N
• Application Number:
  US15/861,390
• Publication Number:
  US 20180144066A1
• Time in Patent Office: 524 Days
• Field of Search: None
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 5/00 Measuring for diagnostic purposes radiation diagnosis A61B6/00; diagnosis by ultrasonic, sonic or infrasonic waves A61B8/00; Identification of persons
• CPC Class Codes: G06F 16/2428 Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06N 5/027 Frames
• CPC Class Codes: G16H 10/20 for electronic clinical trials or questionnaires
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 15/00 ICT specially adapted for medical reports, e.g. generation or transmission thereof
• CPC Class Codes: G16H 40/20 for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: G16H 70/00 : ICT specially adapted for the handling or processing of medical references
• CPC Class Codes: Y02A 90/10 : Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

A method, system and computer program product for generating answers to questions. In one embodiment, the method comprises receiving an input query, identifying a plurality of candidate answers to the query; and for at least one of these candidate answers, identifying at least one proof of the answer. This proof includes a series of premises, and a multitude of documents are identified that include references to the premises. A set of these documents is selected that include references to all of the premises. This set of documents is used to generate one or more scores for the one of the candidate answers. A defined procedure is applied to the candidate answers to determine a ranking for the answers, and this includes using the one or more scores for the at least one of the candidate answers in the defined procedure to determine the ranking for this one candidate answer.

Claims

1. A computer-implemented method of searching through a database of documents for documents to generate a score for a candidate answer to an input query, the method comprising:

indexing, by a computer processor system, the documents in the database, including

for each of the documents, annotating, by the computer processor system, spans of text in said each document that refer to entities with entity types to form entity annotations, and annotating, by the computer processor system, spans of text in said each document that refer to facts with fact types to form relation annotations,

for each of the annotated spans of text that refers to one of the facts, linking, by the computer processor system, said one of the facts to said each annotation, and

recording, by the computer processor system, in an index the entities, the facts, the annotations that refer to said entities, and the annotations that refer to said facts;

receiving an input query at the computer processor system;

conducting, by the computer processor system, a search in a data source to identify a candidate answer to the input query;

determining at the computer processor system a set of the documents in the database using said index and theorem proving for scoring the candidate answer, including;

identifying a plurality of logical proofs of the candidate answer, each of the logical proofs including a conclusion and a sequence of premises that logically prove the conclusion, including

using the candidate answer as the conclusion of each of the logical proofs, and

wherein the sequence of the premises of each of the logical proofs forms a logical proof of the candidate answer;

for each of the logical proofs, identifying one or more documents in the database of documents that establish all the premises of the each of the logical proofs, including

for each of the premises of said each of the logical proofs, searching through the database of documents to identify one or more of the documents in the database of documents that include said each of the premises, wherein for each of the premises of each of the logical proofs, each of the documents identified that include said each of the premises includes a given number of the premises of said each of the logical proofs, and

selecting a plurality of the identified documents in the database of documents, based on the number of the premises of said each of the logical proofs included in each of the identified documents, to form a set of documents for said each of the logical proofs, wherein the set of documents for said each of the logical proofs includes all the premises of said each of the logical; and

selecting, based on specified criteria, one of the sets of documents for the logical proofs as the set of documents for scoring the candidate answer; and

using, by the computer processor system, the set of documents selected for scoring the candidate answer to generate a score for the candidate answer.

2. The method according to claim 1, wherein the indexing the documents in the database includes:

determining which of the entity annotations refer to the same fact; and

determining which of the relation annotations refer to the same fact.

3. The method according to claim 1, wherein the identifying a plurality of logical proofs of the candidate answer includes using the fact and entities in the index to identify each of the logical proofs of the candidate answer.

4. The method according to claim 1, wherein:

the identifying a plurality of logical proofs of the candidate answer includes identifying the premises of the logical proofs; and

the searching through the database of documents to identify one or more of the documents in the database of documents that includes said each of the premises includes using the index to identify the one or more of the documents that include said each of the premises.

5. The method according to claim 1, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof includes selecting for said set of documents a first document of said identified documents that, of the identified documents, has the most of the premises of said each logical proof.

6. The method according to claim 5, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof further includes, after selecting said first document for said set of documents, adding to said set of documents a second document of said identified documents that, of the identified documents, has the second most of the premises of said each logical proof.

7. The method according to claim 6, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof further includes adding to said set of documents one or more additional documents of the identified documents until each of the premises of said each logical proof is included in at least one of the documents of said set of documents.

8. The method according to claim 1, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof includes identifying a number of the premises of said each logical proof in each of the identified documents.

9. The method according to claim 1, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof includes using each of the plurality of logical proofs of the candidate answer to form a respective one of the plurality of sets of documents.

10. The method according to claim 1, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof includes using a defined algorithm to identify a quasi-minimal number of the identified documents to form the set of documents.

11. A computer system for searching through a database of documents for documents to generate a score for a candidate answer to an input query, the computer system comprising:

a memory for storing data; and

one or more processor units operatively connected to the memory to transmit data to and to receive data from the memory, and configured for;

indexing the documents in the database, including

for each of the documents, annotating spans of text in said each document that refer to entities with entity types to form entity annotations, and annotating, by the computer processor system, spans of text in said each document that refer to facts with fact types to form relation annotations,

for each of the annotated spans of text that refers to one of the facts, linking, by the computer processor system, said one of the facts to said each annotation, and

recording in an index the entities, the facts, the annotations that refer to said entities, and the annotations that refer to said facts;

receiving an input query;

conducting a search in a data source to identify a candidate answer to the input query;

determining a set of the documents in the database using said index and theorem proving for scoring the candidate answer, including;

identifying a plurality of logical proofs of the candidate answer, each of the logical proofs including a conclusion and a sequence of premises that logically prove the conclusion, including

using the candidate answer as the conclusion of each of the logical proofs, and

wherein the sequence of the premises of each of the logical proofs forms a logical proof of the candidate answer;

for each of the logical proofs, identifying one or more documents in the database of documents that establish all the premises of the each of the logical proofs, including

for each of the premises of said each of the logical proofs, searching through the database of documents to identify one or more of the documents in the database of documents that include said each of the premises, wherein for each of the premises of each of the logical proofs, each of the documents identified that include said each of the premises includes a given number of the premises of said each of the logical proofs, and

selecting a plurality of the identified documents in the database of documents, based on the number of the premises of said each of the logical proofs included in each of the identified documents, to form a set of documents for said each of the logical proofs, wherein the set of documents for said each of the logical proofs includes all the premises of said each of the logical; and

selecting, based on specified criteria, one of the sets of documents for the logical proofs as the set of documents for scoring the candidate answer; and

using the set of documents selected for scoring the candidate answer to generate a score for the candidate answer.

12. The computer system according to claim 11, wherein the indexing the documents in the database includes:

determining which of the entity annotations refer to the same fact; and

determining which of the relation annotations refer to the same fact.

13. The computer system according to claim 11, wherein the identifying a plurality of logical proofs of the candidate answer includes using the fact and entities in the index to identify each of the logical proofs of the candidate answer.

14. The computer system according to claim 11, wherein:

the identifying a plurality of logical proofs of the candidate answer includes identifying the premises of the logical proofs; and

the searching through the database of documents to identify one or more of the documents in the database of documents that includes said each of the premises includes using the index to identify the one or more of the documents that include said each of the premises.

15. The computer system according to claim 11, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof includes selecting for said set of documents a first document of said identified documents that, of the identified documents, has the most of the premises of said each logical proof.

16. An article of manufacture comprising:

at least one tangible computer readable medium having computer readable program code logic to generate answers to questions, the computer readable program code logic, when executing, performing the following;

indexing the documents in the database, including

for each of the documents, annotating spans of text in said each document that refer to entities with entity types to form entity annotations, and annotating, by the computer processor system, spans of text in said each document that refer to facts with fact types to form relation annotations,

for each of the annotated spans of text that refers to one of the facts, linking, by the computer processor system, said one of the facts to said each annotation, and

recording in an index, the entities, the facts, the annotations that refer to said entities, and the annotations that refer to said facts;

receiving an input query;

conducting a search in a data source to identify a candidate answer to the input query;

determining a set of the documents in the database using said index and theorem proving for scoring the candidate answer, including;

identifying a plurality of logical proofs of the candidate answer, each of the logical proofs including a conclusion and a sequence of premises that logically prove the conclusion, including

using the candidate answer as the conclusion of each of the logical proofs, and

wherein the sequence of the premises of each of the logical proofs forms a logical proof of the candidate answer;

for each of the logical proofs, identifying one or more documents in the database of documents that establish all the premises of the each of the logical proofs, including

for each of the premises of said each of the logical proofs, searching through the database of documents to identify one or more of the documents in the database of documents that include said each of the premises, wherein for each of the premises of each of the logical proofs, each of the documents identified that include said each of the premises includes a given number of the premises of said each of the logical proofs, and

selecting a plurality of the identified documents in the database of documents, based on the number of the premises of said each of the logical proofs included in each of the identified documents, to form a set of documents for said each of the logical proofs, wherein the set of documents for said each of the logical proofs includes all the premises of said each of the logical; and

selecting, based on specified criteria, one of the sets of documents for the logical proofs as the set of documents for scoring the candidate answer; and

using the set of documents selected for scoring the candidate answer to generate a score for the candidate answer.

17. The article of manufacture according to claim 16, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof further includes, after selecting said first document for said set of documents, adding to said set of documents a second document of said identified documents that, of the identified documents, has the second most of the premises of said each logical proof.

18. The article of manufacture according to claim 17, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof further includes adding to said set of documents one or more additional documents of the identified documents until each of the premises of said each logical proof is included in at least one of the documents of said set of documents.

19. The article of manufacture according to claim 16, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof includes identifying a number of the premises of said each logical proof in each of the identified documents.

20. The article of manufacture according to claim 16, wherein the selecting the plurality of the identified documents to form the set of documents for said each logical proof includes using each of the plurality of logical proofs of the candidate answer to form a respective one of the plurality of sets of documents.