Grading sources and managing evidence for intelligence analysis

• US 10,318,870 B2
• Filed: 11/19/2014
• Issued: 06/11/2019
• Est. Priority Date: 11/19/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Adderly, Darryl M., Allen, Corville O., Tucker, Robert K.
• Primary Examiner(s): Waldron, Scott A.
• Assistant Examiner(s): Seck, Ababacar
• Application Number:
  US14/547,246
• Publication Number:
  US 20160140446A1
• Time in Patent Office: 1,665 Days
• Field of Search: None
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Mechanisms for evaluating an evidential statement in a corpus of evidence are provided. An evidential statement is received for determining a level of confidence in a hypothetical ontological link of an ontology. A source of the evidential statement is identified and a grading of the source of the evidential statement is determined based on a source grading measurement value indicative of a degree of reliability and credibility of the source. An indication of trustworthiness of the evidential statement is generated based on the source grading measurement value. A representation of the indication of trustworthiness of the evidential statement is output in association with the evidential statement.

Claims

1. A method, in a data processing system comprising a processor and a memory, for evaluating an evidential statement in a corpus of evidence, comprising:

configuring the data processing system to execute a corpora management engine;

receiving, by the corpora management engine executing on the data processing system, an evidential statement, from a corpus of electronically stored documents, for determining a level of confidence in a hypothetical ontological link of an ontology data structure;

identifying, by the corpora management engine executing on the data processing system, a source of the evidential statement based on a source profile directory data structure associated with the corpus of electronically stored documents;

determining, by the corpora management engine executing on the data processing system, a grading of the source of the evidential statement based on a source grading measurement value indicative of a degree of reliability and credibility of the source calculated based on information in the source profile directory data structure corresponding to the source;

generating, by the corpora management engine executing on the data processing system, an indication of trustworthiness of the evidential statement based on the source grading measurement value; and

managing, by the corpora management engine executing on the data processing system, a corpus of evidence data based on the grading of the source of the evidential statement, wherein managing the corpus comprises comparing the source grading measurement value to one or more predetermined threshold values and performing a corpus management operation based on results of the comparison, and wherein, in response to the comparison indicating that the source grading measurement is below a first threshold, removing evidence data associated with the source from the corpus.

2. The method of claim 1, wherein grading the source of the evidential statement comprises generating a weight value to be applied to confidence scores calculated for answers to questions generated based on evidential statements in the corpus associated with the source.

3. The method of claim 1, further comprising managing a corpus of evidence data based on the grading of the source of the evidential statement, wherein managing the corpus comprises one of maintaining an amount of evidence data from the source in the corpus, increasing the amount of evidence data from the source that is in the corpus, or decreasing the amount of evidence data from the source in the corpus.

4. The method of claim 3, wherein the corpus is managed to temporarily modify the corpus being used to evaluate candidate answers to a question set used to evaluate a hypothetical ontological link of an ontology.

5. The method of claim 1, wherein, in response to the comparison indicating that the source grading measurement value is above a second threshold, ingesting additional evidence data from the source into the corpus.

6. The method of claim 1, wherein, in response to the comparison indicating that the source grading measurement value is above a second threshold, sending a request to a user specifying the source and the source grading measurement value and a recommendation to obtain additional evidence data from the source.

7. The method of claim 1, wherein determining a grading of the source comprises evaluating current and historical characteristics of the source, wherein the historical characteristics of the source comprises information indicative of results from previous utilizations of evidence data from the source when evaluating answers to previously processed questions.

8. The method of claim 7, wherein the current and historical characteristics are maintained in a source profile data structure of a source directory associated with the data processing system, and wherein the source profile data structure stores data indicating a number of previous uses of evidence data from the source and a number of previous positive uses of the evidence data from the source.

9. The method of claim 1, further comprising:

outputting, by the corpora management engine executing on the data processing system, a representation of the indication of trustworthiness of the evidential statement in association with the evidential statement to a link scoring and analyzer engine executing on the data processing system;

generating, by the link scoring and analyzer engine, a score for the hypothetical ontological link at least based on the indication of trustworthiness of the evidential statement; and

modifying, by an ontology modification engine executing on the data processing system, the ontology data structure based on the score for the hypothetical ontological link.

10. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to:

configure the data processing system to execute a corpora management engine;

receive, by the corpora management engine executing on the data processing system, an evidential statement, from a corpus of electronically stored documents, for determining a level of confidence in a hypothetical ontological link of an ontology data structure;

identify, by the corpora management engine executing on the data processing system, a source of the evidential statement based on a source profile directory data structure associated with the corpus of electronically stored documents;

determine, by the corpora management engine executing on the data processing system, a grading of the source of the evidential statement based on a source grading measurement value indicative of a degree of reliability and credibility of the source calculated based on information in the source profile directory data structure corresponding to the source;

generate, by the corpora management engine executing on the data processing system, an indication of trustworthiness of the evidential statement based on the source grading measurement value; and

manage, by the corpora management engine executing on the data processing system, a corpus of evidence data based on the grading of the source of the evidential statement, wherein managing the corpus comprises comparing the source grading measurement value to one or more predetermined threshold values and performing a corpus management operation based on results of the comparison, and wherein, in response to the comparison indicating that the source grading measurement is below a first threshold, removing evidence data associated with the source from the corpus.

11. The computer program product of claim 10, wherein grading the source of the evidential statement comprises generating a weight value to be applied to confidence scores calculated for answers to questions generated based on evidential statements in the corpus associated with the source.

12. The computer program product of claim 10, further comprising managing a corpus of evidence data based on the grading of the source of the evidential statement, wherein managing the corpus comprises one of maintaining an amount of evidence data from the source in the corpus, increasing the amount of evidence data from the source that is in the corpus, or decreasing the amount of evidence data from the source in the corpus.

13. The computer program product of claim 12, wherein the corpus is managed to temporarily modify the corpus being used to evaluate candidate answers to a question set used to evaluate a hypothetical ontological link of an ontology.

14. The computer program product of claim 10, wherein, in response to the comparison indicating that the source grading measurement value is above a second threshold, ingesting additional evidence data from the source into the corpus.

15. The computer program product of claim 10, wherein, in response to the comparison indicating that the source grading measurement value is above a second threshold, sending a request to a user specifying the source and the source grading measurement value and a recommendation to obtain additional evidence data from the source.

16. The computer program product of claim 10, wherein determining a grading of the source comprises evaluating current and historical characteristics of the source, wherein the historical characteristics of the source comprises information indicative of results from previous utilizations of evidence data from the source when evaluating answers to previously processed questions, and wherein the current and historical characteristics are maintained in a source profile data structure of a source directory associated with the data processing system, and wherein the source profile data structure stores data indicating a number of previous uses of evidence data from the source and a number of previous positive uses of the evidence data from the source.

17. The computer program product of claim 10, wherein the computer readable program further causes the data processing system to:

output, by the corpora management engine executing on the data processing system, a representation of the indication of trustworthiness of the evidential statement in association with the evidential statement to a link scoring and analyzer engine executing on the data processing system;

generate, by the link scoring and analyzer engine, a score for the hypothetical ontological link at least based on the indication of trustworthiness of the evidential statement; and

modify, by an ontology modification engine executing on the data processing system, the ontology data structure based on the score for the hypothetical ontological link.

18. An apparatus comprising:

at least one processor; and

at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to;

configure the at least one processor to execute a corpora management engine;

receive, by the corpora management engine executing on the at least one processor, an evidential statement, from a corpus of electronically stored documents, for determining a level of confidence in a hypothetical ontological link of an ontology data structure;

identify, by the corpora management engine executing on the at least one processor, a source of the evidential statement based on a source profile directory data structure associated with the corpus of electronically stored documents;

determine, by the corpora management engine executing on the at least one processor, a grading of the source of the evidential statement based on a source grading measurement value indicative of a degree of reliability and credibility of the source calculated based on information in the source profile directory data structure corresponding to the source;

generate, by the corpora management engine executing on the at least one processor, an indication of trustworthiness of the evidential statement based on the source grading measurement value; and

manage, by the corpora management engine executing on the at least one processor, a corpus of evidence data based on the grading of the source of the evidential statement, wherein managing the corpus comprises comparing the source grading measurement value to one or more predetermined threshold values and performing a corpus management operation based on results of the comparison, and wherein, in response to the comparison indicating that the source grading measurement is below a first threshold, removing evidence data associated with the source from the corpus.

19. The apparatus of claim 18, wherein the instructions further cause the processor to:

output, by the corpora management engine executing on the at least one processor, a representation of the indication of trustworthiness of the evidential statement in association with the evidential statement to a link scoring and analyzer engine executing on the data processing system;

generate, by the link scoring and analyzer engine, a score for the hypothetical ontological link at least based on the indication of trustworthiness of the evidential statement; and

modify, by an ontology modification engine executing on the at least one processor, the ontology data structure based on the score for the hypothetical ontological link.

20. The apparatus of claim 18, wherein, in response to the comparison indicating that the source grading measurement value is above a second threshold, sending a request to a user specifying the source and the source grading measurement value and a recommendation to obtain additional evidence data from the source.