System and method for the automated discovery of unknown unknowns

• US 8,078,559 B2
• Filed: 06/03/2008
• Issued: 12/13/2011
• Est. Priority Date: 06/30/2004
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Talbot, Patrick James, Ellis, Dennis Regan, Sanders, J. Ryan
• Primary Examiner(s): Holmes, Michael B
• Assistant Examiner(s): Wong, Lut
• Application Number:
  US12/132,367
• Publication Number:
  US 20080270336A1
• Time in Patent Office: 1,288 Days
• Field of Search: 706/45, 706/55
• US Class Current: 706/45
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Systems and methods are provided for determining unknown unknowns for at least one assisted decision making system. An assisted decision making system has an associated set of at least one story of interest. The system further comprises a plurality of inferencing algorithms. A given inferencing algorithm is operative to generate story fragments from data stored within an associated knowledge base. A first arbitrator is associated with the first assisted decision making system. The arbitrator accepts a story fragment from the plurality of inferencing algorithms if the story fragment is sufficiently related to a story of interest from the set of at least one story of interest.

Claims

1. A machine for determining potential unknown unknowns for an assisted decision making system, where an unknown unknown is an item of data that is missing from a story of interest and whose absence is not accounted for in the story of interest, comprising:

a processor;

a memory containing;

an information extraction algorithm that extracts evidence from unformatted text segments to produce an evidence template and an associated confidence value from a given text segment;

a plurality of inferencing algorithms, a given inferencing algorithm receiving evidence templates from the information extraction component and generating a story fragment comprising at least one hypothesis, evidence supporting the at least one hypothesis, and respective belief values associated with the at least one hypothesis from the evidence stored in the knowledge base; and

an arbitrator, associated with the assisted decision making system, that compares the at least one hypothesis and the evidence supporting the at least one hypothesis comprising the story fragment to a plurality of hypotheses and associated evidence comprising a story of interest associated with the assisted decision making system to determine if the story fragment is sufficiently related to the story of interest as to represent a potential unknown unknown within the story of interest.

2. The machine of claim 1, further comprising a second arbitrator that compares the at least one hypothesis and the evidence supporting the at least one hypothesis to a plurality of hypotheses and associated evidence comprising a second story of interest associated with the assisted decision making system to determine if the story fragment is sufficiently related to the story of interest as to represent a potential unknown unknown within the second story of interest.

3. The machine of claim 1, the arbitrator fusing a first story fragment from a first decision algorithm that has been determined to represent a potential unknown unknown and a second story fragment from a second decision algorithm that has been determined to represent a potential unknown unknown to produce a combined story fragment.

4. The machine of claim 1, the assisted decision making system having an associated user interface, the arbitrator providing a story fragment to a human analyst through the user interface if it is determined that the story fragment is a potential unknown unknown for the story of interest.

5. The machine of claim 1, each inferencing algorithm further generating a metric that represents a validity of the generated story fragment, the story fragment being provided to the arbitrator only if the generated metric exceeds an internal threshold associated with the inferencing algorithm.

6. The machine of claim 1, further comprising a knowledge base comprising a plurality of executable stories, where an executable story is an executable belief network comprising a plurality of hypotheses, evidence supporting the plurality of hypotheses, and a plurality of text documents from which the evidence supporting the plurality of hypotheses was extracted.

7. The machine of claim 6, the plurality of inferencing algorithms comprising an inductive reasoner that determines appropriate structural changes in the story of interest, including the addition and removal of the plurality of hypotheses comprising the story of interest as well as links connecting pairs of hypotheses within the plurality of hypotheses, in response to new evidence from the information extraction algorithm.

8. The machine of claim 1, the plurality of inferencing algorithms comprising an evidential reasoner that reviews executable stories in the knowledge base to identify story fragments and evidence that may be relevant to the story of interest.

9. The machine of claim 1, wherein the processing data stored in the knowledge base comprises using an abductive reasoning process on new evidence provided to the knowledge base in an hierarchy and determining a plurality of new hypotheses and logical relationships between the new hypotheses from the hierarchy.

10. The machine of claim 1, the information extractor producing an evidence template comprising a plurality of text fields describing an event of interest, including text fields describing at least two of a nature of the event, a cause of the event, a mechanism producing the event, an actor associated with the event, a location of the event, a time of the event, and a magnitude of the event.

11. The machine of claim 10, further comprising at least one evidence classifier that classifies a given evidence template according to the associated plurality of text fields to associate the evidence template with a hypothesis.

12. The machine of claim 1, the unformatted text segments being received from a source external to the machine.

13. A computer implemented method for discovering potential unknown unknowns for an assisted decision making system, where an unknown unknown is an item of data that is missing from a story of interest and whose absence is not accounted for in the story of interest, comprising:

processing unstructured data received at an inferencing algorithm to generate a story fragment comprising at least one hypothesis, evidence supporting the at least one hypothesis, and respective belief values associated with the at least one hypothesis, as well as a metric that represents a validity of the generated story fragment;

comparing the generated metric to an internal threshold of the inferencing algorithm;

comparing the at least one hypothesis and the evidence supporting the at least one hypothesis comprising the story fragment to a plurality of hypotheses and associated evidence comprising a story of interest associated with the assisted decision making system to determine if the story fragment is sufficiently related to the story of interest if the generated metric exceeds the internal threshold of the inferencing algorithm; and

providing the story fragment to a user interface for review by a user as a potential unknown unknown associated with the story of interest if it is sufficiently related to the story of interest.

14. The method of claim 13, further comprising fusing the story fragment and the story of interest according to input from the user provided through the user interface.

15. The method of claim 13, wherein processing unstructured data comprises:

extracting evidence from unformatted text segments to produce an evidence template;

clustering evidence templates as part of an unsupervised clustering algorithm according to characteristics of the evidence, the generated metric comprising an associated density of a cluster of evidence supporting at least one hypothesis in the generated story fragment.

16. The method of claim 15, wherein processing unstructured data further comprises computing a link value between two hypotheses, each hypothesis having an associated set of a plurality of evidence templates, with a given evidence template comprising a plurality of text fields describing an event of interest, the link value being computed according to a comparison of the sets of evidence templates associated with the two hypotheses.

17. A non transitory computer readable medium, operative in a computer processor to discover potential unknown unknowns for an assisted decision making system, where an unknown unknown is an item of data that is missing from a story of interest and whose absence is not accounted for in the story of interest, comprising:

an information extraction component that extracts evidence from unformatted text segments to produce an evidence template and an associated confidence value from a given text segment; and

at least one inference component that generates a story fragment comprising at least one hypothesis, evidence supporting the at least one hypothesis, and respective belief values associated with the at least one hypothesis from the evidence templates from the information extraction component, as well as a metric that represents a validity of the generated story fragment; and

a plurality of arbitrators, representing respective stories of interest associated with the assisted decision making system, that compare the at least one hypothesis and the evidence supporting the at least one hypothesis comprising the story fragment to a plurality of hypotheses and associated evidence comprising each of the plurality of stories of interest associated with the assisted decision making system to determine if the story fragment is sufficiently related to any one of the plurality of stories of interest as to represent a potential unknown unknown within the story of interest.

18. The computer readable medium of claim 17, further comprising a knowledge base comprising a plurality of executable stories, where an executable story is an executable belief network comprising a plurality of hypotheses, evidence supporting the plurality of hypotheses, and a plurality of text documents from which the evidence supporting the plurality of hypotheses was extracted.

19. The computer readable medium of claim 17, the assisted decision making system having an associated user interface, a given arbitrator of the plurality of arbitrators providing a story fragment to a human analyst through the user interface only if it is determined that the story fragment is a potential unknown unknown for the story of interest associated with the arbitrator.