Knowledge discovery appliance

• US 8,484,217 B1
• Filed: 03/10/2011
• Issued: 07/09/2013
• Est. Priority Date: 03/10/2011
• Status: Active Grant

• Current Assignee: Cyveillance, Inc. (ZeroFox Holdings, Inc.)
• Original Assignee: QinetiQ North America Incorporated (QinetiQ Group PLC)
• Inventors: Srivastava, Manoj Kumar, Walker, William Andrews, Olson, Eric Alexander, Smith, Steven Robert
• Primary Examiner(s): NGUYEN, PHONG H
• Application Number:
  US13/045,028
• Time in Patent Office: 852 Days
• Field of Search: 707/737, 707/738, 707/754, 707/748
• US Class Current: 707/738
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9532 Query formulation
• CPC Class Codes: Y02A 90/10 Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

Methods and systems for collecting and processing large volumes of data to determine the relevancy and value thereof comprise: deploying one or more data retrieval modules to interface with one or more data sources within the local network, wherein the one or more data retrieval modules comprise executable program code; executing the one or more data retrieval modules to perform operations comprising: accessing application data representing data stored or communicated through the one or more data sources; and forwarding the application data for analysis; and analyzing the forwarded application data to determine whether it is relevant to the topic, wherein the analyzing is performed by one or more devices within the local network.

Claims

1. A computer-implemented method of analyzing data in a local network to determine relevance to a topic, the method comprising:

deploying one or more data retrieval modules to interface with one or more data sources within the local network, wherein the one or more data retrieval modules comprise executable program code;

executing the one or more data retrieval modules to perform operations comprising;

accessing application data representing data stored or communicated through the one or more data sources;

converting the application data into a normalized format; and

forwarding the normalized application data for analysis; and

analyzing the forwarded application data to determine whether it is relevant to the topic, wherein the analyzing is performed by one or more devices within the local network, the analyzing comprising;

calculating one or more relevancy scores for the forwarded application data;

classifying as relevant forwarded application data having scores above a first threshold;

classifying as non-relevant forwarded application data having scores below a second threshold, wherein the second threshold is lower than the first threshold;

classifying as review-pending forwarded application data having scores between the first threshold and the second threshold;

forwarding the review-pending forwarded application data to one or more analysts;

receiving input from the one or more analysts as to whether at least a portion of the forwarded review-pending application data is relevant; and

modifying one or more automated algorithms used for determining relevance based on the input from the one or more analysts.

2. The method of claim 1, wherein deploying the one or more data retrieval modules further comprises:

installing the one or more data retrieval modules on one or more devices executing processes associated with the one or more data sources.

3. The method of claim 1, wherein forwarding the application data for analysis further comprises:

forwarding the application data to an intermediate datastore, the datastore providing the forwarded application data to one or more processes analyzing the forwarded application data for relevance on a periodic or as-needed basis.

4. The method of claim 1, wherein the application data comprises structured, semi-structured, and unstructured data, and wherein converting the application data into a normalized format comprises converting the structured, semi-structured, and unstructured data into a semi-structured format.

5. The method of claim 1, wherein calculating one or more relevancy scores for the forwarded application data further comprises:

analyzing the forwarded application data to determine one or more answers to one or more questions about the forwarded application data;

generating a first set of metadata about the forwarded application data based on the one or more answers; and

calculating the one or more relevancy scores for the forwarded application data based on the first set of metadata.

6. The method of claim 5, further comprising:

excluding forwarded application data satisfying one or more preliminary exclusion criteria prior to calculating one or more relevancy scores for the forwarded application data.

7. The method of claim 6, further comprising:

generating a second set of metadata about the forwarded application data based on one or more properties of the forwarded application data; and

determining whether the forwarded application data satisfies the one or more preliminary exclusion criteria based on the second set of metadata.

8. The method of claim 6, wherein one or more of the preliminary exclusion criteria, the one or more questions, the first threshold, and the second threshold are configured based on analyst input.

9. The method of claim 8, wherein the one or more of the preliminary exclusion criteria, the one or more questions, the first threshold, and the second threshold are configured based on analyst input such that application data subjected to the one or more of the preliminary exclusion criteria, the one or more questions, the first threshold, and the second threshold is classified as relevant or non-relevant based on how the analyst would classify the application data if performing manual review.

10. The method of claim 1, wherein all steps are performed in real-time, and wherein application data is maintained in volatile memory until classified as relevant.

11. A system for analyzing data in a network to determine relevance to a topic, the system comprising:

an application system comprising one or more devices within the network configured to operate one or more data sources, wherein the one or more devices are additionally configured to execute one or more data retrieval modules, the data retrieval modules configured to;

access application data representing data stored or communicated through the one or more data sources;

converting the application data into a normalized format; and

forwarding the normalized application data for analysis; and

an analysis system comprising one or more devices within the network configured to analyze the forwarded application data to determine whether it is relevant to the topic, the analysis system is configured to;

calculate one or more relevancy scores for the forwarded application data;

classify as relevant forwarded application data having scores above a first threshold;

classifying as non-relevant forwarded application data having scores below a second threshold, wherein the second threshold is lower than the first threshold;

classify as review-pending forwarded application data having scores between the first threshold and the second threshold;

forward the review-pending forwarded application data to one or more analysts;

receive input from the one or more analysts as to whether at least a portion of the forwarded review-pending application data is relevant; and

modify one or more automated algorithms used for determining relevance based on the input from the one or more analysts.

12. The system of claim 11, further comprising:

a datastore system comprising one or more devices within the network configured to receive forwarded application data from the one or more data retrieval modules and provide the forwarded application data to the analysis system.

13. The system of claim 11, wherein the application data comprises structured, semi-structured, and unstructured data, and wherein converting the application data into a normalized format comprises converting the structured, semi-structured, and unstructured data into a semi-structured format.

14. The system of claim 11, wherein calculating one or more relevancy scores for the forwarded application data further comprises:

analyzing the forwarded application data to determine one or more answers to one or more questions about the forwarded application data;

generating a first set of metadata about the forwarded application data based on the one or more answers; and

calculating the one or more relevancy scores for the forwarded application data based on the first set of metadata.

15. The system of claim 14, wherein analyzing the application data to determine whether it is relevant further comprises:

excluding forwarded application data satisfying one or more preliminary exclusion criteria prior to calculating one or more relevancy scores for the forwarded application data.

16. The system of claim 15, wherein analyzing the application data to determine whether it is relevant further comprises:

generating a second set of metadata about the forwarded application data based on one or more properties of the forwarded application data; and

determining whether the forwarded application data satisfies the one or more preliminary exclusion criteria based on the second set of metadata.

17. The system of claim 15, wherein one or more of the preliminary exclusion criteria, the one or more questions, the first threshold, and the second threshold are configured based on analyst input.

18. The system of claim 17, wherein the one or more of the preliminary exclusion criteria, the one or more questions, the first threshold, and the second threshold are configured based on analyst input such that application data subjected to the one or more of the preliminary exclusion criteria, the one or more questions, the first threshold, and the second threshold is classified as relevant or non-relevant based on how the analyst would classify the application data if performing manual review.

19. The system of claim 11, wherein all operations performed by the analysis system are performed in real-time, and wherein application data is maintained in volatile memory in the analysis system until classified as relevant.

20. The system of claim 19, wherein all operations performed by the analysis system are performed by software installed on and executing on a single hardware bundle.

21. A computer-implemented method of analyzing data in a network to determine relevance to a topic, the method comprising:

deploying one or more data retrieval modules to interface with one or more data sources within the network, wherein the one or more data retrieval modules comprise executable program code;

executing the one or more data retrieval modules to perform operations comprising;

accessing application data representing data stored or communicated through the one or more data sources;

converting the application data into a normalized format; and

forwarding the application data for analysis; and

analyzing the forwarded application data to determine whether it is relevant to the topic, wherein the analyzing is performed by one or more devices within the network, the analyzing comprising;

excluding forwarded application data satisfying one or more preliminary exclusion criteria;

analyzing the forwarded application data to determine one or more answers to one or more questions about the forwarded application data;

generating a set of metadata about the forwarded application data based on the one or more answers;

calculating one or more relevancy scores for the forwarded application data based on the first set of metadata;

classifying as relevant forwarded application data having scores above a first threshold;

classifying as non-relevant forwarded application data having scores below a second threshold, wherein the second threshold is lower than the first threshold;

classifying as review-pending forwarded application data having scores between the first threshold and the second threshold;

forwarding the review-pending forwarded application data to one or more analysts;

receiving input from the one or more analysts as to whether the review-pending forwarded application data is relevant; and

modifying one or more automated algorithms used for determining relevance based on the input from the one or more analysts.