External malware data item clustering and analysis

• US 9,965,937 B2
• Filed: 08/29/2014
• Issued: 05/08/2018
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Cohen, David, Ma, Jason, Fu, Bing Jie, Nepomnyashchiy, Ilya, Berler, Steven, Smaliy, Alex, Grossman, Jack, Thompson, James, Boortz, Julia, Sprague, Matthew, Menon, Parvathy, Kross, Michael, Harris, Michael, Borochoff, Adam
• Primary Examiner(s): Zand, Kambiz
• Assistant Examiner(s): FIELDS, COURTNEY D
• Application Number:
  US14/473,920
• Publication Number:
  US 20160344758A1
• Time in Patent Office: 1,348 Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G08B 21/18 Status alarms G08B21/02 takes precedence
• CPC Class Codes: H04L 63/0281 Proxies
• CPC Class Codes: H04L 63/1433 Vulnerability analysis
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

Embodiments of the present disclosure relate to a data analysis system that may automatically generate memory-efficient clustered data structures, automatically analyze those clustered data structures, and provide results of the automated analysis in an optimized way to an analyst. The automated analysis of the clustered data structures (also referred to herein as data clusters) may include an automated application of various criteria or rules so as to generate a compact, human-readable analysis of the data clusters. The human-readable analyzes (also referred to herein as “summaries” or “conclusions”) of the data clusters may be organized into an interactive user interface so as to enable an analyst to quickly navigate among information associated with various data clusters and efficiently evaluate those data clusters in the context of, for example, a fraud investigation. Embodiments of the present disclosure also relate to automated scoring of the clustered data structures.

Claims

1. A computer system comprising:

one or more computer readable storage devices configured to store;

a plurality of computer executable instructions;

a data clustering strategy; and

a plurality of data items including at least;

external domain data items; and

network-related data items associated with captured communications between an internal network and an external network, the network-related data items including at least one of;

external Internet Protocol addresses, external domains, identifiers corresponding to external computerized devices, internal Internet Protocol addresses, identifiers corresponding to internal computerized devices, identifiers corresponding to users of particular computerized devices, or organizational positions associated with users of particular computerized devices; and

one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause the computer system to;

scan one or more threat lists stored external to the internal network, each of the threat lists including information related to previously identified malware threats and information related to those previously identified malware threats including external domain data items;

identify one or more external domain data items included in the one or more threat lists, each of the one or more external domain data items being associated with a malicious domain;

designate each of the identified one or more external domain data items as a seed;

for each of the designated seeds, generate a data item cluster based on the data clustering strategy by at least;

adding the seed to the data item cluster;

identifying one or more of the network-related data items associated with the seed;

adding, to the data item cluster, the one or more identified network-related data items;

identifying an additional one or more data items, including external domain data items and/or network-related data items, associated with any data items of the data item cluster; and

adding, to the data item cluster, the additional one or more data items;

determine to regenerate a particular data item cluster;

regenerate the particular data item cluster by at least;

identifying new one or more data items, including external domain data items and/or network-related data items, associated with any data items of the particular data item cluster, wherein the new one or more data items were not present in the particular data item cluster as initially generated; and

adding, to the particular data item cluster, the new one or more data items;

access a plurality of data item clusters including at least one of the data item cluster or the particular data item cluster, wherein the plurality of data item clusters include data items associated with malware threats;

generate alert scores for at least some of the plurality of data item clusters according to one or more scoring strategies, wherein the alert scores indicate criticalities of the malware threats represented by the plurality of data item clusters; and

cause presentation, in a user interface, of at least a visualization including alerts for at least one of the plurality of data item clusters based on the alert scores, wherein the alerts visually indicate the criticalities of the malware threats represented by the plurality of data item clusters.

2. The computer system of claim 1, wherein determining to regenerate the particular data item cluster:

detecting at least one of;

the new one or more data items or a change in any data item of the particular data item cluster.

3. The computer system of claim 1, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions in order to cause the one or more hardware computer processors to:

scan communications between the internal network and the external network so as to generate additional network-related data items; and

store the additional network-related data items in the one or more computer readable storage devices.

4. The computer system of claim 3, wherein the communications are continuously scanned via a proxy.

5. The computer system of claim 3, wherein identifying the one or more of the network-related data items associated with the seed comprises:

for each of the network-related data items, determining whether the network-related data item is associated with a communication with the malicious domain associated with the seed; and

in response to determining that the network-related data item is associated with a communication with the malicious domain associated with the seed, identifying the network-related data item as being associated with the seed.

6. The computer system of claim 5, wherein the data item clusters are continuously updated.

7. The computer system of claim 1, wherein:

the one or more computer readable storage devices are further configured to store;

a plurality of data cluster analysis rules associated with the data clustering strategy, and

the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions in order to cause the one or more hardware computer processors to;

for each generated data item cluster;

access the plurality of data cluster analysis rules associated with the data clustering strategy;

analyze the data item cluster based on the accessed data cluster analysis rules; and

based on the analysis of the data item cluster;

generate one or more human-readable conclusions regarding the data item cluster.

8. The computer system of claim 1, wherein an alert score indicates a degree of correlation between characteristics of the data item cluster and the one or more scoring strategies.

9. The computer system of claim 8, wherein the degree of correlation is based on both an assessment of risk associated with the particular data cluster and a confidence level in accuracy of the assessment of risk.

10. The computer system of claim 8, wherein a relatively higher alert score indicates a data cluster that is relatively more important for a human analyst to evaluate, and a relatively lower alert score indicated a data cluster that is relatively less important for the human analyst to evaluate.

11. The computer system of claim 8, wherein each alert score for respective data clusters is assigned to a category indicating a high degree of correlation, a medium degree of correlation, or a low degree of correlation.

12. The computer system of claim 11, wherein the high degree of correlation is associated with a first color, the medium degree of correlation is associated with a second color, and the low degree of correlation is associated with a third color.

13. The computer system of claim 7, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions in order to cause the computer system to:

for each data item cluster of the plurality of data item clusters;

generate an alert summary, the alert summary comprising a respective alert score, the one or more human-readable conclusions, the data items associated with the data item cluster, and metadata associated with the data items of the data item cluster.

14. The computer system of claim 13, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions in order to cause the computer system to:

generate a second visualization in the user interface including a list of user-selectable alert indicators, an alert indicator being provided for each of the generated alert summaries, each of the alert indicators providing a summary of information associated with respective generated alert summaries.

15. The computer system of claim 14, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions in order to cause the computer system to:

in response to a selection of an alert indicator by a human analyst;

generate an alert display, the alert display including at least an indication of the alert score and a list of the one or more human-readable conclusions.

16. The computer system of claim 15, wherein the alert display further includes a table of information associated with the data items associated with the data item cluster of the alert, and metadata associated with the data items of the data item cluster of the alert.

17. The computer system of claim 16, wherein the table of information includes a mixture of information of various types.

18. The computer system of claim 17, wherein the table of information includes one or more user interface controls selectable by a human analysis in order to filter according information type and/or time period.

19. The computer system of claim 7, wherein the one or more human-readable conclusions each comprise a phrase or sentence including one or more indications of summary or aggregated data associated with a plurality of the data items of the data item cluster.

20. The computer system of claim 19, wherein generating the one or more human-readable conclusions comprises:

selecting, based on the data cluster type associated with the particular data cluster, one or more conclusion templates; and

populating the one or more conclusion templates with data associated with the particular data cluster.

21. The computer system of claim 1, wherein generating the alert scores for at least some of the plurality of data item clusters further comprises at least one of:

determining a number of domains in a data item cluster of the plurality of data item clusters;

determining a percentage of captured communications in the data item cluster of the plurality of data item clusters categorized as likely malicious;

or

determining an organizational position associated with a user identifier in the data item cluster of the plurality of data item clusters.

22. The computer system of claim 1, wherein causing presentation of the visualization including the alerts further comprises:

determining that an alert score for a data item cluster of the plurality of data item clusters exceeds a threshold; and

causing presentation of an alert for the data item cluster of the plurality of data item clusters.