Internal malware data item clustering and analysis

• US 9,344,447 B2
• Filed: 09/15/2014
• Issued: 05/17/2016
• Est. Priority Date: 07/03/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Cohen, David, Ma, Jason, Fu, Bing Jie, Nepomnyashchiy, Ilya, Berler, Steven, Smaliy, Alex, Grossman, Jack, Thompson, James, Boortz, Julia, Sprague, Matthew, Menon, Parvathy, Kross, Michael, Harris, Michael, Borochoff, Adam
• Primary Examiner(s): Hamilton, Lalita M
• Application Number:
  US14/486,991
• Publication Number:
  US 20160006749A1
• Time in Patent Office: 610 Days
• Field of Search: 705/35, 705/36.R, 705/37, 705/38
• US Class Current: 1/1
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06Q 40/12 Accounting
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

Embodiments of the present disclosure relate to a data analysis system that may automatically generate memory-efficient clustered data structures, automatically analyze those clustered data structures, and provide results of the automated analysis in an optimized way to an analyst. The automated analysis of the clustered data structures (also referred to herein as data clusters) may include an automated application of various criteria or rules so as to generate a compact, human-readable analysis of the data clusters. The human-readable analysis (also referred to herein as “summaries” or “conclusions”) of the data clusters may be organized into an interactive user interface so as to enable an analyst to quickly navigate among information associated with various data clusters and efficiently evaluate those data clusters in the context of, for example, a fraud investigation. Embodiments of the present disclosure also relate to automated scoring of the clustered data structures.

Claims

1. A computer system for protecting a computer network from malware by providing for the efficient analysis of large amounts of malware-related data, the computer system comprising:

one or more computer readable storage devices configured to store;

a plurality of computer executable instructions;

one or more software modules including computer executable instructions, the one or more software modules including a cluster engine module, a user interface engine module and a workflow engine module;

a plurality of data clustering strategies based on rules generated by the cluster engine module; and

a plurality of data cluster types, each data cluster type of the plurality of data cluster types associated with a data clustering strategy;

one or more cluster data sources configured to store;

a plurality of data items including at least;

file data items, each file data item associated with at least one suspected malware file; and

malware-related data items associated with captured communications between an internal network and an external network, the malware-related data items including at least one of;

external Internet Protocol addresses, external domains, external computing devices, internal Internet Protocol addresses, internal computing devices, users of particular computing devices, or organizational positions associated with users of particular computing devices; and

one or more hardware computer processors in communication with the one or more computer readable storage devices and the one or more cluster data sources, and configured to execute the one or more software modules in order to cause the computer system to;

designate, by the cluster engine module, one or more seeds by;

accessing, from the one or more cluster data sources, the file data items;

calculating, for each file data item of the file data items, at least one of a hash of the file data item or a hash of an executed file data item, wherein the executed file data item was generated by an execution of the file data item in a sandboxed environment; and

identifying one or more file data items based at least in part on comparing the at least one hash of the file data item or the executed file data item with a malware threat list of hashes, and designating each of the identified one or more file data items as a seed;

for each of the file data items designated as a seed;

select, by the cluster engine module, a particular data clustering strategy;

identify, by the cluster engine module, one or more malware-related data items determined to be associated with the designated file data item seed based at least on the particular data clustering strategy, wherein the particular data clustering strategy performs at least one of querying the one or more cluster data sources or scanning network traffic to determine at least one of;

external Internet Protocol addresses associated with the designated file data item seed, external domains associated with the designated file data item seed, external computing devices associated with the designated file data item seed, internal Internet Protocol addresses associated with the designated file data item seed, internal computing devices associated with the designated file data item seed users of particular computing devices associated with the designated file data item seed, or organizational positions associated with the determined users of particular computing devices;

generate, by the cluster engine module, a data item cluster based at least on the designated file data item seed, wherein generating the data item cluster comprises;

adding the designated file data item seed to the data item cluster;

identifying one or more of the network indicators that are associated with the seed;

identifying one or more of the network-related data items associated with at least one of the identified one or more of the network indicators;

adding, to the data item cluster, the identified one or more malware-related data items;

identifying an additional one or more data items, including file data items and/or malware-related data items, associated with any data items of the data item cluster;

adding, to the data item cluster, the additional one or more data items; and

storing the one or more data item clusters,

generating by the user interface engine module at least one human-readable conclusion associated with at least one generated data item cluster, wherein generating the at least one human-readable conclusion comprises;

determining a particular data cluster type from the plurality of cluster types based at least on the particular data clustering strategy;

identifying one or more human-readable templates comprising pre-generated text, wherein the human-readable templates are based at least on predefined associations between respective human-readable templates and data cluster types;

automatically analyzing the data item cluster to generate summary data according to rules, scoring algorithms, or other criteria; and

populating the identified one or more human-readable templates with data from the at least one generated data item cluster or summary data of the at least one generated data item cluster;

cause presentation, by the user interface engine module, of the at least one generated data item cluster and the at least one human-readable conclusion including the populated pre-generated text data, in a user interface of a client computing device; and

generate, by the workflow engine and the user interface engine, an interactive workflow process to allow the user to perform at least one of;

select new seeds, operate on existing seeds, generate new data clusters, or regenerate existing clusters.

2. The computer system of claim 1, wherein each of the data items of the data item cluster identify at least an internal computing device, a user of the internal computing device, and an organizational position associated with the user.

3. The computer system of claim 1, wherein the one or more hardware computer processors are further configured to execute the one or more software modules in order to cause the one or more hardware computer processors to:

scan communications between the internal network and the external network so as to identify additional malware-related data items; and

store the additional malware-related data items in the one or more cluster data sources.

4. The computer system of claim 3, wherein the communications are continuously scanned via a proxy.

5. The computer system of claim 1, wherein designation of the one or more seeds further comprises determining one or more network indicators associated with the identified one or more file data items, the one or more network indicators include at least one of an external Internet Protocol address or an external domain.

6. The computer system of claim 5, wherein the one or more of the network indicators that are associated with the seed comprise network indicators that are contacted by the at least one suspected malware file associated with the seed when the at least one suspected malware file is executed.

7. The computer system of claim 1, wherein designation of the one or more seeds further comprises determining whether or not a respective file data item has been marked by a human analyst as a seed.

8. The computer system of claim 7, wherein each of the file data items is processed by the computer system by at least:

initiating an analysis of the file data item including the at least one suspected malware file, wherein the analysis of the file data item generates a plurality of analysis information items including at least one of calculated hashes, file properties, academic analysis information, file execution information, or third-party analysis information;

associating the plurality of analysis information items with the file data item; and

generating a user interface including one or more user selectable portions presenting various of the analysis information items, the user interface usable by the human analyst to determine one or more characteristics of the file data item and to mark the file data item as a seed.

9. The computer system of claim 8, wherein the file data item is marked by a human analyst as a seed via a user interface of the computer system.

10. The computer system of claim 8, wherein initiating analysis of the file data item and generating the plurality of analysis information items comprises:

initiating an internal analysis of the file data item; and

initiating an external analysis of the file data item,

wherein the internal analysis includes analysis performed by the one or more hardware computer processors, and wherein the internal analysis includes at least one of calculation of an MD5 hash of the file data item, calculation of a SHA-1 hash of the file data item, or calculation of a size of the file data item, and

wherein the external analysis includes analysis performed by at least a second computer system, and wherein the external analysis includes execution of the file data item in a sandboxed environment and analysis of the file data item by a third-party malware analysis service.

11. The computer system of claim 1, wherein:

the one or more computer readable storage devices are further configured to store;

a plurality of data cluster analysis rules associated with a respective data clustering strategy, and

the one or more hardware computer processors are further configured to execute the one or more software modules in order to cause the one or more hardware computer processors to;

for each generated data item cluster;

access a plurality of data cluster analysis rules associated with the data clustering strategy;

analyze the data item cluster based on the accessed data cluster analysis rules; and

based at least on the analysis of the data item cluster;

determine an alert score for the data item cluster; and

cause presentation of the determined alert score in the user interface

generate one or more human-readable conclusions regarding the data item cluster.

12. The computer system of claim 11, wherein the alert score indicates a degree of correlation between characteristics of the data item cluster and the accessed data cluster analysis rules.

13. The computer system of claim 12, wherein a relatively higher alert score indicates a data cluster that is relatively more important for a human analyst to evaluate, and a relatively lower alert score indicated a data cluster that is relatively less important for the human analyst to evaluate.

14. The computer system of claim 12, wherein each alert score for respective data clusters is assigned to a category indicating a high degree of correlation, a medium degree of correlation, or a low degree of correlation.

15. The computer system of claim 14, wherein the high degree of correlation is associated with a first color, the medium degree of correlation is associated with a second color, and the low degree of correlation is associated with a third color.

16. The computer system of claim 11, wherein the one or more hardware computer processors are further configured to execute the one or more software modules in order to cause the computer system to:

for each generated data item cluster;

generate one or more human-readable conclusions regarding the data item cluster;

generate an alert, the alert comprising a the alert score, the one or more human-readable conclusions, the data items associated with the data item cluster, and metadata associated with the data items of the data item cluster.

17. The computer system of claim 16, wherein the one or more hardware computer processors are further configured to execute the one or more software modules in order to cause the computer system to:

generate a second user interface including a list of user-selectable alert indicators, an alert indicator being provided for each of the generated alerts, each of the alert indicators providing a summary of information associated with respective generated alerts.

18. The computer system of claim 17, wherein the one or more hardware computer processors are further configured to execute the one or more software modules in order to cause the computer system to:

in response to a selection of an alert indicator by a human analyst;

generate an alert display, the alert display including at least an indication of the alert score and a list of the one or more human-readable conclusions.

19. The computer system of claim 1, wherein the at least human-readable conclusion further comprises a phrase or sentence including one or more indications of summary or aggregated data associated with a plurality of the data items of the at least one generated data item cluster.