External malware data item clustering and analysis

• US 10,798,116 B2
• Filed: 04/24/2018
• Issued: 10/06/2020
• Est. Priority Date: 07/03/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Cohen, David, Ma, Jason, Fu, Bing Jie, Nepomnyashchiy, Ilya, Berler, Steven, Smaliy, Alex, Grossman, Jack, Thompson, James, Boortz, Julia, Sprague, Matthew, Menon, Parvathy, Kross, Michael, Harris, Michael, Borochoff, Adam
• Primary Examiner(s): Dolly, Kendall
• Assistant Examiner(s): Fields, Courtney D
• Application Number:
  US15/961,431
• Publication Number:
  US 20180270264A1
• Time in Patent Office: 896 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06Q 40/12 Accounting
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

Embodiments of the present disclosure relate to a data analysis system that may automatically generate memory-efficient clustered data structures, automatically analyze those clustered data structures, and provide results of the automated analysis in an optimized way to an analyst. The automated analysis of the clustered data structures (also referred to herein as data clusters) may include an automated application of various criteria or rules so as to generate a compact, human-readable analysis of the data clusters. The human-readable analyses (also referred to herein as “summaries” or “conclusions”) of the data clusters may be organized into an interactive user interface so as to enable an analyst to quickly navigate among information associated with various data clusters and efficiently evaluate those data clusters in the context of, for example, a fraud investigation. Embodiments of the present disclosure also relate to automated scoring of the clustered data structures.

Claims

1. A computer system configured to provide a dynamic user interface relating to visualization of alerts of malicious network activity, the computer system comprising:

an electronic data structure configured to store a plurality of clusters of data items, wherein each cluster of data items represents a group of related malicious network activities; and

one or more hardware computer processors configured to execute code in order to cause the computer system to;

access the electronic data structure including the plurality of clusters of data items;

analyze the plurality of clusters of data items to determine, for each cluster of the plurality of clusters;

a type of malicious network activity represented by the cluster, and

a criticality of the malicious network activity represented by the cluster;

further analyze the plurality of clusters of data items to determine respective numbers of clusters of the plurality of clusters having each of a plurality of types of malicious network activity;

provide a dynamic user interface configured to display at least;

a first visualization indicating, for each type of malicious network activity of the plurality of types of malicious network activity, respective portions of the plurality of clusters having the type of malicious network activity; and

a second visualization indicating, for each cluster of the plurality of clusters, an alert corresponding to the cluster, wherein the alert visually indicates that criticality of the malicious network activity represented by the cluster; and

automatically order the alerts indicated in the second visualization based on the respective determined criticalities of malicious network activity represented by the clusters corresponding to the alerts.

2. The computer system of claim 1, wherein the alert visually indicates that criticality of the malicious network activity represented by the cluster by at least one of:

an icon, or a color.

3. The computer system of claim 2, wherein the second visualization further indicates, for each alert, the type of malicious network activity represented by the cluster corresponding to the alert.

4. The computer system of claim 1, wherein the one or more hardware computer processors are further configured to execute code in order to cause the computer system to:

access a plurality of cluster analysis rules; and

for each cluster of the plurality of clusters;

determine at least one of the plurality of cluster analysis rules that is associated with the type of malicious network activity represented by the cluster;

analyze the cluster based on the at least one of the plurality of cluster analysis rules; and

based on the analysis of the cluster, generate one or more human-readable conclusions regarding the cluster.

5. The computer system of claim 4, wherein the second visualization further indicates, for each alert, at least one of the one or more human-readable conclusions regarding the cluster corresponding to the alert.

6. The computer system of claim 4, wherein the criticality of the malicious network activity represented by the cluster is determined based on a correlation between characteristics of the cluster and the at least one of the plurality of cluster analysis rules that is associated with type of malicious network activity represented by the cluster.

7. The computer system of claim 6, wherein the degree of correlation is based on both an assessment of risk associated with the cluster and a confidence level in accuracy of the assessment of risk.

8. The computer system of claim 4, wherein the criticality is represented by a score.

9. The computer system of claim 8, wherein a relatively higher score indicates a cluster that is relatively more important for a human analyst to evaluate, and a relatively lower score indicates a cluster that is relatively less important for the human analyst to evaluate.

10. The computer system of claim 8, wherein the score is selected from high, medium, or low.

11. The computer system of claim 4, wherein generating the one or more human-readable conclusions is further based on one or more conclusion templates that are populated with data associated with the cluster.

12. The computer system of claim 4, wherein the one or more human-readable conclusions each comprise a phrase or sentence including one or more indications of summary or aggregated data associated with a plurality of the data items of the cluster.

13. The computer system of claim 1, wherein the one or more hardware computer processors are further configured to execute code in order to cause the computer system to:

receive, via the first visualization of the dynamic user interface, a user selection of a first type of malicious network activity from the plurality of types of malicious network activity; and

automatically update at least the second visualization of the dynamic user interface to filter the alerts to only those alerts corresponding to clusters associated with the selected first type of malicious network activity.

14. A computer-implemented method comprising:

by one or more hardware computer processors executing code;

communicating with an electronic data structure configured to store a plurality of clusters of data items, wherein each cluster of data items represents a group of related malicious network activities;

accessing the electronic data structure including the plurality of clusters of data items;

analyzing the plurality of clusters of data items to determine, for each cluster of the plurality of clusters;

a type of malicious network activity represented by the cluster, and

a criticality of the malicious network activity represented by the cluster;

further analyzing the plurality of clusters of data items to determine respective numbers of clusters of the plurality of clusters having each of a plurality of types of malicious network activity;

providing a dynamic user interface configured to display at least;

a first visualization indicating, for each type of malicious network activity of the plurality of types of malicious network activity, respective portions of the plurality of clusters having the type of malicious network activity; and

a second visualization indicating, for each cluster of the plurality of clusters, an alert corresponding to the cluster, wherein the alert visually indicates that criticality of the malicious network activity represented by the cluster; and

automatically ordering the alerts indicated in the second visualization based on the respective determined criticalities of malicious network activity represented by the clusters corresponding to the alerts.

15. The computer-implemented method of claim 14, wherein the alert visually indicates that criticality of the malicious network activity represented by the cluster by at least one of:

an icon, or a color.

16. The computer-implemented method of claim 15, wherein the second visualization further indicates, for each alert, the type of malicious network activity represented by the cluster corresponding to the alert.

17. The computer-implemented method of claim 14 further comprising:

by the one or more hardware computer processors executing code;

accessing a plurality of cluster analysis rules; and

for each cluster of the plurality of clusters;

determining at least one of the plurality of cluster analysis rules that is associated with the type of malicious network activity represented by the cluster;

analyzing the cluster based on the at least one of the plurality of cluster analysis rules; and

based on the analysis of the cluster, generating one or more human-readable conclusions regarding the cluster.

18. The computer-implemented method of claim 17, wherein the criticality of the malicious network activity represented by the cluster is determined based on a correlation between characteristics of the cluster and the at least one of the plurality of cluster analysis rules that is associated with type of malicious network activity represented by the cluster.

19. The computer-implemented method of claim 18, wherein the degree of correlation is based on both an assessment of risk associated with the cluster and a confidence level in accuracy of the assessment of risk.

20. The computer-implemented method of claim 14 further comprising:

by the one or more hardware computer processors executing code;

receiving, via the first visualization of the dynamic user interface, a user selection of a first type of malicious network activity from the plurality of types of malicious network activity; and

automatically updating at least the second visualization of the dynamic user interface to filter the alerts to only those alerts corresponding to clusters associated with the selected first type of malicious network activity.