Network intrusion data item clustering and analysis

• US 9,998,485 B2
• Filed: 09/15/2014
• Issued: 06/12/2018
• Est. Priority Date: 07/03/2014
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Cohen, David, Ma, Jason, Fu, Bing Jie, Nepomnyashchiy, Ilya, Berler, Steven, Smaliy, Alex, Grossman, Jack, Thompson, James, Boortz, Julia, Sprague, Matthew, Menon, Parvathy, Kross, Michael, Harris, Michael, Borochoff, Adam
• Primary Examiner(s): Zand, Kambiz
• Assistant Examiner(s): FIELDS, COURTNEY D
• Application Number:
  US14/487,021
• Publication Number:
  US 20160366164A1
• Time in Patent Office: 1,366 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06Q 40/12 Accounting
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

Embodiments of the present disclosure relate to a data analysis system that may automatically generate memory-efficient clustered data structures, automatically analyze those clustered data structures, and provide results of the automated analysis in an optimized way to an analyst. The automated analysis of the clustered data structures (also referred to herein as data clusters) may include an automated application of various criteria or rules so as to generate a compact, human-readable analysis of the data clusters. The human-readable analyses (also referred to herein as “summaries” or “conclusions”) of the data clusters may be organized into an interactive user interface so as to enable an analyst to quickly navigate among information associated with various data clusters and efficiently evaluate those data clusters in the context of, for example, a fraud investigation. Embodiments of the present disclosure also relate to automated scoring of the clustered data structures.

Claims

1. A computer system comprising:

one or more non-transitory computer readable storage devices configured to store;

a plurality of computer executable instructions;

a data clustering strategy; and

a plurality of data items including at least;

intrusion detection system reports, each intrusion detection system report associated with at least a source Internet Protocol address and a destination Internet Protocol address; and

network-related data items associated with captured communications between an internal network and an external network, the network-related data items including at least one of;

external Internet Protocol addresses, external domains, external computerized devices, internal Internet Protocol addresses, internal computerized devices, users of particular computerized devices, intrusion detection system information, network firewall data, or WHOIS information; and

one or more hardware computer processors in communication with the one or more non-transitory computer readable storage devices and configured to execute the plurality of computer executable instructions to cause the computer system to;

receive an intrusion detection system report including a communication between a source Internet Protocol address and a destination Internet Protocol address;

initiate an automated lookup to determine which of the source Internet Protocol address and the destination Internet Protocol address is an external Internet Protocol address, the external Internet Protocol address being external to the internal network;

designate the external Internet Protocol address as a seed;

generate a first data item cluster based on the data clustering strategy by at least;

adding the seed to the first data item cluster;

identifying one or more of the network-related data items associated with the seed; and

adding, to the first data item cluster, the one or more identified network-related data items;

determine to regenerate the first data item cluster;

regenerate the first data item cluster by at least;

identifying one or more new network-related data items associated with at least one of;

the seed, or the one or more identified network-related data items, wherein the one or more new network-related data items were not added to the first data item cluster as initially generated; and

adding, to the first data item cluster, the one or more new network-related data items;

access a plurality data item clusters including the first data item cluster, wherein the plurality of data item clusters include data items associated with malicious network activities;

analyze the plurality of data item clusters;

determine criticalities of the malicious network activity represented by the data item clusters; and

provide a dynamic user interface displaying at least a first visualization including alerts for at least one of the plurality of data item clusters, wherein the alerts visually indicate the criticalities of the malicious network activity represented by the data item clusters.

2. The computer system of claim 1, wherein generating the first data item cluster based on the data clustering strategy further comprises:

identifying additional one or more data items associated with any data items of the first data item cluster; and

adding, to the first data item cluster, the additional one or more data items.

3. The computer system of claim 1, wherein the first data item cluster includes at least the source Internet Protocol address, the destination Internet Protocol address, an internal computerized device associated with an Internet Protocol address in the cluster, a user of the internal computerized device, and WHOIS information associated with the external Internet Protocol address.

4. The computer system of claim 1, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

receive a second intrusion detection system report including a communication between a second source Internet Protocol address and a second destination Internet Protocol address;

initiate an automated lookup to determine which of the second source Internet Protocol address and the second destination Internet Protocol address is a second external Internet Protocol address, the second external Internet Protocol address being external to the internal network;

compare the external Internet Protocol address to the second external Internet Protocol address;

in response to determining, based on the comparison, that the external Internet Protocol address and the second external Internet Protocol address are the same, add the second external Internet Protocol address to the first data item cluster; and

in response to determining, based on the comparison, that the external Internet Protocol address and the second external Internet Protocol address are not the same;

designate the second external Internet Protocol address as a second seed; and

generate a second data item cluster based on the data clustering strategy and the second seed.

5. The computer system of claim 4, wherein the second external Internet Protocol address is added to the first data item cluster only if the intrusion detection system report and the second intrusion detection system report are received on a same day.

6. The computer system of claim 1, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

scan communications between the internal network and the external network so as to generate additional network-related data items; and

store the additional network-related data items in the one or more non-transitory computer readable storage devices.

7. The computer system of claim 1, wherein received intrusion detection system reports are automatically stored in the one or more non-transitory computer readable storage devices, and the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

identify newly received intrusion detection system reports;

initiate automated lookups to determine external Internet Protocol addresses associated with each of the newly received intrusion detection system reports;

designate the determined external Internet Protocol addresses as seeds; and

generate data item clusters based on the data clustering strategy and the seeds.

8. The computer system of claim 7, wherein data item clusters generated based on common external Internet Protocol addresses are merged.

9. The computer system of claim 8, wherein data item clusters generated based on common external Internet Protocol addresses are merged only if the associated intrusion detection system reports are received on a same day.

10. The computer system of claim 1, wherein:

the one or more non-transitory computer readable storage devices are further configured to store;

a plurality of data cluster analysis rules associated with the data clustering strategy, and

the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to;

for the first data item cluster;

access the plurality of data cluster analysis rules associated with the data clustering strategy;

analyze the first data item cluster based on the accessed data cluster analysis rules; and

based on the analysis of the first data item cluster;

determine an alert score for the first data item cluster; and

generate one or more human-readable conclusions regarding the first data item cluster.

11. The computer system of claim 10, wherein the alert score indicates a degree of correlation between characteristics of the first data item cluster and the accessed data cluster analysis rules.

12. The computer system of claim 11, wherein the degree of correlation is based on both an assessment of risk associated with the first data item cluster and a confidence level in accuracy of the assessment of risk.

13. The computer system of claim 11, wherein a relatively higher alert score indicates a data item cluster that is relatively more important for a human analyst to evaluate, and a relatively lower alter score indicated a data item cluster that is relatively less important for the human analyst to evaluate.

14. The computer system of claim 11, wherein each alert score for respective data item clusters is assigned to a category indicating a high degree of correlation, a medium degree of correlation, or a low degree of correlation.

15. The computer system of claim 14, wherein the high degree of correlation is associated with a first color, the medium degree of correlation is associated with a second color, and the low degree of correlation is associated with a third color.

16. The computer system of claim 10, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

for the first data item cluster;

generate an alert, the alert comprising the alert score, the one or more human-readable conclusions, the data items associated with the first data item cluster, and metadata associated with the data items of the first data item cluster.

17. The computer system of claim 16, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

generate a user interface including a list of user-selectable alert indicators, an alert indicator being provided for each of a plurality of generated alerts, each of the alert indicators providing a summary of information associated with respective generated alerts.

18. The computer system of claim 17, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

in response to a selection, by a human analyst, of an alert indicator associated with the alert for the first data item cluster;

generate an alert display, the alert display including at least an indication of the alert score and a list of the one or more human-readable conclusions.

19. The computer system of claim 10, wherein the one or more human-readable conclusions each comprise a phrase or sentence including one or more indications of summary or aggregated data associated with a plurality of the data items of the first data item cluster.

20. The computer system of claim 19, wherein generating the one or more human-readable conclusions comprises:

selecting, based on the data cluster type associated with the first data item cluster, one or more conclusion templates; and

populating the one or more conclusion templates with data associated with the first data item cluster.

21. The computer system of claim 1, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

identify a second data item cluster having at least one network-related data item in common with the first data item cluster; and

merge the first data item cluster and the second data item cluster.

22. The computer system of claim 21, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

determine a type of the first data item cluster; and

determine a type of the second data item cluster,

wherein the first data item cluster and the second data item cluster are merged only if the type of the first data item cluster and the type of the second data item cluster are the same.

23. The computer system of claim 21, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

determine a first time associated with the intrusion detection system report associated with the first data item cluster;

determine a second time associated with an intrusion detection system report associated with the second data item cluster; and

determine a difference between the first time and the second time,

wherein the first data item cluster and the second data item cluster are merged only if the difference between the first time and the second time satisfies a threshold period of time.

24. The computer system of claim 1, wherein the computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

determine types of malicious network activity represented by the data item clusters;

determine numbers of the plurality of data item clusters associated with each type of malicious network activity; and

further provide the dynamic user interface displaying at least a second visualization indicating the numbers of the plurality of data item clusters associated with each type of malicious network activity.

25. A computer system comprising:

one or more non-transitory computer readable storage devices configured to store;

a plurality of computer executable instructions;

a data clustering strategy; and

a plurality of data items including at least;

intrusion detection system reports, each intrusion detection system report associated with at least a source Internet Protocol address and a destination Internet Protocol address; and

network-related data items associated with captured communications between an internal network and an external network, the network-related data items including at least one of;

external Internet Protocol addresses, external domains, external computerized devices, internal Internet Protocol addresses, internal computerized devices, users of particular computerized devices, intrusion detection system information, network firewall data, or WHOIS information; and

one or more hardware computer processors in communication with the one or more non-transitory computer readable storage devices and configured to execute the plurality of computer executable instructions to cause the computer system to;

receive an intrusion detection system report including a communication between a source Internet Protocol address and a destination Internet Protocol address;

initiate an automated lookup to determine which of the source Internet Protocol address and the destination Internet Protocol address is an external Internet Protocol address, the external Internet Protocol address being external to the internal network;

designate the external Internet Protocol address as a seed;

generate a first data item cluster based on the data clustering strategy by at least;

adding the seed to the first data item cluster;

identifying one or more of the network-related data items associated with the seed; and

adding, to the first data item cluster, the one or more identified network-related data items;

determine to regenerate the first data item cluster;

regenerate the first data item cluster by at least;

identifying one or more new network-related data items associated with at least one of;

the seed, or the one or more identified network-related data items, wherein the one or more new network-related data items were not added to the first data item cluster as initially generated; and

adding, to the first data item cluster, the one or more new network-related data items;

access a second data item cluster having at least one network-related data item in common with the first data item cluster;

determine a first time associated with the intrusion detection system report associated with the first data item cluster;

determine a second time associated with an intrusion detection system report associated with the second data item cluster;

determine a difference between the first time and the second time; and

in response to determining that the difference between the first time and the second time satisfies a threshold period of time, merge the first data item cluster and the second data item cluster.