System and method for network data characterization

• US 9,832,216 B2
• Filed: 11/16/2015
• Issued: 11/28/2017
• Est. Priority Date: 11/21/2014
• Status: Active Grant

• Current Assignee: BluVector, Inc. (Comcast Corporation)
• Original Assignee: BluVector, Inc. (Comcast Corporation)
• Inventors: Kaloroumakis, Peter E., Boyle, Michael P., Valentino, Christopher C.
• Primary Examiner(s): Korsak, Oleg
• Application Number:
  US14/941,999
• Publication Number:
  US 20160149943A1
• Time in Patent Office: 743 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 63/0263 Rule management
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

Described herein are embodiments of a system and method for network data characterization and/or classification that overcome the defects of the prior art. These and other advantages are achieved by a method for network data characterization. The method includes receiving network events, wherein at least some of the events include content, converting the received content into data messages, in which at least some of the data messages include the content and metadata describing an event and the content, routing the data messages to a plurality of analyzers based on specified message criteria to which each analyzer subscribes, each of one or more analyzers that received the routed data messages analyzing the content within the data messages in order to characterize the content, in which the one or more analyzers include at least one machine-learning analyzer that classifies the content with a confidence percentage that indicates the probability that the content is malign or the confidence that a prediction that the content is malign is correct, outputting the characterization results of the one or more analyzers, and comparing the output characterization results against a plurality of criteria to determine subsequent action to take based on the characterization results.

Claims

1. A computer-implemented method for network data characterization, comprising:

receiving a network event and related content;

creating a data message comprising the content and metadata describing the content and the related event;

routing the data message to a plurality of analyzers based on the metadata and one or more subscriptions associated with the plurality of analyzers, wherein the plurality of analyzers include at least one machine-learning analyzer;

receiving the routed data message and analyzing the content therein to generate characterization results comprising a content classification and an associated confidence percentage;

outputting the characterization results of the plurality of analyzers; and

comparing the output characterization results against a plurality of criteria to determine subsequent action to take based on the characterization results, wherein;

the characterization results include metadata describing one or more of the following;

at least one of the plurality of analyzers, the classified content, and the characterization results; and

the subscriptions associated with one of the plurality of analyzers includes at least one message type not included in the subscriptions associated with a different one of the plurality of analyzers.

2. The method of claim 1 wherein receiving network events includes receiving network data from one or more network data feeds, including HTTP, FTP and SMTP data feeds.

3. The method of claim 2 further comprising re-assembling the network data into events and content.

4. The method of claim 1 wherein receiving a network events includes receiving content from a command line interface or graphical user interface upload.

5. The method of claim 1 wherein the associated confidence percentage indicates the probability that the content is malign or the confidence that a prediction that the content is malign is correct.

6. The method of claim 1 wherein outputting includes combining the characterization results of the one or more analyzers.

7. The method of claim 1 wherein outputting includes tagging the classified content with tags indicating that the content is malicious, suspicious, benign, unknown or needs further review.

8. The method of claim 7 wherein outputting includes storing the classified, tagged content on a disk.

9. The method of claim 1 wherein outputting includes storing the characterization metadata in a database.

10. The method of claim 1 wherein the subsequent action includes storing the classified content and characterization metadata, modifying network traffic, or forwarding said classified content to external analyzers.

11. The method of claim 1 wherein the one or more subscriptions include at least one of file type, event source, or event destination.

12. The method of claim 1 wherein the machine-learning analyzer utilizes feature-based processing to determine the confidence percentage based on a comparison of features of the content and a classifier derived from features of known malicious and known benign files.

13. The method of claim 1 wherein the routing is further based on resource allocation factors.

14. The method of claim 1 further comprising generating a visualization of the characterization results.

15. The method of claim 14 wherein the visualization is generated based on one or more criteria or thresholds pertaining to the number of events classified as suspicious and requiring the user'"'"'s attention.

16. The method of claim 15, wherein the one or more criteria or thresholds are user-configurable.

17. The method of claim 16 further comprising presenting a prediction of future characterization results based on the change to the one or more threshold levels or other criteria.

18. A sensor for network data characterization, comprising:

one or more processors; and

at least one memory including instructions for execution by the one or more processors that configure the sensor to;

receive a network event and related content;

create a data message comprising the content and metadata describing the content and the related event;

route the data message to a plurality of analyzers based on the metadata and one or more subscriptions associated with the plurality of analyzers, wherein the plurality of analyzers include at least one machine-learning analyzer;

receive the routed data message and analyze the content therein to generate characterization results comprising a content classification and an associated confidence percentage;

output the characterization results of the plurality of analyzers; and

compare the output characterization results against a plurality of criteria to determine subsequent action to take based on the characterization results, wherein;

the characterization results include metadata describing one or more of the following;

at least one of the plurality of analyzers, the classified content, and the characterization results; and

the subscriptions associated with one of the plurality of analyzers includes at least one message type not included in the subscriptions associated with a different one of the plurality of analyzers.

19. The sensor of claim 18 wherein the associated confidence percentage that indicates the probability that the content is malign or the confidence that a prediction that the content is malign is correct.

20. The sensor of claim 18 further comprising one or more interfaces connected to an external network external to an enterprise hosting the one or more characterization sensors and an internal network that is an enterprise internal network.