Streaming Method and System for Processing Network Metadata

• US 20140075557A1
• Filed: 03/14/2013
• Published: 03/13/2014
• Est. Priority Date: 09/11/2012
• Status: Abandoned Application

• Current Assignee: NetFlow Logic Corporation
• Original Assignee: NetFlow Logic Corporation
• Inventors: Velednitsky, Alexander, Balabine, Igor
• Application Number:
  US13/830,924
• Publication Number:
  US 20140075557A1
• Time in Patent Office: Days
• US Class Current: 726/23
• CPC Class Codes: G06F 21/55 Detecting local intrusion or implementing counter-measures
• CPC Class Codes: H04L 2463/144 Detection or countermeasures against botnets
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/18 Protocol analysers
• CPC Class Codes: H04L 47/2475 for supporting traffic characterised by the type of applications
• CPC Class Codes: H04L 63/102 Entity profiles
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

An improved method and system for processing network metadata is described. Network metadata may be processed by dynamically instantiated executable software modules which make policy-based decisions about the character of the network metadata and about presentation of the network metadata to consumers of the information carried by the network metadata. The network metadata may be type classified and each subclass within a type may be mapped to a definition by a unique fingerprint value. The fingerprint value may be used for matching the network metadata subclasses against relevant policies and transformation rules. For template-based network metadata such as NetFlow v9, an embodiment of the invention can constantly monitor network traffic for unknown templates, capture template definitions, and informs administrators about templates for which custom policies and conversion rules do not exist. Conversion modules can efficiently convert selected types and/or subclasses of network metadata into alternative metadata formats.

Claims

1. A method of improved management of a software-defined network, said network including a network controller and transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:

receiving network metadata from a plurality of sources in a data processing system, in at least one data format;

processing said network metadata while said network metadata is in transition on said network between a network device that generated said network metadata and a device that is able to store said network metadata to extract useful information therefrom; and

determining as a result of said metadata processing step, information relating to applications operating on said network; and

using said applications information to enable said network controller to perform more efficient management of said software-defined network.

2. The method as set forth in claim 1, further comprising the steps of:

determining as a result of said metadata processing step, information relating to the users present on said network; and

using said user information to enable said network controller to perform more efficient management of said software-defined network.

3. A method of improved management of a cloud-based virtual computing environment, said environment including a cloud operating system and a cloud environment controller and transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:

receiving network metadata from a plurality of sources in said cloud-based virtual computing environment, in at least one data format;

processing said network metadata while said network metadata is in transition in said environment between a network device that generated said network metadata and a device that is able to store said network metadata to extract useful information therefrom; and

determining as a result of said metadata processing step, information relating to applications operating in said environment; and

using said applications information to enable said cloud environment controller to perform more efficient management of said cloud-based virtual computing environment.

4. The method as set forth in claim 3, further comprising the steps of:

determining as a result of said metadata processing step, information relating to the users present in said environment; and

using said user information to enable said cloud environment controller to perform more efficient management of said cloud-based virtual computing environment.

5. A method of providing on-demand access to network metadata relating to an identified potentially security-related network event, in a network in which devices transmit network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:

processing network metadata in a streaming fashion according to a configured collection of network metadata processing policies;

retaining a time-indexed set of network metadata in a fast-access storage mechanism for a defined period of time;

identifying a potentially security-related network event; and

providing from said time-indexed set a collection of network metadata related in time to said identified potentially security-related network event; and

performing analysis to correlate said collection of network metadata with said identified potentially security-related network event to further characterize said identified potentially security-related network event.

6. The method as set forth in claim 5, further comprising the step of removing selected network metadata from said fast-access storage mechanism to facilitate the arrival of new network metadata thereto.

7. A method of detecting botnet slaves on a network-connected device comprising:

applying an inline cluster analysis algorithm to classify outbound traffic on a network;

said cluster analysis algorithm taking into consideration frequency of communications to the network hosts at identifiable geographic locations and data communication patterns such as, without limitation, application type, flow rate, the number of packets per flow, average packets size in each flow, and traffic rate;

based upon said applying step, identifying outbound traffic on said network that is not part of the general pattern of traffic on said network; and

communicating an alert in the event of outbound traffic on said network that is not part of the general pattern of traffic on said network.

8. A system for improved management of a software-defined network, said network including a network controller and transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface and generate network metadata relating to said network traffic, said management system comprising:

at least one ingress interface for receiving network metadata from a plurality of sources in a software-defined network, in at least one data format;

a processing engine for processing said network metadata while said network metadata is in transition on said network between a network device that generated said network metadata and a device that is able to store said network metadata to extract useful information therefrom;

said processing engine determining information relating to applications operating on said network and using said applications information to enable said network controller to perform more efficient management of said software-defined network.

9. The management system as set forth in claim 8, wherein said processing engine determines as a result of said metadata processing step, information relating to the users present on said network;

and uses said user information to enable said network controller to perform more efficient management of said software-defined network.

10. A system for improved management of a cloud-based virtual computing environment, said environment including a cloud operating system and a cloud environment controller that transmits network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the management system further comprising:

an interface for receiving network metadata from a plurality of sources in said cloud-based virtual computing environment, in at least one data format;

a processing engine for processing said network metadata while said network metadata is in transition in said environment between a network device that generated said network metadata and a device that is able to store said network metadata to extract useful information therefrom;

said processing engine determining as a result of said metadata processing step, information relating to applications operating in said environment; and

using said applications information to enable said cloud environment controller to perform more efficient management of said cloud-based virtual computing environment.

11. The management system as set forth in claim 10, wherein said processing engine determines as a result of said metadata processing step, information relating to the users present in said environment;

and uses said user information to enable said cloud environment controller to perform more efficient management of said cloud-based virtual computing environment.

12. A system for providing on-demand access to network metadata relating to an identified potentially security-related network event, in a network in which devices transmit network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the system comprising:

a processing engine for processing network metadata in a streaming fashion according to a configured collection of network metadata processing policies;

a fast-access storage mechanism for retaining a time-indexed set of network metadata for a defined period of time;

said processing engine identifying a potentially security-related network event and providing from said time-indexed set a collection of network metadata related in time to said identified potentially security-related network event; and

an analysis engine for performing analysis to correlate said collection of network metadata with said identified potentially security-related network event to further characterize said identified potentially security-related network event.

13. The system as set forth in claim 12, further comprising a memory management engine for removing selected network metadata from said fast-access storage mechanism to facilitate the arrival of new network metadata thereto.

14. A system for detecting botnet slaves on a network-connected device comprising:

a processing engine for applying an inline cluster analysis algorithm to classify outbound traffic on a network;

said cluster analysis algorithm taking into consideration frequency of communications to the network hosts at identifiable geographic locations and data communication patterns such as, without limitation, application type, flow rate, the number of packets per flow, average packets size in each flow, and traffic rate;

an analysis engine that, based upon the results of said cluster analysis algorithm, identifies outbound traffic on said network that is not part of the general pattern of traffic on said network; and

an alert generation engine for communicating an alert in the event of outbound traffic on said network that is not part of the general pattern of traffic on said network.