Aggregation of network traffic source behavior data across network-based endpoints

• US 9,591,018 B1
• Filed: 11/20/2014
• Issued: 03/07/2017
• Est. Priority Date: 11/20/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Zakian, Christopher Samuel, Smith, Patrick Devere
• Primary Examiner(s): Simitoski, Michael
• Application Number:
  US14/549,432
• Time in Patent Office: 838 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 2221/2145 Inheriting rights or properties, e.g., propagation of permissions or restrictions within a hierarchy
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

Aggregation of network traffic source behavior data across network endpoints may be implemented. Indications of endpoint-specific network traffic directed to different network endpoints may be received. Aggregate traffic source behavior data may be generated across multiple aggregation levels. One or more traffic aggregation nodes may be implemented for each aggregation level to maintain different respective portions of the aggregate traffic source behavior data. Different granularity of the aggregate traffic source behavior data may be maintained at each of the aggregation levels. An indication of traffic source behavior for traffic sources may be provided such that responsive actions, such as traffic control actions, may be performed with regard to the traffic sources.

Claims

1. A network traffic analysis system, comprising:

a plurality of traffic behavior aggregation nodes that implement a plurality of different aggregation levels for traffic source behavior data for a network traffic analysis system, the plurality of traffic behavior aggregation nodes implemented via one or more computers comprising one or more hardware processors and configured to;

receive respective indications of endpoint-specific network traffic directed to different ones of a plurality of network endpoints from a plurality of traffic sources;

based, at least in part, on the respective indications of the endpoint-specific network traffic from the plurality of traffic sources, generate aggregate traffic source behavior data that is maintained across the plurality of different aggregation levels, wherein a different respective granularity of the aggregate traffic source behavior data is maintained at the plurality of different aggregation levels, wherein the plurality of traffic behavior aggregation nodes maintain different respective portions of the aggregate traffic source behavior data according to the different respective granularity of the plurality of different aggregation levels;

a control plane for the network traffic analysis system, configured to;

identify traffic behavior for a particular traffic source of the plurality of traffic sources based, at least in part, on the aggregate traffic source behavior data at one or more of the plurality of different aggregation levels, wherein to identify the traffic behavior for the particular traffic source of the plurality of traffic sources, the control plane is configured to;

in response to a received request for traffic behavior data;

identify one or more aggregation levels that provide a respective granularity of the aggregate traffic source behavior data that includes the requested traffic behavior data;

identify at least one of the one or more traffic behavior aggregation nodes of the identified one or more aggregation levels to query for the traffic behavior data; and

send a query to the identified at least one traffic behavior aggregation node to obtain the traffic behavior data; and

provide an indication of the identified traffic behavior of the particular traffic source such that a traffic control action is performed with regard to the particular traffic source for one or more network endpoints of the plurality of endpoints, wherein the particular traffic source did not direct endpoint-specific network traffic to at least one of the one or more network endpoints.

2. The system of claim 1, wherein to generate the aggregate traffic source behavior data, at least one of the plurality of traffic behavior aggregation nodes is configured to:

receive traffic source behavior data according to the respective granularity for a particular aggregation level for the at least one traffic behavior aggregation node from another one of the plurality of traffic behavior aggregation nodes;

analyze the traffic source behavior data according to the respective granularity for another aggregation level that is different than the particular aggregation level; and

based, at least in part, on the analysis of the traffic source behavior data, send an aggregated version of the traffic source behavior data to a traffic behavior aggregation node that maintains a respective portion of the aggregate traffic source behavior data as part of the other aggregation level.

3. The system of claim 1, wherein the network traffic analysis system is implemented as part of a provider network, wherein the provider network implements one or more network-based services, wherein the plurality of network endpoints receive network traffic for different resources implemented at the one or more network-based services of the provider network.

4. The system of claim 3, wherein the one or more network-based services of the provider network is a virtual computing service, wherein the different resources are implemented at a plurality of virtual computing hosts for the virtual computing service, and wherein the virtual computing hosts for the resources of the one or more network endpoints perform the traffic control action in response to a determination that the indicated behavior of the particular traffic source violates respective traffic control policies for the resources.

5. A method, comprising:

performing, by one or more computing devices comprising one or more hardware processors;

receiving respective indications of endpoint-specific network traffic directed to different ones of a plurality of network endpoints from a plurality of traffic sources;

based, at least in part, on the respective indications of the endpoint-specific network traffic from the plurality of traffic sources, generating aggregate traffic source behavior data that is maintained across a plurality of different aggregation levels, wherein a different respective granularity of the aggregate traffic source behavior data is maintained at the plurality of different aggregation levels, wherein the plurality of different aggregation levels respectively comprise one or more traffic behavior aggregation nodes that maintain different portions of the aggregate traffic source behavior data according to the respective granularity of the aggregation level; and

based, at least in part, on the aggregate traffic source behavior data at one or more of the plurality of different aggregation levels, providing a traffic behavior indication for one or more traffic sources of the plurality of traffic sources such that a responsive action is performed with regard to at least one of the one or more traffic sources for one or more network endpoints of the plurality of endpoints, wherein providing the traffic behavior indication for the one or more traffic sources of the plurality of traffic sources comprises;

in response to receiving a request for traffic behavior data;

identifying one or more aggregation levels that provide a respective granularity of the aggregate traffic source behavior data that includes the requested traffic behavior data;

identifying at least one of the one or more traffic behavior aggregation nodes of the identified one or more aggregation levels to query for the traffic behavior data; and

sending a query to the identified at least one traffic behavior aggregation nodes to obtain the traffic behavior data.

6. The method of claim 5, wherein providing the traffic behavior indication for the one or more traffic sources of the plurality of traffic sources further comprises:

in response to receiving the request for traffic behavior data;

identifying the at least one of the one or more traffic behavior aggregation nodes of the identified one or more aggregation levels to query for the traffic behavior data according to a respective distribution schema for portions of the aggregate traffic source behavior data maintained at the identified one or more aggregation levels;

and

wherein the traffic behavior indication for the one or more traffic sources is based, at least in part, on the obtained traffic behavior data in a response to the query.

7. The method of claim 6, wherein the request for traffic behavior data pertains to network traffic directed to a particular network endpoint of the plurality of endpoints.

8. The method of claim 5, wherein the one or more computing devices implement a network traffic analysis system for a provider network, wherein the provider network implements one or more network-based services, wherein at least some of the plurality of network endpoints receive network traffic for different resources implemented at the one or more network-based services of the provider network.

9. The method of claim 8, wherein at least one of the one or more network endpoints that is provided with the traffic behavior indication for the one or more traffic sources receives network traffic for a resource that is external to the provider network.

10. The method of claim 5, wherein at least some other ones of the plurality of network endpoints receive network traffic for different resources that are external to the provider network.

11. The method of claim 5, wherein providing the traffic behavior indication for the one or more traffic sources comprises publishing the traffic behavior indication for the one or more traffic sources to a location for consumption by respective network traffic agents for the plurality of network endpoints.

12. The method of claim 5, wherein the responsive action performed with regard to the at least one traffic source initiates a different process for handling network traffic from the at least one traffic source.

13. The method of claim 5, wherein the generating aggregate traffic source behavior data comprises forwarding traffic behavior data from an aggregation node of a particular aggregation level to another aggregation node of the same aggregation level of the plurality of different aggregation levels.

14. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:

receiving respective indications of endpoint-specific network traffic directed to different ones of a plurality of network endpoints from a plurality of traffic sources;

based, at least in part, on the respective indications of the endpoint-specific network traffic from the plurality of traffic sources, generating aggregate traffic source behavior data that is maintained across a plurality of different aggregation levels, wherein a different respective granularity of the aggregate traffic source behavior data is maintained at the plurality of different aggregation levels, wherein the plurality of different aggregation levels respectively comprise one or more traffic behavior aggregation nodes that maintain different portions of the aggregate traffic source behavior data according to the respective granularity of the aggregation level; and

based, at least in part, on the aggregate traffic source behavior data at one or more of the plurality of different aggregation levels, providing a traffic behavior indication for one or more traffic sources of the plurality of traffic sources such that a traffic control action is performed with regard to at least one of the one or more traffic sources for one or more network endpoints of the plurality of endpoints, wherein the at least one traffic source did not direct endpoint-specific network traffic to at least one of the one or more network endpoints, and wherein providing the traffic behavior indication for the one or more traffic sources of the plurality of traffic sources comprises;

in response to receiving a request for traffic behavior data;

identifying one or more aggregation levels that provide a respective granularity of the aggregate traffic source behavior data that includes the requested traffic behavior data;

identifying at least one of the one or more traffic behavior aggregation nodes of the identified one or more aggregation levels to query for the traffic behavior data; and

sending a query to the identified at least one traffic behavior aggregation nodes to obtain the traffic behavior data.

15. The non-transitory, computer-readable storage medium of claim 14, wherein, in generating aggregate traffic source behavior data that is maintained across a plurality of different aggregation levels, the program instructions cause the one or more computing devices to implement:

for each of different ones of the plurality of traffic behavior aggregation nodes;

receiving, at the traffic behavior aggregation node, traffic source behavior data according to the respective granularity for a particular aggregation level for the traffic behavior aggregation node from another one of the plurality of traffic behavior aggregation nodes;

analyzing, at the traffic behavior aggregation node, the traffic source behavior data according to the respective granularity for another aggregation level that is different than the particular aggregation level; and

based, at least in part, on the analyzing of the traffic source behavior data, sending an aggregated version of the traffic source behavior data to a traffic behavior aggregation node that maintains a respective portion of the aggregate traffic source behavior data as part of the other aggregation level.

16. The non-transitory, computer-readable storage medium of claim 14, wherein, in providing the traffic behavior indication for the one or more traffic sources of the plurality of traffic sources, the program instructions cause the one or more computing devices to implement:

in response to receiving the request for traffic behavior data;

identifying the at least one of the one or more traffic behavior aggregation nodes of the identified one or more aggregation levels to query for the traffic behavior data according to a respective distribution schema for portions of the aggregate traffic source behavior data maintained at the identified one or more aggregation levels;

wherein the traffic behavior indication for the one or more traffic sources is based, at least in part, on the obtained traffic behavior data in a response to the query.

17. The non-transitory, computer-readable storage medium of claim 14, wherein the one or more computing devices implement a network traffic analysis system for a provider network, wherein the provider network implements one or more network-based services, wherein at least some of the plurality of network endpoints receive network traffic for different resources implemented at the one or more network-based services of the provider network.

18. The non-transitory, computer-readable storage medium of claim 17, wherein at least one of the one or more network endpoints that is provided with the traffic behavior indication for the one or more traffic sources receives network traffic for a resource that is external to the provider network.

19. The non-transitory, computer-readable storage medium of claim 17, wherein the one or more network-based services of the provider network is a virtual computing service, wherein the different resources are implemented at a plurality of virtual computing hosts for the virtual computing service, and wherein the virtual computing hosts for the resources of the one or more network endpoints perform the traffic control action in response to a determination that the indicated behavior of the particular traffic source violates respective traffic control policies for the resources.

20. The non-transitory, computer-readable storage medium of claim 14, wherein the responsive action performed with regard to the at least one traffic source blocks subsequent network traffic from the at least one traffic source to the one or more network endpoints.