BANDWIDTH METERING IN LARGE-SCALE NETWORKS

• US 20140351106A1
• Filed: 05/21/2013
• Published: 11/27/2014
• Est. Priority Date: 05/21/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: FURR, MICHAEL BROOKE, HENDRIE, CHRISTOPHER IAN, MILLER, KEVIN CHRISTOPHER, MURPHY, RYAN DAVID, SHANTHARAJ, SANDEEP
• Granted Patent: US 9,672,503 B2
• Time in Patent Office: Days
• US Class Current: 705/34
• CPC Class Codes: G06Q 20/14 specially adapted for billing systems
• CPC Class Codes: G06Q 20/145 Payments according to the detected use or quantity
• CPC Class Codes: G06Q 30/04 Billing or invoicing
• CPC Class Codes: H04L 12/14 Charging , metering or billing arrangements for data wireline or wireless communications
• CPC Class Codes: H04L 12/1403 Architecture for metering, charging or billing
• CPC Class Codes: H04L 12/1407 Policy-and-charging control [PCC] architecture
• CPC Class Codes: H04L 12/1435 volume-based
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/122 of virtualised topologies, e.g. software-defined networks [SDN] or network function virtualisation [NFV]
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 61/2514 between local and global IP addresses
• CPC Class Codes: H04M 15/31 Distributed metering or calculation of charges
• CPC Class Codes: H04M 15/41 Billing record details, i.e. parameters, identifiers, structure of call data record [CDR]
• CPC Class Codes: H04M 15/43 Billing software details
• CPC Class Codes: H04M 15/61 based on the service used
• CPC Class Codes: H04M 15/66 Policy and charging system
• CPC Class Codes: H04M 15/8214 Data or packet based
• CPC Class Codes: H04M 15/825 based on the number of used channels, e.g. bundling channels, frequencies or CDMA codes

Abstract

Methods and apparatus for bandwidth metering in large-scale networks are disclosed. Metadata for a network transmission involving a virtualized resource at a host of a provider network, including endpoint address information and a traffic metric, is determined at a metering component. The metadata is aggregated at another metering component and provided to a traffic classification node. The traffic classification node generates a categorized usage record for the network transmission, based at least in part on network topology information associated with the provider network. The categorized usage record is used to determine a billing amount for the network transmission.

Claims

1. A system, comprising a plurality of computing devices configured to:

determine, at a first metering component on a host of a provider network comprising a plurality of hosts, networking metadata comprising (a) endpoint address information and (b) a traffic metric, wherein the networking metadata is associated with one or more network transmissions for which at least one endpoint comprises a virtualized resource instantiated at the host;

provide, by the first metering component to a second metering component on the host, at least a subset of the networking metadata determined at the first metering component;

aggregate networking metadata from at least the first metering component at the second metering component in accordance with an on-host aggregation policy;

transmit, from the second metering component to a traffic classification node of the provider network, aggregated networking metadata;

generate, at the traffic classification node, a set of categorized usage records based at least in part on aggregated networking metadata obtained from at least a subset of the plurality of hosts and based at least in part on a representation of a network topology associated with the provider network, wherein a particular usage record of the set of categorized usage records indicates a particular billable usage category to be associated with the one or more network transmissions;

provide the set of categorized usage records from the traffic classification node to a billing node of the provider network; and

determine, using the set of categorized usage records, a billing amount to be charged for the one or more network transmissions at the billing node.

2. The system as recited in claim 1, wherein the first metering component comprises a kernel-mode component of a management software stack at the host, and wherein the second metering component comprises a user-mode component of the management software stack.

3. The system as recited in claim 1, wherein the one or more network transmissions comprise a set of network packets for which one endpoint comprises the virtualized resource instantiated at the host, wherein a second endpoint of the set has a particular Internet Protocol (IP) address, wherein the address endpoint information comprises the particular IP address, wherein the traffic metric comprises a number of bytes transmitted in the set of network packets, and wherein the networking metadata includes identification information of the virtualized resource distinguishing the virtualized resource from a different virtualized resource instantiated on the host.

4. The system as recited in claim 1, wherein the on-host aggregation policy comprises one or more of (a) a chunk size determination policy usable to determine an amount of aggregated networking metadata to be transmitted to the classification node, (b) a chunk scheduling policy usable to determine a schedule in accordance with which the aggregated networking metadata is to be transmitted to the classification node, (c) a classification node selection policy, (d) a compression policy for transmission of the aggregated networking metadata, (e) a grouping policy usable to combine networking metadata for a set of IP addresses prior to transmitting the aggregated networking metadata, or (f) a sampling policy usable to select a subset of the aggregated networking metadata to be transmitted to the classification node.

5. The system as recited in claim 1, wherein the plurality of computing devices are configured to:

collect, at one or more topology observer nodes of the provider network, network configuration information comprising routing information associated with at least a portion of the provider network; and

transmit the network configuration information and an associated timestamp to a topology authority node configured to, generate the representation of the network topology based at least in part on the routing information and the associated timestamp.

6. The system as recited in claim 1, wherein the particular billable usage category comprises at least one of:

(a) a local-provider-network category, (b) an inter-region-provider-network category (c) an extra-provider-network category, (d) a category associated with a particular multi-tenant service implemented at the provider network, (e) a category associated with a private network established within the provider network on behalf of a client, or (f) a category associated with a direct physical network link established at an edge node of the provider network to connect a client network with the provider network.

7. A method, comprising:

determining, at a first metering component on a host of a provider network comprising a plurality of hosts, networking metadata comprising (a) endpoint address information and (b) a traffic metric, wherein the networking metadata is associated with one or more network transmissions for which at least one endpoint comprises a virtualized resource instantiated at the host;

aggregating networking metadata from at least the first metering component at a second metering component in accordance with an aggregation policy;

generating, at a traffic classification node of the provider network, one or more categorized usage records corresponding to the one or more network transmissions, based at least in part on aggregated networking metadata obtained from at least the second metering component and based at least in part on a representation of a network topology associated with the provider network; and

determining, using the one or more categorized usage records, a billing amount to be charged for the one or more network transmissions.

8. The method as recited in claim 7, wherein the first metering component comprises a kernel-mode component of a management software stack at the host, and wherein the second metering component comprises a user-mode component of the management software stack.

9. The method as recited in claim 7, wherein the one or more network transmissions comprise a set of network packets for which one endpoint comprises the virtualized resource instantiated at the host, wherein a second endpoint of the set has a particular Internet Protocol (IP) address, wherein the address endpoint information comprises the particular IP address, wherein the traffic metric comprises a number of bytes transmitted in the set of network packets, and wherein the networking metadata includes identification information of the virtualized resource distinguishing the virtualized resource from a different virtualized resource instantiated on the host.

10. The method as recited in claim 7, wherein the aggregation policy comprises one or more of (a) a chunk size determination policy usable to determine an amount of aggregated networking metadata to be transmitted to the classification node, (b) a chunk scheduling policy usable to determine a schedule in accordance with which the aggregated networking metadata is to be transmitted to the classification node, (c) a classification node selection policy, (d) a compression policy for transmission of the aggregated networking metadata, (e) a grouping policy usable to combine networking metadata for a set of IP addresses prior to transmitting the aggregated networking metadata, or (f) a sampling policy usable to select a subset of the aggregated networking metadata to be transmitted to the classification node.

11. The method as recited in claim 7, further comprising:

collecting, at one or more topology observer nodes of the provider network, networking configuration information associated with at least a portion of the provider network; and

transmitting the networking configuration information and an associated timestamp to a topology authority node configured to generate the representation of the network topology based at least in part on the networking configuration information and the associated timestamp.

12. The method as recited in claim 7, wherein the one or more categorized usage records include a particular usage record indicating a billable usage category of the one or more network transmissions, wherein the billable usage category comprises at least one of:

(a) a local-provider-network category, (b) an inter-region-provider-network category, (c) an extra-provider-network category, (d) a category associated with a particular multi-tenant service implemented at the provider network, (e) a category associated with a private network established within the provider network on behalf of a client, or (f) a category associated with a direct physical network link established at an edge node of the provider network to connect a client network with the provider network.

13. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:

generate a plurality of networking metadata records at a host of a provider network, wherein a particular networking metadata record of the plurality of networking metadata records corresponds to one or more network transmissions detected at the host, wherein the particular networking metadata record comprises (a) endpoint address information of the one or more network transmissions and (b) a traffic metric;

aggregate the plurality of networking metadata at the host based at least in part on the endpoint address information; and

transmit aggregated networking metadata from the host to a traffic classification node of the provider network, wherein the traffic classification node is configured to generate a categorized usage record corresponding to the one or more network transmissions based at least in part on a representation of a network topology associated with the provider network.

14. The non-transitory computer-accessible storage medium as recited in claim 13, wherein the program instructions when executed on the one or more processors communicate with a kernel-mode component executing at the host to generate the particular networking metadata record.

15. The non-transitory computer-accessible storage medium as recited in claim 13, wherein the one or more network transmissions comprise a set of network packets for which one endpoint comprises a virtualized resource instantiated at the host, wherein a second endpoint of the set has a particular Internet Protocol (IP) address, wherein the address endpoint information comprises the particular IP address, wherein the traffic metric comprises a number of bytes transmitted in the set of network packets, and wherein the particular networking metadata record includes identification information of the virtualized resource distinguishing the virtualized resource from a different virtualized resource instantiated on the host.

16. The non-transitory computer-accessible storage medium as recited in claim 13, wherein the program instructions when executed on the one or more processors aggregate the plurality of networking metadata records based at least in part on an aggregation policy comprising one or more of (a) a chunk size determination policy usable to determine an amount of aggregated networking metadata to be transmitted to the classification node, (b) a chunk scheduling policy usable to determine a schedule in accordance with which the aggregated networking metadata is to be transmitted to the classification node, (c) a classification node selection policy, (d) a compression policy for transmission of the aggregated networking metadata, (e) a grouping policy usable to combine networking metadata for a set of IP addresses prior to transmitting the aggregated networking metadata, or (f) a sampling policy usable to select a subset of the aggregated networking metadata to be transmitted to the classification node.

17. The non-transitory computer-accessible storage medium as recited in claim 13, wherein the program instructions when executed on the one or more processors:

determine whether networking metadata records are to be collected corresponding to each network transmission during a particular time period, based at least in part on a count of a number of distinct endpoint addresses associated with network transmissions detected during another time period; and

in response to a determination that networking metadata records corresponding to each network transmission are not to be collected, utilize a sampling methodology to generate one or more networking metadata records during the particular time period.

18. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:

obtain a representation of a network topology corresponding to at least a portion of a provider network, wherein the representation is generated by a topology authority of the provider network and is based at least in part on network configuration information collected by one or more topology observer nodes of the provider network;

receive, from metering components at one or more virtualization hosts of the provider network, a plurality of networking metadata records associated with network transmissions detected at the one or more virtualization hosts, wherein a particular networking metadata record of the plurality of networking metadata records comprises endpoint address information and a traffic metric;

generate a set of categorized usage records based at least in part on the plurality of networking metadata records and based at least in part on the representation of the network topology, wherein the set of categorized usage records is usable for determining billing amounts associated with the network transmissions detected at the one or more virtualization hosts.

19. The non-transitory computer-accessible storage medium as recited in claim 18, wherein the network configuration information comprises one or more timestamps associated with a routing change, and wherein the representation of the network topology is time-indexed.

20. The non-transitory computer-accessible storage medium as recited in claim 18, wherein a particular categorized usage record of the set comprises an indication of a billable usage category comprising at least one of:

(a) a local-provider-network category, (b) an inter-region-provider-network category, (c) an extra-provider-network category, (d) a category associated with a particular multi-tenant service implemented at the provider network, (e) a category associated with a private network established within the provider network on behalf of a client, or (f) a category associated with a direct physical network link established at an edge node of the provider network to connect a client network with the provider network.