Bandwidth metering in large-scale networks

• US 9,672,503 B2
• Filed: 05/21/2013
• Issued: 06/06/2017
• Est. Priority Date: 05/21/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Furr, Michael Brooke, Hendrie, Christopher Ian, Murphy, Ryan David, Miller, Kevin Christopher, Shantharaj, Sandeep
• Primary Examiner(s): Zeender, Ryan
• Assistant Examiner(s): Racic, Milena
• Application Number:
  US13/898,570
• Publication Number:
  US 20140351106A1
• Time in Patent Office: 1,477 Days
• Field of Search: 705 30, 705 34
• CPC Class Codes: G06Q 20/14 specially adapted for billing systems
• CPC Class Codes: G06Q 20/145 Payments according to the detected use or quantity
• CPC Class Codes: G06Q 30/04 Billing or invoicing
• CPC Class Codes: H04L 12/14 Charging , metering or billing arrangements for data wireline or wireless communications
• CPC Class Codes: H04L 12/1403 Architecture for metering, charging or billing
• CPC Class Codes: H04L 12/1407 Policy-and-charging control [PCC] architecture
• CPC Class Codes: H04L 12/1435 volume-based
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/122 of virtualised topologies, e.g. software-defined networks [SDN] or network function virtualisation [NFV]
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/20 the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
• CPC Class Codes: H04L 61/2514 between local and global IP addresses
• CPC Class Codes: H04M 15/31 Distributed metering or calculation of charges
• CPC Class Codes: H04M 15/41 Billing record details, i.e. parameters, identifiers, structure of call data record [CDR]
• CPC Class Codes: H04M 15/43 Billing software details
• CPC Class Codes: H04M 15/61 based on the service used
• CPC Class Codes: H04M 15/66 Policy and charging system
• CPC Class Codes: H04M 15/8214 Data or packet based
• CPC Class Codes: H04M 15/825 based on the number of used channels, e.g. bundling channels, frequencies or CDMA codes

Abstract

Methods and apparatus for bandwidth metering in large-scale networks are disclosed. Metadata for a network transmission involving a virtualized resource at a host of a provider network, including endpoint address information and a traffic metric, is determined at a metering component. The metadata is aggregated at another metering component and provided to a traffic classification node. The traffic classification node generates a categorized usage record for the network transmission, based at least in part on network topology information associated with the provider network. The categorized usage record is used to determine a billing amount for the network transmission.

Claims

1. A system, comprising:

a plurality of computing devices comprising one or more hardware processors and memory and configured to;

determine, at a first metering component on a host of a provider network comprising a plurality of hosts, networking metadata comprising (a) endpoint address information and (b) a traffic metric, wherein the networking metadata is associated with one or more network transmissions for which at least one endpoint comprises a virtualized resource instantiated at the host;

provide, by the first metering component to a second metering component on the host, at least a subset of the networking metadata determined at the first metering component;

aggregate at least the subset of the networking metadata from at least the first metering component at the second metering component in accordance with an on-host aggregation policy;

transmit, from the second metering component to a traffic classification node of the provider network, aggregated networking metadata based on the aggregation of at least the subset of the networking metadata;

generate, at the traffic classification node, a set of categorized usage records based at least in part on aggregated networking metadata obtained from at least a subset of the plurality of hosts and based at least in part on a representation of a network topology associated with the provider network, wherein the representation of the network topology indicates different networking configurations of the provider network that correspond to different time periods, and wherein a particular usage record of the set of categorized usage records indicates a particular billable usage category to be associated with the one or more network transmissions;

provide the set of categorized usage records from the traffic classification node to a billing node of the provider network;

determine, using the set of categorized usage records, a billing amount to be charged for the one or more network transmissions at the billing node; and

transmit the billing amount to a client device.

2. The system as recited in claim 1, wherein the first metering component comprises a kernel-mode component of a management software stack at the host, and wherein the second metering component comprises a user-mode component of the management software stack.

3. The system as recited in claim 1, wherein the one or more network transmissions comprise a set of network packets for which one endpoint comprises the virtualized resource instantiated at the host, wherein a second endpoint of the set has a particular Internet Protocol (IP) address, wherein the address endpoint information comprises the particular IP address, wherein the traffic metric comprises a number of bytes transmitted in the set of network packets, and wherein the networking metadata includes identification information of the virtualized resource distinguishing the virtualized resource from a different virtualized resource instantiated on the host.

4. The system as recited in claim 1, wherein the on-host aggregation policy comprises one or more of (a) a chunk size determination policy usable to determine an amount of aggregated networking metadata to be transmitted to the classification node, (b) a chunk scheduling policy usable to determine a schedule in accordance with which the aggregated networking metadata is to be transmitted to the classification node, (c) a classification node selection policy, (d) a compression policy for transmission of the aggregated networking metadata, (e) a grouping policy usable to combine networking metadata for a set of IP addresses prior to transmitting the aggregated networking metadata, or (f) a sampling policy usable to select a subset of the aggregated networking metadata to be transmitted to the classification node.

5. The system as recited in claim 1, wherein the plurality of computing devices are configured to:

collect, at one or more topology observer nodes of the provider network, network configuration information comprising routing information associated with at least a portion of the provider network; and

transmit the network configuration information and an associated timestamp to a topology authority node configured to, generate the representation of the network topology based at least in part on the routing information and the associated timestamp.

6. The system as recited in claim 1, wherein the particular billable usage category comprises at least one of:

(a) a local-provider-network category, (b) an inter-region-provider-network category (c) an extra-provider-network category, (d) a category associated with a particular multi-tenant service implemented at the provider network, (e) a category associated with a private network established within the provider network on behalf of a client, or (f) a category associated with a direct physical network link established at an edge node of the provider network to connect a client network with the provider network.

7. The system as recited in claim 1, wherein the plurality of computing devices are configured to:

determine whether networking metadata records are to be collected corresponding to each network transmission during a particular time period, based at least in part on a count of a number of distinct endpoint addresses associated with network transmissions detected during another time period; and

in response to a determination that networking metadata records corresponding to each network transmission are not to be collected, utilize a sampling methodology to generate one or more networking metadata records during the particular time period.

8. The system as recited in claim 1, wherein the representation of the network topology is time-indexed and is based at least in part on network configuration information, and wherein the network configuration information comprises one or more timestamps associated with a routing change.

9. A method, comprising:

determining, at a first metering component on a host of a provider network comprising a plurality of hosts, networking metadata comprising (a) endpoint address information and (b) a traffic metric, wherein the networking metadata is associated with one or more network transmissions for which at least one endpoint comprises a virtualized resource instantiated at the host;

providing, by the first metering component to a second metering component on the host, at least a subset of the networking metadata determined at the first metering component;

aggregating at least the subset of networking metadata from at least the first metering component at the second metering component in accordance with an on-host aggregation policy;

transmitting, from the second metering component to a traffic classification node of the provider network, aggregated networking metadata based on the aggregation of at least the subset of the networking metadata;

generating, at the traffic classification node of the provider network, one or more categorized usage records corresponding to the one or more network transmissions, based at least in part on the aggregated networking metadata obtained from at least the second metering component and based at least in part on a representation of a network topology associated with the provider network, wherein the representation of the network topology indicates different networking configurations of the provider network that correspond to different time periods;

providing the one or more categorized usage records from the traffic classification node to a billing node of the provider network;

determining, using the one or more categorized usage records, a billing amount to be charged for the one or more network transmissions at the billing node; and

transmitting the billing amount to a client device.

10. The method as recited in claim 9, wherein the first metering component comprises a kernel-mode component of a management software stack at the host, and wherein the second metering component comprises a user-mode component of the management software stack.

11. The method as recited in claim 9, wherein the one or more network transmissions comprise a set of network packets for which one endpoint comprises the virtualized resource instantiated at the host, wherein a second endpoint of the set has a particular Internet Protocol (IP) address, wherein the address endpoint information comprises the particular IP address, wherein the traffic metric comprises a number of bytes transmitted in the set of network packets, and wherein the networking metadata includes identification information of the virtualized resource distinguishing the virtualized resource from a different virtualized resource instantiated on the host.

12. The method as recited in claim 9, wherein the aggregation policy comprises one or more of (a) a chunk size determination policy usable to determine an amount of aggregated networking metadata to be transmitted to the classification node, (b) a chunk scheduling policy usable to determine a schedule in accordance with which the aggregated networking metadata is to be transmitted to the classification node, (c) a classification node selection policy, (d) a compression policy for transmission of the aggregated networking metadata, (e) a grouping policy usable to combine networking metadata for a set of IP addresses prior to transmitting the aggregated networking metadata, or (f) a sampling policy usable to select a subset of the aggregated networking metadata to be transmitted to the classification node.

13. The method as recited in claim 9, further comprising:

collecting, at one or more topology observer nodes of the provider network, networking configuration information associated with at least a portion of the provider network; and

transmitting the networking configuration information and an associated timestamp to a topology authority node configured to generate the representation of the network topology based at least in part on the networking configuration information and the associated timestamp.

14. The method as recited in claim 9, wherein the one or more categorized usage records include a particular usage record indicating a billable usage category of the one or more network transmissions, wherein the billable usage category comprises at least one of:

(a) a local-provider-network category, (b) an inter-region-provider-network category, (c) an extra-provider-network category, (d) a category associated with a particular multi-tenant service implemented at the provider network, (e) a category associated with a private network established within the provider network on behalf of a client, or (f) a category associated with a direct physical network link established at an edge node of the provider network to connect a client network with the provider network.

15. The method as recited in claim 9, further comprising:

determining whether networking metadata records are to be collected corresponding to each network transmission during a particular time period, based at least in part on a count of a number of distinct endpoint addresses associated with network transmissions detected during another time period; and

in response to a determination that networking metadata records corresponding to each network transmission are not to be collected, utilizing a sampling methodology to generate one or more networking metadata records during the particular time period.

16. The method as recited in claim 9, wherein the representation of the network topology is time-indexed and is based at least in part on network configuration information, and wherein the network configuration information comprises one or more timestamps associated with a routing change.