Distributed metering and monitoring system

• US 9,251,481 B2
• Filed: 06/13/2011
• Issued: 02/02/2016
• Est. Priority Date: 06/13/2011
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Richter, Owen E., Lundell, Lukas M., Portell, Charles Michael, Walker, Bryan M., Parthasarathy, Sonali
• Primary Examiner(s): Joo, Joshua
• Application Number:
  US13/159,042
• Publication Number:
  US 20120317274A1
• Time in Patent Office: 1,695 Days
• Field of Search: 709220-224
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal

Abstract

The distributed metering and monitoring service (DMMS) system provides a way to gather and maintain metrics data which remains distributed, until requested. The DMMS system uses messaging queues to scale the number of servers that may be monitored and metered to a hyperscale of greater than 10,000 servers. The DMMS system determines how many servers (nodes) to assign to a cluster, and uses a metric aggregator to collect and store metrics data for the nodes. The DMMS system creates message queues for the instances, injects instance identifiers into the cluster state data and metrics data, listens for request messages for metering information for instances, retrieves the metrics data for users identified by the instance identifiers stored locally at the nodes, and calculates the metering information for the instance.

Claims

1. A distributed metering and monitoring service (DMMS) system, the DMMS system comprising:

a first memory coupled to a first processor, wherein the first memory comprises;

messaging queue logic executable by the first processor that when executed causes the first processor to;

receive a request message;

route the request message;

receive a reply message; and

route the reply message;

logic executable by the first processor that when executed causes the first processor to;

receive a request message comprising DMMS parameters for which to collect metrics data, and calculate metering information;

determine a number of nodes and node identifiers of the nodes networked in communication via a network, each node comprising;

instances responsive to respective user interactions through a user interface, including instances for users of the instances, including a first instance for a first user of the first instance;

collect the metrics data for the nodes, including metrics data for a first node, including the first instance; and

generate the reply message comprising the metering information, including metering information for the first user of the first instance;

the nodes of the number of nodes, wherein each of the nodes comprise;

a second memory coupled to a second processor, wherein the second memory comprises;

cluster state data and the metrics data for the node identifiers;

a hypervisor operable to manage the instances, and instance state information stored in the second memory and executed by the second processor, wherein the hypervisor assigns each of the instances, including the first instance, to a node from the number of nodes, including the first node;

metering logic executable by the second processor that when executed causes the second processor to;

 

generate the reply message comprising the metering information for the instances assigned to the nodes;

 

inject, for each node, an instance identifier and/or the instance state information into the cluster state data and metrics data; and

 

register, according to a configurable frequency threshold, each of the instances with the messaging queue logic by using the instance identifier to create a message queue for each instance.

2. The system of claim 1,

wherein the hypervisor assigns the instance identifier that identifies the node assigned to each instance by a node id, and wherein the instances consume networked resources in the network;

and

wherein the metering logic is further executable by the second processor to retrieve instance state information from the hypervisor for the instances identified by the instance identifier.

3. The system of claim 2, wherein the DMMS system further comprises:

a communications interface in communications with a virtual machine controller system, comprising;

a third memory coupled to a third processor, wherein the third memory comprises;

information regarding a link between the instance identifier and user identifiers, including a first user identifier for the first user.

4. The system of claim 2, wherein the metering logic is further operable to cause the second processor to:

determine, for each node, the instances assigned to the node using the instance identifier, including the first instance;

listen for receipt of the request message, and when the request message is received;

retrieve the metrics data from the second memory for the instance identifier;

calculate the metering information, including the metering information for the first instance;

request instance information from the hypervisor regarding the instances.

5. The system of claim 4, wherein the frequency threshold to register each of the instances with the messaging queue logic is calculated based on:

a calculated average frequency at which users including the first user cause the nodes to generate the instances;

an average the duration of the instances; and

a performance capacity of the nodes.

6. The system of claim 4, wherein the metering information, for each instance, comprises:

storage utilization including duration of storage utilization, frequency of access to storage utilized, and size of retrieval and storage requests;

compute utilization including duration of use of networked resources; and

infrastructure utilization including duration of use of an amount of bandwidth of network utilized.

7. The system of claim 4, wherein the request message comprises a request for metering information for the first instance, and when the request message is received the metering logic calculates the metering information by determining first instance metrics data from the metrics data for the first instance.

8. The system of claim 4, wherein the metrics data includes discrete events metrics data and continuous events metrics data that identify discrete events and continuous events, respectively, wherein the metrics data comprises:

storage metrics;

compute metrics; and

infrastructure metrics.

9. The system of claim 8, wherein the instances consume networked resources consumed during discrete events and continuous events that occur in the network responsive to the user interactions through the user interface, and wherein the cluster state data and the metrics data, include data that records the discrete events and the continuous events.

10. The system of claim 1, wherein the DMMS logic stored in memory that calculates the metering information for each of the users, including the first user of the first instance by further causing the first processor to:

identify for a continuous event metric a start time of interest and end time of interest for which to retrieve and/or collect continuous event metrics data.

11. The system of claim 10, wherein the DMMS logic stored in memory that associates the continuous event metrics data with discrete event metrics data, by further causing the first processor to:

identify a subset of instances corresponding to the continuous event metric start time of interest and end time of interest;

query the message queue for continuous data sets associated with discreet metrics.

12. The system of claim 8, further comprising:

metric aggregator logic executable by the second processor that when executed causes the second processor to assign the nodes to clusters,

wherein the discrete events comprise;

a start time and stop time, including a number of networked resources consumed by the instances, a state of the clusters, and state of the nodes.

13. The system of claim 8, wherein the continuous events comprise:

a state of continuously available including connectivity, bandwidth of the network, and compute throughput capacity.

14. The system of claim 1, wherein the request message comprises a request for time-series metrics data or metering data for instances.

15. The system of claim 1, wherein the cluster state data and metrics data comprises time-series metrics.

16. The system of claim 2, wherein the instances for each of the nodes are virtual machines managed by the hypervisor of each of the nodes respectively.

17. The system of claim 2, wherein the instance identifier of each node is a routing identifier used by the message queue logic.

18. The system of claim 1, wherein the DMMS logic stored in a memory that further causes the first processor to:

determine a subset of the metrics data to collect in order to calculate the DMMS parameters.

19. The system of claim 1, wherein each of the instances identifies a user for whom the metering logic calculates the metering information.

20. The system of claim 2, wherein the metering logic employs libvirt to communicate with the hypervisor of each of the nodes to retrieve instance information about the instances assigned to the nodes.

21. A distributed metering and monitoring service (DMMS) system, the DMMS system comprising:

a first memory coupled to a first processor, wherein the first memory comprises;

messaging queue logic executable by the first processor that when executed causes the first processor to;

receive a request message;

route the request message;

receive a reply message; and

route the reply message;

logic executable by the first processor that when executed causes the first processor to;

receive a request message comprising DMMS parameters for which to collect metrics data, and calculate metering information;

determine a number of nodes and node identifiers of nodes networked in communication via a network, each node comprising;

instances responsive to respective user interactions through a user interface, including instances for users of the instances, including a first instance for a first user of the first instance;

collect the metrics data for the nodes, including metrics data for a first node and the first instance; and

generate the reply message comprising the metering information, including metering information for the first user of the first instance;

the nodes of the number of nodes, wherein each of the nodes comprise;

a second memory coupled to second processor, wherein the second memory comprises;

cluster state data and the metrics data for the node identifiers;

a hypervisor operable to manage the instances, and instance state information stored in the second memory and executed by the second processor, wherein the hypervisor assigns an instance identifier that identifies a node from the number of nodes as an assigned node identified by a node identifier;

metering logic executable by the second processor that when executed causes the second processor to;

retrieve, using the instance identifier for each of the instances from the hypervisor for the instances, the metrics data; and

generate the reply message comprising the metering information for the instances assigned the assigned nodes;

wherein the messaging queue logic comprises a cluster-to-cluster ratio threshold calculation that is based on a performance capacity of clusters including a first cluster.

22. The system of claim 21, wherein the DMMS parameters identify the users of the instances, including the user of the first instance, and wherein the metering information is calculated for each of the users by using the instance identifier for each of the instances linked to the metrics data of the nodes by the metering logic to identify a subset of metrics data collected by the metric aggregator logic to retrieve.

23. The system of claim 21, further comprising logic stored in memory, wherein the performance capacity of the clusters including the first cluster and a second cluster is calculated when the logic further causes the first processor to:

determine a number of clusters, including the first cluster and the second cluster, available to network with the nodes;

determine a frequency to collect the metrics data in order to accurately calculate the DMMS parameters;

evaluate each cluster to determine the performance capacity for each of clusters, including;

a node count limit, wherein the performance capacity is based on the number of nodes, the metrics data to collect in order to calculate the DMMS parameters, and the frequency to perform the metrics data collection.

24. The system of claim 21, wherein the metric aggregator logic stored in memory that monitors the nodes of a cluster and collects node level information.

25. The system of claim 21, wherein the cluster-to-node ratio threshold calculation indicates the number of nodes to assign to a shared messaging queue from a plurality of shared messaging queues for the nodes of respective clusters.

26. A distributed metering and monitoring service (DMMS) system, the DMMS system comprising:

a memory coupled to a processor, wherein the memory comprises;

a hypervisor that is operable to;

manage instances, and instance state information stored in the memory and executed by the processor, wherein the hypervisor assigns each of the instances, including a first instance, to a node from a number of nodes, including a first node;

metric aggregator logic;

metering logic operable to cause the processor to;

determine the number of nodes networked in communication via a network, each node comprising;

the instances responsive to respective user interactions through a user interface, wherein the instances include instances for users of the instances, including the first instance for a first user of the first instance;

determine a number of clusters by assigning the nodes to respective clusters including a first cluster, including a first subset of the nodes including the first node comprising the first instance assigned to the first cluster, wherein each cluster defines node identifiers for each of the nodes assigned to the respective clusters wherein the metric aggregator logic when executed causes the processor to assign the nodes to the respective clusters including the first cluster;

determine a configurable frequency threshold that is calculated by using a probable frequency and a performance capacity;

determine, for each node, the instances of the node; and

register, based on the determined configurable frequency threshold, each of the instances with a respective messaging queue in communication with the respective clusters comprising messaging queue logic.

27. A distributed metering and monitoring service (DMMS) system, the DMMS system comprising:

a memory coupled to a processor, wherein the memory comprises;

a hypervisor that is operable to;

manage instances, and instance state information stored in the memory and executed by the processor;

metering logic operable to cause the processor to;

determine, by the processor, a number of nodes networked in communication via a network, each of the number of nodes comprising;

the instances responsive to respective user interactions through a user interface;

determine, by the processor, a number of clusters to use to assign the nodes to respective clusters;

determine, by the processor, the instances of the node of the number of nodes; and

determine, by the processor, a configurable registration frequency threshold to use to register each of the instances of the node of the number of nodes with a respective messaging queue in communication with the cluster comprising messaging queue logic, wherein the registration frequency threshold is calculated based on a model of the nodes that identifies;

a probable frequency at which users cause the nodes to generate the instances; and

a performance capacity wherein each of the clusters is in communications with a respective messaging queue.

28. A method for a distributed metering and monitoring service (DMMS) system having a first memory coupled to a first processor, the method comprising:

receiving, into the first memory by executing messaging queue logic by the first processor, a request message;

routing, from the first memory by executing the messaging queue logic by the first processor, the request message;

receiving, from the first memory by executing the messaging queue logic by the first processor, a reply message;

routing, into the first memory by executing the messaging queue logic by the first processor, the reply message;

receiving, by executing the first processor, a request message comprising DMMS parameters for which to collect metrics data, and calculate metering information;

determining, by executing the first processor, a number of nodes and node identifiers of nodes networked in communication via a network,

wherein each node comprises;

instances responsive to respective user interactions through a user interface, including instances for users of the instances, including a first instance for a first user of the first instance,

wherein each of the nodes of the number of nodes comprises;

a second memory coupled to a second processor, wherein the second memory comprises;

cluster state data and the metrics data for the node identifiers;

a hypervisor operable to manage the instances, and instance state information stored in the second memory and executed by the second processor, wherein the hypervisor assigns each of the instances, including the first instance, to a node from the number of nodes, including the first node;

metering logic executable by the second processor that when executed causes the second processor to;

 

generate the reply message comprising the metering information for the instances assigned to the nodes;

 

inject, for each node, an instance identifier and/or the instance state information into the cluster state data and metrics data; and

 

register, according to a configurable frequency threshold, each of the instances with the messaging queue logic by using the instance identifier of each node to create a message queue for each instance;

collecting, by executing the first processor, the metrics data for the nodes, including metrics data for a first node and the first instance; and

generating, by executing the first processor, the reply message comprising the metering information, including metering information for the first user of the first instance.

29. A method for a distributed metering and monitoring service (DMMS) system having a first memory coupled to a first processor, the method comprising:

receiving, into the first memory by executing messaging queue logic by the first processor, a request message;

routing, from the first memory by executing the messaging queue logic by the first processor, the request message;

receiving, from the first memory by executing the messaging queue logic by the first processor, a reply message;

routing, into the first memory by executing the messaging queue logic by the first processor, the reply message;

receiving, by executing the first processor, a request message comprising DMMS parameters for which to collect metrics data, and calculate metering information;

determining, by executing the first processor, a number of nodes and node identifiers of nodes networked in communication via a network,

wherein each node comprises;

instances responsive to respective user interactions through a user interface, including instances for users of the instances, including a first instance for a first user of the first instance,

wherein each of the nodes of the number of nodes comprises;

a second memory coupled to second processor, wherein the second memory comprises;

cluster state data and the metrics data for the node identifiers,

a hypervisor operable to manage the instances, and instance state information stored in the second memory and executed by the second processor, wherein the hypervisor assigns an instance identifier that identifies a node from the number of nodes as an assigned node identified by a node identifier,

 

metering logic executable by the second processor that when executed causes the second processor to;

 

retrieve, using the instance identifier for each of the instances from the hypervisor for the instances, the metrics data, and

 

generate the reply message comprising the metering information for the instances assigned the assigned nodes;

collecting, by executing the first processor, the metrics data for the nodes, including metrics data for a first node and the first instance; and

generating, by executing the first processor, the reply message comprising the metering information, including metering information for the first user of the first instance.