Self-healing and dynamic optimization of VM server cluster management in multi-cloud platform

• US 10,616,070 B2
• Filed: 05/07/2019
• Issued: 04/07/2020
• Est. Priority Date: 11/09/2015
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Yang, Chen-Yui, Lu, David H., Baker, Scott, Srdar, Anthony M., Bourge, Gabriel
• Primary Examiner(s): Hussain, Imad
• Application Number:
  US16/404,921
• Publication Number:
  US 20190260646A1
• Time in Patent Office: 336 Days
• Field of Search: 709224
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0823 characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability for optimising operational conditions of wireless networks H04W24/02
• CPC Class Codes: H04L 41/0895 Configuration of virtualised networks or elements, e.g. virtualised network function or OpenFlow elements
• CPC Class Codes: H04L 41/0896 Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities flow or congestion control using dynamic resource allocation, e.g. in-call renegotiation, H04L47/76
• CPC Class Codes: H04L 41/0897 by horizontal or vertical scaling of resources, or by migrating entities, e.g. virtual resources or entities
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 41/5025 by proactively reacting to service quality change, e.g. by reconfiguration after service quality degradation or upgrade
• CPC Class Codes: H04L 43/0876 Network utilisation, e.g. volume of load or congestion level
• CPC Class Codes: H04L 43/16 Threshold monitoring

Abstract

Virtual machine server clusters are managed using self-healing and dynamic optimization to achieve closed-loop automation. The technique uses adaptive thresholding to develop actionable quality metrics for benchmarking and anomaly detection. Real-time analytics are used to determine the root cause of KPI violations and to locate impact areas. Self-healing and dynamic optimization rules are able to automatically correct common issues via no-touch automation in which finger-pointing between operations staff is prevalent, resulting in consolidation, flexibility and reduced deployment time.

Claims

1. A method, comprising:

supporting a group of statistics for selecting metric statistics for managing a virtual machine server cluster, the group of statistics comprising each of average, maximum value, minimum value, last value, standard, sum of historical values, sum of squares of historical values, and count of values;

supporting a group of predetermined quality metric types for classifying quality metrics;

classifying a quality metric into a selected one of the group of predetermined quality metric types;

selecting a statistic for monitoring the quality metric, the selecting being based on the classifying the quality metric into the selected one of the group of predetermined quality metric types, the statistic being selected from the group of statistics;

accumulating values for one or more partial sums from performance monitoring data relating to the quality metric, the partial sums being selected to calculate a value of the statistic;

calculating the value of the statistic from the partial sums accumulated from the performance monitoring data relating to the quality metric;

determining an adaptive threshold range for the quality metric based on the value of the statistic and based on the classifying the quality metric into the selected one of the group of predetermined quality metric types;

determining that a monitoring value for the quality metric is outside the adaptive threshold range for the quality metric;

performing a self-healing and dynamic optimization task based on the determining that the monitoring value is outside the adaptive threshold range;

determining that a value of one of the partial sums accumulated from the performance monitoring data relating to the quality metric exceeds a limit imposed to prevent arithmetic overflow of a value storage; and

dividing values of each partial sum accumulated from the performance monitoring data relating to the quality metric by two.

2. The method of claim 1, wherein the group of predetermined quality metric types includes each of a load metric type, a utilization metric type, a process efficiency metric type and a response time metric type.

3. The method of claim 1, wherein:

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a utilization metric type; and

performing the self-healing and dynamic optimization task comprises adding a resource if the monitoring value is above an upper threshold of the adaptive threshold range and removing a resource if the monitoring value is below a lower threshold of the adaptive threshold range.

4. The method of claim 1, wherein

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a utilization metric type;

and

performing the self-healing and dynamic optimization task comprises performing optimizing resource tuning.

5. The method of claim 1, wherein

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a load metric type;

and

performing the self-healing and dynamic optimization task comprises adjusting a system load.

6. The method of claim 1, wherein:

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a process efficiency metric type; and

performing the self-healing and dynamic optimization task comprises performing virtual machine life cycle management.

7. The method of claim 1, further comprising:

associating a single, normalized timestamp with all the partial sum values that are accumulated in a single execution of a history update function.

8. The method of claim 1, wherein the partial sums include a sum value, a sum-of-the-squares value, and a count value.

9. A computer-readable storage device having stored thereon computer readable instructions, wherein execution of the computer readable instructions by a processor causes the processor to perform operations comprising:

supporting a group of statistics for selecting metric statistics for managing a virtual machine server cluster, the group of statistics comprising each of average, maximum value, minimum value, last value, standard, sum of historical values, sum of squares of historical values, and count of values;

supporting a group of predetermined quality metric types for classifying quality metrics comprising each of a load metric type, a utilization metric type, a process efficiency metric type and a response time metric type;

classifying a quality metric into a selected one of the group of predetermined quality metric types;

selecting a statistic for monitoring the quality metric, the selecting being based on the classifying the quality metric into the selected one of the group of predetermined quality metric types, the statistic being selected from the group of statistics;

accumulating values for one or more partial sums from performance monitoring data relating to the quality metric, the partial sums being selected to calculate a value of the statistic;

calculating the value of the statistic from the partial sums accumulated from the performance monitoring data relating to the quality metric;

determining an adaptive threshold range for the quality metric based on the value of the statistic and based on the classifying the quality metric into the selected one of the group of predetermined quality metric types;

determining that a monitoring value for the quality metric is outside the adaptive threshold range for the quality metric;

performing a self-healing and dynamic optimization task based on the determining that the monitoring value is outside the adaptive threshold range;

determining that a value of one of the partial sums accumulated from the performance monitoring data relating to the quality metric exceeds a limit imposed to prevent arithmetic overflow of a value storage; and

dividing values of each partial sum accumulated from the performance monitoring data relating to the quality metric by two.

10. The computer-readable storage device of claim 9, wherein

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a utilization metric type;

and

performing the self-healing and dynamic optimization task comprises adding a resource if the monitoring value is above an upper threshold of the adaptive threshold range and removing a resource if the monitoring value is below a lower threshold of the adaptive threshold range.

11. The computer-readable storage device of claim 9, wherein

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a utilization metric type;

and

performing the self-healing and dynamic optimization task comprises performing optimizing resource tuning.

12. The computer-readable storage device of claim 9, wherein

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a load metric type;

and

performing the self-healing and dynamic optimization task comprises adjusting a system load.

13. The computer-readable storage device of claim 9, further comprising:

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a process efficiency metric type; and

performing the self-healing and dynamic optimization task comprises performing virtual machine life cycle management.

14. The computer-readable storage device of claim 9, wherein the operations further comprise:

associating a single, normalized timestamp with all the partial sum values that are accumulated in a single execution of a history update function.

15. The computer-readable storage device of claim 9, wherein the partial sums include a sum value, a sum-of-the-squares value, and a count value.

16. A system for managing a virtual machine server cluster in a multi-cloud platform, comprising:

a processor resource;

a performance measurement interface connecting the processor resource to the virtual machine server cluster; and

a computer-readable storage device having stored thereon computer readable instructions, wherein execution of the computer readable instructions by the processor resource causes the processor resource to perform operations comprising;

supporting a group of statistics for selecting metric statistics, the group of statistics comprising each of average, maximum value, minimum value, last value, standard, sum of historical values, sum of squares of historical values, and count of values;

supporting a group of predetermined quality metric types for classifying quality metrics;

classifying a quality metric into a selected one of the group of predetermined quality metric types;

selecting a statistic for monitoring the quality metric, the selecting being based on the classifying the quality metric into the selected one of the group of predetermined quality metric types, the statistic being selected from the group of statistics;

accumulating values for one or more partial sums from performance monitoring data relating to the quality metric, the partial sums being selected to calculate a value of the statistic;

calculating the value of the statistic from the partial sums accumulated from the performance monitoring data relating to the quality metric;

determining an adaptive threshold range for the quality metric based on the value of the statistic and based on the classifying the quality metric into the selected one of the group of predetermined quality metric types;

determining that a monitoring value for the quality metric is outside the adaptive threshold range for the quality metric;

performing a self-healing and dynamic optimization task based on the determining that the monitoring value is outside the adaptive threshold range;

determining that a value of one of the partial sums accumulated from the performance monitoring data relating to the quality metric exceeds a limit imposed to prevent arithmetic overflow of a value storage; and

dividing values of each partial sum accumulated from the performance monitoring data relating to the quality metric by two.

17. The system of claim 16, wherein the group of predetermined quality metric types includes each of a load metric type, a utilization metric type, a process efficiency metric type and a response time metric type.

18. The system of claim 16, wherein:

classifying a quality metric into a selected one of the group of predetermined quality metric types comprises classifying the quality metric into a utilization metric type; and

performing the self-healing and dynamic optimization task comprises adding a resource if the monitoring value is above an upper threshold of the adaptive threshold range and removing a resource if the monitoring value is below a lower threshold of the adaptive threshold range.

19. The system of claim 16, wherein the partial sums include a sum value, a sum-of-the-squares value, and a count value.