Methods and systems that diagnose and manage undesirable operational states of computing facilities

• US 10,454,801 B2
• Filed: 06/02/2017
• Issued: 10/22/2019
• Est. Priority Date: 06/02/2017
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Lui, Rui
• Primary Examiner(s): Taylor, Nicholas R
• Assistant Examiner(s): Roy, Sanjoy K
• Application Number:
  US15/612,365
• Publication Number:
  US 20180351838A1
• Time in Patent Office: 872 Days
• CPC Class Codes: H04L 41/064 involving time analysis
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

The current document is directed to automatically, semi-automatically, and/or manually monitoring a computing facility to detect and address undesirable operational states in computing facilities, including large distributed computing systems. The currently disclosed monitoring methods and systems employ case-based inference to diagnose and ameliorate undesirable operational states. In disclosed implementations, a database is maintained to store and provide access to records of previously handled undesirable operational states and the actions taken to remediate the undesirable operational state is maintained in order to facilitate case-based reasoning inference.

Claims

1. A computing-facility monitoring-and-management system comprising:

one or more processors;

one or more memories;

one or more mass-storage devices;

operational state data stored in one or more of the one or more memories, the operational state data comprising previously encountered undesirable computing-facility operational states and administrative steps taken to address the previously encountered undesirable computing-facility operational states; and

computer instructions, stored in one or more of the one or more memories, wherein the instructions, when executed by one or more of the one or more processors, cause the computing-facility monitoring-and-management system to at least;

determine that a current operational state of the computing-facility monitoring-and-management system is undesirable, based on the current operational state being a threshold distance outside of an n-dimensional volume of desirable operational states for the computing-facility monitoring-and-management system;

identify, within the operational state data, information associated with one or more previously determined undesirable operational states similar to the current operational state of the computing-facility;

use the identified information to determine a set of one or more administrative steps to address the determined current undesirable operational state; and

carry out the identified administrative steps to drive the computing facility into a desirable operational state.

2. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system is one of:

an automated computing-facility monitoring-and-management system; and

a semi-automated computing-facility monitoring-and-management system.

3. The computing-facility monitoring-and-management system of claim 1 wherein the instructions further cause the computing-facility monitoring-and-management system to at least:

determine that the current operational state of the computing-facility monitoring-and-management system is undesirable by generating an alert that triggers administrative intervention by the computing-facility monitoring-and-management system.

4. The computing-facility monitoring-and-management system of claim 1 wherein the instructions further cause the computing-facility monitoring-and-management system to at least:

generate a snapshot vector that represents the operational state of the computing facility;

generate values of one or more desirability metrics from the snapshot vector; and

comparing the generated values to corresponding threshold values to determine that the current operational state of the computing-facility monitoring-and-management system is undesirable.

5. The computing-facility monitoring-and-management system of claim 1 wherein the instructions further cause the computing-facility monitoring-and-management system to at least:

generate a snapshot vector that represents the current operational state of the computing facility; and

retrieve, from the operational state data, the information associated with the one or more previously determined undesirable operational states based on a similarity metric computed from the generated snapshot vector and a respective snapshot vector for the one or more previously determined undesirable operational states indicates similarity to the current operational state of the computing facility.

6. The computing-facility monitoring-and-management system of claim 1 wherein the instructions further cause the computing-facility monitoring-and-management system to at least:

generate a snapshot vector that represents the current operational state of the computing facility;

retrieve, from the operational state data, the information associated with the one or more previously determined undesirable operational states based on an alert identifier within the information matching an alert identifier associated with an alert that indicates that the current operational state of the computing-facility monitoring-and-management system is undesirable; and

select a subset of the information based on a similarity metric computed from the generated snapshot vector and a snapshot vector included in the information that indicates that the previously determined undesirable operational state is similar to the current operational state of the computing facility.

7. The computing-facility monitoring-and-management system of claim 1 wherein the instructions further cause the computing-facility monitoring-and-management system to at least:

generate a snapshot vector that represents the current operational state of the computing facility;

retrieve, from the operational state data, the information associated with the one or more previously determined undesirable operational states based on an alert identifier within the information matching an alert identifier associated with an alert that indicates that the current operational state of the computing-facility monitoring-and-management system is undesirable; and

select a subset of the information based on;

a similarity metric computed from the generated snapshot vector and a snapshot vector included in the information that indicates that the previously determined undesirable operational state is similar to the current operational state of the computing facility, and a

a desirability metric included in the information that indicates that the administrative steps taken to address the previously encountered undesirable computing-facility operational state were effective.

8. The computing-facility monitoring-and-management system of claim 1 wherein the information associated with the one or more previously determined undesirable operational states describes a previously encountered undesirable computing-facility operational state and administrative steps taken to address the previously encountered undesirable computing-facility operational state, and the set of one or more administrative steps comprises the administrative steps taken to address the previously encountered undesirable computing-facility operational state.

9. The computing-facility monitoring-and-management system of claim 1 wherein the information associated with the one or more previously determined undesirable operational states describes a previously encountered undesirable computing-facility operational state and administrative steps taken to address the previously encountered undesirable computing-facility operational state, and the instructions further cause the computing-facility monitoring-and-management system to at least:

extract the administrative steps taken to address the previously encountered undesirable computing-facility operational state from the information associated with the one or more previously determined undesirable operational states;

modify the extracted administrative steps; and

use the modified extracted administrative steps as the set of one or more administrative steps to address the determined current undesirable operational state.

10. The computing-facility monitoring-and-management system of claim 1 wherein administrative steps carried out to drive the computing facility into a desirable operational state include one or more of:

a computing-facility configuration change;

a computing-facility parameter change; and

a computing facility policy change.

11. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system computes a snapshot vector to represent the current operational state of the computing facility by setting each component of the snapshot vector to a current value of a computing-facility characteristic or parameter corresponding to the snapshot-vector component.

12. The computing-facility monitoring-and-management system of claim 11 wherein the computing-facility monitoring-and-management system computes a similarity metric that represents a comparison of two snapshot vectors as the length of a vector obtained by subtracting one of the snapshot vectors from the other of the snapshot vectors.

13. The computing-facility monitoring-and-management system of claim 11 wherein the the instructions further cause the computing-facility monitoring-and-management system to at least:

set a desirability metric to a value that is proportional to a distance between the snapshot vector and the n-dimensional volume of desirable operational states.

14. The computing-facility monitoring-and-management system of claim 1 wherein the operational state data includes one or more information sets, each information set including:

a snapshot vector representing a previously encountered undesirable computing-facility operational state;

an encoding of one or more administrative steps applied to address the previously encountered undesirable computing-facility operational state; and

a desirability metric computed for a computing-facility operational state following application of the one or more administrative steps to the computing facility.

15. The computing-facility monitoring-and-management system of claim 14 wherein the information further includes an alert identifier.

16. A method that carries out administrative steps to drive a computing facility into a desirable operational state, the method comprising:

determining, by a computing-facility monitoring-and-management system, that a current operational state of the computing facility is undesirable, based on the current operational state being a threshold distance outside of an n-dimensional volume of desirable operational states for the computing-facility monitoring-and-management system;

identifying, by the computing-facility monitoring-and-management system, one or more information sets stored by the computing-facility monitoring-and-management system that contain operational state data associated with one or more previously determined undesirable operational states similar to the current undesirable operational state of the computing-facility;

using, by the computing-facility monitoring-and-management system, the identified one or more information sets to determine a set of one or more administrative steps to address the determined current undesirable operational state; and

carrying out, by the computing-facility monitoring-and-management system, the identified administrative steps to drive the computing facility into a desirable operational state.

17. The method of claim 16 further comprising:

generating a snapshot vector, by the computing-facility monitoring-and-management system, to represent the current operational state of the computing facility by setting each component of the snapshot vector to a current value of a computing-facility characteristic or parameter corresponding to the snapshot-vector component.

18. The method of claim 17 further comprising:

generating, by the computing-facility monitoring-and-management system, a similarity metric that represents a comparison of two snapshot vectors as the length of a vector obtained by subtracting one of the snapshot vectors from the other of the snapshot vectors.

19. The method of claim 17 further comprising:

setting, by the computing-facility monitoring-and-management system, a desirability metric to a value that is proportional to a distance between the snapshot vector and the n-dimensional volume.

20. Computer instructions encoded in a physical data-storage device within a computing-facility monitoring-and-management system having one or more processors, one or more memories, one or more mass-storage device, and operational state data stored in the one or more memories, the operational state data comprising previously encountered undesirable computing-facility operational states and administrative steps taken to address the previously encountered undesirable computing-facility operational states, wherein the instructions, when executed by one or more of the one or more processors, cause the computing-facility monitoring-and-management system to at least:

determine that a current operational state of the computing facility is undesirable, based on the current operational state being a threshold distance outside of an n-dimensional volume of desirable operational states for the computing-facility monitoring-and-management system;

identify one or more information sets stored by the computing-facility monitoring-and-management system that contain operational state data, the information being associated with one or more previously determined undesirable operational states similar to the current undesirable operational state of the computing-facility;

use the identified one or more information sets to determine a set of one or more administrative steps to address the determined current undesirable operational state; and

carry out the identified administrative steps to drive the computing facility into a desirable operational state.