METHODS AND SYSTEMS THAT DIAGNOSE AND MANAGE UNDESIRABLE OPERATIONAL STATES OF COMPUTING FACILITIES

• US 20180351838A1
• Filed: 06/02/2017
• Published: 12/06/2018
• Est. Priority Date: 06/02/2017
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Lui, Rui
• Granted Patent: US 10,454,801 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 41/064 involving time analysis
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route

Abstract

The current document is directed to automatically, semi-automatically, and/or manually monitoring a computing facility to detect and address undesirable operational states in computing facilities, including large distributed computing systems. The currently disclosed monitoring methods and systems employ case-based inference to diagnose and ameliorate undesirable operational states. In disclosed implementations, a database is maintained to store and provide access to records of previously handled undesirable operational states and the actions taken to remediate the undesirable operational state is maintained in order to facilitate case-based reasoning inference.

Claims

1. A computing-facility monitoring-and-management system comprising:

one or more processors;

one or more memories;

one or more mass-storage devices;

stored information about previously encountered undesirable computing-facility operational states and administrative steps taken to address the previously encountered undesirable computing-facility operational states; and

computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the computing-facility monitoring-and-management system to

determine that a current operational state of the computing-facility monitoring-and-management system is undesirable,

identify information within the stored information about one or more previously determined undesirable operational states similar to the current operational state of the computing-facility,

use the identified information to determine a set of one or more administrative steps to address the determined current undesirable operational state, and

carry out the identified administrative steps to drive the computing facility into a desirable operational state.

2. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system is one of:

an automated computing-facility monitoring-and-management system; and

a semi-automated computing-facility monitoring-and-management system.

3. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system determines that the current operational state of the computing-facility monitoring-and-management system is undesirable by:

generating an alert that triggers administrative intervention by the computing-facility monitoring-and-management system.

4. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system determines that the current operational state of the computing-facility monitoring-and-management system is undesirable by:

generating a snapshot vector that represents the operational state of the computing facility;

generating values of one or more desirability metrics from the snapshot vector; and

comparing the generated values to corresponding threshold values.

5. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system identifies information within the stored information about one or more previously determined undesirable operational states similar to the current operational state of the computing-facility by:

generating a snapshot vector that represents the current operational state of the computing facility; and

retrieving information within the stored information about one or more previously determined undesirable operational states similar to the operational state of the computing-facility by

retrieving each set of stored information about a previously determined undesirable operational state for which a similarity metric computed from the generated snapshot vector and a snapshot vector included in the set of stored information indicates that the previously determined undesirable operational state is similar to the current operational state of the computing facility.

6. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system identifies information within the stored information about one or more previously determined undesirable operational states similar to the current operational state of the computing-facility by:

generating a snapshot vector that represents the current operational state of the computing facility;

retrieving a first set of information sets within the stored information about one or more previously determined undesirable operational states similar to the operational state of the computing-facility by retrieving each set of stored information about a previously determined undesirable operational state for which an alert identifier within the set of information is equivalent to an alert identifier associated with an alert that indicated that the current operational state of the computing-facility monitoring-and-management system is undesirable; and

selecting a second set of information sets from the first set of information sets for which a similarity metric computed from the generated snapshot vector and a snapshot vector included in the information set indicates that the previously determined undesirable operational state is similar to the current operational state of the computing facility.

7. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system identifies information within the stored information about one or more previously determined undesirable operational states similar to the current operational state of the computing-facility by:

generating a snapshot vector that represents the current operational state of the computing facility;

retrieving a first set of information sets within the stored information about one or more previously determined undesirable operational states similar to the operational state of the computing-facility by retrieving each set of stored information about a previously determined undesirable operational state for which an alert identifier within the set of information is equivalent to an alert identifier associated with an alert that indicated that the current operational state of the computing-facility monitoring-and-management system is undesirable; and

selecting a second set of information sets from the first set of information sets, for each information set of which

a similarity metric computed from the generated snapshot vector and a snapshot vector included in the information set indicates that the previously determined undesirable operational state is similar to the current operational state of the computing facility, and a

a desirability metric included in the information set indicates that the administrative steps taken to address the previously encountered undesirable computing-facility operational state were effective.

8. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system uses the identified information to determine a set of one or more administrative steps to address the determined current undesirable operational state by:

selecting an information set from the identified information that describes a previously encountered undesirable computing-facility operational state and administrative steps taken to address the previously encountered undesirable computing-facility operational state; and

selecting the administrative steps taken to address the previously encountered undesirable computing-facility operational state as the set of one or more administrative steps to address the determined current undesirable operational state.

9. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system uses the identified information to determine a set of one or more administrative steps to address the determined current undesirable operational state by:

selecting an information set from the identified information that describes a previously encountered undesirable computing-facility operational state and administrative steps taken to address the previously encountered undesirable computing-facility operational state;

extracting the administrative steps taken to address the previously encountered undesirable computing-facility operational state from the selected information set;

modifying the extracted administrative steps; and

using the modified extracted administrative steps as the set of one or more administrative steps to address the determined current undesirable operational state.

10. The computing-facility monitoring-and-management system of claim 1 wherein administrative steps carried out to drive the computing facility into a desirable operational state include one or more of:

a computing-facility configuration change;

a computing-facility parameter change; and

a computing facility policy change.

11. The computing-facility monitoring-and-management system of claim 1 wherein the computing-facility monitoring-and-management system computes a snapshot vector to represent the current operational state of the computing facility by setting each component of the snapshot vector to a current value of a computing-facility characteristic or parameter corresponding to the snapshot-vector component.

12. The computing-facility monitoring-and-management system of claim 11 wherein the computing-facility monitoring-and-management system computes a similarity metric that represents a comparison of two snapshot vectors as the length of a vector obtained by subtracting one of the snapshot vectors from the other of the snapshot vectors.

13. The computing-facility monitoring-and-management system of claim 12 wherein the computing-facility monitoring-and-management system computes a value for a desirability metric corresponding to a snapshot vector by:

when the snapshot vector represents a point in an n-dimensional operational-state space corresponding to an acceptable operational state, setting the value of the desirability metric to a first value; and

when the snapshot vector represents a point in an n-dimensional operational-state space corresponding to an unacceptable operational state, setting the value of the desirability metric to a second value proportional to a distance, in the n-dimensional operational-state space, from the point in an n-dimensional operational-state space represented by the snapshot vector and a point in the n-dimensional operational-state space corresponding to an acceptable operational state.

14. The computing-facility monitoring-and-management system of claim 1 wherein the stored information includes one or more information sets, each information set including:

a snapshot vector representing a previously encountered undesirable computing-facility operational state;

an encoding of one or more administrative steps applied to address the previously encountered undesirable computing-facility operational state; and

a desirability metric computed for a computing-facility operational state following application of the one or more administrative steps to the computing facility.

15. The computing-facility monitoring-and-management system of claim 14 wherein the information set further includes an alert identifier.

16. A method that carries out administrative steps to drive a computing facility into a desirable operational state, the method comprising:

determining, by a computing-facility monitoring-and-management system, that a current operational state of the computing facility is undesirable,

identifying, by the computing-facility monitoring-and-management system, one or more information sets stored by the computing-facility monitoring-and-management system that contain information about one or more previously determined undesirable operational states similar to the current undesirable operational state of the computing-facility;

using, by the computing-facility monitoring-and-management system, the identified one or more information sets to determine a set of one or more administrative steps to address the determined current undesirable operational state, and

carrying out, by the computing-facility monitoring-and-management system, the identified administrative steps to drive the computing facility into a desirable operational state.

17. The method of claim 16 further comprising:

generating a snapshot vector, by the computing-facility monitoring-and-management system, to represent the current operational state of the computing facility by setting each component of the snapshot vector to a current value of a computing-facility characteristic or parameter corresponding to the snapshot-vector component.

18. The method of claim 17 further comprising:

generating, by the computing-facility monitoring-and-management system, a similarity metric that represents a comparison of two snapshot vectors as the length of a vector obtained by subtracting one of the snapshot vectors from the other of the snapshot vectors.

19. The method of claim 17 further comprising:

generating, by the computing-facility monitoring-and-management system, a value for a desirability metric corresponding to a snapshot vector by

when the snapshot vector represents a point in an n-dimensional operational-state space corresponding to an acceptable operational state, setting the value of the desirability metric to a first value; and

when the snapshot vector represents a point in an n-dimensional operational-state space corresponding to an unacceptable operational state, setting the value of the desirability metric to a second value proportional to a distance, in the n-dimensional operational-state space, from the point in an n-dimensional operational-state space represented by the snapshot vector and a point in the n-dimensional operational-state space corresponding to an acceptable operational state.

20. Computer instructions encoded in a physical data-storage device within a computing-facility monitoring-and-management system having one or more processors, one or more memories, one or more mass-storage device;

and stored information about previously encountered undesirable computing-facility operational states and administrative steps taken to address the previously encountered undesirable computing-facility operational states, the computer instructions, when executed by one or more of the one or more processors, controlling the computing-facility monitoring-and-management system to;

determine that a current operational state of the computing facility is undesirable,

identify one or more information sets stored by the computing-facility monitoring-and-management system that contain information about one or more previously determined undesirable operational states similar to the current undesirable operational state of the computing-facility;

use the identified one or more information sets to determine a set of one or more administrative steps to address the determined current undesirable operational state, and

carry out the identified administrative steps to drive the computing facility into a desirable operational state.