METHOD AND SYSTEM FOR CLUSTERING AND PRIORITIZING EVENT MESSAGES

• US 20150370799A1
• Filed: 06/30/2014
• Published: 12/24/2015
• Est. Priority Date: 06/24/2014
• Status: Abandoned Application

• Current Assignee: VMware, Inc. (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Kushmerick, Nicholas, Lin, Junyuan
• Application Number:
  US14/319,057
• Publication Number:
  US 20150370799A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities
• CPC Class Codes: H04L 43/16 Threshold monitoring

Abstract

The current document is directed to methods and systems for processing, classifying, and efficiently storing large volumes of event messages generated in modern computing systems. In a disclosed implementation, received event messages are assigned to clusters based on metrics computed for the event messages. In addition, a significance value is determined for each received event message. When the significance value exceeds a threshold value, one or more actions are taken, including marking an event record corresponding to the event message, storing an event record corresponding to the event message in a significant-event log, and generating a notice or alarm.

Claims

1. An event-message clustering system comprising:

one or more processors;

one or more memories; and

computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the event-message clustering system to

receive event messages, and

process each of the received event messages by

determining a cluster to which to assign the event message,

extracting data values from the event message,

computing a significance value for the event message,

generating an event record corresponding to the event message that includes the extracted data values, and

storing the event record within, or associated with, the selected cluster in a physical data-storage device.

2. The event-message clustering system of claim 1 wherein the significance value computed for an event message is a numeric value that reflects one or more of:

a dissimilarity of the event-message type that includes the received event message to other event-message types;

a frequency that event messages of the event-message type that includes the received event message are received; and

a temporal proximity of event messages of the event-message type that includes the received event message to critical events.

3. The event-message clustering system of claim 2 wherein the dissimilarity of the event-message type that includes the received event message to other event-message types is computed from one or more of:

a distance separating a point in feature-vector space corresponding to the type of the received event message from another point in feature-vector space;

a ratio of densities of points in feature-vector space;

a difference between a pair-wise similarity computed for a group of event messages that includes the received event message and a pair-wise similarity computed for the group without the received event message; and

a difference between a pair-wise similarity computed for a group of event-message types that include the type of the received event message and a pair-wise similarity computed for the group without the type of the received event message.

4. The event-message clustering system of claim 3 wherein, in computing the distance separating the point in feature-vector space corresponding to the type of the received event message from another point in feature-vector space, the other point in feature-vector space corresponds to one of;

a centroid of a cluster of event-message types;

a feature vector associated with a cluster; and

a k^th-nearest-neighbor event message.

5. The event-message clustering system of claim 3 wherein the ratio of densities of points in feature-vector space is the ratio of an average density of points in feature-vector space to a density of points in feature-vector-space neighborhood of the point in feature-vector space corresponding to the type of the received event message.

6. The event-message clustering system of claim 2 wherein the frequency that event messages of the event-message type that includes the received event message are received is computed by:

accessing a number of event records within the stored event records, each stored event record including a time indication; and

determining, from the accessed number of event records, an average number of event messages of the event-message type that includes the received event message that are received for each interval of time.

7. The event-message clustering system of claim 2 wherein the temporal proximity of event messages of the event-message type that includes the received event message to critical events is computed by:

accessing a number of event records within the stored event records, each stored event record including a time indication, within temporal neighborhoods about time points of critical system events; and

determining, from the accessed number of event records, an average number of event messages of the event-message type that includes the received event message that are received within the temporal neighborhoods.

8. The event-message clustering system of claim 7 further including one or more of:

determining, from the accessed number of event records, an average highest number of event messages of the event-message type that includes the received event message that are received that occur within the temporal neighborhoods.

9. The event-message clustering system of claim 1 wherein, following computation of a significance value for the event message, the event-message clustering system:

compares the computed significance value to a threshold value; and

when the computed significance value is greater than the threshold value,

generates an event record corresponding to the event message,

places an indication in the event record to indicate that the event record corresponds to a significant event, and

stores the event record within, or associated with, the selected cluster in a physical data-storage device.

10. The event-message clustering system of claim 1 wherein, following computation of a significance value for the event message, the event-message clustering system:

compares the computed significance value to a threshold value; and

when the computed significance value is greater than the threshold value,

generates an event record corresponding to the event message, and

stores the event record within an event log for significant event messages.

11. The event-message clustering system of claim 1 wherein, following computation of a significance value for the event message, the event-message clustering system:

compares the computed significance value to a threshold value; and

when the computed significance value is greater than the threshold value,

generates one or more of a notice and alarm, and

transmits the generated notice or alarm to one or more of an automated system-administration subsystem and human system administrator.

12. A method that processes event messages, carried out within an event-message clustering system, the event-message clustering system having one or more processors, one or more memories, and computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the event-message clustering system to receive event messages and process each of the received event messages, the method comprising:

receiving event messages, and

processing each of the received event messages by

determining a cluster to which to assign the event message,

extracting data values from the event message,

computing a significance value for the event message,

generating an event record corresponding to the event message that includes the extracted data values, and

storing the event record within, or associated with, the selected cluster in a physical data-storage device.

13. The method of claim 12 wherein the significance value computed for an event message is a numeric value that reflects one or more of:

a dissimilarity of the event-message type that includes the received event message to other event-message types;

a frequency that event messages of the event-message type that includes the received event message are received; and

a temporal proximity of event messages of the event-message type that includes the received event message to critical events.

14. The method of claim 13 wherein the dissimilarity of the event-message type that includes the received event message to other event-message types is computed from one or more of:

a distance separating a point in feature-vector space corresponding to the type of the received event message from another point in feature-vector space;

a ratio of densities of points in feature-vector space;

a difference between a pair-wise similarity computed for a group of event messages that includes the received event message and a pair-wise similarity computed for the group without the received event message; and

a difference between a pair-wise similarity computed for a group of event-message types that include the type of the received event message and a pair-wise similarity computed for the group without the type of the received event message.

15. The method of claim 14 wherein, in computing the distance separating the point in feature-vector space corresponding to the type of the received event message from another point in feature-vector space, the other point in feature-vector space corresponds to one of:

a centroid of a cluster of event-message types;

a feature vector associated with a cluster; and

a k^th-nearest-neighbor event message.

16. The method of claim 14 wherein the ratio of densities of points in feature-vector space is the ratio of an average density of points in feature-vector space to a density of points in feature-vector-space neighborhood of the point in feature-vector space corresponding to the type of the received event message.

17. The method of claim 13 wherein the frequency that event messages of the event-message type that includes the received event message are received is computed by:

accessing a number of event records within the stored event records, each stored event record including a time indication; and

determining, from the accessed number of event records, an average number of event messages of the event-message type that includes the received event message that are received for each interval of time.

18. The method of claim 13 wherein the temporal proximity of event messages of the event-message type that includes the received event message to critical events is computed by:

accessing a number of event records within the stored event records, each stored event record including a time indication, within temporal neighborhoods about time points of critical system events; and

determining, from the accessed number of event records, an average number of event messages of the event-message type that includes the received event message that are received within the temporal neighborhoods.

19. The method of claim 18 further including one or more of:

determining, from the accessed number of event records, an average highest number of event messages of the event-message type that includes the received event message that are received that occur within the temporal neighborhoods.

20. The method of claim 12 further including, following computation of a significance value for the event message:

comparing the computed significance value to a threshold value; and

when the computed significance value is greater than the threshold value,

generating an event record corresponding to the event message,

placing an indication in the event record to indicate that the event record corresponds to a significant event, and

storing the event record within, or associated with, the selected cluster in a physical data-storage device.

21. The method of claim 12 further including, following computation of a significance value for the event message:

comparing the computed significance value to a threshold value; and

when the computed significance value is greater than the threshold value,

generating an event record corresponding to the event message, and

storing the event record within an event log for significant event messages.

22. The method of claim 12 further including, following computation of a significance value for the event message:

comparing the computed significance value to a threshold value; and

when the computed significance value is greater than the threshold value,

generating one or more of a notice and alarm, and

transmitting the generated notice or alarm to one or more of an automated system-administration subsystem and human system administrator.

23. Computer instructions stored in a physical device that, when executed on one or more processors of an event-message clustering system that additionally includes one or more memories, control the event-message clustering system to:

receive event messages; and

process each of the received event messages by

determining a cluster to which to assign the event message,

extracting data values from the event message,

computing a significance value for the event message,

generating an event record corresponding to the event message that includes the extracted data values, and

storing the event record within, or associated with, the selected cluster in a physical data-storage device.