METHOD AND SYSTEM FOR CLUSTERING EVENT MESSAGES AND MANAGING EVENT-MESSAGE CLUSTERS

• US 20150370885A1
• Filed: 06/30/2014
• Published: 12/24/2015
• Est. Priority Date: 06/24/2014
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Kushmerick, Nicholas, Lin, Junyuan
• Granted Patent: US 10,120,928 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 41/40 using virtualisation of network functions or resources, e.g. SDN or NFV entities

Abstract

The current document is directed to methods and systems for processing, classifying, and efficiently storing large volumes of event messages generated in modern computing systems. In a disclosed implementation, received event messages are assigned to event-message clusters based on non-parameter tokens identified within the event messages. A parsing function is generated for each cluster that is used to extract data from incoming event messages and to prepare event records from event messages that more efficiently and accessible store event information. The parsing functions also provide an alternative basis for assignment of event massages to clusters.

Claims

7. The event-message clustering system of claim 7 wherein, when, during periodic monitoring, the event-message clustering system identifies a cluster for which a parsing function has not yet been generated and with which more than a threshold number of event messages are associated, the event-message clustering system generates a parsing function for the cluster and associates the parsing function with the cluster.

8. The event-message clustering system of claim 7 wherein, when, during periodic monitoring, the event-message clustering system identifies a first cluster associated with a first parsing function and a second cluster with a second parsing function for which the first parsing function can be successfully applied to event messages assigned to the second cluster and the second parsing function can be successfully applied to event messages assigned to the first cluster, the event-message clustering system merges the two clusters into a single cluster.

9. The event-message clustering system of claim 7 wherein, when, during periodic monitoring, the event-message clustering system identifies a first cluster to which a number of event messages have been assigned that is less than a threshold fraction of the number of events assigned to a second cluster, the event-message clustering system analyzes the first and second clusters to determine whether or not to merge the first and second cluster into a single cluster.

10. The event-message clustering system of claim 7 wherein, when, during periodic monitoring, the event-message clustering system identifies a cluster associated with a parsing function that cannot be successfully applied to more than a threshold number of event messages assigned to the cluster, the event-message clustering system splits the cluster into a pair of clusters that includes the cluster and a new cluster and assigns the more than a threshold number of event messages to the new cluster.

11. The event-message clustering system of claim 7 wherein determining a cluster to which to assign the event message further includes:

normalizing the event message to identify parameter tokens within the event message;

computing, using non-parameter tokens within the event message, a metric to represent the event message; and

using the metric to select an event-message cluster to which to assign the event message.

12. The event-message clustering system of claim 7 wherein determining a cluster to which to assign the event message further includes:

identifying a parsing function associated with a cluster that can be successfully applied to the event message; and

selecting the identified cluster as the event-message cluster to which to assign the event message.

13. A method that processes event messages, carried out within an event-message clustering system, the event-message clustering system having one or more processors, one or more memories, and computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the event-message clustering system to receive event messages and process each of the received event messages, the method comprising:

receiving event messages, and

processing each of the received event messages by

determining a cluster to which to assign the event message,

employing a parsing function associated with the determined cluster to extract data values from the event message,

generating an event record corresponding to the event message that includes the extracted data values, and

storing the event record within, or associated with, the selected cluster in a physical data-storage device.

17. The method of claim 13 further including:

maintaining the clusters by periodically monitoring the clusters and carrying out maintenance operations.

18. The method of claim 17 further including:

when, during periodic monitoring, a cluster for which a parsing function has not yet been generated and with which more than a threshold number of event messages are associated is identified,

generating a parsing function for the cluster and associates the parsing function with the cluster.

19. The method of claim 17 further including:

when, during periodic monitoring, a first cluster associated with a first parsing function and a second cluster with a second parsing function are identified for which the first parsing function can be successfully applied to event messages assigned to the second cluster and the second parsing function can be successfully applied to event messages assigned to the first cluster,

merging the two clusters into a single cluster.

20. The method of claim 17 further including:

when, during periodic monitoring, a first cluster to which a number of event messages have been assigned that is less than a threshold fraction of the number of events assigned to a second cluster,

analyzing the first and second clusters to determine whether or not to merge the first and second cluster into a single cluster.

21. The method of claim 17 further including:

when, during periodic monitoring, a cluster associated with a parsing function that cannot be successfully applied to more than a threshold number of event messages assigned to the cluster is identified,

splitting the cluster into a pair of clusters that includes the cluster and a new cluster and assigns the more than a threshold number of event messages to the new cluster.

22-1. The computer instructions of claim 22 that further control the event-message clustering system to:

generate a parsing function for a cluster by;

determining the non-variable portions common to a number of event messages assigned to the cluster; and

generating a regular expression that includes literals representing the determined non-variable portions common to a number of event messages assigned to the cluster and that includes any-substring-matching sub-regular expressions to represent the variable portions.