Clustering and labeling streamed data

• US 10,698,926 B2
• Filed: 04/20/2017
• Issued: 06/30/2020
• Est. Priority Date: 04/20/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Zhou, Shandan, Subramanian, Karthikeyan, Chowdhury, Murtaza Muidul Huda, Natarajan, Gowtham
• Primary Examiner(s): Stevens, Robert
• Application Number:
  US15/493,036
• Publication Number:
  US 20180307740A1
• Time in Patent Office: 1,167 Days
• Field of Search: 707738
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/285 Clustering or classification

Abstract

Aspects extend to methods, systems, and computer program products for clustering streamed or batch data. Aspects of the invention include dynamic clustering and labeling of streamed data and/or batch data, including failures and error logs (user, platform, etc.), latency logs, warning logs, information logs, Virtual Machine (VM) creation data logs, template logs, etc., for use in analysis (e.g., error log analysis). A clustering system can learn from previously identified patterns and use that information to group newer information dynamically as it gets generated. The clustering system can leverage streamed data and/or batch data domain knowledge for preprocessing. In one aspect, a clustering system uses a similarity measure. Based on (e.g., users'"'"' configuration of) a similarity threshold, the cluster system (e.g., automatically) assigns/clusters streamed data and/or batch data into groups.

Claims

1. A computer system, comprising:

one or more hardware processors;

system memory coupled to the one or more hardware processors, the system memory storing instructions that are executable by the one or more hardware processors;

the one or more hardware processors executing the instructions stored in the system memory to;

receive streamed log data over a network connection;

pre-process the streamed log data to remove any customer and request related information;

filter words that include less than a specified number of characters and that have a frequency of occurrence above a specified threshold to select relevant features from within the streamed log data, wherein the relevant features are relevant to a monitored event at a device where the streamed log data originated;

for one or more previously formed log pattern clusters, determine a similarity between the relevant features and the one or more previously formed log pattern clusters; and

assign the streamed log data to a log pattern cluster based on the determined similarity.

2. The computer system of claim of claim 1, wherein the one or more hardware processors executing the instructions stored in the system memory to receive streamed log data over the network connection comprises the one or more hardware processors executing the instructions stored in the system memory to receive streamed error log data from the device;

wherein the one or more hardware processors executing the instructions stored in the system memory to select relevant features from within the streamed log data comprises the one or more hardware processors executing the instructions stored in the system memory to select an error pattern from within the streamed error log data;

wherein the one or more hardware processors executing the instructions stored in the system memory to determine a similarity between the relevant features and the one or more previously formed log pattern clusters comprises the one or more hardware processors executing the instructions stored in the system memory to determine a similarity between the error pattern and the one or more previously formed log pattern clusters; and

wherein the one or more hardware processors executing the instructions stored in the system memory to assign the streamed log data to the log pattern cluster based on the determined similarities comprises the one or more hardware processors executing the instructions stored in the system memory to assign the error pattern to the log pattern cluster based on the determined similarities.

3. The computer system of claim 1, wherein the one or more hardware processors executing the instructions stored in the system memory to select relevant features from within the streamed log data comprise the one or more hardware processors executing the instructions stored in the system memory to:

tokenize the streamed log data into a plurality of words;

for one or more words in the plurality of words, calculate the frequency of occurrence of the word within the plurality of words; and

based on the calculated frequencies, determine a sub-plurality of more unique words from among the plurality of words that form a relevant feature.

4. The computer system of claim 1, wherein the one or more hardware processors executing the instructions stored in the system memory to determine a similarity between the relevant features and the one or more previously formed log pattern clusters comprises the one or more hardware processors executing the instructions stored in the system memory to:

calculate a similarity between the relevant features and the one or more previously formed log pattern clusters; and

determine if the calculated similarity satisfies a similarity threshold.

5. The computer system of claim 4, further comprising the one or more hardware processors executing the instructions stored in the system memory to access the similarity threshold from user configuration.

6. The computer system of claim 4, further comprising the one or more hardware processors executing the instructions stored in the system memory to, from among any previously formed log pattern clusters having calculated similarities satisfying the similarity threshold, select a previously formed log pattern cluster having a highest calculated similarity to the relevant features;

and

wherein the one or more hardware processors executing the instructions stored in the system memory to assign the streamed log data to the log pattern cluster based on the determined similarities comprise the one or more hardware processors executing the instructions stored in the system memory to assign the streamed log data to a previously formed log pattern cluster having the highest calculated similarity to the relevant features.

7. The computer system of claim 1, wherein the one or more hardware processors executing the instructions stored in the system memory to assign the streamed log data to the log pattern cluster based on the determined similarities comprise the one or more hardware processors executing the instructions stored in the system memory to assign the streamed log data to a previously formed log pattern cluster.

8. The computer system of claim 7, further comprising the one or more hardware processors executing the instructions stored in the system memory to alter a label for the previously formed log pattern cluster based on words contained in the relevant features.

9. The computer system of claim 1, wherein the one or more hardware processors executing the instructions stored in the system memory to assign the streamed log data to the log pattern cluster based on the determined similarities comprise the one or more hardware processors executing the instructions stored in the system memory to:

create a new log pattern cluster based on the relevant features; and

assign the streamed log data to the new log pattern cluster.

10. The computer system of claim 9, wherein the one or more hardware processors executing the instructions stored in the system memory to determine a similarity between the relevant features and the one or more previously formed log pattern clusters comprise the one or more hardware processors executing the instructions stored in the system memory to:

calculate a similarity between the relevant features and the one or more previously formed log pattern clusters; and

determine that the calculated similarity does not satisfy a similarity threshold.

11. The computer system of claim 9, further comprising the one or more hardware processors executing the instructions stored in the system memory to:

access a label length from user configuration; and

construct a label for the new log pattern cluster by summarizing words contained in the relevant features in accordance with the label length.

12. The computer system of claim 1, wherein the one or more hardware processors executing the instructions stored in the system memory to receive the streamed log data over the network connection comprises the one or more hardware processors executing the instructions stored in the system memory to receive the streamed log data from a Virtual Machine.

13. A method for use at a computer system, the method for clustering streamed data, the method comprising:

receiving streamed log data from the computer system over a network connection;

pre-processing the streamed log data to remove any customer and request related information;

filtering words that include less than a specified number of characters and that have a frequency of occurrence above a specified threshold to select relevant features from within the streamed log data, wherein the relevant features are relevant to a monitored event at the computer system;

determining a similarity between the relevant features and a first previously formed log pattern cluster;

determining a similarity between the relevant features and a second previously formed log pattern cluster; and

automatically assigning the streamed log data to a log pattern cluster based on the determined similarities between the relevant features, the first previously formed log pattern cluster, and the second previously formed log pattern cluster.

14. The method of claim 13, wherein selecting the relevant features from within the streamed log data comprises:

tokenizing the streamed log data into a plurality of words;

for one or more words in the plurality of words, calculating a frequency of occurrence of the word within the plurality of words; and

based on the calculated frequencies, determining a sub-plurality of more unique words from among the plurality of words that form the relevant feature, including ignoring words that both;

include less than a specified number of characters and have a frequency of occurrence above the specified threshold.

15. The method of claim 13, wherein:

determining the similarity between the relevant features and the first previously formed log pattern cluster comprises;

calculating a similarity between the relevant features and the first previously formed log pattern cluster;

accessing a similarly threshold from user configuration; and

determining if the calculated similarity satisfies the similarity threshold;

the method further comprises, from among any other previously formed log pattern clusters having calculated similarities satisfying the similarity threshold, selecting a previously formed log pattern cluster having a highest calculated similarity to the relevant features; and

assigning the streamed log data to the log pattern cluster based on the determined similarities comprises assigning the streamed log data to the previously formed log pattern cluster having the highest calculated similarity to the relevant features.

16. The method of claim 13, wherein assigning the streamed log data to the log pattern cluster based on the determined similarities comprises assigning the streamed log data to the first previously formed log pattern cluster;

and

further comprising altering a label for the first previously formed log pattern cluster based on words contained in the relevant features.

17. The method of claim 13, wherein determining the similarity between the relevant features and the first previously formed log pattern cluster comprises:

calculating a similarity between the relevant features and the first previously formed log pattern cluster; and

determining that the calculated similarity does not satisfy a similarity threshold;

wherein assigning the streamed log data to the log pattern cluster based on the determined similarities comprises;

creating a new log pattern cluster representative of the relevant features based on determining that the calculated similarity does not satisfy a similarity threshold for both the first and the second previously formed log pattern cluster; and

assigning the streamed data to the new log pattern cluster.

18. A computer program product for use at a computer system, the computer program product for implementing a method for clustering streamed data, the computer program product comprises one or more computer storage devices having stored thereon computer-executable instructions that, when executed at a processor, cause the computer system to:

receive streamed error log data from a device over a network connection;

pre-process the streamed error log data to remove any customer and request related information;

filter words that include less than a specified number of characters and that have a frequency of occurrence above a specified threshold to select relevant features from within the streamed error log data, wherein the relevant features are relevant to a monitored event at the device;

for each of a plurality of previously formed error log pattern clusters, determine a similarity between the relevant features and the previously formed error log pattern clusters; and

assign the streamed log data to an error log pattern cluster based on the determined similarities.