Scalable log analytics

• US 9,244,755 B2
• Filed: 05/20/2013
• Issued: 01/26/2016
• Est. Priority Date: 05/20/2013
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Huang, Mark, Lin, Junyuan
• Primary Examiner(s): KUDIRKA, JOSEPH R
• Application Number:
  US13/897,994
• Publication Number:
  US 20140344622A1
• Time in Patent Office: 981 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36

Abstract

Large amounts of unstructured log data generated by software and infrastructure components of a computing system are processed and analyzed in real time to identify anomalies and potential problems within the computing system. A log analytics module reduces both the volume and level of detail of log data by first classifying log messages into message types based on their content similarity. The log analytics module may then further reduce data by grouping bursts of log messages into log events. Patterns within these log events, such as the collection and number of different message types that comprise the event, can be used to identify anomalous events.

Claims

1. A method for providing real-time analysis of log messages for a computer infrastructure, the method comprising:

receiving a plurality of log messages including a first log message;

generating a sketch associated with the first log message, wherein the the sketch includes a tuple of fingerprint values generated by processing a subset of words of the first log message through a fingerprint function;

determining a message type for the first log message based on a comparison of the generated sketch to a plurality of sketches stored in an index, wherein log messages of a same message type have similar sketches;

determining a first log event associated with one or more of the plurality of log messages occurring with a time interval, wherein the first log event comprises a first composition of message types corresponding to the one or more of the plurality of log messages associated with the first log event;

determining an event type for the first log event based on a comparison of the first composition of message types to a plurality of compositions of message types stored in the index; and

determining an anomalous log event within the plurality of log messages based on the event type for the first log event.

2. The method of claim 1, wherein the fingerprint function comprises a hash function and the fingerprint values comprise integers generated by the hash function.

3. The method of claim 1, wherein log events of a same event type have similar compositions of message types.

4. The method of claim 1, wherein the determining the event type for the first log event comprises:

upon determining the first composition of message types is similar to at least one of the plurality of compositions of message types, assigning the event type to be a same event type corresponding to the similar composition of message types.

5. The method of claim 1, wherein the determining the event type of the first log event comprises:

upon determining the first composition of message types is not similar to any of the plurality of compositions of message types, assigning a new event type for the first log event and inserting the first composition of message types into the index.

6. The method of claim 1, wherein the first composition of message types for the first log event comprises a list of pairs of a message type identifier and corresponding frequency of occurrence.

7. The method of claim 6,

wherein the index comprises a plurality of hash tables configured to map compositions of message types to event types;

and

wherein the determining the event type of the first log event comprises performing lookups in the plurality of hash tables according to each pair of message type identifier and corresponding frequency of occurrence.

8. The method of claim 1, wherein the determining the anomalous log event within the plurality of log messages further comprises:

determining the anomalous log event that differs in composition of message types based on the event type for the first log event.

9. The method of claim 1, wherein the determining the anomalous log event within the plurality of log messages further comprises:

determining the anomalous log event based on the event type for the first log event and further based on frequency of occurrence over time.

10. A non-transitory computer-readable storage medium comprising instructions that, when executed in a computing device, provide real-time analysis of log messages for a computer infrastructure, by performing the steps of:

receiving a plurality of log messages including a first log message;

generating a sketch associated with the first log message, wherein the sketch includes a tuple of fingerprint values generated by processing a subset of words of the first log message through a fingerprint function;

determining a message type for the first log message based on a comparison of the generated sketch to a plurality of sketches stored in an index, wherein log messages of a same message type have similar sketches;

determining a first log event associated with one or more of the plurality of log messages occurring with a first time interval, wherein the first log event comprises a first composition of message types corresponding to the one or more of the plurality of log messages associated with the first log event;

determining an event type for the first log event based on a comparison of the first composition of message types to a plurality of compositions of message types stored in the index; and

determining an anomalous log event within the plurality of log messages based on the event type for the first log event.

11. The non-transitory computer-readable storage medium of claim 10, wherein the fingerprint function comprises a hash function and the fingerprint values comprise integers generated by the hash function.

12. The non-transitory computer-readable storage medium of claim 10, wherein log events of a same event type have similar compositions of message types.

13. The non-transitory computer-readable storage medium of claim 10, wherein the determining the event type for the first log event comprises:

upon determining the first composition of message types is similar to at least one of the plurality of compositions of message types, assigning the event type to be a same event type corresponding to the similar composition of message types.

14. The non-transitory computer-readable storage medium of claim 10, wherein the determining the event type of the first log event comprises:

upon determining the first composition of message types is not similar to any of the plurality of compositions of message types, assigning a new event type for the first log event and inserting the first composition of message types into the index.

15. The non-transitory computer-readable storage medium of claim 10, wherein the first composition of message types for the first log event comprises a list of pairs of a message type identifier and corresponding frequency of occurrence.

16. The non-transitory computer-readable storage medium of claim 15,

wherein the index comprises a plurality of hash tables configured to map compositions of message types to event types;

and

wherein the determining the event type of the first log event comprises performing lookups in the plurality of hash tables according to each pair of message type identifier and corresponding frequency of occurrence.

17. The non-transitory computer-readable storage medium of claim 10, wherein the determining the anomalous log event within the plurality of log messages further comprises:

determining the anomalous log event that differs in composition of message types based on the event type for the first log event.

18. The non-transitory computer-readable storage medium of claim 10, wherein the determining the anomalous log event within the plurality of log messages further comprises:

determining the anomalous log event based on the event type for the first log event and further based on frequency of occurrence over time.

19. A computer system for providing real-time analysis of log messages for a computer infrastructure, the computer system comprising:

a system memory;

a storage device having (i) a plurality of log messages including a first log message and (ii) an index having a plurality of sketches and compositions of message types; and

a processor programmed to carry out the steps of;

generating a sketch associated with the first log message, wherein the sketch includes a tuple of fingerprint values generated by processing a subset of words of the first log message through a fingerprint function;

determining a message type for the first log message based on a comparison of the generated sketch to a plurality of sketches stored in the index, wherein log messages of a same message type have similar sketches;

determining a first log event associated with one or more of the plurality of log messages occurring with a first time interval, wherein the first log event comprises a first composition of message types corresponding to the one or more of the plurality of log messages associated with the first log event;

determining an event type for the first log event based on a comparison of the first composition of message types to a plurality of compositions of message types stored in the index; and

determining an anomalous log event within the plurality of log messages based on the event type for the first log event.

20. The computer system of claim 19, wherein the processor programmed to carry out the step of determining the event type for the first log event is further programmed to carry out the steps of:

upon determining the first composition of message types is similar to at least one of the plurality of compositions of message types, assigning the event type to be a same event type corresponding to the similar composition of message types.