PROBABILITY-DISTRIBUTION-BASED LOG-FILE ANALYSIS

• US 20160277268A1
• Filed: 03/17/2015
• Published: 09/22/2016
• Est. Priority Date: 03/17/2015
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Brown, Darren, McLaughlin, Matt Roy, Herlocker, Jon, Kushmerick, Nicholas, Lin, Junyuan
• Granted Patent: US 11,048,608 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3075 the data filtering being achieved in order to maintain consistency among the monitored data, e.g. ensuring that the monitored data belong to the same timeframe, to the same system or component
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

The current document is directed to systems, and methods incorporated within the systems, that carry out probability-distribution-based analysis of log-file entries. A monitoring subsystem within a distributed computer system uses probability-distribution-based analysis of log-file entries to detect changes in the state of the distributed computer system. A log-file-analysis subsystem within a distributed computer system uses probability-distribution-based analysis of log-file entries to identify subsets of log-file entries that predict anomalies and impending problems in the distributed computer system. In many implementations, a numerical comparison of probability distributions of log-file-entry types is used to detect state changes in the distributed computer system.

Claims

1. A log-file analysis system comprising:

one or more processors;

one or more memories; and

computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the log-file analysis system to generate a first probability distribution of all or a subset of the event types in a first set of log-file entries,

generate a second probability distribution of all or a subset of the event types in a second set of log files,

generate a numeric divergence metric from the first and second probability distributions, and

when the numeric divergence metric is greater than a threshold value, generate one of a displayed indication, alarm, event, or other physical-state transformation to indicate that the first and second sets of log-file entries exhibit more than a threshold level of difference.

2. The log-file analysis system of claim 1 wherein the log-file analysis system generates a probability distribution of all or a subset of the event types in a set of log-file entries by:

determining and storing the count of log-file entries for each event type of for each event type within the subset of event types;

determining the total number N of log-file entries in the set of log-file entries; and

determining and storing, in memory, the probability for each event type or each event type within the set of event types as the count of log-file entries for the event type divided by N.

3. The log-file analysis system of claim 1 wherein numeric divergence metric is the Jensen-Shannon divergence metric.

4. A log-file analysis subsystem within a computer system having one or more processors, one or more memories, and computer instructions, stored in one or more of the one or more memories that, when executed by one or more of the one or more processors, control the log-file analysis system to monitor a state of the computer system by repeatedly:

generating a probability distribution of all or a subset of the event types in one or more log files for a time interval to represent the state of a monitored computer system for the time interval, and

storing the generated probability distribution in association with an indication of the time interval.

5. The log-file analysis system of claim 4 wherein the log-file analysis system generates a probability distribution of all or a subset of the event types in one or more log files for a time interval by:

selecting a set of log-file entries represents events from the one or more log files;

determining and storing the count of log-file entries for each event type or each event type within the subset of event types;

determining the total number N of log-file entries in the selected set of log-file entries; and

determining and storing, in memory, the probability for each event type or each event type within the subset of event types as the count of log-file entries for the event type divided by N.

6. The log-file analysis subsystem of claim 4 wherein monitoring the state of the computer system by the log-file analysis system further includes:

after generating and storing each probability distribution following generation and storing of an initial set of probability distributions,

computing a divergence metric from the two most recently generated and stored probability distributions; and

when divergence metric is greater than a threshold value, raising an alarm to indicate, or displaying an indication of;

a significant system-state change.

7. The log-file analysis subsystem of claim 4 wherein the divergence metric is the Jensen-Shannon divergence metric.

8. The log-file analysis subsystem of claim 4 monitoring the state of the computer system by the log-file analysis system further includes:

using the stored probability distributions collected over a first time interval spanning multiple shorter, secondary time intervals to generate a typical probability distribution for each of a set of time intervals selected from the multiple shorter, secondary time intervals; and

at subsequent secondary time intervals,

generating a probability distribution for the event types of log entries selected from the most recently completed secondary time interval,

computing a Jensen-Shannon divergence metric for the probability distribution generated from the most recently completed secondary time interval and the typical probability distribution for the most recently completed secondary time interval, and

when Jensen-Shannon divergence metric is greater than a threshold value, raising an alarm to indicate, or displaying an indication of, a system-state change.

9. The log-file analysis subsystem of claim 4 wherein the divergence metric is the Jensen-Shannon divergence metric.

10. The log-file analysis subsystem of claim 4 wherein monitoring the state of the computer system by the log-file analysis system further includes:

for each of a number of different subsets of the event types for which the log-file analysis subsystem has generated and stored probability distributions for different time intervals,

computing a Jensen-Shannon divergence metric for the probability distributions for different pairs of time intervals, and

computing a measure of the variance of the Jensen-Shannon divergence metrics computed for the probability distributions for different pairs of the time intervals; and

selecting, as a basis for a monitoring fingerprint, a subset of the event types having the greatest computed variance.

11. The log-file analysis subsystem of claim 4 wherein the divergence metric is the Jensen-Shannon divergence metric.

12. A method that monitors a state of a distributed computer system that includes multiple, network interconnected discrete computer systems, each having one or more processors, one or more memories, and one or more data-storage devices, one or more of the discrete computer systems including computer instructions, stored in one or more of the one or more memories of the discrete computer system, that, when executed by one or more of the one or more processors, control the discrete computer system to carry out the method comprising:

repeatedly

generating a probability distribution of all or a subset of the event types in one or more log files for a time interval to represent the state of a monitored computer system for the time interval, and

storing the generated probability distribution in association with an indication of the time interval in one or more of one or more memories and/or data-storage devices.

13. The method of claim 12 wherein generating a probability distribution of all or a subset of the event types in one or more log files for a time interval further includes:

selecting a set of log-file entries represents events from the one or more log files;

determining and storing the count of log-file entries for each event type or each event type within the subset of event types;

determining the total number N of log-file entries in the selected set of log-file entries; and

determining and storing, in memory, the probability for each event type or each event type within the subset of event types as the count of log-file entries for the event type divided by N.

14. The method of claim 12 further including:

after generating and storing each probability distribution following generation and storing of an initial set of probability distributions,

computing a divergence metric from the two most recently generated and stored probability distributions; and

when divergence metric is greater than a threshold value, raising an alarm to indicate, or displaying an indication of, a system-state change.

15. The method of claim 14 wherein the divergence metric is the Jensen-Shannon divergence metric.

16. The method of claim 12 further including:

using the stored probability distributions collected over a first time interval spanning multiple shorter, secondary time intervals to generate a typical probability distribution for each of a set of time intervals selected from the multiple shorter, secondary time intervals; and

at subsequent secondary time intervals,

generating a probability distribution for the event types of log entries selected from the most recently completed secondary time interval,

computing a Jensen-Shannon divergence metric for the probability distribution generated from the most recently completed secondary time interval and the typical probability distribution for the most recently completed secondary time interval, and

when Jensen-Shannon divergence metric is greater than a threshold value, raising an alarm to indicate, or displaying an indication of, a system-state change.

17. The method of claim 16 wherein the divergence metric is the Jensen-Shannon divergence metric.

18. The method of claim 12 further including:

for each of a number of different subsets of the event types for which the log-file analysis subsystem has generated and stored probability distributions for different time intervals,

computing a Jensen-Shannon divergence metric for the probability distributions for different pairs of time intervals, and

computing a measure of the variance of the Jensen-Shannon divergence metrics computed for the probability distributions for different pairs of the time intervals; and

selecting, as a basis for a monitoring fingerprint, a subset of the event types having the greatest computed variance.

19. The method of claim 18 wherein the divergence metric is the Jensen-Shannon divergence metric.

20. A computer-readable device that stores a set of computer instructions that, when executed on one or more processors of a computer system that additionally includes one or more memories, controls the computer system to:

generate a first probability distribution of all or a subset of the event types in a first set of log-file entries,

generate a second probability distribution of all or a subset of the event types in a second set of log files,

generate a numeric divergence metric from the first and second probability distributions, and

when the numeric divergence metric is greater than a threshold value, generate one of a displayed indication, alarm, event, or other physical-state transformation to indicate that the first and second sets of log-file entries exhibit more than a threshold level of difference.