Streaming and sampling in real-time log analysis

• US 8,850,263 B1
• Filed: 09/14/2012
• Issued: 09/30/2014
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Siddiqui, Muhammad Ali, Borst, Alexander S., Yourtee, Kendra A., Vance, Amos Dylan, Kaufmann, Miles C.
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US13/619,344
• Time in Patent Office: 746 Days
• Field of Search: 714/20, 714/47.3
• US Class Current: 714/20
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence

Abstract

Technologies are described herein for incorporating streaming and/or sampling in real-time log analysis. Representative samples of log data are extracted from the log files on a number of monitored hosts and streamed in real-time to log processors for processing. The log processors accumulate and process the representative samples of log data, and track a data completeness value representing an indication of a proportion of total log data represented by the representative samples received. The representative samples of log data are merged and collated. Estimated metrics are calculated from the merged and collated representative samples and the data completeness, and the estimated metrics are published to consumers in near real-time.

Claims

1. A computer-implemented method of providing real-time log analysis comprising:

hashing, by one or more monitored hosts, a value in log messages comprising log files on the one or more monitored hosts;

tagging, by the one or more monitored hosts, each of the log messages with the hashed value;

extracting, by the one or more monitored hosts, representative samples of log data from the log files, each of the representative samples of log data comprising at least a portion of a log message extracted from the log files based on the tagged hashed value;

streaming, by the one or more monitored hosts, the representative samples of log data to a plurality of log processors;

processing, by the plurality of log processors, the representative samples of log data;

determining, by the plurality of log processors, a data completeness of the representative samples of log data processed, the data completeness comprising an indication of a proportion of total log data represented by the representative samples of log data;

merging and collating, by a data accumulation computer, the representative samples of log data;

generating, by the data accumulation computer, an estimated metric value from the merged and collated representative samples of log data based on the data completeness; and

publishing, by the data accumulation computer, the estimated metric value to consumers.

2. The computer-implemented method of claim 1, wherein extracting the representative samples of log data from the log files comprises applying one or more sampling rules to log messages in the log files, each of the one or more sampling rules comprising a pattern to be matched to at least a portion of the data in the log messages.

3. The computer-implemented method of claim 2, further comprising recognizing, by the data accumulation computer, patterns in representative samples of log data and generating updated sampling rules to be sent to the one or more monitored hosts.

4. The computer-implemented method of claim 1, wherein the streaming of the representative samples of log data to the plurality of log processors is performed in real-time and wherein the generation and publishing of the estimated metrics are performed in near real-time.

5. The computer-implemented method of claim 1, wherein each of the plurality of log processors is configured to receive representative samples of log data comprising one or more specific hashed values.

6. The computer-implemented method of claim 1, wherein the data completeness is determined based on information received from the one or more monitored hosts.

7. A computer-implemented method of providing real-time log analysis comprising:

sampling, at one or more monitored hosts, log data from log files on the one or more monitored hosts;

streaming, by the one or more monitored hosts, the sampled log data to at least one log processor;

processing, by the at least one log processor, the sampled log data;

determining, by the at least one log processor, a data completeness of the sampled log data stored processed, the data completeness comprising an indication of a proportion of total log data represented by the sampled log data;

merging and collating, by a data accumulation computer, the sampled log data;

generating, by the data accumulation computer, an estimated metric value from the merged and collated sampled log data based on the data completeness; and

publishing, by the data accumulation computer, the estimated metric value to consumers.

8. The computer-implemented method of claim 7, wherein the sampled log data is streamed in log data samples, each log data sample comprising at least a portion of a log message sampled from the log files.

9. The computer-implemented method of claim 8, further comprising:

hashing, by the one or more monitored hosts, a value in the log message comprising each log data sample;

tagging, by the one or more monitored hosts, each log data sample with the hashed value; and

streaming, by the one or more monitored hosts, each of the log data samples to a target log processor based on the tagged hashed value.

10. The computer-implemented method of claim 7, wherein sampling the log data from the log files comprises utilizing a hashing algorithm to extract a representative sample of log messages from the log files.

11. The computer-implemented method of claim 7, wherein sampling the log data from the log files comprises utilizing a statistical sampling method to extract a representative sample of log messages from the log files.

12. The computer-implemented method of claim 7, wherein sampling the log data from the log files comprises applying one or more sampling rules to log messages in the log files, each of the one or more sampling rules comprising a pattern to be matched to at least a portion of the data in the log messages.

13. The computer-implemented method of claim 12, wherein the one or more sampling rules comprise regular expressions.

14. The computer-implemented method of claim 12, further comprising recognizing, by the data accumulation computer, patterns in representative samples of log data and generating updated sampling rules to be sent to the one or more monitored hosts.

15. A computer-readable storage medium having computer-executable instructions stored thereon that, when executed by a host computer, cause the host computer to:

extract representative samples of log data from log files on the host computer; and

periodically transmit the representative samples of log data from the host computer to one or more log processors, wherein the one or more log processors are configured to process the representative samples of log data and to determine a data completeness of the representative samples of log data processed, and wherein the processed representative samples of log data and the data completeness are utilized to generate an estimated metric value that is published to consumers in near real-time.

16. The computer-readable storage medium of claim 15, wherein extracting representative samples of log data from the log files comprises utilizing a hashing algorithm to extract a representative sample of log messages from the log files.

17. The computer-readable storage medium of claim 15, wherein extracting representative samples of log data from the log files comprises utilizing a statistical sampling method to extract a representative sample of log messages from the log files.

18. The computer-readable storage medium of claim 17, wherein extracting the representative sample of log messages comprises extracting a configured percentage of log messages written to the log files.

19. The computer-readable storage medium of claim 15, wherein extracting representative samples of log data from the log files comprises applying one or more sampling rules to log messages in the log files, each of the one or more sampling rules comprising a pattern to be matched to at least a portion of the data in the log messages.

20. The computer-readable storage medium of claim 19, wherein the one or more sampling rules comprise regular expressions.

21. The computer-readable storage medium of claim 15, wherein each of the representative samples of log data comprise at least a portion of a log message sampled from the log files, and wherein the representative samples of log data are streamed from the host computer the one or more log processors using a multicast protocol.

22. The computer-readable storage medium of claim 21, having further computer-executable instructions stored thereon that cause the host computer to:

hash a value in the log message comprising each representative sample of log data;

tag each representative sample of log data log with the hashed value; and

stream each of the representative samples of log data to a target log processor based on the tagged hashed values.

23. A system for incorporating streaming in real-time log analysis, the system comprising:

a host computer;

an agent executing on the host computer and configured to tag log messages in log files on the host computer with a tag value,

extract log data from the log files, the extracted log data comprising at least a portion of a log message extracted from the log files, and

stream the extracted log data to a log processor;

at least one server computer;

the log processor executing on the at least one server computer and configured to retrieve the extracted log data from the stream,

process the extracted log data by accumulating the log data over a configured interval, and

determine a data completeness of the processed log data, the data completeness indicating a proportion of total log data for the configured interval retrieved by the log processor; and

an accumulation task executing on the at least one server computer and configured to

merge and collate the processed log data,

generate an estimated metric value from the merged and collated log data based on the data completeness; and

publish the estimated metric value to consumers in near real-time.

24. The system of claim 23, wherein the extracted log data is streamed from the host computer the log processor using a multicast protocol.

25. The system of claim 23, wherein the tag values comprise values hashed from a value in each of the log messages.

26. The system of claim 23, wherein the agent is further configured to stream the extracted log data to one of a plurality of log processors the tag values.

27. The system of claim 23, wherein extracting the log data comprises extracting a representative sample of log data from the log files based on the tag values.

28. The system of claim 23, wherein extracting the log data comprises extracting a representative sample of log data from the log files utilizing a statistical sampling method.

29. The system of claim 23, wherein extracting the log data comprises applying one or more sampling rules to log messages in the log files, each of the one or more sampling rules comprising a pattern to be matched to at least a portion of the data in the log messages.

30. The system of claim 29, wherein the one or more sampling rules comprise regular expressions.

31. The system of claim 29, wherein the accumulation task is further configured to recognize patterns in the extracted log data and generate updated sampling rules to be sent to the agent.