Identifying Anomalous Conditions in Machine Data

• US 20160147589A1
• Filed: 01/31/2016
• Published: 05/26/2016
• Est. Priority Date: 09/07/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Noel, Cary Glen, Pakkirisamy, Kirubakaran, Raitz, Alex, Tsai, Pierre
• Granted Patent: US 9,442,789 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0712 in a virtual computing platform, e.g. logically partitioned systems
• CPC Class Codes: G06F 11/0721 within a central processing unit [CPU]
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

Embodiments are directed towards the visualization of machine data received from computing clusters. Embodiments may enable improved analysis of computing cluster performance, error detection, troubleshooting, error prediction, or the like. Individual cluster nodes may generate machine data that includes information and data regarding the operation and status of the cluster node. The machine data is received from each cluster node for indexing by one or more indexing applications. The indexed machine data including the complete data set may be stored in one or more index stores. A visualization application enables a user to select one or more analysis lenses that may be used to generate visualizations of the machine data. The visualization application employs the analysis lens to produce visualizations of the computing cluster machine data.

Claims

1. A computer-implemented method comprising:

receiving machine data from one or more data sources, the machine data related to performance aspects of one or more information technology systems;

parsing the received machine data to determine event boundaries within the received machine data to generate a plurality of time stamped events, thereby transforming the received machine data into the plurality of time stamped events, the time stamp for each event extracted from the parsed machine data associated with that event;

analyzing the plurality of time stamped events using heuristics to identify an occurrence of an event pattern;

comparing the occurrence of the event pattern to one or more registered event patterns to identify whether the event pattern is an anomalous pattern, the one or more registered event patterns indicative of performance aspects of the one or more information technology systems; and

generating a notification based upon the identification.

2. The method of claim 1, wherein the one or more registered event patterns are specified by a user.

3. The method of claim 1, wherein the analyzing data in the subset of events uses scripts to analyze the event pattern.

4. The method of claim 1, wherein the analyzing data in the subset of events uses regular expressions to analyze the event pattern.

5. The method of claim 1, wherein the notification is an email message.

6. The method of claim 1, wherein the notification is a text message.

7. The method of claim 1, wherein the notification is an audio user-interface alarm.

8. The method of claim 1, wherein the notification is a visual user-interface alarm.

9. The method of claim 1, wherein the machine data includes log files.

10. The method of claim 1, wherein the event pattern comprises a pattern represented in a heat map, and wherein the heat map indicates a metric for each information technology system of the one or more information technology systems.

11. The method of claim 1, wherein the event pattern comprises a pattern represented in a heat map, wherein the heat map indicates a metric for each information technology system of the one or more information technology systems, and wherein analyzing data in a set of time stamped events further comprises determining whether the heat map is expanding or contracting by comparing the heat map to previous heat maps.

12. The method of claim 1, further comprising:

upon identification of the occurrence of the event pattern, generating a visualization representing one or more information technology systems of the one or more information technology systems that generated events associated with the event pattern, the visualization also indicating a metric for the represented one or more information technology systems.

13. The method of claim 1, wherein the analysis to identify the occurrence of the event pattern is used to discover an occurrence of an abnormal condition.

14. The method of claim 1, wherein the analysis to identify the occurrence of the event pattern is used to discover that an error has occurred.

15. The method of claim 1, wherein the analysis to identify the occurrence of an event pattern includes determining that the occurrence of the event pattern happened during a limited-duration time window associated with the registered event pattern.

16. The method of claim 1, wherein the analysis to identify the occurrence of an event pattern includes determining that the occurrence of the event pattern happened during a user selected time window associated with the registered event pattern.

17. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:

receiving machine data from a computing cluster, the computing cluster including a plurality of computational nodes;

parsing the received machine data to determine event boundaries within the received machine data to generate a plurality of time stamped events, the time stamp for each event extracted from the parsed machine data associated with that event;

analyzing the plurality of time stamped events using heuristics to identify an occurrence of an event pattern;

comparing the occurrence of the event pattern to one or more registered event patterns that are indicative of a previously determined or known problem for operation of the computing cluster to identify a particular registered event pattern that is the same as or similar to the occurrence of the event pattern; and

generating a notification based upon the identification.

18. The one or more non-transitory computer-readable storage media of claim 17, wherein the one or more registered event patterns are specified by a user.

19. The one or more non-transitory computer-readable storage media of claim 17, wherein the one or more sequences of instructions, which when executed by the one or more processors further cause performance of:

upon identification of the occurrence of the event pattern, generating a visualization representing one or more information technology systems of the one or more information technology systems that generated events associated with the event pattern, the visualization also indicating a metric for the represented one or more information technology systems.

20. The one or more non-transitory computer-readable storage media of claim 17, wherein the analysis to identify the occurrence of the event pattern is used to discover an occurrence of an abnormal condition.

21. The one or more non-transitory computer-readable storage media of claim 17, wherein the analysis to identify the occurrence of the event pattern is used to discover that an error has occurred.

22. The one or more non-transitory computer-readable storage media of claim 17, wherein the analysis to identify the occurrence of an event pattern includes determining that the occurrence of the event pattern happened during a limited-duration time window associated with the registered event pattern.

23. The one or more non-transitory computer-readable storage media of claim 17, wherein the analysis to identify the occurrence of an event pattern includes determining that the occurrence of the event pattern happened during a user selected time window associated with the registered event pattern.

24. An apparatus comprising:

a subsystem, implemented at least partially in hardware, that receives machine data from a computing cluster, the computing cluster including a plurality of computational nodes;

a subsystem, implemented at least partially in hardware, that parses the received machine data to determine event boundaries within the received machine data to generate a plurality of time stamped events, the time stamp for each event extracted from the parsed machine data associated with that event;

a subsystem, implemented at least partially in hardware, that analyzes the plurality of time stamped events using heuristics to identify an occurrence of an event pattern;

a subsystem, implemented at least partially in hardware, that compares the occurrence of the event pattern to one or more registered event patterns that are indicative of a previously determined or known problem for operation of the computing cluster to identify a particular registered event pattern that is the same as or similar to the occurrence of the event pattern;

a subsystem, implemented at least partially in hardware, that generates a notification based upon the identification.

25. The apparatus of claim 24, wherein the one or more registered event patterns are specified by a user.

26. The apparatus of claim 24, wherein the analysis to identify the occurrence of the event pattern subsystem is used to discover an occurrence of an abnormal condition.

27. The apparatus of claim 24, wherein the analysis to identify the occurrence of the event pattern subsystem is used to discover that an error has occurred.

28. The apparatus of claim 24, further comprising:

a subsystem, implemented at least partially in hardware, that, upon identification of the occurrence of the event pattern, generates a visualization representing one or more information technology systems of the one or more information technology systems that generated events associated with the event pattern, the visualization also indicating a metric for the represented one or more information technology systems.

29. The apparatus of claim 25, wherein the analysis to identify the occurrence of an event pattern subsystem determines that the occurrence of the event pattern happened during a limited-duration time window associated with the registered event pattern.

30. The apparatus of claim 25, wherein the analysis to identify the occurrence of an event pattern subsystem determines that the occurrence of the event pattern happened during a user selected time window associated with the registered event pattern.