Processing of performance data and raw log data from an information technology environment

• US 10,353,957 B2
• Filed: 01/31/2017
• Issued: 07/16/2019
• Est. Priority Date: 04/30/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Bingham, Brian, Fletcher, Tristan, Bhide, Alok Anant
• Primary Examiner(s): Nguyen, Cam Linh T
• Application Number:
  US15/421,370
• Publication Number:
  US 20170255711A1
• Time in Patent Office: 896 Days
• Field of Search: 707721, 707722
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/301 where the computing system is a virtual computing platform, e.g. logically partitioned systems virtual machines G06F9/45533; logical partitioning of resources G06F9/5077
• CPC Class Codes: G06F 11/3024 where the computing system component is a central processing unit [CPU]
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3037 where the computing system component is a memory, e.g. virtual memory, cache accessing, addressing or allocating within memory systems or architectures G06F12/00; checking stores for correct operation G11C29/00
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/9017 using directory or table look-up use of a directory or look-up table in file systems G06F16/13
• CPC Class Codes: G06F 16/90328 using search space presentation or visualization, e.g. category or range presentation and selection
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors

Abstract

The disclosed system and method acquire and store performance measurements relating to performance of a component in an information technology (IT) environment and log data produced by the IT environment, in association with corresponding time stamps. The disclosed system and method correlate at least one of the performance measurements with at least one of the portions of log data.

Claims

1. A method comprising:

acquiring, by a computer system, a plurality of performance measurements for a performance metric associated with at least one hardware or software component of an information technology (IT) environment;

acquiring, by the computer system, a plurality of portions of raw log data from at least one log file, the portions of raw log data representing activity of at least one hardware or software component of the IT environment;

storing, by the computer system, the acquired performance measurements in a first format;

storing, by the computer system, the acquired portions of raw log data from the at least one log file in a second format different from the first format; and

correlating, by the computer, at least one of the stored performance measurements with at least one of the stored portions of raw log data from the at least one log file, based on a correlation criterion, wherein said correlating includes

in response to a user-specified search query including the correlation criterion, applying the search query to the stored performance measurements in the first format and the stored portions of log data in the second format different from the first format,

causing display, via a graphical user interface, of a performance measurement that satisfies the correlation criterion, and

causing display, via the graphical user interface, of a portion of raw log data, from the at least one log file, that satisfies the correlation criterion.

2. A method as recited in claim 1, wherein the correlation criterion is a time-based criterion.

3. A method as recited in claim 1, wherein the correlation criterion is not a time-based criterion.

4. A method as recited in claim 1, wherein the correlation criterion is a user-specified search criterion.

5. A method as recited in claim 1, wherein the correlation criterion is a user-specified search criterion that is not a time-based criterion.

6. A method as recited in claim 1, wherein said correlating comprises causing concurrent display of the performance measurements that satisfy the correlation criterion and a listing of raw log data that satisfy the correlation criterion.

7. A method as recited in claim 1, wherein the correlation criterion relates to a machine in the IT environment.

8. A method as recited in claim 1, further comprising:

obtaining the correlation criterion from a user input to a graphical user interface element that enables a user to input a search criterion as said correlation criterion but does not enable input of a complete search query.

9. A method as recited in claim 1, further comprising:

obtaining the correlation criterion from a user'"'"'s selection from a drop-down list in a graphical user interface.

10. A method as recited in claim 1, wherein the correlation criterion relates to a machine in the IT environment and is specified by a user'"'"'s selection from a drop-down list of machines included in the IT environment.

11. A method as recited in claim 1, further comprising, prior to said correlating:

obtaining the correlation criterion from a search query input by a user into a query box in a text-based search query language.

12. A method as recited in claim 1, wherein

said storing comprises storing each of the acquired performance measurements and each of the acquired portions of raw log data with a time-stamp in a time-series data store;

said correlating further comprises identifying at least one of the stored performance measurements and at least one of the stored portions of raw log data that have time stamps that satisfy a user-specified time criterion.

13. A method as recited in claim 1, wherein the performance measurements have been determined by direct measurement of a hardware or software component in the IT environment.

14. A method as recited in claim 1, wherein the plurality of portions of raw log data are derived from a text-based log file.

15. A method as recited in claim 1, wherein the plurality of performance measurements are not from a log file and are acquired independently of the plurality of portions of raw log data.

16. A method as recited in claim 1, wherein the plurality of performance measurements are acquired independently of the plurality of portions of log data, and wherein the performance measurements have been determined by direct measurement of a hardware or software component in the IT environment and the plurality of portions of raw log data are from a text-based log file.

17. A method as recited in claim 1, wherein the plurality of performance measurements are acquired independently of the plurality of portions of log data by direct measurement of a hardware or software component in the IT environment, and the plurality of portions of raw log data are acquired independently of the performance measurements.

18. A method as recited in claim 1, wherein acquiring the plurality of performance measurements comprises acquiring, via an application programming interface (API), the plurality of performance measurements from a third-party software application that collects the performance measurements.

19. A method as recited in claim 1, wherein the performance measurements are stored in a first time-series data store, and the portions of raw log data are stored in a second time-series data store separate from the first time-series data store.

20. A method as recited in claim 1, wherein the performance measurements are stored in a time-series data store in the first format, and the portions of raw log data are stored in said time-series data store in the second format different from the first format.

21. A method as recited in claim 1, wherein the performance measurements are stored in a first time-series data store in the first format, and the portions of raw log data are stored in a second time-series data store separate from the first time-series data store in the second format different from the first format.

22. A method as recited in claim 1, wherein the performance measurements are stored in a first time-series data store in the first format, and the portions of raw log data are stored in a second time-series data store separate from the first time-series data store in the second format different from the first format;

the method further comprising;

receiving a user-specified search query in a query language for searching the first and second time-series data stores, the user-specified search query containing the correlation criterion;

wherein said correlating includes, in response to the user-specified search query, searching the first time-series data store for performance data that satisfy the correlation criterion and searching the second time-series data store for log data that satisfy the correlation criterion.

23. A method as recited in claim 1, wherein the performance measurements are stored in a first time-series data store in the first format, and the portions of raw log data are stored in a second time-series data store separate from the first time-series data store in the second format different from the first format;

the method further comprising;

receiving the search query, wherein the correlation criterion comprises a user-specified value or a user-specified range of values, for a user-specified field;

wherein said correlating includes, in response to the user-specified search query, searching the first time-series data store for performance data that satisfy the correlation criterion and searching the second time-series data store for log data that satisfy the correlation criterion.

24. A method as recited in claim 1, further comprising:

acquiring structure data indicative of structure characteristics of the IT environment;

storing the acquired structure data indicative of structure characteristics of the IT environment; and

correlating a performance characteristic of the IT environment with a structure characteristic of the IT environment, based on the stored performance measurements and stored structure data.

25. A method as recited in claim 1, further comprising:

acquiring structure data indicative of structure characteristics of the IT environment, wherein the structure data is derived from log data from the IT environment;

storing the acquired structure data indicative of structure characteristics of the IT environment; and

correlating a performance characteristic of the IT environment with a structure characteristic of the IT environment, based on the stored performance measurements and stored structure data.

26. A method as recited in claim 1, wherein the performance metric comprises a performance metric for at least one hardware or software resource of a computer system.

27. A method as recited in claim 1, wherein the performance metric comprises a performance metric for at least one virtual machine or virtual machine host.

28. A method as recited in claim 1, wherein the performance metric comprises a performance metric for a virtual machine cluster.

31. A method as recited in claim 1, wherein the correlation criterion relates to a machine in the IT environment.

32. A method as recited in claim 1, wherein the correlation criterion includes a time-based criterion and relates to a machine in the IT environment.

29. A non-transitory machine-readable storage medium for use in a processing system of a data intake and query system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:

acquiring a plurality of performance measurements for a performance metric associated with at least one hardware or software component of an information technology (IT) environment;

acquiring a plurality of portions of raw log data from at least one log file, the portions of raw log data representing activity of at least one hardware or software component of the IT environment;

storing the acquired performance measurements in a first format;

storing the acquired portions of raw log data from the at least one log file in a second format different from the first format; and

correlating at least one of the stored performance measurements with at least one of the stored portions of raw log data from the at least one log file, based on a correlation criterion, wherein said correlating includes

in response to a user-specified search query including the correlation criterion, applying the search query to the stored performance measurements in the first format and the stored portions of log data in the second format different from the first format,

causing display, via a graphical user interface, of a performance measurement that satisfies the correlation criterion, and

causing display, via the graphical user interface, of a portion of raw log data, from the at least one log file, that satisfies the correlation criterion.

30. A system comprising:

a communication device through which to communicate on a computer network;

and

at least one processor operatively coupled to the communication device and configured to perform operations including acquiring a plurality of performance measurements for a performance metric associated with at least one hardware or software component of an information technology (IT) environment;

acquiring a plurality of portions of raw log data from at least one log file, the portions of raw log data representing activity of at least one hardware or software component of the IT environment;

storing the acquired performance measurements in a first format;

storing the acquired portions of raw log data from the at least one log file in a second format different from the first format; and

correlating at least one of the stored performance measurements with at least one of the stored portions of raw log data from the at least one log file, based on a correlation criterion, wherein said correlating includes

in response to a user-specified search query including the correlation criterion, applying the search query to the stored performance measurements in the first format and the stored portions of log data in the second format different from the first format,

causing display, via a graphical user interface, of a performance measurement that satisfies the correlation criterion, and

causing display, via the graphical user interface, of a portion of raw log data, from the at least one log file, that satisfies the correlation criterion.