TECHNIQUES FOR INGESTING METRICS DATA

• US 20180089328A1
• Filed: 10/31/2016
• Published: 03/29/2018
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Bath, Amrittpal Singh, Shah, Pratiksha, Kandaswamy, Murugan, Patel, Vishal
• Granted Patent: US 10,606,856 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/8373 Query execution
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 2200/24 : involving graphical user interfaces [GUIs]
• CPC Class Codes: H04L 43/08 : Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 67/02 : based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/025 : for remote control or remote monitoring of applications

Abstract

The disclosed embodiments include a method performed by a data intake and query system. The method includes ingesting collected data including metrics data including key values and numerical values, where each numerical value is indicative of a measured characteristic of a computing resource (e.g., device), and populating a first portion of a metric-series index (msidx) file with the key values and a second portion of the msidx file with the numerical values. The first portion is distinct from the second portion. The method further includes generating metrics from the metrics data, where each metric has dimensions populated with at least some of the key values and has one of the numerical values. The method further includes indexing the metrics by at least one of the dimensions.

Claims

1. A method, comprising:

ingesting, by a data intake and query system, collected data including metrics data including a plurality of key values and a plurality of numerical values, each numerical value being indicative of a measured characteristic of a computing resource;

populating, by the data intake and query system, a first portion of a metric-series index (msidx) file with the plurality of key values and a second portion of the msidx file with the plurality of numerical values, the first portion being distinct from the second portion;

generating, by the data intake and query system, a plurality of metrics from the metrics data, each metric having a plurality of dimensions populated with at least some of the plurality of key values and having one of the plurality of numerical values; and

indexing, by the data intake and query system, the plurality of metrics by at least one of the plurality of dimensions.

2. The method of claim 1, further comprising:

receiving, by the data intake and query system, a query including criteria, the query being input by a user and expressed in a pipelined search language;

evaluating, by the data intake and query system, the query by applying the criteria to the first portion of the msidx file to obtain query results indicative of metrics that satisfy the criteria; and

displaying, on a display device, the query results or data indicative of the query results

3. The method of claim 1, wherein each numerical value is a floating point value.

4. The method of claim 1, wherein the metrics data is semi-structured data or structured data.

5. The method of claim 1, wherein each characteristic of a computing resource is a utilization of a processor, a temperature of an electronic component, or a voltage reading of an electronic component.

6. The method of claim 1, wherein the metrics data is received by the data intake and query system over a computer network from a plurality of remote computer systems.

7. The method of claim 1, wherein each of the plurality of dimensions is either a required dimension or an optional dimension, each metric includes a value for each required dimension, and only some of the plurality of metrics include values for some optional dimensions.

8. The method of claim 1, wherein the plurality of dimensions includes a plurality of required dimensions, and each metric includes a value for each required dimension.

9. The method of claim 1, wherein the plurality of dimensions includes a time dimension including a value indicative of when the measured characteristic of the metric was measured and a name dimension indicative of the measured characteristic and the computing resource of the one numerical value.

10. The method of claim 1, wherein the plurality of dimensions includes optional dimensions, and only some of the plurality of metrics include values for some optional dimensions.

11. The method of claim 1, wherein:

the plurality of dimensions includes a host dimension, a manufacturer dimension, and a model dimension; and

only some of the plurality of metrics include values for the host dimension, the manufacturer dimension, or the model dimension.

12. The method of claim 1, wherein the plurality of dimensions includes a user specified dimension selected by a user prior to ingesting the metrics data.

13. The method of claim 1, wherein at least some of the plurality of numerical values are indicative of a time series of measured characteristics of a single computing resource.

14. The method of claim 1, wherein the collected data includes raw data, the method further comprising:

generating, by the data intake and query system, a plurality of events indexed by timestamps, each of the plurality of events including a respective segment of the raw machine data.

15. The method of claim 1, wherein the collected data includes raw data, the method further comprising:

generating, by the data intake and query system, a plurality of events indexed by timestamps, each of the plurality of events including a respective segment of the raw machine data; and

correlating, by the data intake and query system, the plurality of events and the plurality of metrics.

16. The method of claim 1, further comprising, prior to ingesting the collected data:

causing, by the data intake and query system, a universal forwarder to collect the collected data, the collected data being selected from a group consisting of;

only raw data;

raw data and structured metrics data; and

only structured metrics data; and

receiving, by the data intake and query system, the collected data over a computer network.

17. The method of claim 1, further comprising, prior to ingesting the collected data:

causing, by the data intake and query system, a heavy forwarder to collect and locally index the collected data, the collected data being selected from a group consisting of;

only raw data;

raw data and structured metrics data; and

only structured metrics data; and

receiving, by the data intake and query system, the collected data over a computer network.

18. The method of claim 1, further comprising, prior to ingesting the collected data:

causing, by the data intake and query system, a script running on a remote computer system to collect the metrics data from an application running on the remote computer system, the application including the script; and

receiving, by the data intake and query system, the metrics data over an HTTP-based connection of a computer network.

19. The method of claim 1, further comprising, prior to ingesting the collected data:

causing, by the data intake and query system, a background process of a remote computer system to collect the metrics data from an application running on the remote computer system, the background process being independent of the application; and

receiving, by the data intake and query system, the metrics data over a computer network.

20. The method of claim 1, further comprising, prior to ingesting the collected data:

causing, by the data intake and query system, a StatsD collector running on a remote computer system to collect metrics data from the remote computing system; and

receiving, by the data intake and query system, the metrics data over a computer network.

21. The method of claim 1, further comprising, prior to ingesting the collected data:

causing, by the data intake and query system, a StatsD collector running on a remote computer system to collect and aggregate metrics data from the remote computing system to produce the collected data; and

receiving, by the data intake and query system, the collected data over a computer network.

22. The method of claim 1, further comprising, prior to ingesting the collected data:

receiving, by the data intake and query system, a command causing the data intake and query system to obtain the structured metrics data using an add-on reusable software component.

23. The method of claim 1, further comprising, prior to ingesting the collected data:

calling, by the data intake and query system, an application programming interface, API, of a remote computer system to send the metrics data to the data intake and query system; and

receiving, by the data intake and query system, the metrics data over a computer network.

24. The method of claim 1, further comprising, prior to ingesting the collected data:

receiving, by the data intake and query system, a command from a user to modify an add-on reusable software component;

causing, by the data intake and query system, the modified add-on reusable software component to obtain the structured metrics data from a remote computer system; and

receiving, by the data intake and query system, the structured metrics data over a computer network.

25. The method of claim 1, further comprising, prior to ingesting the collected data:

scheduling, by the data intake and query system, transfer of the metrics data collected by a plurality of collectors from a plurality of remote computer system; and

receiving, by the data intake and query system, the structured metrics data over a computer network.

26. A method, comprising:

ingesting, by a data intake and query system, raw data and metrics data including a plurality of key values and a plurality of numerical values, each numerical value being indicative of a measured characteristic of a computing resource;

populating, by the data intake and query system, a first portion of a metric-series index (msidx) file with the plurality of key values and a second portion of the msidx file with the plurality of numerical values, the first portion being distinct from the second portion;

generating, by the data intake and query system, a plurality of metrics from the metrics data, each metric having a plurality of dimensions populated with at least some of the plurality of key values and having one of the plurality of numerical values;

generating, by the data intake and query system, a plurality of events including a timestamp and a segment of the raw data; and

indexing, by the data intake and query system, the plurality of metrics by at least one of the plurality of dimensions; and

indexing, by the data intake and query system, the plurality of events by the timestamps.

27. The method of claim 26, further comprising:

receiving, by the data intake and query system, a query having a scope that includes at least one of the plurality of metrics or the plurality of events, the query being input by a user and expressed in a pipelined search language;

obtaining, by the data intake and query system, results satisfying the received query; and

displaying, on a display device, the results or data indicative of the results.

28. The method of claim 26, wherein each numerical value is a floating point value.

29. The method of claim 26, wherein the metrics data is semi-structured data or structured data.

30. A data intake and query system comprising:

a processor; and

memory containing instructions that, when executed by the electronic device, cause the data intake and query system to;

ingest collected data including metrics data including a plurality of key values and a plurality of numerical values, each numerical value being indicative of a measured characteristic of a computing resource;

populate a first portion of a metric-series index (msidx) file with the plurality of key values and a second portion of the msidx file with the plurality of numerical values, the first portion being distinct from the second portion;

generate a plurality of metrics from the metrics data, each metric having a plurality of dimensions populated with at least some of the plurality of key values and having one of the plurality of numerical values; and

index the plurality of metrics by at least one of the plurality of dimensions.