Quantization of data streams of instrumented software

• US 10,394,693 B2
• Filed: 10/31/2017
• Issued: 08/27/2019
• Est. Priority Date: 10/08/2014
• Status: Active Grant

• Current Assignee: SignalFx LLC (Splunk Inc.)
• Original Assignee: SignalFx, Inc. (Splunk Inc.)
• Inventors: Liu, Phillip, Mukherji, Arijit, Raman, Rajesh, Grandy, Kris, Lindamood, Jack
• Primary Examiner(s): Nguyen, Duy Khuong T
• Assistant Examiner(s): Hebert, Theodore E
• Application Number:
  US15/799,049
• Publication Number:
  US 20180046567A1
• Time in Patent Office: 665 Days
• Field of Search: 717130
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3644 by instrumenting at runtime
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

A data analysis system processes data generated by instrumented software. The data analysis system receives data streams generated by instances of instrumented software executing on systems. The data analysis system also receives metadata describing data streams. The data analysis system receives an expression based on the metadata. The data analysis system receives data of data streams for each time interval and computes the result of the expression based on the received data values. The data analysis system repeats these steps for each time interval. The data analysis system may quantize data values of data streams for each time interval by generating an aggregate value for the time interval based on data received for each data stream for that time interval. The data analysis system evaluates the expression using the quantized data for the time interval.

Claims

1. A method for processing data streams generated by instrumented software, the method comprising:

receiving a plurality of input data streams, each input data stream received from a separate instance of instrumented software executing on at least one external system, each input data stream providing values of a metric, the values generated by the instances of instrumented software at variable time intervals;

receiving a request to periodically evaluate an expression based on the plurality of input data streams;

for each input data stream, identifying a first function for aggregating values of the metric of the input data stream;

generating a plurality of quantized data streams based on the input data streams, each quantized data stream comprising data values occurring periodically at a first fixed time interval, the generating comprising, for each input data stream and for each first fixed time interval;

storing values of the metric of the input data stream received during the fixed time interval in a buffer;

determining a data value of the quantized data stream for the fixed time interval based on the values stored in the buffer by applying the identified first function for aggregating values over the values of the input data stream received within the first fixed time interval and stored in the buffer; and

sending the determined data value as the data value of the quantized data stream for the first fixed time interval;

periodically evaluating the expression based on data values of at least two of the plurality of quantized data streams to generate an output data stream, the evaluating comprising repeatedly;

applying a second function to the data values of the quantized data streams, each data value associated with a second fixed time interval, the second function aggregating values over the quantized data streams; and

sending the output data stream for display via a user interface.

2. The method of claim 1, wherein the expression determines aggregate values over groups of quantized data streams, each group associated with a value of a metadata attribute describing the data streams.

3. The method of claim 1, wherein the data of an input data stream is generated by an instruction executed by the instrumented software, the instruction associated with one of a counter or a gauge.

4. The method of claim 1, the method further comprising:

sending the output data stream for display as a chart updated for the second fixed time interval.

5. The method of claim 1, further comprising:

reusing the buffer for storing the data values of the data stream received during a subsequent fixed time interval.

6. The method of claim 1, further comprising:

determining a rollup time interval, wherein the rollup time interval is larger than the first fixed time interval; and

determining a rollup data stream corresponding to each quantized data stream, the determining comprising;

determining an aggregate value by applying the identified first function to all data values of the quantized data stream generated within the rollup time interval, and

storing the aggregate value as a value of the rollup data stream for the rollup time interval.

7. The method of claim 1, wherein the plurality of input data streams comprises a first input data stream and a second input data stream, and a number of data values of a first input data stream received during a particular fixed time interval is different from a number of data values received for a second input data stream during the particular fixed time interval.

8. The method of claim 1, wherein a number of data values of an input data stream received during a particular fixed time interval is different from a number of data values received for the input data stream during a subsequent fixed time interval.

9. The method of claim 1, wherein identifying the first function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represent a count value; and

determining that the identified first function for the data stream is a sum function.

10. The method of claim 1, wherein identifying the first function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represents a sum value; and

determining that the identified first function for the data stream is a sum function.

11. The method of claim 1, wherein identifying the first function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represent a latest value; and

determining that the identified first function for the data stream selects a latest value of a set of data values received in a time interval.

12. The method of claim 1, wherein identifying the first function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represent an average value; and

determining that the identified first function for the data stream is an average function.

13. The method of claim 1, wherein identifying the first function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represent an average value, each data value comprising a tuple having the average value and a count of data values used for the average; and

determining that the identified first function for the data stream for a time interval;

determines a product of each average value and count value received during the time interval,

determines a sum of the product values, and

determines a ratio of the sum of the product values and a sum of all count values.

14. The method of claim 1, wherein identifying the first function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represent an average value, each data value comprising a tuple having a sum of data values and a count of data values used for the sum; and

determining that the identified first function for the data stream for a time interval;

determines a sum of all the sum values received during the time interval, and

determines a ratio of the sum values and a sum of all count values.

15. The method of claim 1, wherein identifying the first function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represent a maximum value; and

determining that the identified first function for the data stream determines a maximum of data values received within a time interval.

16. The method of claim 1, wherein identifying the first function for aggregating values of the metric of the input data stream comprises:

receiving information indicating that a metric for a data stream represent a minimum value; and

determining that the identified first function for the data stream determines a minimum of data values received within a time interval.

17. The method of claim 1, wherein the expression determines one of:

a percentile value, an average value, a minimum value, a maximum value, a sum value, or a moving average value.

18. A computer readable non-transitory storage medium storing instructions for processing data generated by instrumented software, the instructions executable by a processor to perform operations comprising:

receiving a plurality of input data streams, each input data stream received from a separate instance of instrumented software executing on at least one external system, each input data stream providing values of a metric generated by the instances of instrumented software at variable time intervals;

receiving a request to periodically evaluate an expression based on the plurality of input data streams;

for each input data stream, identifying an aggregation function based on the metric of the input data stream;

generating a plurality of quantized data streams based on the plurality of input data streams, each quantized data stream comprising data values occurring periodically at a first fixed time interval, the generating comprising, for each first fixed time interval, for each input data stream;

storing values of the metric of the input data stream received during the first fixed time interval in a buffer;

determining an aggregate value for the input data stream based on the values stored in the buffer by applying the identified aggregation function over data values of the input data stream received within the first fixed time interval; and

sending the determined data value as the data value of the quantized data stream for the first fixed time interval;

periodically evaluating the expression based on data values of the plurality of at least two of the plurality of quantized data streams to generate an output data stream, the evaluating comprising repeatedly;

applying a second function to the data values of the quantized data streams, each data value associated with a second fixed time interval, the second function aggregating values over the quantized data streams; and

sending the output data stream for display via a user interface.

19. A computer-implemented system for processing data generated by instrumented software, the system comprising:

a computer processor; and

a computer readable non-transitory storage medium storing instructions thereon, the instructions executable by a processor to perform operations comprising;

receiving a plurality of input data streams, each input data stream received from a separate instance of instrumented software executing on at least one external system, each input data stream providing values of a metric generated by the instances of instrumented software at variable time intervals;

receiving a request to periodically evaluate an expression based on the plurality of input data streams;

for each input data stream, identifying an aggregation function based on the metric of the input data stream;

generating a plurality of quantized data streams based on the plurality of input data streams, each quantized data stream comprising data values occurring periodically at a first fixed time interval, the generating comprising, for each first fixed time interval, for each input data stream;

storing values of the metric of the input data stream received during the first fixed time interval in a buffer;

determining an aggregate value for the input data stream based on the values stored in the buffer by applying the identified aggregation function over data values of the input data stream received within the first fixed time interval; and

sending the determined data value as the data value of the quantized data stream for the first fixed time interval;

periodically evaluating the expression based on data values of at least two of the plurality of quantized data streams to generate an output data stream, the evaluating comprising repeatedly;

applying a second function to the data values of the quantized data streams, each data value associated with a second fixed time interval, the second function aggregating values over the quantized data streams; and

sending the output data stream for display via a user interface.