Real-time processing of data streams received from instrumented software

• US 10,394,692 B2
• Filed: 01/25/2016
• Issued: 08/27/2019
• Est. Priority Date: 01/29/2015
• Status: Active Grant

• Current Assignee: SignalFx LLC (Splunk Inc.)
• Original Assignee: SignalFx, Inc. (Splunk Inc.)
• Inventors: Liu, Phillip, Mukherji, Arijit, Raman, Rajesh
• Primary Examiner(s): Brophy, Matthew J
• Application Number:
  US15/005,489
• Publication Number:
  US 20160224459A1
• Time in Patent Office: 1,310 Days
• Field of Search: None
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/3644 by instrumenting at runtime
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

An analysis system receives data streams generated by instances of instrumented software executing on external systems. The analysis system evaluates an expression using data values of the data streams over a plurality of time intervals. For example, the analysis system may aggregate data values of data streams for each time interval. The analysis system determines whether or not a data stream is considered for a time interval based on when the data value arrives during the time interval. The analysis system determines a maximum expected delay value for each data stream being processed. The analysis system evaluates the expression using data values that arrive before their maximum expected delay values. The analysis system also determines a failure threshold value for a data stream. If a data value of a data stream fails to arrive before the failure threshold value, the analysis system marks the data stream as dead.

Claims

1. A method for processing data generated by instrumented software, the method comprising:

receiving, by an analysis system, information identifying a set of data streams, each data stream generated by an instance of instrumented software executing on an external system;

receiving a specification of an expression, the expression processing data values of the set of data streams, each of the data values associated with a time interval of a plurality of time intervals for evaluating the expression, each time interval having a beginning point and an end point; and

for each of the plurality of time intervals for evaluating the expression;

for each data stream of the set of data streams, determining a maximum expected delay for the data stream, the maximum expected delay representing a limit on arrival delay within the time interval for a data value of the data stream from a corresponding instance of instrumented software to the analysis system, wherein the maximum expected delay is less than a length of the time interval,

monitoring the set of data streams for arrival of data values within the time interval,

modifying the set of data streams for the time interval by excluding a data stream if the data value of the data stream fails to arrive within the maximum expected delay for the data stream after the beginning point of the time interval,

evaluating the expression using the arrived data values of the modified set of data streams for the time interval, and

prior to the end point of the time interval, sending a value of the evaluated expression for presentation.

2. The method of claim 1, wherein the expression aggregates the data values of the set of data streams, the data values associated with a time interval.

3. The method of claim 1, wherein the maximum expected delay is determined based on historical arrival delays of data values of the data stream.

4. The method of claim 1, wherein the maximum expected delay for a time interval is determined based on the arrival delay of the data value of the data stream for a previous time interval.

5. The method of claim 1, wherein the maximum expected delay is determined to be a moving aggregate value based on arrival delays of at least three previous data values of the data stream.

6. The method of claim 5, wherein the maximum expected delay is determined to be the moving aggregate value increased by a factor.

7. The method of claim 1, further comprising:

storing data values of data streams that arrive after the maximum expected delay in a persistent data store.

8. The method of claim 1, wherein data streams in the set of data streams dynamically change from one time interval to a subsequent time interval.

9. The method of claim 1, further comprising:

determining a failure threshold value for a data stream; and

marking the data stream as dead if a data value of the data stream fails to arrive before the failure threshold value.

10. The method of claim 9, further comprising:

excluding a data stream marked dead from the set of data streams monitored for arrival of data values for one or more subsequent time intervals.

11. The method of claim 9, further comprising:

marking the data stream live if a data value is subsequently received for the data stream; and

including the data stream in the set of data streams monitored for arrival of data values for one or more subsequent time intervals.

12. The method of claim 1, further comprising:

configuring results of evaluation of the expression for presentation by a real-time chart that is updated for every time interval.

13. A computer readable non-transitory storage medium storing instructions for processing data generated by instrumented software, the instructions when executed by a processor cause the processor to perform the steps of:

receiving, by an analysis system, information identifying a set of data streams, each data stream generated by an instance of instrumented software executing on an external system;

receiving a specification of an expression, the expression processing data values of the set of data streams, each of the data values associated with a time interval of a plurality of time intervals for evaluating the expression, each time interval having a beginning point and an end point; and

for each of the plurality of time intervals for evaluating the expression;

for each data stream of the set of data streams, determining a maximum expected delay for the data stream, the maximum expected delay representing a limit on arrival delay within the time interval for a data value of the data stream from a corresponding instance of instrumented software to the analysis system, wherein the maximum expected delay is less than a length of the time interval,

monitoring the set of data streams for arrival of data values within the time interval,

modifying the set of data streams for the time interval by excluding a data stream if the data value of the data stream fails to arrive within the maximum expected delay for the data stream after the beginning point of the time interval,

evaluating the expression using the arrived data values of the modified set of data streams for the time interval, and

prior to the end point of the time interval, sending a value of the evaluated expression for presentation.

14. The computer readable non-transitory storage medium of claim 13, wherein the expression aggregates the data values of the set of data streams, the data values associated with a time interval.

15. The computer readable non-transitory storage medium of claim 13, wherein the maximum expected delay is determined based on historical arrival delays of data values of the data stream.

16. The computer readable non-transitory storage medium of claim 13, wherein the maximum expected delay is determined to be a moving aggregate value based on arrival delays of at least three previous data values of the data stream.

17. The computer readable non-transitory storage medium of claim 13, wherein the instructions further cause the processor to perform the steps of:

determining a failure threshold value for a data stream; and

marking the data stream as dead if a data value of the data stream fails to arrive before the failure threshold value.

18. The computer readable non-transitory storage medium of claim 17, wherein the instructions further cause the processor to perform the steps of:

excluding a data stream marked dead from the set of data streams monitored for arrival of data values for one or more subsequent time intervals.

19. The computer readable non-transitory storage medium of claim 17, wherein the instructions further cause the processor to perform the steps of:

marking the data stream live if a data value is subsequently received for the data stream; and

including the data stream in the set of data streams monitored for arrival of data values for one or more subsequent time intervals.

20. A computer-implemented system for processing data generated by instrumented software, the system comprising:

a computer processor; and

a computer readable non-transitory storage medium storing instructions thereon, the instructions when executed by a processor cause the processor to perform the steps of;

receiving, by an analysis system, information identifying a set of data streams, each data stream generated by an instance of instrumented software executing on an external system;

receiving a specification of an expression, the expression processing data values of the set of data streams, each of the data values associated with a time interval of a plurality of time intervals for evaluating the expression, each time interval having a beginning point and an end point; and

for each of the plurality of time intervals for evaluating the expression;

for each data stream of the set of data streams, determining a maximum expected delay for the data stream, the maximum expected delay representing a limit on arrival delay within the time interval for a data value of the data stream from a corresponding instance of instrumented software to the analysis system, wherein the maximum expected delay is less than a length of the time interval,

monitoring the set of data streams for arrival of data values within the time interval,

modifying the set of data streams for the time interval by excluding a data stream if the data value of the data stream fails to arrive within the maximum expected delay for the data stream after the beginning point of the time interval,

evaluating the expression using the arrived data values of the modified set of data streams for the time interval, and

prior to the end point of the time interval, sending a value of the evaluated expression for presentation.