Representing result data streams based on execution of data stream language programs

• US 10,409,568 B2
• Filed: 12/18/2017
• Issued: 09/10/2019
• Est. Priority Date: 12/19/2014
• Status: Active Grant

• Current Assignee: SignalFx LLC (Splunk Inc.)
• Original Assignee: SignalFx, Inc. (Splunk Inc.)
• Inventors: Raman, Rajesh, Mukherji, Arijit, Grandy, Kris, Liu, Phillip
• Primary Examiner(s): Nguyen, Duy Khuong T
• Assistant Examiner(s): Hebert, Theodore E
• Application Number:
  US15/845,993
• Publication Number:
  US 20180307471A1
• Time in Patent Office: 631 Days
• Field of Search: 707822
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3086 where the reporting involves the use of self describing data formats, i.e. metadata, markup languages, human readable formats
• CPC Class Codes: G06F 11/34 Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3604 Software analysis for verifying properties of programs byte-code verification G06F9/44589
• CPC Class Codes: G06F 16/164 File meta data generation
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/835 Timestamp
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/88 Monitoring involving counting
• CPC Class Codes: G06F 8/443 Optimisation
• CPC Class Codes: G06F 9/466 Transaction processing
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

An instrumentation analysis system processes data streams by executing instructions specified using a data stream language program. The data stream language allows users to specify a search condition using a find block for identifying the set of data streams processed by the data stream language program. The set of identified data streams may change dynamically. The data stream language allows users to group data streams into sets of data streams based on distinct values of one or more metadata attributes associated with the input data streams. The data stream language allows users to specify a threshold block for determining whether data values of input data streams are outside boundaries specified using low/high thresholds. The elements of the set of data streams input to the threshold block can dynamically change. The low/high threshold values can be specified as data streams and can dynamically change.

Claims

1. A method comprising:

receiving data from a plurality of input data streams;

storing metadata describing the plurality of input data streams, the metadata for each input data stream including one or more attributes associated with the input data stream;

receiving a set of instructions specified using a data stream language program, the set of instructions comprising instructions for processing the plurality of input data streams;

generating one or more result data streams by executing the set of instructions for processing the plurality of input data streams;

for each of the one or more result data streams;

determining a set of values of attributes describing the result data stream based on a last grouping command from a plurality of grouping commands specified in the data stream language program, wherein each of the plurality of grouping commands groups data of the input data streams based on a respective set of metadata attributes;

storing the set of values as metadata describing the result data stream;

generating an identifier for the result data stream and associating the identifier with the metadata describing the result data stream; and

storing data of the result data stream in association with the identifier;

receiving a dynamic expression for finding a set of result data streams based on at least one attribute value of the dynamic expression;

identifying at least one input data stream of the plurality of input data streams and at least one result data stream of the one or more result data streams based on the dynamic expression, wherein the at least one result data stream is described by the at least one attribute value of the dynamic expression;

generating an output based on data values in the identified at least one input data stream and the at least one result data stream; and

sending the output for display via a user interface.

2. The method of claim 1, wherein the set of result data streams is associated with a publish block, the publish block associated with a metric name, further comprising:

using the metric name of the publish block as an attribute describing each of the result data streams in the set of result data streams; and

storing the metric name attribute with the metadata describing each of the result data streams in the set of result data streams.

3. The method of claim 1, wherein one or more input data streams from the plurality of input data streams are generated by instances of instrumented software executing on an external system.

4. The method of claim 1, wherein one or more result data streams are obtained as a result of execution of instructions of another data stream language program.

5. The method of claim 1, wherein generating the output is a result of execution of instructions of another data stream language program.

6. A computer readable non-transitory storage medium storing instructions executable by a processor to perform operations comprising:

receiving data from a plurality of input data streams;

storing metadata describing the plurality of input data streams, the metadata for each input data stream including one or more attributes associated with the input data stream;

receiving a set of instructions specified using a data stream language program, the set of instructions comprising instructions for processing the plurality of input data streams;

generating one or more result data streams by executing the set of instructions for processing the plurality of input data streams;

for each of the one or more result data streams;

determining a set of values of attributes describing the result data stream based on a last grouping command from a plurality of grouping commands specified in the data stream language program, wherein each of the grouping commands groups data of the input data streams based on a respective set of metadata attribute;

storing the set of values as metadata describing the result data stream;

generating an identifier for the result data stream and associating the identifier with the metadata describing the result data stream; and

storing data of the result data stream in association with the identifier;

receiving a dynamic expression for finding a set of result data streams based on at least one attribute value of the dynamic expression;

identifying at least one input data stream of the plurality of input data streams and at least one result data stream of the one or more result data streams based on the dynamic expression, wherein the at least one result data stream is described by the at least one attribute value of the dynamic expression;

generating an output based on data values in the identified at least one input data stream and the at least one result data stream; and

sending the output for display via a user interface.

7. The computer readable non-transitory storage medium of claim 6, wherein the set of result data streams is associated with a publish block, the publish block associated with a metric name, further comprising:

using the metric name of the publish block as an attribute describing each of the result data streams in the set of result data streams; and

storing the metric name attribute with the metadata describing each of the result data streams in the set of result data streams.

8. The computer readable non-transitory storage medium of claim 6, wherein one or more input data streams from the plurality of input data streams are generated by instances of instrumented software executing on an external system.

9. The computer readable non-transitory storage medium of claim 6, wherein one or more result data streams are obtained as a result of execution of instructions of another data stream language program.

10. The computer readable non-transitory storage medium of claim 6, wherein generating the output is a result of execution of instructions of another data stream language program.

11. A computer-implemented system comprising:

a computer processor; and

a computer readable non-transitory storage medium storing instructions thereon, the instructions executable by a processor to perform operations comprising;

receiving data from a plurality of input data streams;

storing metadata describing the plurality of input data streams, the metadata for each input data stream including one or more attributes associated with the input data stream;

receiving a set of instructions specified using a data stream language program, the set of instructions comprising instructions for processing the plurality of input data streams;

generating one or more result data streams by executing the set of instructions for processing the plurality of input data streams;

for each of the one or more result data streams;

determining a set of values of attributes describing the result data stream based on a last grouping command from a plurality of grouping commands specified in the data stream language program, wherein each of the plurality of grouping commands groups data of the input data streams based on a respective set of metadata attribute;

storing the set of values as metadata describing the result data stream;

generating an identifier for the result data stream and associating the identifier with the metadata describing the result data stream; and

storing data of the result data stream in association with the identifier;

receiving a dynamic expression for finding a set of result data streams based on at least one attribute value of the dynamic expression;

identifying at least one input data stream of the plurality of input data streams and at least one result data stream of the one or more result data streams based on the dynamic expression, wherein the at least one result data stream is described by the at least one attribute value of the dynamic expression;

generating an output based on data values in the identified at least one input data stream and the at least one result data stream; and

sending the output for display via a user interface.

12. The system of claim 11, wherein the set of result data streams is associated with a publish block, the publish block associated with a metric name, further comprising:

using the metric name of the publish block as an attribute describing each of the result data streams in the set of result data streams; and

storing the metric name attribute with the metadata describing each of the result data streams in the set of result data streams.

13. The system of claim 11, wherein one or more input data streams from the plurality of input data streams are generated by instances of instrumented software executing on an external system.

14. The system of claim 11, wherein one or more result data streams are obtained as a result of execution of instructions of another data stream language program.

15. The system of claim 11, wherein generating the output is a result of execution of instructions of another data stream language program.