REAL-TIME DATA ANALYTICS FOR STREAMING DATA

• US 20180006924A1
• Filed: 06/30/2016
• Published: 01/04/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Haye, Donnie, Kline, Eric V., Komatsu, Jeffrey G., Spielberg, Anthony C., Steele, JR., Benjamin J., Wargo, John M., Yashchin, Emmanuel, Zulpa, Paul A.
• Granted Patent: US 10,171,334 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/16 Threshold monitoring

Abstract

A computer-implemented method includes receiving a plurality of values of a data stream, where the plurality of values are generated by a monitored device and received in real time. One or more statistical moments of the data stream are updated, by a computer processing device, based on each value of the plurality of values. Each value of the plurality of values is discarded, after updating the one or more statistical moments of the data stream based on that value. A threshold is set for the data stream based on the one or more statistical moments. It is detected that the threshold has been passed by the data stream. A remedial action is performed on the monitored device, responsive to the threshold being passed, where the remedial action is associated with the threshold.

Claims

1. A computer-implemented method, comprising:

receiving a plurality of values of a data stream, the plurality of values being generated by a monitored device and being received in real time;

updating, by a computer processing device, one or more statistical moments of the data stream based on each value of the plurality of values;

discarding each value of the plurality of values, after updating the one or more statistical moments of the data stream based on the value;

setting a threshold for the data stream based on the one or more statistical moments;

detecting that the threshold has been passed by the data stream; and

performing a remedial action on the monitored device, responsive to the threshold being passed, the remedial action being associated with the threshold.

2. The computer-implemented method of claim 1, wherein the setting the threshold for the data stream based on the one or more statistical moments comprises:

associating a first statistical model with a first data type, the first data type being variable data;

associating a second statistical model with a second data type, the second data type being attribute data type;

setting an active statistical model as one of the first statistical model and the second statistical model;

detecting a data type of the data stream;

determining that the data type of the data stream differs from the data type associated with the active statistical model;

changing the active statistical model, responsive to the data type of the data stream differing from the data type associated with the active statistical model;

applying the changed active statistical model to the one or more statistical moments to calculate the threshold.

3. The computer-implemented method of claim 1, wherein the setting the threshold for the data stream based on the one or more statistical moments comprises:

applying a first statistical model to the one or more statistical moments to calculate a first proposed threshold;

applying a second statistical model to the data stream to calculate a second proposed threshold;

determining that the first proposed threshold and the second proposed threshold disagree;

determining which of the first proposed threshold and the second proposed threshold is valid; and

selecting the threshold from among the first proposed threshold and the second proposed threshold, based on the determining which of the first proposed threshold and the second proposed threshold is valid.

4. The computer-implemented method of claim 3, wherein the determining which of the first proposed threshold and the second proposed threshold is valid is based on at least one of a standard of error and a coefficient of variance of the data stream.

5. The computer-implemented method of claim 3, wherein the first statistical model is associated with variable data, and the second statistical model is associated with attribute data.

6. The computer-implemented method of claim 1, wherein the one or more statistical moments comprise at least one of a mean, a variance, a skewness, and a kurtosis of the data stream.

7. The computer-implemented method of claim 1, wherein the threshold is dynamic, and wherein the setting the threshold for the data stream based on the one or more statistical moments is performed repeatedly to update the threshold.

8. A system comprising:

a memory having computer readable instructions; and

one or more processors for executing the computer readable instructions, the computer readable instructions comprising;

receiving a plurality of values of a data stream, the plurality of values being generated by a monitored device and being received in real time;

updating one or more statistical moments of the data stream based on each value of the plurality of values;

discarding each value of the plurality of values, after updating the one or more statistical moments of the data stream based on the value;

setting a threshold for the data stream based on the one or more statistical moments;

detecting that the threshold has been passed by the data stream; and

performing a remedial action on the monitored device, responsive to the threshold being passed, the remedial action being associated with the threshold.

9. The system of claim 8, wherein the setting the threshold for the data stream based on the one or more statistical moments comprises:

associating a first statistical model with a first data type, the first data type being variable data;

associating a second statistical model with a second data type, the second data type being attribute data type;

setting an active statistical model as one of the first statistical model and the second statistical model;

detecting a data type of the data stream;

determining that the data type of the data stream differs from the data type associated with the active statistical model;

changing the active statistical model, responsive to the data type of the data stream differing from the data type associated with the active statistical model;

applying the changed active statistical model to the one or more statistical moments to calculate the threshold.

10. The system of claim 8, wherein the setting the threshold for the data stream based on the one or more statistical moments comprises:

applying a first statistical model to the one or more statistical moments to calculate a first proposed threshold;

applying a second statistical model to the data stream to calculate a second proposed threshold;

determining that the first proposed threshold and the second proposed threshold disagree;

determining which of the first proposed threshold and the second proposed threshold is valid; and

selecting the threshold from among the first proposed threshold and the second proposed threshold, based on the determining which of the first proposed threshold and the second proposed threshold is valid.

11. The system of claim 10, wherein the determining which of the first proposed threshold and the second proposed threshold is valid is based on at least one of a standard of error and a coefficient of variance of the data stream.

12. The system of claim 10, wherein the first statistical model is associated with variable data, and the second statistical model is associated with attribute data.

13. The system of claim 8, wherein the one or more statistical moments comprise at least one of a mean, a variance, a skewness, and a kurtosis of the data stream.

14. The system of claim 8, wherein the threshold is dynamic, and wherein the setting the threshold for the data stream based on the one or more statistical moments is performed repeatedly to update the threshold.

15. A computer program product for analyzing streaming data, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:

receiving a plurality of values of a data stream, the plurality of values being generated by a monitored device and being received in real time;

updating one or more statistical moments of the data stream based on each value of the plurality of values;

discarding each value of the plurality of values, after updating the one or more statistical moments of the data stream based on the value;

setting a threshold for the data stream based on the one or more statistical moments;

detecting that the threshold has been passed by the data stream; and

performing a remedial action on the monitored device, responsive to the threshold being passed, the remedial action being associated with the threshold.

16. The computer program product of claim 15, wherein the setting the threshold for the data stream based on the one or more statistical moments comprises:

associating a first statistical model with a first data type, the first data type being variable data;

associating a second statistical model with a second data type, the second data type being attribute data type;

setting an active statistical model as one of the first statistical model and the second statistical model;

detecting a data type of the data stream;

determining that the data type of the data stream differs from the data type associated with the active statistical model;

changing the active statistical model, responsive to the data type of the data stream differing from the data type associated with the active statistical model;

applying the changed active statistical model to the one or more statistical moments to calculate the threshold.

17. The computer program product of claim 15, wherein the setting the threshold for the data stream based on the one or more statistical moments comprises:

applying a first statistical model to the one or more statistical moments to calculate a first proposed threshold;

applying a second statistical model to the data stream to calculate a second proposed threshold;

determining that the first proposed threshold and the second proposed threshold disagree;

determining which of the first proposed threshold and the second proposed threshold is valid; and

selecting the threshold from among the first proposed threshold and the second proposed threshold, based on the determining which of the first proposed threshold and the second proposed threshold is valid.

18. The computer program product of claim 17, wherein the determining which of the first proposed threshold and the second proposed threshold is valid is based on at least one of a standard of error and a coefficient of variance of the data stream.

19. The computer program product of claim 17, wherein the first statistical model is associated with variable data, and the second statistical model is associated with attribute data.

20. The computer program product of claim 15, wherein the one or more statistical moments comprise at least one of a mean, a variance, a skewness, and a kurtosis of the data stream.