DETECTING TRENDS IN REAL TIME ANALYTICS

• US 20080140471A1
• Filed: 12/12/2006
• Published: 06/12/2008
• Est. Priority Date: 12/12/2006
• Status: Active Grant

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Selby, David A., Ramsey, Mark S., Todd, Stephen J.
• Granted Patent: US 7,840,377 B2
• Time in Patent Office: Days
• US Class Current: 705/7
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202

Abstract

A system, method and program product for processing a stream of data to detect trends in real time. A system is provided comprising: a real time statistical processing system for updating a running value each time a new data event is obtained, wherein the running value is calculated based on a previously calculated running value and a difference DV between a current data event value (V_i) and a previous data event value (V_i−1); and an analysis system that analyzes the running value after it is updated to detect trends.

Claims

1. A system for processing a stream of data events, comprising:

a real time statistical processing system for updating a first and a second running value each time a new data event is obtained, wherein the first and second running values are calculated based on a previously calculated running value and a difference DV between a current data event value (V_i+1) and a previous data event value (V_i), wherein the first running value RDV_i=(1−

K)*DV+K*RDV<sub>i−

1</sub> and the second running value RDVV_i=(1−

K)*DV²+K*RDVV<sub>i−

1</sub> in which K is a half-life based smoothing factor; and

an analysis system for analyzing each running value after it is updated to detect trends in the stream of data events.

4. The system of claim 1, wherein a third running value RSDD comprising a running standard deviation is computed by the real time statistical processing system as follows:

RSDD=sqrt(RDVV_i−

RDV_i²).

5. The system of claim 1, wherein the real time statistical processing system includes a cache for storing each previously calculated running value and each previous data event value.

6. The system of claim 1, further comprising a first level analysis system for generating an alert in real time when at least one running value breaks a threshold.

7. The system of claim 6, further comprising a second level analysis system that analyses each running value using historical data in a non-real time fashion.

2. (canceled)

3. (canceled)

8. A computer readable medium storing a computer program product, which when executed, processes a stream of data events to detect trends, the program product comprising:

program code configured for updating a running value each time a new data event is obtained, wherein the running value RDVV_i is calculated based on a previously calculated running value and a difference DV between a current data event value (V<sub>i−

1) and a previous data event value (Vi</sub>) and is calculated as (1−

K)*DV²+K*RDVV<sub>i−

1</sub>, where K is a half-life based smoothing factor;

program code configured for analyzing the running value after it is updated to detect trends in the stream of data events; and

program code configured for outputting an alarm.

9. The computer readable medium of claim 8, further comprising program code configured for determining a running mean calculated as follows:

RDV_i=(1−

K)*DV+K*RDV<sub>i−

1</sub>,

wherein K is a half-life based smoothing factor that can be selected in any manner.

11. The computer readable medium of claim 9, further comprising program code configured for determining a running standard deviation calculated as follows:

RSDD=sqrt(RDVV_i−

RDV_i²)

12. The computer readable medium of claim 8, further comprising program code configured for storing the previously calculated running value and the previous data event value in a cache.

13. The computer readable medium of claim 8, wherein the program code configured for analyzing the running value includes a first level analysis that generates an alert in real time when the running value breaks a threshold.

14. The computer readable medium of claim 13, wherein the program code configured for analyzing the running value includes a second level analysis that analyses the running value using historical data in a non-real time fashion.

10. (canceled)

15. A method of processing a stream of data events to detect trends, comprising:

obtaining a new data event value;

updating a running value based on a previously calculated running value and a difference DV between the new data event value (V_i) and a previous data event value (V<sub>i−

1</sub>), wherein the running value is calculated as (1−

K)*DV²+K*RDVV<sub>i−

1</sub>, where K is a half-life based smoothing factor; and

analyzing the running value after it is updated to detect trends in the stream of data events.

16. The method of claim 15, further comprising updating a running mean calculated as follows:

RDV_i=(1−

K)*DV+K*RDV<sub>i−

1</sub>,

wherein K is a half-life based smoothing factor that can be selected in any manner.

18. The method of claim 16, further comprising updating a running standard deviation calculated as follows:

RSDD=sqrt(RDVV_i−

RDV_i²)

19. The method of claim 15, further comprising storing the running value and the new data event value in a cache for use in a next calculation.

20. The method of claim 15, wherein analyzing the running value includes:

generating an alert in real time when the running value breaks a threshold using a first level analysis; and

analyzing the running value using historical data in a non-real time fashion

17. (canceled)