Detecting trends in real time analytics

• US 7,840,377 B2
• Filed: 12/12/2006
• Issued: 11/23/2010
• Est. Priority Date: 12/12/2006
• Status: Expired due to Fees

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Selby, David A., Ramsey, Mark S., Todd, Stephen J.
• Primary Examiner(s): West; Jeffrey R
• Application Number:
  US11/609,440
• Publication Number:
  US 20080140471A1
• Time in Patent Office: 1,442 Days
• Field of Search: 702/81, 702/84, 702/176, 702/177, 702/179, 702/182, 702/183, 702/187
• US Class Current: 702/179
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202

Abstract

A system, method and program product for processing a stream of data to detect trends in real time. A system is provided comprising: a real time statistical processing system for updating a running value each time a new data event is obtained, wherein the running value is calculated based on a previously calculated running value and a difference DV between a current data event value (V_i) and a previous data event value (V_i−1); and an analysis system that analyzes the running value after it is updated to detect trends.

Claims

1. A system for processing a stream of data events, comprising:

at least one computing device, including;

a real time statistical processing system for updating a first and a second running value each time a new data event is obtained, wherein the first and second running values are calculated based on a previously calculated running value and a difference DV between a current data event value (V_i) and a previous data event value (V<sub>i−

1</sub>), wherein the first running value RDV_i=(1−

K)*DV+K*RDV<sub>i−

1</sub> and the second running value RDVV_i=(1−

K)*DV²+K*RDVV<sub>i−

1</sub> in which K is a half-life based smoothing factor that dictates a period over which a decline in data event values is analyzed, and wherein a third running value RSDD comprises a running standard deviation computed by the real time statistical processing system as follows;

RSDD=sqrt(RDVV_i−

RDV_i²); and

an analysis system for analyzing each running value after it is updated to detect trends in the stream of data events.

2. The system of claim 1, wherein the real time statistical processing system includes a cache for storing each previously calculated running value and each previous data event value.

3. The system of claim 1, further comprising a first level analysis system for generating an alert in real time when at least one running value breaks a threshold.

4. The system of claim 3, further comprising a second level analysis system that analyzes each running value using historical data in a non-real time fashion.

5. The system of claim 1, wherein the detect trends in the stream of data events identifies potential fraudulent activity in a financial setting.

6. The system of claim 1, wherein the detect trends in the stream of data events tracks network usage.

7. The system of claim 1, wherein the detect trends in the stream of data events identifies opportunities in a business setting.

8. The system of claim 1, wherein the detect trends in the stream of data events identifies problems in a business setting.

9. The system of claim 1, wherein the detect trends in the stream of data events identifies risks in a business setting.

10. The system of claim 1, wherein the data events are selected from a group consisting of:

business data, research data, scientific data, technical data and financial data.

11. A non-transitory computer readable medium storing a computer program product, which when executed by a computing device, processes a stream of data events to detect trends, the program product comprising:

program code configured for updating a running value each time a new data event is obtained, wherein the running value RDVV_i is calculated based on a previously calculated running value and a difference DV between a current data event value (V_i) and a previous data event value (V<sub>i−

1</sub>) and is calculated as (1−

K)*DV²+K*RDVV<sub>i−

1</sub>, where K is a half-life based smoothing factor that dictates a period over which a decline in data event values is analyzed;

program code configured for determining a running mean calculated as follows;

RDV_i=(1−

K)*DV+K*RDV<sub>i−

1</sub>;

program code configured for determining a running standard deviation calculated as follows;

RSDD=sqrt(RDVV_i=RDV_i²);

program code configured for analyzing the running value, running mean, and running standard deviation to detect trends in the stream of data events; and

program code configured for outputting an alarm.

12. The non-transitory computer readable medium of claim 11, further comprising program code configured for storing the previously calculated running value and the previous data event value in a cache.

13. The non-transitory computer readable medium of claim 11, wherein the program code configured for analyzing the running value includes a first level analysis that generates an alert in real time when the running value breaks a threshold.

14. The non-transitory computer readable medium of claim 13, wherein the program code configured for analyzing the running value includes a second level analysis that analyzes the running value using historical data in a non-real time fashion.

15. A method of processing a stream of data events to detect trends, comprising:

obtaining, using a computing device, a new data event value;

updating, using the computing device, a running value based on a previously calculated running value and a difference DV between the new data event value (V_i) and a previous data event value (V<sub>i−

1</sub>), wherein the running value is calculated as (1−

K)*DV²+K*RDVV<sub>i−

1</sub>, where K is a half-life based smoothing factor that dictates a period over which a decline in data event values is analyzed;

updating a running mean calculated as follows;

RDV_i=(1−

K)*DV+K*RDV<sub>i−

1</sub>;

updating a running standard deviation calculated as follows;

RSDD=sqrt(RDVV_i−

RDV_i²); and

analyzing, using the computing device, the running value, running mean, and running standard deviation to detect trends in the stream of data events.

16. The method of claim 15, further comprising storing the running value and the new data event value in a cache for use in a next calculation.

17. The method of claim 15, wherein analyzing the running value includes:

generating an alert in real time when the running value breaks a threshold using a first level analysis; and

analyzing the running value using historical data in a non-real time fashion.