TREND GROUPS AND RUNNING PROFILES IN REAL TIME ANALYTICS

• US 20080141281A1
• Filed: 12/12/2006
• Published: 06/12/2008
• Est. Priority Date: 12/12/2006
• Status: Active Grant

• Current Assignee: X Corp. (f/k/a Twitter, Inc.) (X Holdings Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Selby, David A., Ramsey, Mark S., Todd, Stephen J.
• Granted Patent: US 7,751,909 B2
• Time in Patent Office: Days
• US Class Current: 719/318
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes

Abstract

A system. method and program product that utilizes running profiles and trend groups to analyze streams of event data in real time. A system is provided having a group profile processing system for generating a running group profile from a plurality of data event values, wherein each of the plurality of data event values is obtained from an associated one of the plurality of data streams; an event normalization system for normalizing each of the plurality of data event values based on the running group profile to generate a normalized data event value for the data event values obtained from each of the plurality of data streams; an event profile processing system for generating a running event profile for each normalized data event value; and an analysis system that evaluates each data stream by comparing a normalized data event value with an associated running event profile.

Claims

1. A real time data event processing system for processing a plurality of data streams, comprising:

a group profile processing system for generating a running group profile from a plurality of data event values, wherein each of the plurality of data event values is obtained from an associated one of the plurality of data streams;

an event normalization system for normalizing each of the plurality of data event values based on the running group profile to generate a normalized data event value for the data event values obtained from each of the plurality of data streams;

an event profile processing system for generating a running event profile for each normalized data event value; and

an analysis system that evaluates each data stream by comparing a normalized data event value with an associated running event profile.

2. The real time data event processing system of claim 1, wherein the running group profile is calculated based on a previously calculated running group profile.

3. The real time data event processing system of claim 1, wherein each running event profile is calculated based on a normalized data event value and a previously calculated running event profile.

4. The real time data event processing system of claim 3, wherein each running event profile is calculated using an algorithm selected from the group consisting of:

a running median, a running mean, a hybrid of a running median and running mean, a middle filter, an exponential smoothing algorithm, and a complex exponential smoothing algorithm.

5. The real time data event processing system of claim 3, wherein each running event profile includes a plurality of values including a value selected from the group consisting of:

a percentile and a standard deviation.

6. The real time data event processing system of claim 1, wherein the event normalization system calculates a normalized data event value NX(e, t) for a data stream e at a time t according to the equation:

NX(e, t)=X(e, t)/M(G, t),

wherein X(e, t) is a data event value for data stream e at time t and M(G, t) is a running group profile value for a group G of data events at time t for a plurality of data streams.

7. The real time data event processing system of claim 1, wherein the analysis system includes exception detection logic that generates an exception when a difference between a normalized data event value and an associated running event profile exceeds a predetermined threshold.

8. The real time data event processing system of claim 1, wherein the analysis system includes exception detection logic that generates an exception when a difference between a normalized data event value and an associated running event profile exceeds a factor of N times a normalized running standard deviation, wherein N is a predetermined value.

9. A computer program product stored on a computer readable medium, which when executed, processes a plurality of data streams, comprising:

program code configured for generating a running group profile from a plurality of data event values, wherein each of the plurality of data event values is obtained from an associated one of the plurality of data streams;

program code configured for normalizing each of the plurality of data event values based on the running group profile to generate a normalized data event value for the data event values obtained from each of the plurality of data streams;

program code configured for generating a running event profile for each normalized data event value; and

program code configured for evaluating each data stream.

10. The computer program product of claim 9, wherein the running group profile is calculated based on a previously calculated running group profile.

11. The computer program product of claim 9, wherein each running event profile is calculated based on a normalized data event value and a previously calculated running event profile.

12. The computer program product of claim 11, wherein each running event profile is calculated using an algorithm selected from the group consisting of:

a running median, a running mean, a hybrid of a running median and running mean, a middle filter, an exponential smoothing algorithm, and a complex exponential smoothing algorithm.

13. The computer program product of claim 11, wherein each running event profile includes a plurality of values including a value selected from the group consisting of:

a percentile and a standard deviation.

14. The computer program product of claim 9, wherein each normalized data event value NX(e, t) for a data stream e at a time t is calculated according to the equation:

NX(e, t)=X(e, t)/M(G, t),

wherein X(e, t) is a data event value for data stream e at time t and M(G, t) is a running group profile value for a group G of data events at time t for a plurality of data streams.

15. The computer program product of claim 9, wherein the data stream is evaluated by comparing a normalized data event value with an associated running event profile.

16. The computer program product of claim 9, wherein the data stream is evaluated by determining if a difference between a normalized data event value and an associated running event profile exceeds a factor of N times a normalized running standard deviation, wherein N is a predetermined value.

17. A method for processing a plurality of data streams, comprising:

generating a running group profile from a plurality of data event values, wherein each of the plurality of data event values is obtained from an associated one of the plurality of data streams;

normalizing each of the plurality of data event values based on the running group profile to generate a normalized data event value for the data event values obtained from each of the plurality of data streams;

generating a running event profile for each normalized data event value; and

evaluating each data stream by comparing a normalized data event value with an associated running event profile.

18. The method of claim 17, wherein the running group profile is calculated based on a previously calculated running group profile.

19. The method of claim 17, wherein each running event profile is calculated based on a normalized data event value and a previously calculated running event profile.

20. The method of claim 19, wherein each running event profile is calculated using an algorithm selected from the group consisting of:

a running median, a running mean, a hybrid of a running median and running mean, a middle filter, an exponential smoothing algorithm, and a complex exponential smoothing algorithm.

21. The method of claim 19, wherein each running event profile includes a plurality of values including a value selected from the group consisting of:

a percentile and a standard deviation.

22. The method of claim 17, wherein each normalized data event value NX(e, t) for a data stream e at a time t is calculated according to the equation:

NX(e, t)=X(e, t)/M(G, t),

wherein X(e, t) is a data event value for data stream e at time t and M(G, t) is a running group profile value for a group G of data events at time t for a plurality of data streams.

23. The method of claim 17, further comprising evaluating each data stream by determining if a difference between a normalized data event value and an associated running event profile exceeds a factor of N times a normalized running standard deviation, wherein N is a predetermined value.