Method and apparatus for performing real time anomaly detection

• US 8,713,190 B1
• Filed: 09/08/2006
• Issued: 04/29/2014
• Est. Priority Date: 09/08/2006
• Status: Expired due to Fees

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Goodall, Colin, Agarwal, Deepak, DuMouchel, William H.
• Primary Examiner(s): Choudhury, Azizul
• Assistant Examiner(s): NAJEE-ULLAH, TARIQ S
• Application Number:
  US11/518,291
• Time in Patent Office: 2,790 Days
• Field of Search: 709/231
• US Class Current: 709/231
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods

Abstract

A method and apparatus for anomaly detection in a data stream are disclosed. In one embodiment, the present method detects an anomalous condition in a data stream, by calculating at least one expected base event count for at least one event in the data stream for a time interval, obtaining an actual event count for the at least one event in the data stream, applying at least one shrinkage factor to the at least one expected base event count to obtain at least one actual estimated event count and detecting the anomalous condition in accordance with the actual event count and the at least one actual estimated event count.

Claims

1. A method for detecting an anomalous condition in a data stream, comprising:

calculating, by a processor, an expected base event count for an event in the data stream for a time interval, wherein the data stream represents data with cross-classified events, wherein each cross-classified event is an event having at least two categories;

obtaining, by the processor, an actual event count for the event in the data stream for the time interval;

applying, by the processor, a shrinkage factor to a ratio of the actual event count and the expected base event count to obtain an actual estimated event count, wherein the shrinkage factor uses an N parameter family of functions that comprises a family of gamma functions, where N is an integer not greater than two, wherein the shrinkage factor is obtained using a kalman filter gamma-poisson shrinker; and

detecting, by the processor, the anomalous condition in accordance with the actual event count and the actual estimated event count.

2. The method of claim 1, wherein the data stream represents packets in a communication network.

3. The method of claim 1, wherein the data stream represents medical records.

4. The method of claim 1, wherein the data stream represents financial records.

5. The method of claim 1, wherein the data stream represents chemical reaction records.

6. A non-transitory computer-readable medium to store a plurality of instructions which, when executed by a processor, cause the processor to perform operations for detecting an anomalous condition in a data stream, the operations comprising:

calculating an expected base event count for an event in the data stream for a time interval, wherein the data stream represents data with cross-classified events, wherein each cross-classified event is an event having at least two categories;

obtaining an actual event count for the event in the data stream for the time interval;

applying a shrinkage factor to a ratio of the actual event count and the expected base event count to obtain an actual estimated event count, wherein the shrinkage factor uses an N parameter family of functions that comprises a family of gamma functions, where N is an integer not greater than two, wherein the shrinkage factor is obtained using a kalman filter gamma-poisson shrinker; and

detecting the anomalous condition in accordance with the actual event count and the actual estimated event count.

7. The non-transitory computer-readable medium of claim 6, wherein the data stream represents packets in a communication network.

8. The non-transitory computer-readable medium of claim 6, wherein the data stream represents medical records.

9. The non-transitory computer-readable medium of claim 6, wherein the data stream represents financial records.

10. The non-transitory computer-readable medium of claim 6, wherein the data stream represents chemical reaction records.

11. An apparatus for detecting an anomalous condition in a data stream comprising:

a processor; and

a computer-readable medium in communication with the processor, to store a plurality of instructions which, when executed by the processor, cause the processor to perform operations, the operations comprising;

calculating an expected base event count for an event in the data stream for a time interval, wherein the data stream represents data with cross-classified events, wherein each cross-classified event is an event having at least two categories;

obtaining an actual event count for the event in the data stream for the time interval;

applying a shrinkage factor to a ratio of the actual event count and the expected base event count to obtain an actual estimated event count, wherein the shrinkage factor uses an N parameter family of functions that comprises a family of gamma functions, where N is an integer not greater than two, wherein the shrinkage factor is obtained using a kalman filter gamma-poisson shrinker; and

detecting the anomalous condition in accordance with the actual event count and the actual estimated event count.