Anomaly detection

• US 20050193281A1
• Filed: 01/28/2005
• Published: 09/01/2005
• Est. Priority Date: 01/30/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ide, Tsuyoshi, Yoda, Kunikazu, Kashima, Hisashi, Etoh, Hiroaki, Hirade, Ryo
• Granted Patent: US 7,346,803 B2
• Time in Patent Office: Days
• US Class Current: 714/4.1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A system such as a Web-based system in which a plurality of computers interact with each other is monitored to detect online an anomaly. Transactions of a service provided by each of a plurality of computers to another computer are collected, a matrix of correlations between nodes in the system is calculated from the transactions, and a feature vector representing anode activity balance is obtained from the matrix. The feature vector is monitored using a probability model to detect a transition to an anomalous state.

Claims

1. ) An anomaly detection system comprising means for monitoring a program for a fault on the basis of a probabilistic model to automatically detect a fault, said program being run through cooperation between computers in a computer system forming a network consisting of a plurality of computers, said anomaly detection system further comprising:

a plurality of computers providing services associated with said program;

an agent device associating and recording transactions with a service, said transactions being processes of said service performed by each of said plurality of computers in response to a call from a program running on another of said plurality of computers; and

an anomaly monitoring server connected to said network, comprising, a transaction collecting section collecting transactions recorded by said agent device from each of said plurality of computers;

a correlation matrix calculating section outputting a node correlation matrix calculated from said transactions collected from each of said plurality of computers;

an activity vector calculating section calculating an activity vector by solving an eigenequation specific to said node correlation matrix;

a probability estimating section estimating the probability of occurrence of said activity vector; and

a fault detecting section calculating an outlier measure of the activity vector from a probability density estimated by said probability estimating section to automatically detect a fault in the program being run through cooperation between said plurality of computers.

2. ) The anomaly detection system according to claim 1, wherein said service associated with said program and provided by each of said plurality of computers includes one or more items selected from a group consisting of an IP address, a port number, and a transaction type.

3. ) The anomaly detection system according to claim 2, comprising a fault detecting section detecting a fault at the application layer of said program by using a service associated with said program and provided by each of said plurality of computers, said service including one or more items selected from a group including a source IP address, a destination IP address, a destination port number, and a transaction type at the destination port.

4. ) The anomaly detection system according to claim 1, wherein said fault detecting section automatically detects a fault by transforming said activity vector into a cosine measure and calculating an outlier measure.

5. ) The anomaly detection system according to claim 1, wherein said probability estimating section uses an online EM algorithm to learn a probability distribution online.

6. ) The anomaly detection system according to claim 1, wherein said agent device is provided in said plurality of computers.

7. ) The anomaly detection system according to claim 1, wherein said agent device is provided in a network device in said computer system.

8. ) The anomaly detection system according to claim 1, wherein previously captured data about a transaction of each of said computers is used as data for calculating said node correlation matrix to determine the time point and service in which anomaly has occurred.

31. ) A computer program product comprising a computer usable medium having computer readable program code means embodied therein for causing anomaly detection, the computer readable program code means in said computer program product comprising computer readable program code means for causing a computer to effect the functions of claim 1.

9. ) An anomaly detection system for detecting a fault in a program being run through cooperation between computers in a computer system forming a network consisting of a plurality of computers, said anomaly detection system comprising:

a plurality of computers providing services associated with said program;

an agent device associating and recording transactions with a service, said transactions being processes of said service performed by each of said plurality of computers in response to a call from a program running on another of said plurality of computers; and

an anomaly monitoring server connected to said network, comprising, a transaction collecting section collecting transactions recorded by said agent device from each of said plurality of computers;

a correlation matrix calculating section outputting a node correlation matrix calculated from said transactions collected from each of said plurality of computers;

an activity vector calculating section calculating an activity vector by solving an eigenequation specific to said node correlation matrix; and

a fault detecting section providing a notification of a fault in a program being run through cooperation between said plurality of computers, depending on said activity vector.

10. ) The anomaly detection system according to claim 9, wherein said service associated with said program and provided by each of said plurality of computers includes one or more items selected from a group including of an IP address, a port number, and a transaction type.

11. ) The anomaly detection system according to claim 10, comprising a fault detecting section detecting a fault at the application layer of said program by using a service associated with said program and provided by each of said plurality of computers, said service including one or more items selected from a group consisting of a source IP address, a destination IP address, a destination port number, and a transaction type at the destination port.

12. ) The anomaly detection system according to claim 9, wherein said agent device is provided in each of said plurality of computers.

13. ) The anomaly detection system according to claim 9, wherein said agent device is provided in a network device in said computer system.

14. ) The anomaly detection system according to claim 9, wherein previously captured data about a transaction of each of said computers is used as data for calculating said node correlation matrix to determine the time point and service in which anomaly has occurred.

32. ) A computer program product comprising a computer usable medium having computer readable program code means embodied therein for causing anomaly detection, the computer readable program code means in said computer program product comprising computer readable program code means for causing a computer to effect the functions of claim 9.

15. ) An anomaly detection system comprising means for monitoring a program for a fault on the basis of a probabilistic model to automatically detect a fault, said program being run through cooperation between computers in a computer system forming a network consisting of a plurality of computers, said anomaly detection system further comprising:

an agent module having transaction recording means for associating and recording transactions with a service, said transactions being processes of said service performed by each of said plurality of computers in response to a call from a program running on another of said plurality of computers; and

an anomaly monitoring server connected to said network, comprising, transaction collecting means for collecting transactions recorded by said agent module from each of said plurality of computers;

correlation matrix calculating means for outputting a node correlation matrix calculated from said transactions collected from each of said plurality of computers;

activity vector calculating means for calculating an activity vector by solving an eigenequation specific to said node correlation matrix;

probability estimating means for estimating the probability of occurrence of said activity vector; and

fault detecting means for calculating an outlier measure of the activity vector from a probability density estimated by said probability estimating means to automatically detect a fault in the program being run through cooperation between said plurality of computers.

16. ) The anomaly detection system according to claim 15, comprising fault detecting means for detecting a fault at the application layer of said program by using a service associated with said program and provided by each of said plurality of computers, said service including one or more items selected from a group including a source IP address, a destination IP address, a destination port number, and a transaction type at the destination port.

17. ) The anomaly detection system according to claim 15, wherein said fault detecting means automatically detects a fault by transforming said activity vector into a cosine measure and calculating an outlier measure.

18. ) The anomaly detection system according to claim 15, wherein said probability estimating means uses an online EM algorithm to learn a probability distribution online.

33. ) A computer program product comprising a computer usable medium having computer readable program code means embodied therein for causing anomaly detection, the computer readable program code means in said computer program product comprising computer readable program code means for causing a computer to effect the functions of claim 15.

19. ) An anomaly detection method comprising monitoring a program for a fault on the basis of a probabilistic model to automatically detect a fault, said program being run through cooperation between computers in a computer system forming a network consisting of a plurality of computers, said anomaly detection method further comprising:

a recording step performed by an agent module of associating and recording transactions with a service, said transactions being processes of said service performed by each of said plurality of computers in response to a call from aprogram running on another of said plurality of computers; and

a step performed by an anomaly monitoring server connected to said network, comprising, a transaction collecting step of collecting transactions recorded by said agent, module from each of said plurality of computers;

a correlation matrix calculating step of outputting a node correlation matrix calculated from said transactions collected from each of said plurality of computers;

an activity vector calculating step of calculating an activity vector by solving an eigenequation specific to said node correlation matrix;

a probability estimating step of estimating the probability of occurrence of said activity vector; and

a fault detecting step of calculating an outlier measure of the activity vector from a probability density estimated in said probability estimating step to automatically detect a fault in the program being run through cooperation between said plurality of computers.

20. ) The anomaly detection method according to claim 19, comprising a fault detecting step of detecting a fault at the application layer of said program by using a service associated with said program and provided by each of said plurality of computers, said service including one or more items selected from a group including a source IP address, a destination IP address, a destination port number, and a transaction type at the destination port.

21. ) The anomaly detection method according to claim 19, wherein said fault detecting step automatically detects a fault by transforming said activity vector into a cosine measure and calculating an outlier measure.

22. ) The anomaly detection method according to claim 19, wherein said probability estimating step uses an online EM algorithm to learn a probability distribution online.

29. ) An article of manufacture comprising a computer usable medium having computer readable program code means embodied therein for causing anomaly detection, the computer readable program code means in said article of manufacture comprising computer readable program code means for causing a computer to effect the steps of claim 19.

30. ) A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for anomaly detection, said method steps comprising the steps of claim 19.

23. ) An anomaly detection program comprising a step of monitoring a program for a fault on the basis of a probabilistic model to automatically detect a fault, said program being run through cooperation between computers in a computer system forming a network consisting of a plurality of computers, said anomaly detection program further comprising:

a recording step performed by an agent module of associating and recording transactions with a service, said transactions being processes of said service performed by each of said plurality of computers in response to a call from a program running on another of said plurality of computers; and

a step performed by an anomaly monitoring server connected to said network, comprising, a transaction collecting step of collecting transactions recorded by said agent module from each of said plurality of computers;

a correlation matrix calculating step of outputting a node correlation matrix calculated from said transactions collected from each of said plurality of computers;

an activity vector calculating step of calculating an activity vector by solving an equation specific to said node correlation matrix;

a probability estimating step of estimating the probability of occurrence of said activity vector; and

a fault detecting step of calculating an outlier measure of the activity vector from a probability density estimated in said probability estimating step to automatically detect a fault in the program being run through cooperation between said plurality of computers.

24. ) The anomaly detection program according to claim 23, comprising a fault detecting step of detecting a fault at the application layer of said program by using a service associated and provided with said program running on said plurality of computers, said service including one or more items selected from a group including a source IP address, a destination IP address, a destination port number, and a transaction type at the destination port.

25. ) The anomaly detection program according to claim 23, wherein said fault detecting step automatically detects a fault by transforming said activity vector into a cosine measure and calculating an outlier measure.

26. ) The anomaly detection program according to claim 23, wherein said probability estimating step uses an online EM algorithm to learn a probability distribution online.

27. ) A storage medium on which the anomaly detection program according to claim 23 is stored.

28. ) An anomaly monitoring server for monitoring a program for a fault on the basis of a probabilistic model to automatically detect a fault, said program being run through cooperation between computers in a computer system forming a network consisting of a plurality of computers, said anomaly monitoring server comprising:

a transaction collecting section collecting transactions which are processes of a service performed by each of said plurality of computers in response to a call froma program running on another of said plurality of computers;

a correlation matrix calculating section outputting a node correlation matrix calculated from said transactions collected from each of said plurality of computers;

an activity vector calculating section calculating an activity vector by solving an eigenequation specific to said node correlation matrix;

a probability estimating section estimating the probability of occurrence of said activity vector; and

a fault detecting section calculating an outlier measure of the activity vector from a probability density estimated by said probability estimating section to automatically detect a fault in the program being run through cooperation between said plurality of computers.