Detection of abnormal behavior using probabilistic distribution estimation

• US 7,561,991 B2
• Filed: 02/17/2004
• Issued: 07/14/2009
• Est. Priority Date: 02/18/2003
• Status: Expired due to Fees

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Yamanishi, Kenji, Matsunaga, Yuko
• Primary Examiner(s): Shah; Kamini S
• Assistant Examiner(s): Alhija; Saif A
• Application Number:
  US10/778,178
• Publication Number:
  US 20040167893A1
• Time in Patent Office: 1,974 Days
• Field of Search: 703/2
• US Class Current: 703/2
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models

Abstract

Supplied with a string of vector data as input data, a probabilistic distribution estimation apparatus estimates, by using a stochastic model having hidden variables, a probabilistic distribution in which each data occurs by successively reading the train of vector data. Specifically, the probabilistic distribution estimation apparatus reads values of parameters of the stochastic model having the hidden variables for a value of the input data, calculates, by using the stochastic model, a certainty in which the input data occurs, renews the parameters in response to new read data with past data forgotten, and produce several parameter'"'"'s values. By using the parameter'"'"'s values received from the probabilistic distribution estimation apparatus, an abnormality detection unit calculates an information amount of data as an abnormal behavior degree to produce the abnormal behavior degree.

Claims

1. An abnormal behavior detection apparatus for detecting abnormalities in data output from a system, comprising:

a probabilistic distribution estimation apparatus for responding to, as input data, a string of vector data representing behavior to estimate, using a stochastic model, a probabilistic distribution occurred in each data by successively reading said string of vector data, said probabilistic distribution estimation apparatus comprising a parameter storage unit for storing all of parameters for the stochastic model having hidden variables, certainty calculation means for calculating, in response to said input data, a certainty occurred in said input data and that represents a level of assuredness that said input data is correct and does not contain errors, as the probabilistic distribution, using said stochastic model by reading the parameters of said stochastic model from said parameter storage unit, parameter renewal means for renewing the parameters in said parameter storage unit in accordance with new read data and with past data previously read using the certainty calculated by said certainty calculation means and using each parameter of said stochastic model read from said parameter storage unit, and parameter output means for outputting the parameters for the stochastic model stored in said parameter storage unit, wherein the parameters in said parameter storage unit are renewed based on a parameter renewal rule that unitizes both the certainty and the parameters of said stochastic model, said parameter renewal rule being an oblivious type algorithm for calculating a conditional expected value of a statistical amount weighted with an oblivion coefficient indicative of an accuracy of the past data as time progresses, the oblivious type algorithm be used to treat current data and the past data with appropriate weights in data analysis; and

abnormality detection means for receiving the parameters for the stochastic model output from said parameter output means and for calculating a score indicative of an abnormal behavior degree with respect to the new read data by using the parameters of the probabilistic distribution estimated by said probabilistic distribution estimation apparatus to produce the abnormal behavior degree of said new read data, the abnormal behavior degree being a degree where the new read data is out of whole patterns,

the abnormal behavior degree being an indicator of a level of abnormalities that exist in the data output from the system.

2. An abnormal behavior detection apparatus as claimed in claim 1, wherein further comprises a behavior model variation degree calculation unit for calculating, by using the parameters of the probabilistic distribution estimated by said probabilistic distribution estimation apparatus, a variation degree of a behavior mode as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of new data.

3. An abnormal behavior detection apparatus according to claim 1, further comprising:

session means for processing the input data into the string of vector data.

4. An abnormal behavior detection apparatus as claimed in claim 3, wherein further comprises a behavior model variation degree calculation unit for calculating, by using the parameters of the probabilistic distribution estimated by said probabilistic distribution estimation apparatus, a variation degree of a behavior mode as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of new data.

5. An abnormal behavior detection apparatus according to claim 1, wherein the stochastic model has a continuous time distribution and hidden variables.

6. An abnormal behavior detection apparatus as claimed in claim 5, wherein further comprises a behavior model variation degree calculation unit for calculating, by using the parameters of the probabilistic distribution estimated by said probabilistic distribution estimation apparatus, a variation degree of a behavior mode as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of new data.

7. An abnormal behavior detection apparatus according to claim 1, further comprising:

session means for processing the input data into the string of vector data.

8. An abnormal behavior detection apparatus as claimed in claim 7, wherein further comprises a behavior model variation degree calculation unit for calculating, by using the parameters of the probabilistic distribution estimated by said probabilistic distribution estimation apparatus, a variation degree of a behavior mode as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of new data.

9. An abnormal behavior detection apparatus according to claim 1, wherein a finite mixed distribution of hidden Markov models is used to estimate a probabilistic distribution occurred in each data.

10. An abnormal behavior detection apparatus according to claim 9, further comprising:

session means for processing the input data into the string of vector data.

25. An abnormal behavior detection apparatus according to claim 1, wherein the abnormal behavior degree is utilized to detect a network failure, and wherein an alarm message is outputted when the network failure is detected.

26. An abnormal behavior detection apparatus according to claim 1, wherein the abnormal behavior degree corresponds to a customer'"'"'s behavior that is obtained by reading history of Internet usage made by the customer.

27. An abnormal behavior detection apparatus according to claim 1, wherein the abnormal behavior degree is utilized to detect an invasion of resources of a computer using a system call pattern where a program internally executes within the computer.

11. A method of detecting abnormal behavior of data output from a system, comprising the steps of:

inputting a string of vector data representing behavior as input data;

calculating, using a model in which each data occurs is input thereto by successively reading the string of vector data, a certainty occurred in the input data and that represents a level of assuredness that said input data is correct and does not contain errors, as a probabilistic distribution, on the basis of parameters of said model;

renewing, by using the certainty obtained from the calculating step and the parameters of said stochastic model, the parameters in response to new read data and with past data previously read, the renewing being made based on a parameter renewal rule that utilizes both the certainty and the parameters of said model, said parameter renewal rule being an oblivious type algorithm for calculating a conditional excepted value of a statistical amount weighted with an oblivion coefficient indicative of an accuracy of the past data as time progresses, the oblivious type algorithm be used to treat current data and the past data with appropriate weights in data analysis; and

calculating, by using parameters of an estimated probabilistic distribution, an abnormal behavior degree of new read data using a score indicative of the abnormal behavior degree with respect to the new read data to produce the abnormal behavior degree of said new read data, the abnormal behavior degree being a degree where the new read data is out of whole patterns,

the abnormal behavior degree being an indicator of a level of abnormalities that exist in the data output from the system.

12. A method as claimed in claim 11, wherein further comprises the step of calculating, by using the parameters of said estimated probabilistic distribution, a variation degree of a behavior model as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of data, and wherein the model is a stochastic model having hidden variables as a probabilistic distribution.

13. A method of detecting abnormal behavior to claim 11, further comprising the step of:

carrying out session for converting the input data into a string of vector data when said input data have no structure of vector data.

14. A method as claimed in claim 13, wherein further comprises the step of calculating, by using the parameters of said estimated probabilistic distribution, a variation degree of a behavior model as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of data.

15. A method of detecting abnormal behavior to claim 11, wherein the calculating step comprises the steps of:

calculating, by using parameters of an estimated probabilistic distribution, a first posteriori probability of a state corresponding to the hidden variables by reading reference data different from the input data;

calculating, by using the parameters of the estimated probabilistic distribution, a second posteriori probability of a state corresponding to the hidden variables by reading new read data as the input data; and

calculating, as a variation of a posteriori probability, a difference between the first and the second posteriori probabilities to produce the variation of the posteriori probability.

28. A method according to claim 11, wherein the abnormal behavior degree is utilized to detect a network failure, and wherein an alarm message is outputted when the network failure is detected.

29. A method according to claim 11, wherein the abnormal behavior degree corresponds to a customer'"'"'s behavior that is obtained by reading history of Internet usage made by the customer.

30. A method according to claim 11, wherein the abnormal behavior degree is utilized to detect an invasion of resources of a computer using a system call pattern where a program internally executes within the computer.

16. A method of detecting abnormal behavior of data output from a system, comprising the steps of:

inputting a string of vector data representing behavior as input data;

calculating, using a time series model having a continuous time distribution and hidden variables as a probabilistic distribution in which each data occurs by successively reading the string of vector data, a certainty occurred in said input data and that represents a level of assuredness that said input data is correct and does not contain errors, as the probabilistic distribution, on the basis of parameters of said time series model;

renewing, by using said certainty obtained from the calculating step and the parameters of said time series model, the parameters in response to new read data and with past data previously read, the renewing being made based on a parameter renewal rule that utilizes both the certainty and the parameters of said time series model, said parameter renewal rule being an oblivious type algorithm for calculating a conditional expected value of a statistical amount weighted with an oblivion coefficient indicative of accuracy of the past data as time progresses, the oblivious type algorithm be used to treat current data and the past data with appropriate weights in data analysis; and

calculating, by using parameters of an estimated probabilistic distribution, an abnormal behavior degree of the new read data using a score indicative of the abnormal behavior degree with respect to the new read data to produce the abnormal behavior degree of said new read data, the abnormal behavior degree being a degree where the new read data is out of whole patterns,

the abnormal behavior degree being an indicator of a level of abnormalities that exist in the data output from the system.

17. A method as claimed in claim 16, wherein further comprises the step of calculating, by using the parameters of said estimated probabilistic distribution, a variation degree of a behavior model as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of data.

18. A method of detecting abnormal behavior according to claim 16, further comprising the step of:

carrying out session for converting the input data into a string of vector data when said input data have no structure of vector data.

19. A method as claimed in claim 18, wherein further comprises the step of calculating, by using the parameters of said estimated probabilistic distribution, a variation degree of a behavior model as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of data.

20. A method of detecting abnormal behavior according to claim 16, wherein the calculating step comprises the steps of:

calculating, by using parameters of an estimated probabilistic distribution, a first posteriori probability of a state corresponding to the hidden variables by reading reference data different from the input data;

calculating, by using the parameters of the estimated probabilistic distribution, a second posteriori probability of a state corresponding to the hidden variables by reading new read data as the input data; and

calculating, as a variation of a posteriori probability, a difference between the first and the second posteriori probabilities to produce the variation of the posteriori probability.

21. A computer readable medium storing an abnormal behavior detection program for execution by a computer for determining an abnormality behavior of data output from a system, wherein, when executing the behavior detection program, the computer performs the steps of:

performing a probabilistic distribution estimation for responding to, as input data, a string of vector data representing behavior to estimate, using a stochastic model, a probabilistic distribution occurred in each data by successively reading said string of vector data, said probabilistic distribution estimation step comprising a parameter storage step for storing all of parameters for the stochastic model having hidden variables, a certainty calculation step for calculating, in response to said input data, a certainty occurred in said input data and that represents a level of assuredness that said input data is correct and does not contain errors, as the probabilistic distribution, using said stochastic model by reading the parameters of said stochastic model from said parameter storage step, and a parameter renewal step for renewing the parameters in said parameter storage step in accordance with new read data and with past data previously read using the certainty calculated by said certainty calculation step and using each parameter of said stochastic model read from said parameter storage step, the renewing being made based on a parameter renewal rule that utilizes both the certainty and the parameters of said stochastic model, said parameter renewal rule being an oblivious type algorithm for calculating a conditional expected value of a statistical amount weighted with an oblivion coefficient indicative of accuracy of the past data as time progresses, the oblivious type algorithm be used to treat current data and the past data with appropriate weights in data analysis; and

performing an abnormality detection for calculating a score indicative of an abnormal behavior degree with respect to the new read data by using the parameters of the probabilistic distribution estimated by said probabilistic distribution estimation step to produce the abnormal behavior degree of said new read data, the abnormal behavior degree being a degree where the new read data is out of whole patters,

the abnormal behavior degree being an indicator of a level of abnormalities that exist in the data output from the system.

22. A computer readable medium as claimed in claim 21, wherein said abnormal behavior detection program further makes said computer operate as a behavior model variation degree calculation unit for calculating, by using the parameters of the probabilistic distribution estimated by said probabilistic distribution estimation step, a variation degree of a behavior mode as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of new data.

23. A computer readable medium according to claim 21, wherein said probabilistic distribution estimation step further comprises a session step for processing the input data into the string of vector data.

24. A computer readable medium as claimed in claim 23, wherein said abnormal behavior detection program further makes said computer operate as a behavior model variation degree calculation unit for calculating, by using the parameters of the probabilistic distribution estimated by said probabilistic distribution estimation step, a variation degree of a behavior mode as a time-average of said abnormal behavior degrees for a predetermined width by reading a plurality of new data.

31. A computer readable medium according to claim 21, wherein the abnormal behavior degree is utilized to detect a network failure, and wherein an alarm message is outputted when the network failure is detected.

32. A computer readable medium according to claim 21, wherein the abnormal behavior degree corresponds to a customer'"'"'s behavior that is obtained by reading history of Internet usage made by the customer.

33. A computer readable medium according to claim 21, wherein the abnormal behavior degree is utilized to detect an invasion of resources of a computer using a system call pattern where a program internally executes within the computer.