Computer-Implemented System And Method For Detecting Anomalies Using Sample-Based Rule Identification

• US 20160042287A1
• Filed: 08/10/2014
• Published: 02/11/2016
• Est. Priority Date: 08/10/2014
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Inventors: Greene, Daniel H., Price, Robert, Eldardiry, Hoda, Kallur Palli Kumar, Sricharan
• Granted Patent: US 10,140,576 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 21/00 Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06Q 50/00 Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism healthcare informatics G16H
• CPC Class Codes: G06Q 50/22 Social work or social welfare, e.g. community support activities or counselling services
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: H04L 63/14 for detecting or protecting against malicious traffic
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A computer-implemented system and method for detecting anomalies using sample-based rule identification is provided. Data for data is maintained analytics in a database. A set of anomaly rules is defined. A rare pattern in the data is statistically identified. The identified rare pattern is labeled as at least one of anomaly and non-anomaly based on verification by a domain expert. The set of anomaly rules is adjusted based on the labeled anomaly. Other anomalies in the data are detected and classified by applying the adjusted set of anomaly rules to the data.

Claims

1. A computer-implemented system for detecting anomalies using sample-based rule identification, comprising:

data for data analytics in a database;

a set of anomaly rules;

a statistical anomaly detector configured to statistically identify a rare pattern in the data;

a label module configured to label the identified rare pattern as at least one of anomaly and non-anomaly based on verification by a domain expert;

a sample-based rule identifier configured to adjust the set of anomaly rules based on the labeled anomaly;

a detecting and classifying other anomalies in the data by applying the adjusted set of anomaly rules to the data; and

a processor to execute the modules.

2. A system according to claim 1, further comprising:

a non-anomaly recognizer configured to recognize the identified rare pattern as the labeled non-anomaly; and

the statistical anomaly detector further configured to modify the set of anomaly rules not to comprise the labeled non-anomaly.

3. A system according to claim 1, further comprising:

an anomaly recognizer configured to recognize the identified rare pattern as the labeled anomaly; and

a new rule convertor configured to convert the labeled anomaly to a new anomaly rule and adding the new rule to the set of the anomaly rules.

4. A system according to claim 3, further comprising:

a space identifier configured to identify a space for transformation;

an anomaly introducer configured to introduce one labeled anomaly as an anomaly example;

a logic module configured to obtain a statistical anomaly logic of the labeled anomaly;

a transformation module configured to execute the transformation based on the statistical anomaly logic; and

a new rule composer configured to obtain the new anomaly rule.

5. A system according to claim 3, further comprising:

a space identifier configured to identify a space for transformation;

an anomaly introducer configured to introduce multiple labeled anomalies as anomaly examples;

a logic module configured to obtain a statistical anomaly logic for each of the anomaly samples;

a transformation module configured to execute the transformation based on the statistical anomaly logics;

a new rule composer configured to obtain each new anomaly rule for each anomaly example; and

a condition module configured to add a condition to the new anomaly rule that the obtained new anomaly rules are similar.

6. A system according to claim 3, further comprising:

an algorithm selector configured to select a classification algorithm; and

a modification module configured to modify the classification algorithm comprising at least one of;

a criterial module configured to modify criteria of fit; and

a term module configured to modify a regularization term.

7. A system according to claim 1, further comprising:

an anomaly threshold comprised in the set of anomaly rules;

a score of the labeled non-anomaly; and

a rule refiner configured to refine the set of anomaly rules by comparing the score of the labeled non-anomaly to the anomaly threshold and raising the anomaly threshold if the score of the labeled non-anomaly is below the anomaly threshold.

8. A system according to claim 1, further comprising at least one of:

a multivariate detector configured to statistically detect rare incidents in the data compared to the rest of the population in the data;

a temporal detector configured to statistically detect rare trends in the data with regard to time; and

a correlation detector configured to statistically detect rare events with regard to correlation of events in the data.

9. A system according to claim 1, further comprising:

a pattern generator configured to define a pattern based on the set of anomaly rules;

a matcher configured to match the pattern with the data in the database; and

a rule-based detector configured to identify the matched data as anomalies.

10. A computer-implemented method for detecting anomalies using sample-based rule identification, comprising the steps of:

maintaining data for data analytics in a database;

statistically identifying a rare pattern in the data;

labeling the identified rare pattern as at least one of anomaly and non-anomaly based on verification by a domain expert;

defining a set of anomaly rules based on the labeled anomaly; and

detecting and classifying other anomalies in the data by applying the set of anomaly rules to the data,

wherein a non-transitory computer readable storage medium storing code for executing on a computer system to perform the method steps.

11. A computer-implemented method for detecting anomalies using sample-based rule identification, comprising the steps of:

maintaining data for data analytics in a database;

defining a set of anomaly rules;

statistically identifying a rare pattern in the data;

labeling the identified rare pattern as at least one of anomaly and non-anomaly based on verification by a domain expert;

adjusting the set of anomaly rules based on the labeled anomaly; and

detecting and classifying other anomalies in the data by applying the adjusted set of anomaly rules to the data,

wherein a non-transitory computer readable storage medium storing code for executing on a computer system to perform the method steps.

12. A method according to claim 11, further comprising:

recognizing the identified rare pattern as the labeled non-anomaly; and

modifying the set of anomaly rules not to comprise the labeled non-anomaly.

13. A method according to claim 11, further comprising:

recognizing the identified rare pattern as the labeled anomaly; and

converting the labeled anomaly to a new anomaly rule and adding the new rule to the set of the anomaly rules.

14. A method according to claim 13, further comprising:

identifying a space for transformation;

introducing one labeled anomaly as an anomaly example;

obtaining a statistical anomaly logic of the labeled anomaly;

executing the transformation based on the statistical anomaly logic; and

obtaining the new anomaly rule.

15. A method according to claim 13, further comprising:

identifying a space for transformation;

introducing multiple labeled anomalies as anomaly examples;

obtaining a statistical anomaly logic for each of the anomaly samples;

executing the transformation based on the statistical anomaly logics;

obtaining each new anomaly rule for each anomaly example; and

adding a condition to the new anomaly rule that the obtained new anomaly rules are similar.

16. A method according to claim 13, further comprising:

selecting a classification algorithm; and

modifying the classification algorithm comprising at least one of;

modifying criteria of fit; and

modifying a regularization term.

17. A method according to claim 11, further comprising:

defining an anomaly threshold comprised in the set of anomaly rules;

defining a score of the labeled non-anomaly; and

refining the set of anomaly rules by comparing the score of the labeled non-anomaly to the anomaly threshold and raising the anomaly threshold if the score of the labeled non-anomaly is below the anomaly threshold.

18. A method according to claim 11, further comprising at least one of:

statistically detecting rare incidents in the data compared to the rest of the population in the data;

statistically detecting rare trends in the data with regard to time; and

statistically detecting rare events with regard to correlation of events in the data.

19. A method according to claim 11, further comprising:

defining a pattern based on the set of anomaly rules;

matching the pattern with the data in the database; and

identifying the matched data as anomalies.

20. A method according to claim 11, wherein the data analytics are conducted for controlling manipulative malicious activities in at least one of the fields of social welfare, credit card, transportation systems, the Internet networks, and healthcare systems.