Computer-implemented process and system employing outlier score detection for identifying and detecting scenario-specific data elements from a dynamic data source

• US 10,264,027 B2
• Filed: 07/28/2017
• Issued: 04/16/2019
• Est. Priority Date: 11/04/2014
• Status: Active Grant

• Current Assignee: Corelight, Inc.
• Original Assignee: Patternex, Inc. (Corelight, Inc.)
• Inventors: Veeramachaneni, Uday, Korrapati, Vamsi, Bassias, Constantinos, Arnaldo, Ignacio
• Primary Examiner(s): Su, Sarah
• Application Number:
  US15/662,323
• Publication Number:
  US 20170339192A1
• Time in Patent Office: 627 Days
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 2463/102 applying security measure for e-commerce data processing systems or methods, specially adapted for e-commerce G06Q30/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

Methods and apparatuses employing outlier score detection method and apparatus for identifying and detecting threats to an enterprise or e-commerce system are disclosed, including grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; extracting one or more features from the grouped log lines into one or more features tables; using one or more statistical models on the one or more features tables to identify statistical outliers; using the one or more features tables to create one or more rules for identifying threats to the enterprise or e-commerce system; and using the one or more rules on incoming enterprise or e-commerce system data traffic to detect threats to the enterprise or e-commerce system. Other embodiments are described and claimed.

Claims

1. A method for identifying and detecting scenario-specific data elements from a dynamic data source, comprising threats to an enterprise or e-commerce system, the method comprising:

grouping scenario-specific data elements into grouped log lines, the scenario-specific data elements belonging to one or more scenario-specific data element parameters from one or more dynamic data sources and/or from incoming data traffic to the one or more dynamic data sources;

extracting one or more features from the grouped log lines into one or more features tables, said features formed using a feature generator associated with the dynamic data sources;

using one or more statistical models on the one or more features tables to identify statistical outliers;

identifying said statistical outliers for further investigation by a human security analyst using a combination of outlier detection modules, coordinating output from said combination of a plurality of outlier detection modules, at least a subset of said outlier detection modules operating an outlier detection algorithm distinct from the outlier detection algorithms operating on other outlier detection modules within said combination of outlier detection modules;

using the one or more features tables to create one or more adaptive rules for performing at least one of;

further refining statistical models for identification of statistical outlier; and

preventing access by categorized threats to the dynamic data sources,

wherein the method results in improved security to the enterprise or e-commerce system.

2. The method of claim 1, wherein said scenario-specific data elements for the one or more dynamic data source comprises a malicious threat to an enterprise or e-commerce system.

3. The method of claim 2, wherein at least a subset of said combination of outlier detection modules assigns to said features from the grouped log lines a score of feature incongruity.

4. The method of claim 3, further comprising the step of associating a ranking for said features according to said score of feature incongruity.

5. The method of claim 3, wherein said combination of outlier detection modules have varying robustness properties, wherein said score of said feature incongruity potentially ranges across a variety of different measurement ranges.

6. The method of claim 3, wherein said combination of outlier detection modules have varying robustness properties, wherein said score of said feature incongruity potentially ranges across a variety of different measurement directions.

7. The method of claim 6, wherein a subset of said combination of outlier detection modules identify said statistical outliers according to a highest score.

8. The method of claim 6, wherein a subset of said combination of outlier detection modules identify said statistical outliers according to a lowest score.

9. The method of claim 6, further comprising steps of first computing a negative logarithm of scores of feature incongruity in the event that selected ones of said combination of outlier detection modules generate scores varying by tens of orders of magnitude.

10. The method of claim 6, further comprising of shifting said scores to make certain that they have positive support if needed.

11. The method of claim 6, wherein said combination of outlier detection modules further projects said scores of feature incongruity into a common probability space.

12. The method of claim 11, further comprising a step of modeling said scores of feature incongruity using a Weibull distribution modeling process.

13. The method of claim 12, wherein said Weibull distribution modeling process comprises the step of, for a given score, S, corresponding said outlier probability to a cumulative density function in S:

F(S)=P(X<

S).

14. The method of claim 11, further comprising steps of first computing a negative logarithm of scores of feature incongruity in the event that there is need for tail modeling.

15. The method of claim 14, further comprising a step of obtaining a final score for said probabilities by remodeling said probabilities using a Weibull distribution.

16. The method of claim 15, where said probability space ranges in value from 0 to a value representing a maximum number available to a computing apparatus.

17. The method of claim 16, further comprising a step of remapping said scores of feature incongruity having been mapped into said common probability space to a probability space defined by [0, 1-max_tail_score], where the value of 1-max_tail_score relates to an associated computing machinery, and further comprising the step of computing on said associated computing machinery, a tail score for each of said feature vectors according to the formula:

score=log(1/(1−

P)).

18. The method of claim 1, further comprising a step of combining the output of said combination of outlier detection modules for improving process robustness through compensation of individual biases associated with each of said outlier detection modules.

19. An apparatus for identifying and detecting threats to an enterprise or e-commerce system, the apparatus comprising:

one or more hardware processors;

system memory coupled to the one or more hardware processors;

one or more non-transitory memory units coupled to the one or more hardware processors; and

threat identification and detection code stored on the one or more non-transitory memory units that when executed by the one or more hardware processors are configured to perform a method which results in improved security to the enterprise or e-commerce system, the method comprising;

grouping scenario-specific data elements into grouped log lines, said scenario-specific data elements belonging to one or more scenario-specific data element parameters from one or more dynamic data sources and/or from incoming data traffic to the dynamic data sources;

extracting one or more features from the grouped log lines into one or more features tables, said features formed using a feature generator associated with the dynamic data sources;

using one or more statistical models on the one or more features tables to identify statistical outliers;

identifying said statistical outliers for further investigation by a human security analyst using a combination of outlier detection modules, coordinating output from said combination of a plurality of outlier detection modules, at least a subset of said outlier detection modules operating an outlier detection algorithm distinct from the outlier detection algorithms operating on other outlier detection modules within said combination of outlier detection modules; and

using the one or more features tables to create one or more adaptive rules for performing at least one of;

further refining statistical models for identification of statistical outliers; and

preventing access by categorized threats to the dynamic data sources.

20. The method of claim 19, wherein said scenario-specific data elements for the one or more dynamic data source comprises a malicious threat to an enterprise or e-commerce system.