COPULA OPTIMIZATION METHOD AND APPARATUS FOR IDENTIFYING AND DETECTING THREATS TO AN ENTERPRISE OR E-COMMERCE SYSTEM AND OTHER APPLICATIONS

• US 20170272471A1
• Filed: 06/02/2017
• Published: 09/21/2017
• Est. Priority Date: 11/04/2014
• Status: Active Grant

• Current Assignee: Corelight, Inc.
• Original Assignee: Patternex, Inc. (Corelight, Inc.)
• Inventors: Veeramachaneni, Uday, Korrapati, Vamsi, Bassias, Constantinos, Arnaldo, Ignacio
• Granted Patent: US 10,044,762 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2221/2101 Auditing as a secondary aspect
• CPC Class Codes: G06N 5/047 Pattern matching networks; Rete networks
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 2463/102 applying security measure for e-commerce data processing systems or methods, specially adapted for e-commerce G06Q30/00
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

Methods and apparatuses employing copula optimization in building multivariate statistical models for identifying and detecting threats to an enterprise or e-commerce system are disclosed, including grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system; extracting one or more features from the grouped log lines into one or more features tables; using one or more statistical models on the one or more features tables to identify statistical outliers and using the one or more rules on incoming enterprise or e-commerce system data traffic to detect threats to the enterprise or e-commerce system. Other embodiments are described and claimed.

Claims

1. A method for identifying and detecting threats to an enterprise or e-commerce system, the method comprising:

grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system;

extracting one or more features from the grouped log lines into one or more features tables;

using one or more statistical models on the one or more features tables to identify statistical outliers; and

for the Copula statistical model, estimating the marginal probability distribution of a feature using a nonparametric kernel density determination using a Gaussian kernel estimation step, said Gaussian kernel estimation step comprising the step of setting a bandwidth of said Gaussian kernel, and further setting said bandwidth using a Scott'"'"'s rule of thumb bandwidth setting process.

2. The method of claim 1, further comprising the step of applying a transformation module to at least a subset of said features for generating non-ordinal variables, in the event that said feature generator produces a plurality of discrete ordinal variables.

3. The method of claim 2, further comprising the step of adding Gaussian noise to said discrete ordinal variables for producing an essentially continuous valued feature, said essentially continuous valued feature exhibiting an essentially uniform cumulative distribution function.

4. The method of claim 3, further comprising the step of adding said Gaussian noise according to the expression

x^c_i=x_i+η

(0,n_p)

where, x^c_i equals a customized noise value, x_i represents the individual value of a feature, n_p represents a variance of said Gaussian distribution η

for adding said noise and having a value determined by n_p=P_s/SNR, where SNR represents a desired signal-to-noise ratio and P_s represents a signal power.

5. The method of claim 4, further comprising the step of estimating P_s for each x_i separately to yield a customized noise value, x^c_i.

6. A system for identifying and detecting threats to an enterprise or e-commerce system, comprising:

a processor memory for storing instructions for identifying and detecting threats to an enterprise or e-commerce system;

a computer process for executing said instructions for identifying and detecting threats to an enterprise or e-commerce system, said instructions comprising;

instructions for grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system;

instructions for extracting one or more features from the grouped log lines into one or more features tables;

instructions for using one or more statistical models on the one or more features tables to identify statistical outliers; and

instructions for applying a Copula statistical model for estimating the marginal probability distribution of a feature using a nonparametric kernel density determination using a Gaussian kernel estimation step, said Gaussian kernel estimation step comprising the step of setting a bandwidth of said Gaussian kernel, and further setting said bandwidth using a Scott'"'"'s rule of thumb bandwidth setting process.

7. The system of claim 6, further comprising instructions for applying a transformation module to at least a subset of said features for generating non-ordinal variables, in the event that said feature generator produces a plurality of discrete ordinal variables.

8. The system of claim 7, further comprising instructions for adding Gaussian noise to said discrete ordinal variables for producing an essentially continuous valued feature, said essentially continuous valued feature exhibiting an essentially uniform cumulative distribution function.

9. The system of claim 8, further comprising instructions for adding said Gaussian noise according to the expression

x^c_i=x_i+η

(0,n_p)

where, x^c_i equals a customized noise value, x_i represents the individual value of a feature, n_p represents a variance of said Gaussian distribution η

for adding said noise and having a value determined by n_p=P_s/SNR, where SNR represents a desired signal-to-noise ratio and P_s represents a signal power.

10. The system of claim 9, further comprising instructions for estimating P_s for each x_i separately to yield a customized noise value, x^c_i.

11. A networked enterprise or e-commerce system comprising a system for identifying and detecting threats to an enterprise or e-commerce system, comprising:

a plurality of computing systems networked over a common communications network for communicating with one another in a secure computing environment;

a system for identifying and detecting threats to said enterprise or e-commerce system, comprising;

a processor memory for storing instructions for identifying and detecting threats to an enterprise or e-commerce system;

a computer process for executing said instructions for identifying and detecting threats to an enterprise or e-commerce system, said instructions comprising;

instructions for grouping log lines belonging to one or more log line parameters from one or more enterprise or e-commerce system data sources and/or from incoming data traffic to the enterprise or e-commerce system;

instructions for extracting one or more features from the grouped log lines into one or more features tables;

instructions for using one or more statistical models on the one or more features tables to identify statistical outliers; and

instructions for applying a Copula statistical model for estimating the marginal probability distribution of a feature using a nonparametric kernel density determination using a Gaussian kernel estimation step, said Gaussian kernel estimation step comprising the step of setting a bandwidth of said Gaussian kernel, and further setting said bandwidth using a Scott'"'"'s rule of thumb bandwidth setting process.

12. The networked enterprise or e-commerce system of claim 11, wherein said system for identifying and detecting threats further comprises instructions for applying a transformation module to at least a subset of said features for generating non-ordinal variables, in the event that said feature generator produces a plurality of discrete ordinal variables.

13. The networked enterprise or e-commerce system of claim 12, wherein said system for identifying and detecting threats further comprises further comprises instructions for adding Gaussian noise to said discrete ordinal variables for producing an essentially continuous valued feature, said essentially continuous valued feature exhibiting an essentially uniform cumulative distribution function.

14. The networked enterprise or e-commerce system of claim 13, wherein said system for identifying and detecting threats further comprises further comprising instructions for adding said Gaussian noise according to the expression

x^c_i=x_i+η

(0,n_p)

where, x^c_i equals a customized noise value, x_i represents the individual value of a feature, n_p represents a variance of said Gaussian distribution η

for adding said noise and having a value determined by n_p=P_s/SNR, where SNR represents a desired signal-to-noise ratio and P_s represents a signal power.

15. The networked enterprise or e-commerce system of claim 14, wherein said system for identifying and detecting threats further comprises further comprising instructions for estimating P_s for each x_i separately to yield a customized noise value, x^c_i.