IDENTIFYING DEVIATIONS IN DATA

• US 20160352767A1
• Filed: 01/24/2014
• Published: 12/01/2016
• Est. Priority Date: 01/24/2014
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: Owhadi, Eric
• Granted Patent: US 10,560,469 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

In an example, metrics that cause a deviation in data may be identified by collecting the data for selected metrics stored in a plurality of tables. A metric vector is constructed based on the data for the selected metrics. A probability density may be calculated for the metric vector that indicates a deviation value for the metric vector relative to other metric vectors. Moreover, an outlier metric from the metric vector that causes the deviation value for the metric vector may be identified.

Claims

1. A method to identify metrics that cause a deviation in data, comprising:

collecting, by a processor, the data for selected metrics stored in a plurality of tables;

constructing a metric vector based on the data for the selected metrics;

calculating a probability density for the metric vector that indicates a deviation value for the metric vector relative to other metric vectors; and

identifying an outlier metric from the metric vector that causes the deviation value for the metric vector.

2. The method of claim 1, wherein the calculating of the probability density for the metric vector includes implementing a Multivariate Gaussian Distribution algorithm.

3. The method of claim 2, wherein the identifying of the outlier metric includes:

selecting a maximum outlier product from the multiplication of (x−

μ

)^TΣ

<sup>−

1</sup> and (x−

μ

), where x is the metric vector, μ

is a mean distribution vector, and Σ

is a covariance matrix; and

determining the outlier metric based on the maximum outlier product.

4. The method of claim 3, wherein the determining of the outlier metric includes comparing a first metric from the maximum outlier product that is equal to (x−

μ

)^TΣ

<sup>−

1</sup> to a second metric from the maximum outlier product that is equal to (x−

μ

).

5. The method of claim 4, wherein the second metric is identified as the outlier metric that solely cause the deviation value responsive to the second metric being greater than the first metric, and the second metric is identified as correlating with the another metric to cause the deviation value responsive to the second metric being less than the first metric.

6. The method of claim 5, wherein identifying the another metric includes:

calculating C_i,j=Σ

_i,j/√

{square root over (Σ

_i,iΣ

_j,j)} to determine the another metric, where C is a correlation matrix representing the degree of correlation between the second metric i and the another metric j;

computing the maximum value of |C_i,j|; and

reporting the another metric j and a direct or inverse correlation between the second metric i and the another metric j represented by a sign of C_i,j.

7. The method of claim 1, wherein the constructing of the metric vector includes:

normalizing the data for the selected metrics within designated time boundaries; and

creating a pivot for each node of the selected metrics,

wherein the normalizing of the data for the selected metrics and the creating of the pivot for each node of the selected metrics are implemented by an in-memory join syntax.

8. The method of claim 1, including triggering an alert responsive to the calculated probability density value reaching a predetermined threshold.

9. The method of claim 1, including generating a report, wherein the report includes a column for the calculated probability density and a column for the identified outlier metric.

10. A system to identify metrics that cause a deviation in data, comprising:

a data collector engine, executed by a processor, to collect the data for selected metrics stored in a plurality of tables;

a vector generating engine to construct a metric vector based on the data for the selected metrics;

a probability engine to calculate a probability density for the metric vector using a Multivariate Gaussian Distribution algorithm, wherein the probability density indicates a deviation value for the metric vector relative to other metric vectors; and

an outlier engine to identify an outlier metric from the metric vector that causes the deviation value for the metric vector.

11. The system of claim 10, wherein to identify the outlier metric, the outlier engine is to;

select a maximum outlier product from the multiplication of (x−

μ

)^TΣ

<sup>−

1</sup> and (x−

μ

), where x is the metric vector, μ

is a mean distribution vector, and Σ

is a covariance matrix; and

determine the outlier metric based on the maximum outlier product.

12. The system of claim 11, wherein to determine the outlier metric the outlier engine is to:

compare a first metric from the maximum outlier product that is equal to (x−

u)^TΣ

<sup>−

1</sup> to a second metric from the maximum outlier product that is equal to (x−

μ

);

in response to the second metric being greater than the first metric, identify the second as the outlier metric that solely cause the deviation value; and

in response to the second metric being less than the first metric, identify the second metric as correlating with the another metric to cause the deviation value.

13. The system of claim 12, wherein to identify the another metric, the outlier engine is to:

calculate C_i,j=Σ

_i,j/√

{square root over (Σ

_i,iΣ

_j,j)} to determine the another metric, where C is a correlation matrix representing the degree of correlation between the second metric i and the another metric j;

compute the maximum value of |C_i,j|; and

report the another metric j and a direct or inverse correlation between the second metric i and the another metric j represented by a sign of C_i,j.

14. The system of claim 10, wherein to construct a metric vector, the vector generating engine is to:

normalize the data for the selected metrics within designated time boundaries; and

create a pivot for each node of the selected metrics,

wherein the normalizing of the data for the selected metrics and the creating of the pivot for each node of the selected metrics are implemented by an in-memory join syntax.

15. A non-transitory computer readable medium including machine readable instructions executable by a processor to:

collect data for selected metrics stored in a plurality of tables;

construct a metric vector based on the data for the selected metrics;

calculate a probability density for the metric vector using a Multivariate Gaussian Distribution algorithm, wherein the probability density indicates a deviation value for the metric vector relative to other metric vectors; and

identity an outlier metric from the metric vector that causes the deviation value for the metric vector.