Anomaly detection in dynamically evolving data and systems
First Claim
Patent Images
1. A method comprising:
- a) obtaining, from a traffic analyzer, data comprising a plurality M of N-dimensional data points, wherein N≥
3 and M»
N;
b) defining a collection Q of N-dimensional data points 1≤
Q«
M as a metadata point with dimension N such that the data includes M/Q metadata points;
c) by the traffic analyzer, generating from the metadata points a metadata statistics matrix of size (M/Q)×
N; and
d) by a computer,processing the metadata statistics matrix into a Markov kernel matrix, processing the Markov kernel matrix to obtain r eigenvalues and associated eigenvectors, wherein r«
N,forming a r-dimensional embedded space comprising r-dimensional data points using the r eigenvalues and associate eigenvectors,receiving Q newly arrived N-dimensional points that include N features that form a respective N-dimensional metadata point,embedding the newly arrived N-dimensional metadata point into the r-dimensional embedded space to obtain a new r-dimensional data point;
determining that the new r-dimensional data point is an anomaly based on a density value, andblocking the anomaly,using the M/Q metadata points instead of M data points when Q>
1, and of r-dimensional data points instead of N-dimensional data points wherein r«
N, to increase significantly a speed of detection rate of anomalies and to enhance significantly performance of the computer, respectively.
4 Assignments
0 Petitions
Accused Products
Abstract
Detection of abnormalities in multi-dimensional data is performed by processing the multi-dimensional data to obtain a reduced dimension embedding matrix, using the reduced dimension embedding matrix to form a lower dimension (of at least 2D) embedded space, applying an out-of-sample extension procedure in the embedded space to compute coordinates of a newly arrived data point and using the computed coordinates of the newly arrived data point and Euclidean distances to determine whether the newly arrived data point is normal or abnormal.
-
Citations
12 Claims
-
1. A method comprising:
-
a) obtaining, from a traffic analyzer, data comprising a plurality M of N-dimensional data points, wherein N≥
3 and M»
N;b) defining a collection Q of N-dimensional data points 1≤
Q«
M as a metadata point with dimension N such that the data includes M/Q metadata points;c) by the traffic analyzer, generating from the metadata points a metadata statistics matrix of size (M/Q)×
N; andd) by a computer, processing the metadata statistics matrix into a Markov kernel matrix, processing the Markov kernel matrix to obtain r eigenvalues and associated eigenvectors, wherein r«
N,forming a r-dimensional embedded space comprising r-dimensional data points using the r eigenvalues and associate eigenvectors, receiving Q newly arrived N-dimensional points that include N features that form a respective N-dimensional metadata point, embedding the newly arrived N-dimensional metadata point into the r-dimensional embedded space to obtain a new r-dimensional data point; determining that the new r-dimensional data point is an anomaly based on a density value, and blocking the anomaly, using the M/Q metadata points instead of M data points when Q>
1, and of r-dimensional data points instead of N-dimensional data points wherein r«
N, to increase significantly a speed of detection rate of anomalies and to enhance significantly performance of the computer, respectively. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A system comprising:
-
a) a traffic analyzer used to provide data comprising a plurality M of Tridimensional data points, wherein N≥
3 and M»
N, wherein a collection Q, 1≤
Q«
M, of data points forms a N-dimensional metadata point such that the data includes M/Q metadata points, the traffic analyzer further used to generate from the metadata points a statistics matrix of size (M/Q)×
N; andb) a computer for executing a computer program stored on a computer readable medium, the computer program dedicated to, process the metadata statistics matrix into a Markov kernel matrix, process the Markov kernel matrix to obtain r eigenvalues and associated eigenvectors, wherein r«
N,form a r-dimensional embedded space comprising r-dimensional data points using the r eigenvalues and associate eigenvectors, receive Q newly arrived N-dimensional points that include N features that form a respective N-dimensional metadata point, embed the newly arrived N-dimensional metadata point into the r-dimensional embedded space to obtain a new r-dimensional data point; determine that the new r-dimensional data point is an anomaly based on a density value, and block the anomaly, using the M/Q metadata points instead of M data points when Q>
1, and of r-dimensional data points instead of N-dimensional data points wherein r«
N, to increase significantly a speed of detection rate of anomalies and to enhance significantly performance of the computer, respectively.- View Dependent Claims (8, 9, 10, 11, 12)
-
Specification