Anomaly detection in data perspectives

• US 7,162,489 B2
• Filed: 12/12/2005
• Issued: 01/09/2007
• Est. Priority Date: 06/23/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chickering, David M., Folting, Allan, Heckerman, David E., Thiesson, Bo, Vigesaa, Eric Barber
• Primary Examiner(s): Nghiem; Michael
• Assistant Examiner(s): PRETLOW, DEMETRIUS R
• Application Number:
  US11/299,539
• Publication Number:
  US 20060106560A1
• Time in Patent Office: 393 Days
• Field of Search: 707/102
• US Class Current: 707/600
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06N 7/00 Computing arrangements based on specific mathematical models
• CPC Class Codes: Y10S 707/957 Multidimensional
• CPC Class Codes: Y10S 707/958 Data cubes
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

The present invention leverages curve fitting data techniques to provide automatic detection of data anomalies in a “data tube” from a data perspective, allowing, for example, detection of data anomalies such as on-screen, drill down, and drill across data anomalies in, for example, pivot tables and/or OLAP cubes. It determines if data substantially deviates from a predicted value established by a curve fitting process such as, for example, a piece-wise linear function applied to the data tube. A threshold value can also be employed by the present invention to facilitate in determining a degree of deviation necessary before a data value is considered anomalous. The threshold value can be supplied dynamically and/or statically by a system and/or a user via a user interface. Additionally, the present invention provides an indication to a user of the type and location of a detected anomaly from a top level data perspective.

Claims

1. An automatic data perspective anomaly detection system:

a data tube component that receives at least one data perspective and processes the at least one data perspective into at least one data tube; and

an anomaly detection component that receives and processes the at least one data tube utilizing a curve fitting process to determine any data anomalies, the curve fitting process approximates a function capable of estimating data in the data tube, the estimated data employed as predicted data to determine a deviation score for data in the data tube.

2. The system of claim 1, the at least one data tube comprises a slice of data from the data perspective having only one varying data dimension.

3. The system of claim 1, the anomaly detection component comprises a threshold input to determine an acceptable amount of deviation, data determined to surpass the threshold input are output as anomalies.

4. The system of claim 1, the at least one data perspective selected from n-dimensional views of a data set wherein n=1, 2 or 3.

5. The system of claim 1, the function employed is a piece-wise linear function applied to the data tube.

6. An automatic data perspective anomaly detection component comprising:

a data tube component for generating a plurality of data tubes;

an anomaly detection component comprising a curve fitting function component that receives a data tube from the data tube component to determine a suitable curve fitting function to represent data from the data tube;

a data deviation score component that utilizes the curve fitting function to;

derive predicted values for the data from the data tube;

compare the predicted values to actual values for data in the data tubes; and

determine deviation scores based on an amount of deviation from the predicted value; and

an anomaly determination component that receives the deviation scores and utilizes a threshold input to detect anomalous data that surpasses a threshold value.

7. The component of claim 6, the curve fitting function component receives the data tube from the data tube component and automatically determines an appropriate function to represent data from the data tube.

8. The component of claim 6, the curve fitting function component receives an optional user-specified function to utilize as an appropriate function.

9. The component of claim 6, the date tube component generates the data tube from data perspectives.

10. The component of claim 9, the data perspectives are selected from n-dimensional views of a data set wherein n=1, 2 or 3.

11. The component of claim 10, the data perspectives are selected from at least one of pivot tables and OLAP cubes.

12. The component of claim 10, the data in the data tube is continuous dimension data fit to a piece-wise linear function.

13. The component of claim 6, data is determined to be anomalous if a difference between an actual value and a predicted value is greater than the threshold.

14. A method of facilitating automatic data perspective anomaly detection comprising:

receiving tube data representative of a slice of data from a data perspective where only one dimension varies;

applying a curve fitting function to the data to automatically detect anomalies in the tube data, the curve fitting function provides an estimate of the tube data, the estimate employed as predicted data to determine a deviation score for the tube data; and

displaying the anomalies on an interface component.

15. The method of claim 14, the curve fitting function being at least one of:

derived and user-specified.

16. The method of claim 14, facilitating anomaly detection by using threshold deviation values supplied by at least one of a system and a user.

17. The method of claim 16, the curve fitting function is determined through one of an auto-regression process and a user-provided function.

18. The method of claim 17, a deviation score is determined based on a predicted value of the data given the curve fitting function and the actual data values.

19. The method of claim 18, the deviation score comprises a threshold value for determining the amount of deviation allowed before a data value is considered erroneous.

20. The method of claim 19, the threshold value being one of a static and dynamic value determined by at least one of a system and a user.