ANOMALY DETECTION IN DATA PERSPECTIVES

• US 20050288883A1
• Filed: 06/23/2004
• Published: 12/29/2005
• Est. Priority Date: 06/23/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chickering, David M., Folting, Allan, Heckerman, David E., Thiesson, Bo, Vigesaa, Eric Barber
• Granted Patent: US 7,065,534 B2
• Time in Patent Office: Days
• US Class Current: 702/102
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06N 7/00 Computing arrangements based on specific mathematical models
• CPC Class Codes: Y10S 707/957 Multidimensional
• CPC Class Codes: Y10S 707/958 Data cubes
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

The present invention leverages curve fitting data techniques to provide automatic detection of data anomalies in a “data tube” from a data perspective, allowing, for example, detection of data anomalies such as on-screen, drill down, and drill across data anomalies in, for example, pivot tables and/or OLAP cubes. It determines if data substantially deviates from a predicted value established by a curve fitting process such as, for example, a piece-wise linear function applied to the data tube. A threshold value can also be employed by the present invention to facilitate in determining a degree of deviation necessary before a data value is considered anomalous. The threshold value can be supplied dynamically and/or statically by a system and/or a user via a user interface. Additionally, the present invention provides an indication to a user of the type and location of a detected anomaly from a top level data perspective.

Claims

1. A system that facilitates data perspective analysis, comprising:

a component that receives at least one data perspective; and

an anomaly detection component that automatically analyzes the data perspective to detect at least one data anomaly via a curve fitting process applied to continuous and/or discrete data from a data tube;

the data tube comprising a data slice that includes at least one data cell of the data perspective in which only one data dimension varies.

2. The system of claim 1, the curve fitting process comprising a process that employs, at least in part, a piece-wise linear function.

3. The system of claim 2, the piece-wise linear function comprising a function that utilizes, at least in part, a regression tree.

4. The system of claim 1, the curve fitting process comprising a process that employs, at least in part, a probability model for predicting values in the data perspective, where the probability model depends in a non-trivial way on the position of value within the data perspective.

5. The system of claim 4, the probability model comprising an auto-regressive model.

6. The system of claim 1, the data anomaly comprising an anomaly based on a substantial deviation of a data value from other data values found within the data tube.

7. The system of claim 6, the substantial deviation based on at least one deviation score that exceeds a given threshold.

8. The system of claim 7, the deviation score based, at least in part, on a value of a data cell compared to a predicted value for the data cell derived from a piece-wise linear function representative of a data tube containing the data cell.

9. The system of claim 7, the deviation score based, at least in part, on a value of a data cell compared to a predicted value for the data cell derived from a probability model for predicting discrete values in the data perspective, where the probability model depends in a non-trivial way on the position of value within the data perspective.

10. The system of claim 7, the given threshold comprising at least one selected from the group consisting of a dynamic threshold and a static threshold.

11. The system of claim 10, the given threshold comprising at least one selected from the group consisting of a user-defined threshold and a system-defined threshold.

12. The system of claim 11, further comprising a user interface component that provides a plurality of selectable user-defined thresholds for utilization with different types of data anomalies.

13. The system of claim 1, the data perspective comprising at least one selected from the group consisting of a pivot table and an on-line analytical processing (OLAP) cube.

14. The system of claim 1 further comprising a user interface component that indicates the data anomaly to at least one user.

15. The system of claim 14, the user interface component indicates the data anomaly via at least one selected from the group consisting of a visual indication and an audible indication.

16. The system of claim 14, the user interface component facilitates indications of the data anomaly via utilization of at least one selected from the group consisting of highlighting at least one immediate anomaly and outlining at least one hidden anomaly.

17. The system of claim 14, the user interface component comprising a user interface with a user input control to adjust a level of indication based on a degree of at least one data anomaly.

18. The system of claim 14, the user interface comprising component facilitates indications of the data anomaly via automatic on-screen displaying of at least one data anomaly.

37. A computer readable medium having stored thereon computer executable components of the system of claim 1.

39. A device employing the system of claim 1 comprising at least one selected from the group consisting of a computer, a server, and a handheld electronic device.

19. A method for facilitating data perspective analysis, comprising:

receiving at least one data perspective;

establishing a data tube from the data perspective;

the data tube comprising a data slice that includes at least one data cell of the data perspective in which only one data dimension varies;

determining a curve fitting function representative of continuous and/or discrete data from the data tube;

calculating a deviation score based, at least in part, on a differential of an actual data value and a predicted data value given via the curve fitting function; and

detecting data anomalies via evaluation of the deviation score and a detection criterion.

20. The method of claim 19, the curve fitting function comprising a user-selectable curve fitting function.

21. The method of claim 19, further comprising:

classifying the data anomalies according to their accessibility; and

displaying the data anomalies to a user utilizing a set of anomaly accessibility indicators.

22. The method of claim 21, further comprising:

limiting the data anomalies displayed to a user utilizing a user-selectable data anomaly count, k, that restricts the data anomalies to a top k anomalies based on their deviation scores.

23. The method of claim 21, further comprising:

automatically displaying on-screen at least one data anomaly to a user.

24. The method of claim 21, the anomaly accessibility indicators comprising at least one indicator selected from the group consisting of an on-screen indicator, a drill down indicator, and a drill across indicator.

25. The method of claim 19, the data perspective comprising at least one selected from the group consisting of a pivot table and an on-line analytical processing (OLAP) cube.

26. The method of claim 19, the detection criterion comprising a threshold value.

27. The method of claim 26, the evaluation of the deviation score comprising:

determining if a deviation score surpasses the threshold value.

28. The method of claim 26, the threshold value comprising at least one selected from the group consisting of a dynamic threshold value and a static threshold value.

29. The method of claim 28, the threshold value comprising at least one selected from the group consisting of a user-defined threshold value and a system-defined threshold value.

30. The method of claim 29, further comprising:

adjusting the user-defined threshold value according to a type of data anomaly.

31. The method of claim 26, the curve fitting process comprising a process that employs, at least in part, a piece-wise linear function.

32. The method of claim 31, the piece-wise linear function comprising a function that utilizes, at least in part, a regression tree.

33. The method of claim 26, the curve fitting process comprising a process that employs, at least in part, a probability model for predicting discrete values in the data perspective, where the probability model depends in a non-trivial way on the position of value within the data perspective.

34. The method of claim 33, the probability model comprising a function that utilizes, at least in part, an auto-regressive model.

38. A device employing the method of claim 19 comprising at least one selected from the group consisting of a computer, a server, and a handheld electronic device.

35. A system that facilitates data perspective analysis, comprising:

means for receiving at least one data perspective; and

means for automatically analyzing the data perspective to detect at least one data anomaly via a curve fitting process applied to continuous and/or discrete data from a data tube;

the data tube comprising a data slice that includes at least one data cell of the data perspective in which only one data dimension varies.

36. A data packet, transmitted between two or more computer components, that facilitates data perspective analysis, the data packet comprising, at least in part, information relating to a data perspective analysis system that utilizes, at least in part, a curve fitting process applied to continuous and/or discrete data from a data tube;

the data tube comprising a data slice that includes at least one data cell of a data perspective in which only one data dimension varies.