Method, system, and computer program product for outlier detection

• US 20040039548A1
• Filed: 08/23/2002
• Published: 02/26/2004
• Est. Priority Date: 08/23/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Selby, David A., Thomas, Vincent
• Granted Patent: US 7,050,932 B2
• Time in Patent Office: Days
• US Class Current: 702/179
• CPC Class Codes: G06Q 40/02 Banking, e.g. interest calculation or account maintenance credit or loans G06Q40/03
• CPC Class Codes: G06Q 40/08 Insurance

Abstract

A random sampling of a subset of a data population is taken and the sampled data is used to build a predictive model using a cubic or multiquadric radial basis function, and then “scores” (i.e., predictions) are generated for each data point in the entire data population. This process is repeated on additional random sample subsets of the same data population. After a predetermined number of random sample subsets have been modeled and scores for all data points in the population are generated for each of the models, the average score and variation for each predicted data point is calculated. The data points are subjected to rank ordering by their variance, thereby allowing those data points having a high variance to be identified as outliers.

Claims

1. A method of detecting outliers within a multidimensional data set, comprising the steps of:

identifying a subset of said multidimensional data set;

building a predictive model based on said identified subset;

generating predicted values for each data point in said multidimensional data set based on said predictive model;

repeating said identifying, building, and generating steps a predetermined number of iterations to generate a set of predicted values for each data point in said multidimensional data set based on each built predictive model;

calculating an average predicted value for each data point in said multidimensional data set based on the generated predicted values;

calculating the variance for each data point in said multidimensional data set;

ranking the data points in said multidimensional data set based on the calculated variances; and

identifying as outliers any data points in said multidimensional having variances that exceed a predetermined outlier threshold.

2. The method as set forth in claim 1, wherein said generated set of predicted values comprises a probability distribution.

3. The method of claim 1, wherein said multi-dimensional data set comprises insurance data, and wherein the presence of identified outliers indicates insurance fraud.

4. The method of claim 1, wherein said predictive model is built using a cubic radial basis function.

5. The method of claim 1, wherein said predictive model is built using a multi-quadric radial basis function.

6. A system of detecting outliers within a multidimensional data set, comprising:

means for identifying a subset of said multidimensional data set;

means for building a predictive model based on said identified subset;

means for generating predicted values for each data point in said multidimensional data set based on said predictive model;

means for repeating said identifying, building, and generating steps a predetermined number of iterations to generate a set of predicted values for each data point in said multidimensional data set based on each built predictive model;

means for calculating an average predicted value for each data point in said multidimensional data set based on the generated predicted values;

means for calculating the variance for each data point in said multidimensional data set;

means for ranking the data points in said multidimensional data set based on the calculated variances; and

means for identifying as outliers any data points in said multidimensional having variances that exceed a predetermined outlier threshold.

7. The system as set forth in claim 6, wherein said generated set of predicted values comprises a probability distribution.

8. The system of claim 6, wherein said multi-dimensional data set comprises insurance data, and wherein the presence of identified outliers indicates insurance fraud.

9. The system of claim 7, wherein said predictive model is built using a cubic radial basis function.

10. The system of claim 7, wherein said predictive model is built using a multi-quadric radial basis function.

11. A computer program product recorded on computer readable medium for detecting outliers within a multidimensional data set, comprising:

computer readable means for identifying a subset of said multidimensional data set;

computer readable means for building a predictive model based on said identified subset;

computer readable means for generating predicted values for each data point in said multidimensional data set based on said predictive model;

computer readable means for repeating said identifying, building, and generating steps a predetermined number of iterations to generate a set of predicted values for each data point in said multidimensional data set based on each built predictive model;

computer readable means for calculating an average predicted value for each data point in said multidimensional data set based on the generated predicted values;

computer readable means for calculating the variance for each data point in said multidimensional data set;

computer readable means for ranking the data points in said multidimensional data set based on the calculated variances; and

computer readable means for identifying as outliers any data points in said multidimensional having variances that exceed a predetermined outlier threshold.

12. The computer program product as set forth in claim 11, wherein said generated set of predicted values comprises a probability distribution.

13. The computer program product of claim 11, wherein said multi-dimensional data set comprises insurance data, and wherein the presence of identified outliers indicates insurance fraud.

14. The computer program product of claim 11, wherein said predictive model is built using a cubic radial basis function.

15. The computer program product of claim 11, wherein said predictive model is built using a multi-quadric radial basis function.