Method and system for aggregating data distribution models

• US 6,847,924 B1
• Filed: 06/19/2000
• Issued: 01/25/2005
• Est. Priority Date: 06/19/2000
• Status: Expired due to Term

• Current Assignee: Teradata US, Inc. (Teradata Corporation)
• Original Assignee: NCR Corporation
• Inventors: Aldridge, Bruce E., Durrant, Douglas J.
• Primary Examiner(s): Broda, Samuel
• Application Number:
  US09/596,635
• Time in Patent Office: 1,681 Days
• Field of Search: 703 1- 2, 703 6- 12, 702179-181
• US Class Current: 703/2
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]

Abstract

A system and method for creating an aggregated data model from a plurality data distribution models having bins approximating data elements in a plurality of data distributions is disclosed. Each bin of each data distribution model has a polynomial formula for approximating data elements in a respective data distribution. A method for creating the aggregated data model comprises: determining a start point of the aggregated data model having the minimum value and an end point having a maximum value of all of the bins of all of the data distribution models; setting a start point of a first bin of the aggregated data model; setting an end point of a last bin of the aggregated data model; determining a total number of points for the aggregated data model; approximating the data elements described by each data distribution model, each approximated data element comprising one point in the aggregated data model; sorting the points from minimum to maximum; distributing the points into one or more bins in the aggregated data model; and determining a polynomial formula for the points for each bin of the aggregated data model.

Claims

1. A system for creating an aggregated data model from a plurality of data distribution models, each data distribution model describing a data distribution having one or more data elements, each data element having a value, each data distribution model having one or more bins, each bin comprising a start point having a value, an end point having a value, a value indicating the number of data elements for each bin, and a polynomial formula associated with each bin, the polynomial formula approximating the data elements for the respective bin, said system comprising:

a processor; and

a computer program executable on said processor, the computer program adapted to perform the following steps;

(a) determining which start point has the minimum value and which end point has the maximum value of all of the bins of all of the data distribution models;

(b) setting a start point of a first bin of the aggregated data model to said start point determined to have the minimum value;

(c) setting an end point of a last bin of the aggregated data model to said end point determined to have the maximum value;

(d) determining a total number of a plurality of points for the aggregated data model by adding the values indicating the number of data elements from all bins from all data distribution models;

(e) approximating the data elements in the data distribution described by each data distribution model using the start point, polynomial formula, and number of data elements for each bin in each respective data distribution model, each approximated data element comprising one of said points in the aggregated data model;

(f) sorting the points from minimum to maximum;

(g) distributing the points into one or more bins in the aggregated data model such that a substantially equal number of points are in each bin of the aggregated data model; and

(h) determining a polynomial formula with the sorted data elements for each bin of the aggregated data model.

2. The system of claim 1, wherein the computer program is further for determining the end point for each bin in the aggregated data model.

3. The system of claim 1, wherein the computer program is adapted to perform the step of distributing the points into the one or more bins of the aggregated data model according to the following formula:

(g)(1) if the number of points in the aggregated data model is equally divisible into the number of bins, the end point of the first bin is equal to the value of the ith point in the aggregated data model, wherein i is the number of points in each bin determined by dividing the points equally into the number of bins, wherein the value of the end point of each bin is equal to the value of the ith point after the last point in the proceeding bin, wherein the start point of each bin is equal to the point after the last point of the previous bin, else

(g)(2) if the number of data elements in the points is not equally divisible by the number of bins, then the number of points in each bin is determined by dividing the number of points by the number of bins, and then adding one to the count of the points in each of a number of bins equal to the remainder after dividing, wherein the bins that have one added to the count is determined according to the following formula;

for k from 1 to r

bin_add=INT((n*k)/(r+1))

next k

wherein bin_add is the sequential bin number to add one to the count of points to include therein, n is the total number of bins in the aggregated data model, r is the remainder from dividing the number of points in the data distribution by the number of bins, and INT is a function for rounding the result of the bracketed formula to produce an integer result.

4. The system of claim 1, wherein the computer program is for performing separately for each bin of the aggregated data model, the steps of approximating the data elements for each bin, determining the end point for each bin, and determining the polynomial formula for each bin.

5. The system of claim 1, wherein each data distribution model is the result of the computer program performing the following steps:

(A) sorting the data elements in each data distribution from minimum to maximum;

(B) computing the number of data elements in each data distribution;

(C) determining the value of the start point and the value of the end point of each bin by dividing the data elements into a plurality of substantially equal sized bins for each data distribution;

(D) counting the number of data elements in each bin for each data distribution; and

(E) computing each distribution model for each data distribution, each distribution model comprising, for each bin, the start point of the bin, the end point of the bin, and the number of data elements in the bin.

6. The system of claim 5, wherein the computer program is adapted to perform the following steps for determining the start points and end points of the bins for each data distribution model:

(C)(1) selecting as the start point of the first bin the value of the data element having the minimum value in the sorted data distribution;

(C)(2) if the number of data elements in the data distribution is equally divisible into the number of bins, the end point of the first bin is equal to the value of the ith data element in the data distribution, wherein i is the number of data elements in each bin determined by dividing the data elements equally into the number of bins, wherein the value of the end point of each bin following the first bin is equal to the value of the ith data element after the last data element in the proceeding bin, wherein the start point of each bin is equal to the data element after the last data element of the previous bin, and

(C)(3) if the number of data elements in the data distribution is not equally divisible by the number of bins, then the number of data elements in each bin is determined by dividing the number of data elements by the number of bins, and then adding one to the count of the data elements in each of a number of bins equal to the remainder after dividing, wherein the bins that have one added to the count is determined according to the following formula;

for k from 1 to r

bin_add=INT((n*k)/(r+1))

next k

wherein bin_add is the sequential bin number to add one to the count of data elements to include therein, n is the total number of bins in the data distribution model, r is the remainder from dividing the number of data elements in the data distribution by the number of bins, and INT is a function for rounding the result of the bracketed formula to produce an integer result.

7. The system of claim 6, wherein the computer program is further for performing the step of counting by counting, for each bin, each data element satisfying the following formula:

start point<

element value<

=end point

wherein the bin start point is the start point of the respective bin, element value is the value of each data element in each bin, and end point is the end point of the respective bin.

8. The system of claim 7, comprising a storage medium for storing each data distribution model by storing, for each bin, the start point, the end point, the number of data elements, and the parameters of the polynomial formula.

9. The system of claim 1, wherein the computer program is further for performing one or more statistical analysis using the aggregated data model.

10. The system of claim 9, wherein the statistical analysis performed comprises determining the range of the points of the aggregated data model analyzed by subtracting end point of the last bin in the aggregated data model from the start point of the first bin in the aggregated data model.

11. The system of claim 9, wherein the statistical analysis performed comprises determining the inter quantile range of the points of the aggregated data model.

12. The system of claim 9, wherein the statistical analysis performed comprises determining the median value of the aggregated data model by determining a number j computed by dividing the number of bins by 2, and then reading the value of the end point of the jth bin as the median value if the number of bins in the aggregated data model is equally divisible by 2 or by reading the interpolated value using the polynomial function of the mid point of the jth bin if the number of bins in the aggregated data model is not equally divisible by 2.