Scalable computation of data

• US 20040210562A1
• Filed: 04/18/2003
• Published: 10/21/2004
• Est. Priority Date: 04/18/2003
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cheung, David F., Lee, Yuchun, Kennedy, Ruby L.
• Granted Patent: US 7,870,148 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/3331 Query processing

Abstract

Techniques for producing a cross tabulation are described. The techniques involve issuing a plurality of queries to a database. The queries are for each of at least one sublevel of data for each of at least one dimension of data associated with records in the database. The queries provide sublists of sorted identifiers for each one of the queries. The technique determines occurrences of intersections of levels of one dimension with levels of another dimension of the data associated with records in the database by traversing the sublists to detect intersections of the dimensions.

Claims

1. A method of producing a cross tabulation, the method comprising:

issuing a plurality of queries to a database, the queries being for multiple sublevels of data for multiple dimensions of data associated with records in the database to provide a sublist of sorted record identifiers for each one of the queries; and

determining occurrences of intersections of levels of one dimension with levels of another dimension of the data associated with records in the database by traversing the sub-lists to detect intersections of the dimensions.

2. The method of claim 1 further comprising:

sorting by record identifiers results from the queries into the corresponding plurality of sub-lists.

3. The method of claim 1 wherein determining comprises computing an aggregate of the intersections and populating a cross-tabulation structure.

4. The method of claim 1 wherein the number of dimensions is two dimensions and the two dimensions are first and second dimensions.

5. The method of claim 1 wherein the process produces a cross-tabulation of counts of occurrences of records in the database that have intersecting levels of data for different dimensions of records in the fields.

6. The method of claim 1 wherein issuing issues queries of the form “

Select <

Audience ID(s)>

from<

DB table>

where<

query condition>

order by<

Audience ID(s)>

”

to the database to retrieve audience IDs that satisfy the query.

7. The method of claim 6 wherein the audience ID is a customer identifier and dimensions of data are attributes of customers.

8. The method of claim 1 wherein the audience ID'"'"'are customer ID'"'"'s and attributes are age and income and the query searches the database to retrieve Customer ID'"'"'s over sublevels of ages and sublevels of income.

9. The method of claim 1 wherein aggregating provides values to indicate how many records exist in each combination of dimension sublevels.

10. The method of claim 4 wherein issuing issues N queries to produce N sub-lists of audience ID'"'"'s that are in an first dimension bracket 1 to first dimension bracket N and issues M queries to produce M sub-lists of audience ID'"'"'s that are in second dimension bracket 1 to second dimension bracket M.

11. The method of claim 10 wherein computing an aggregate of occurrences finds a cross tabulation between qualifying first dimension and second dimension to populates a N×

M cross-tabulation structure.

12. The method of claim 11 wherein computing an aggregate further comprises:

merging each sub-list using corresponding cursors to track an occurrence of a audience ID in the sub-lists that matches a current audience ID in a master list of audience ID'"'"'s, and for each occurrence of a matching audience ID accumulating in a corresponding cell of the cross-tabulation structure a count of audience ID'"'"'s forms an intersection of first and second dimension sublevels.

13. The method of claim 11 wherein each cell of the structure is an intersection corresponding to the sublevel under each dimension of the N×

M structure, and the cell is populated with a value that represents the number of times that there was an intersection common audience r ID between a sublevel of the first dimension and a sublevel of the second dimension.

14. The method of claim 1 wherein determining is performed over groups of cells.

15. The method of claim 1 wherein multiple segments for the same dimension are periodically pre-computed for the entire population.

16. The method of claim 1 determining further comprises calculating sums, average counts, minimum counts, maximum counts, a standard deviation, or other calculations of another variable for intersecting dimensions.

17. The method of claim 15 wherein the variable that calculating is performed on is returned in as part of the master list and is referenced for subsequent computations.

18. The method of claim 1 wherein if the levels are mutually exclusive determining an occurrence in one sublist allows the process to skip remaining sublists in dimension and move to finding an occurrence in a succeeding dimension.

19. The method of claim 1 wherein if a dimension has multiple hierarchical levels that can be rolled up from lower levels, only queries for the lowest level of granularity are executed.

20. The method of claim 1 wherein the dimensions of data are recency of purchase, frequency of purchase and an aggregate of amount of purchases so called “

RFM”

characteristics of customer historical behavior.

21. A computer program product residing on a computer readable medium for producing a cross tabulation structure, the computer program comprising instructions for causing a computer to:

issue a plurality of queries to a database, the queries being for multiple sublevels of data for multiple dimensions of data associated with records in the database to provide a sublist of sorted record identifiers for each one of the queries;

determine occurrences of intersections of levels of one dimension with levels of another dimension of the data associated with records in the database by traversing the sub-lists to detect intersections of the dimensions; and

indicate in a cross-tabulation structure each time an intersection of one dimension with levels of another dimension of the data is found.

22. The method of claim 21 further comprising instructions to:

sort by record identifiers results from the queries into the corresponding plurality of sub-lists.

23. The computer program product of claim 21 wherein instructions to indicate comprise instructions to compute an aggregate of the intersections and populate the cross-tabulation structure with the aggregate.

24. The computer program product of claim 21 wherein the number of dimensions is two dimensions and the two dimensions are first and second dimensions.

25. The computer program product of claim 21 wherein instructions to issue queries issues queries of the form “

Select<

Audience ID(s)>

from <

DB table>

where <

query condition>

order by <

Audience ID(s)>

”

to the database to retrieve audience IDs that satisfy the query.

26. The computer program product of claim 21 wherein the audience ID is a customer identifier and dimensions of data are attributes of customers.

27. The computer program product of claim 21 wherein the audience ID'"'"'are customer ID'"'"'s and attributes are age and income and the query searches the database to retrieve Customer ID'"'"'s over sublevels of ages and sublevels of income.

28. The computer program product of claim 21 wherein instructions to issue, issues N queries to produce N sub-lists of audience ID'"'"'s that are in an first dimension bracket 1 to first dimension bracket N and issues M queries to produce M sub-lists of audience ID'"'"'s that are in second dimension bracket 1 to second dimension bracket M.

29. The computer program product of claim 23 wherein instructions to compute, further comprise instructions to compute an aggregate of occurrences to find a cross tabulation between qualifying first dimension and second dimension to populates a N×

M cross-tabulation structure.

30. The computer program product of claim 29 wherein instructions to compute an aggregate further comprises instructions to:

merge each sub-list using corresponding cursors to track an occurrence of a audience ID in the sub-lists that matches a current audience ID in a master list of audience ID'"'"'s, and for each occurrence of a matching audience ID accumulating in a corresponding cell of the cross-tabulation structure a count of audience ID'"'"'s forms an intersection of first and second dimension sublevels.

31. The computer program product of claim 21 wherein instructions to determine are performed over groups of cells.

32. The computer program product of claim 21 wherein multiple segments for the same dimension are periodically pre-computed for the entire population.

33. The computer program product of claim 21 wherein instructions to determine further comprise instructions to calculate sums, average counts, minimum counts, maximum counts, a standard deviation, or other calculations of another variable for intersecting dimensions.

34. The computer program product of claim 33 wherein the variable that calculating is performed on is returned in as part of the master list and is referenced for subsequent computations.

35. The computer program product of claim 21 wherein if the levels are mutually exclusive instructions to determine an occurrence in one sublist allows the process to skip remaining sublists in the dimension and move to find an occurrence in a succeeding dimension.

36. The computer program product of claim 21 wherein if a dimension has multiple hierarchical levels that can be rolled up from lower levels, only queries for the lowest level of granularity are executed.

37. The computer program product of claim 21 wherein the dimensions of data are recency of purchase, frequency of purchase and an aggregate of amount of purchases so called “

RFM”

characteristics of customer historical behavior.

38. Apparatus, comprising:

a processor

a memory coupled to the processor, and

a computer storage medium, storing a computer program product for producing a cross tabulation structure, the computer program comprising instructions which when executed in memory by the processor, causing the apparatus to;

issue a plurality of queries to a database, the queries being for multiple sublevels of data for multiple dimensions of data associated with records in the database to provide a sublist of sorted record identifiers for each one of the queries;

determine occurrences of intersections of levels of one dimension with levels of another dimension of the data associated with records in the database by traversing the sub-lists to detect intersections of the dimensions; and

indicate in a cross-tabulation structure each time an intersection of one dimension with levels of another dimension of the data is found.

39. The apparatus of claim 38 wherein instructions to indicate comprise instructions to compute an aggregate of the intersections and populate the cross-tabulation structure with the aggregate.

40. The apparatus of claim 38 wherein instructions to issue, issues N queries to produce N sub-lists of audience ID'"'"'s that are in an first dimension bracket 1 to first dimension bracket N and issues M queries to produce M sub-lists of audience ID'"'"'s that are in second dimension bracket 1 to second dimension bracket M.

41. The apparatus of claim 39 wherein instructions to compute, further comprise instructions to compute an aggregate of occurrences to find a cross tabulation between qualifying first dimension and second dimension to populates a N×

M cross-tabulation structure.

42. The apparatus of claim 41 wherein instructions to compute an aggregate further comprises instructions to:

merge each sub-list using corresponding cursors to track an occurrence of a audience ID in the sub-lists that matches a current audience ID in a master list of audience ID'"'"'s, and for each occurrence of a matching audience ID accumulating in a corresponding cell of the cross-tabulation structure a count of audience ID'"'"'s forms an intersection of first and second dimension sublevels.

43. The apparatus of claim 38 wherein instructions to determine are performed over groups of cells.

44. The apparatus of claim 38 wherein multiple segments for the same dimension are periodically pre-computed for the entire population.

45. The apparatus of claim 38 wherein instructions to determine further comprise instructions to calculate sums, average counts, minimum counts, maximum counts, a standard deviation, or other calculations of another variable for intersecting dimensions.

46. The apparatus of claim 45 wherein the variable that calculating is performed on is returned in as part of the master list and is referenced for subsequent computations.

47. The apparatus of claim 38 wherein if the levels are mutually exclusive instructions to determine an occurrence in one sublist allows the process to skip remaining sublists in the dimension and move to find an occurrence in a succeeding dimension.

48. The apparatus of claim 38 wherein if a dimension has multiple hierarchical levels that can be rolled up from lower levels, only queries for the lowest level of granularity are executed.