Space- and time-efficient management and summarization of data using intermediate summary structure and hierarchical multidimensional histogram

• US 8,175,845 B1
• Filed: 06/20/2007
• Issued: 05/08/2012
• Est. Priority Date: 09/11/2003
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Strauss, Martin J., Muthukrishnan, Shanmugavelayutham
• Primary Examiner(s): Charioui, Mohamed
• Assistant Examiner(s): Desta, Elias
• Application Number:
  US11/820,800
• Time in Patent Office: 1,784 Days
• Field of Search: 702/180, 702/187, 707/1, 707/2, 707/4, 707/102, 707/104.1, 382/168
• US Class Current: 702/180
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: Y10S 707/957 Multidimensional
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

A method maintains a multidimensional histogram for a data array having a data array size, the method having a processing time substantially less than proportional to the data array size. The method involves (104) receiving a data update that indicates a change to data in the data array; (106) with the data update, updating an intermediate data structure having a size substantially smaller than the data array size, so that the updated intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update; (110) collecting a number of substantially-largest-coefficient linear combinations of then-current data, the number being small compared with the data array size; and (114) forming the multidimensional histogram as a histogram to an intermediate data array re synthesized from the collected linear combinations. Another method prepares a multidimensional histogram for a data array, the data array characterized by a data array size and including data, the method having an execution time proportional to the data array size and using an amount of storage space substantially smaller than the data array size. The method involves (802) receiving data from the data array; (802) transforming the data into linear combinations of data items, or of approximations of the linear combinations of data items; (804) collecting a moderate number of substantially-largest-coefficient linear combinations of the data; and (806) forming the multidimensional histogram as a histogram to the collected linear combinations.

Claims

1. A method of maintaining a multidimensional histogram for a data array having a data array size, the method having a processing time substantially less than proportional to the data array size, the method comprising:

receiving, at a computer, a data update that indicates a change to data in the data array;

updating, at the computer, an intermediate data structure having a size substantially smaller than the data array size based on the data update to generate an updated intermediate data structure, so that the updated intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update;

collecting a number of largest coefficient linear combinations of then-current data, the number being small compared with the data array size, each of the number of largest coefficient linear combinations being a tensor product of Haar wavelets of data in the data array;

discarding, at the computer, one of the number of largest coefficient linear combinations based on comparing a square of the largest coefficient of the linear combination with at least one criterion;

determining whether a multidimensional histogram should be formed in response to reception of the data update; and

forming the multidimensional histogram as a histogram to an intermediate data array re-synthesized from the collected linear combinations, the forming in response to determining that the multidimensional histogram should be formed in response to the reception of the data update.

2. The method of claim 1, wherein:

the forming the multidimensional histogram is executed unconditionally in response to reception of the data update.

3. An apparatus for maintaining a multidimensional histogram for a data array having a data array size, the apparatus configured to process the multidimensional histogram with a processing time substantially less than proportional to the data array size, the apparatus comprising:

means for receiving a data update that indicates a change to data in the data array;

means for updating an intermediate data structure having a size substantially smaller than the data array size based on the data update to generate an updated intermediate data structure, so that the updated intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update;

means for collecting a number of largest coefficient linear combinations of then-current data, the number being small compared with the data array size, each of the number of largest coefficient linear combinations being a tensor product of Haar wavelets of data in the data array;

means for discarding one of the number of largest coefficient linear combinations based on comparing a square of the largest coefficient of the linear combination with at least one criterion;

means for determining whether a multidimensional histogram should be formed in response to reception of the data update; and

means for forming the multidimensional histogram as a histogram to an intermediate data array re-synthesized from the collected linear combinations, the forming in response to determining that the multidimensional histogram should be formed in response to the reception of the data update.

4. The apparatus of claim 3, wherein:

the means for forming the multidimensional histogram executes unconditionally in response to reception of the data update.

5. A method of maintaining a multidimensional histogram for a data array having a data array size comprising:

receiving, at a computer, a data update that indicates a change to data in the data array;

updating, at the computer, an intermediate data structure having a size substantially smaller than the data array size based on the data update to generate an updated intermediate data structure, so that the updated intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update;

collecting a number of linear combinations of then-current data, the number being small compared with the data array size, each of the number of largest coefficient linear combinations being a tensor product of Haar wavelets of data in the data array;

discarding, at a computer, one of the number of largest coefficient linear combinations based on comparing a square of the largest coefficient of the linear combination with at least one criterion;

determining whether a multidimensional histogram should be formed in response to reception of the data update; and

forming the multidimensional histogram as a histogram to an intermediate data array synthesized from the collected linear combinations, the forming in response to determining that the multidimensional histogram should be formed in response to the reception of the data update.

6. The method of claim 5, wherein:

the forming the multidimensional histogram is executed unconditionally in response to reception of the data update.

7. A non-transitory computer readable medium comprising computer program instructions capable of maintaining a multidimensional histogram for a data array having a data array size and capable of being executed in a computer processor and defining the steps comprising:

receiving a data update that indicates a change to data in the data array;

updating an intermediate data structure having a size substantially smaller than the data array size based on the data update to generate an updated intermediate data structure, so that the updated intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update;

collecting a number of largest coefficient linear combinations of then-current data, the number being small compared with the data array size, each of the number of largest coefficient linear combinations being a tensor product of Haar wavelets of data in the data array;

discarding one of the number of largest coefficient linear combinations based on comparing a square of the largest coefficient of the linear combination with at least one criterion;

determining whether a multidimensional histogram should be formed in response to reception of the data update; and

forming the multidimensional histogram as a histogram to an intermediate data array re-synthesized from the collected linear combinations, the forming in response to determining that the multidimensional histogram should be formed in response to the reception of the data update.

8. The non-transitory computer readable medium of claim 7, wherein:

forming the multidimensional histogram is executed unconditionally in response to reception of the data update.