Space- and time-efficient management and summarization of data using intermediate summary structure and hierarchical multidimensional histogram

• US 8,600,704 B2
• Filed: 03/22/2012
• Issued: 12/03/2013
• Est. Priority Date: 09/11/2003
• Status: Expired due to Fees

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Muthukrishnan, Shanmugavelayutham, Strauss, Martin J.
• Primary Examiner(s): Kundu, Sujoy
• Assistant Examiner(s): Desta, Elias
• Application Number:
  US13/427,393
• Publication Number:
  US 20120239346A1
• Time in Patent Office: 621 Days
• Field of Search: 702/180, 702/187, 707/1, 707/2, 707/4, 707/102, 707/104.1, 382/168
• US Class Current: 702/180
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: Y10S 707/957 Multidimensional
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

A method maintains a multidimensional histogram for a data array having a data array size, the method having a processing time substantially less than proportional to the data array size. The method involves receiving a data update that indicates a change to data in the data array; with the data update, updating an intermediate data structure having a size substantially smaller than the data array size, so that the updated intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update; collecting a number of substantially-largest-coefficient linear combinations of then-current data, the number being small compared with the data array size; and forming the multidimensional histogram as a histogram to an intermediate data array re synthesized from the collected linear combinations.

Claims

1. A method of maintaining a multidimensional histogram for a data array having a data array size, the method comprising:

collecting, at a computer, a number of largest coefficient linear combinations of then-current data, the number being smaller than the data array size, each of the largest coefficient linear combinations being a tensor product of Haar wavelets of data in the data array;

discarding, at the computer, one of the largest coefficient linear combinations based on comparing a square of the one of the largest coefficient linear combinations with a criterion; and

forming, at the computer, a multidimensional histogram for an intermediate data structure, the forming based on the collecting and the discarding.

2. The method of claim 1 further comprising:

receiving a data update that indicates a change to data in the data array.

3. The method of claim 2 further comprising:

updating the intermediate data structure based on the data update, so that the intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update and has a size smaller than the data array size.

4. The method of claim 2 further comprising:

determining whether the multidimensional histogram should be formed in response to the receiving a data update, wherein the forming is in response to the determining.

5. The method of claim 3 wherein the forming the multidimensional histogram is executed unconditionally in response to the receiving the data update.

6. The method of claim 3 wherein the intermediate data structure comprises a histogram sketch of the data in the data array, the histogram sketch including an identification sketch of linear combinations of the data in the data array, the identification sketch to identify linear combinations whose coefficients are large.

7. The method of claim 3 wherein the intermediate data structure comprises a histogram sketch of the data in the data array, the histogram sketch including a count sketch of linear combinations of data in the data array, the count sketch to approximate the coefficients of linear combinations.

8. An apparatus for maintaining a multidimensional histogram for a data array having a data array size, the apparatus comprising:

a processor; and

a memory to store computer program instructions, the computer program instructions when executed on the processor cause the processor to perform operations comprising;

collecting a number of largest coefficient linear combinations of then-current data, the number being smaller than the data array size, each of the largest coefficient linear combinations being a tensor product of Haar wavelets of data in the data array;

discarding one of the largest coefficient linear combinations based on comparing a square of the one of the largest coefficient linear combinations with a criterion; and

forming a multidimensional histogram for an intermediate data structure, the forming based on the collecting and the discarding.

9. The apparatus of claim 8, the operations further comprising:

receiving a data update that indicates a change to data in the data array.

10. The apparatus of claim 9, the operations further comprising:

updating the intermediate data structure based on the data update, so that the intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update and has a size smaller than the data array size.

11. The apparatus of claim 9, the operations further comprising:

determining whether the multidimensional histogram should be formed in response to the receiving a data update, wherein the forming is in response to the determining.

12. The apparatus of claim 10 wherein the forming the multidimensional histogram is executed unconditionally in response to the receiving the data update.

13. The apparatus of claim 10 wherein the intermediate data structure comprises a histogram sketch of the data in the data array, the histogram sketch including an identification sketch of linear combinations of the data in the data array, the identification sketch to identify linear combinations whose coefficients are large.

14. The apparatus of claim 10 wherein the intermediate data structure comprises a histogram sketch of the data in the data array, the histogram sketch including a count sketch of linear combinations of data in the data array, the count sketch to approximate the coefficients of linear combinations.

15. A non-transitory computer readable medium storing computer program instructions for maintaining a multidimensional histogram for a data array having a data array size, the computer program instructions, when executed on a processor, cause the processor to perform a operations comprising:

collecting a number of largest coefficient linear combinations of then-current data, the number being smaller than the data array size, each of the largest coefficient linear combinations being a tensor product of Haar wavelets of data in the data array;

discarding one of the largest coefficient linear combinations based on comparing a square of the one of the largest coefficient linear combinations with a criterion;

forming a multidimensional histogram for an intermediate data structure, the forming based on the collecting and the discarding.

16. The non-transitory computer readable medium of claim 15, the operations further comprising:

receiving a data update that indicates a change to data in the data array.

17. The non-transitory computer readable medium of claim 16, the operations further comprising:

updating the intermediate data structure based on the data update, so that the intermediate data structure remains an at-least-approximate representation of the data in the data array as changed by the data update and has a size smaller than the data array size.

18. The non-transitory computer readable medium of claim 16, the operations further comprising:

determining whether the multidimensional histogram should be formed in response to the receiving a data update, wherein the forming is in response to the determining.

19. The non-transitory computer readable medium of claim 17, wherein the forming the multidimensional histogram is executed unconditionally in response to the receiving the data update.

20. The non-transitory computer readable medium of claim 17, wherein the intermediate data structure comprises a histogram sketch of the data in the data array, the histogram sketch including one of an identification sketch of linear combinations of the data in the data array and a count sketch of linear combinations of data in the data array, wherein the identification sketch is configured to identify linear combinations whose coefficients are large and the count sketch is configured to approximate the coefficients of linear combinations.