Database aggregation query result estimator

• US 7,363,301 B2
• Filed: 10/07/2005
• Issued: 04/22/2008
• Est. Priority Date: 01/12/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Narasayya, Vivek R., Chaudhuri, Surajit, Datar, Mayur D., Motwani, Rajeev
• Primary Examiner(s): ALAM, SHAHID AL
• Application Number:
  US11/246,355
• Publication Number:
  US 20060036600A1
• Time in Patent Office: 928 Days
• Field of Search: 707/2, 707/3, 707/4, 707/5, 707/10, 707/101, 707/102
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/2462 Approximate or statistical queries
• CPC Class Codes: G06F 2216/03 Data mining
• CPC Class Codes: Y10S 707/957 Multidimensional
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

Aggregation queries are performed by first identifying outlier values, aggregating the outlier values, and sampling the remaining data after pruning the outlier values. The sampled data is extrapolated and added to the aggregated outlier values to provide an estimate for each aggregation query. Outlier values are identified by selecting values outside of a selected sliding window of data having the lowest variance. An index is created for the outlier values. The outlier data is removed from the window of data, and separately aggregated. The remaining data without the outliers is then sampled to provide a statistically relevant sample that is then aggregated and extrapolated to provide an estimate for the remaining data. This sampled estimate is combined with the outlier aggregate to form an estimate for the entire set of data.

Claims

1. A method of providing an estimated aggregation result for a set of values having first values and second values, the values of the set comprising values in a database, the method comprising:

obtaining a variance of the first values in different groups of first values, and selecting from among the groups a group of first values having a lower variance than one or more other different of the groups of first values, along with corresponding second values for that group, to use for obtaining the first result of aggregating the first values;

obtaining a first result of aggregating the first values and a second result of aggregating the second values, wherein the first values comprise non-outlier values with respect to the set of values, wherein the second values comprise identified outlier values with respect to the set of values;

sampling the non-outlier first values and aggregating the sampled first values;

aggregating the outlier second values;

obtaining a variance of the first values in different groups of first values; and

for a query to the database, estimating the aggregation result for the set of values based on the first and second results, and storing the estimation for further use in conjunction with the database.

2. The method as set forth in claim 1, further comprising:

extrapolating the first result of aggregating the first values based on a sampling frequency associated with sampled first values used for the aggregating.

3. The method as set forth in claim 1, further comprising:

aggregating the first and second values separately.

4. The method as set forth in claim 1, wherein estimating the aggregation result for the set of values based on the first and second results further comprises:

combining the first result of aggregating the first values and the second result of aggregating the second values.

5. The method as set forth in claim 1, further comprising:

sorting at least a portion of the set of values comprising the first and second values.

6. One or more volatile and/or nonvolatile computer-readable medium storing information to enable a computer to perform a process of providing an estimated aggregation result for a set of values having first values and second values, the values of the set comprising values in a database, the process comprising:

obtaining a variance of the first values in different groups of first values, and selecting from among the groups a group of first values having a lower variance than one or more other different of the groups of first values, along with corresponding second values for that group, to use for obtaining the first result of aggregating the first values;

obtaining a first result of aggregating the first values and a second result of aggregating the second values, wherein the first values comprise non-outlier values with respect to the set of values, wherein the second values comprise identified outlier values with respect to the set of values;

sampling the non-outlier first values and aggregating the sampled first values;

aggregating the outlier second values;

obtaining a variance of the first values in different groups of first values; and

for a query to the database, estimating the aggregation result for the set of values based on the first and second results, and storing the estimation for further use in conjunction with the database.