Separating uploads into aggregate and raw data storage

• US 7,587,388 B2
• Filed: 07/28/2005
• Issued: 09/08/2009
• Est. Priority Date: 07/28/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Auerbach, Alan B., Murphy, Shawn M.
• Primary Examiner(s): Wong; Don
• Assistant Examiner(s): DANG, THANH HA T
• Application Number:
  US11/191,322
• Publication Number:
  US 20070027843A1
• Time in Patent Office: 1,503 Days
• Field of Search: 707/2, 707/3, 707/1, 707/200
• US Class Current: 1/1
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system for sampling raw data from a reporting application and segmenting portions of the sampled data into at least one of a set of raw data stores. The system enables ad hoc querying to be done against a data warehouse using the set of raw data stores and in conjunction with an aggregate store. A reporting aspect of the system is responsible for locating the appropriate store when responding to a query. The system also segments data based on anticipated usage of the raw data and appropriately places them into a raw data store having an access speed that corresponds to the anticipated usage level of the data.

Claims

1. A method of data warehousing incoming data from a reporting application, the method comprising:

receiving raw data from the reporting application;

selecting a time marker, wherein the time marker relates to a product cycle milestone;

sampling the received raw data at a first sampling rate when a rate of incoming raw data exceeds a threshold level associated with the time marker;

storing a first sampled data portion in a first memory medium, the first sampled data portion sampled at the first sampling rate;

changing to a second sampling rate when the rate of incoming raw data falls below the threshold level associated with the time marker, the second sampling rate lower than the first sampling rate;

sampling the received raw data at the second sampling rate; and

storing a second sampled data portion in a second memory medium, the second sampled data portion sampled at the second sampling rate, wherein the first memory medium operates at a higher access speed than the second memory medium.

2. The method of claim 1, wherein selecting the time marker comprises selecting the time marker based on a usage characteristic of the reporting application.

3. The method of claim 2, wherein the first sampled data portion is stored in a high speed access medium when the usage characteristic is high and a low speed access medium when the usage characteristic is low.

4. The method of claim 2, wherein the usage characteristic is an incoming data traffic pattern.

5. The method of claim 1, further comprising monitoring the received raw data for a usage pattern and selecting the time marker based on the usage pattern.

6. The method of claim 1, further comprising monitoring the received raw data for a set of data characteristics relating to a metric and selecting a content of the first and second sampled portions based on the set of data characteristics.

7. The method of claim 6, wherein selecting the content of the first and second sampled portions comprises selecting a control set of reporting application data and randomly selecting a variable set of reporting application data.

8. The method of claim 7, further comprising monitoring the control set and the variable set and selecting the time marker based on a statistically significant deviation between the control set and the variable set.

9. The method of claim 1, further comprising:

receiving a query;

checking a directory table to determine whether an information corresponding to the query is in one of an aggregate data store and a set of raw data stores;

determining a one of the set of raw data stores from which to retrieve the information to answer the query;

estimating a duration to retrieve the information based on a characteristic of the one of the set of raw data stores, wherein the characteristic is an access speed of the one of the set of raw data stores; and

retrieving the information from the one of the set of raw data stores.

10. The method of claim 9, further comprising creating a cube if query data is stored in the one of the set of raw data stores and loading the cube with data from the one of the set of raw data stores, and wherein the query is answered through a data warehouse using the cube.

11. A computer-readable storage medium having computer-executable instructions for performing operations comprising:

receiving raw data from a reporting application;

selecting a time marker, wherein the time marker relates to a product cycle milestone;

monitoring a rate of data received to determine when an amount of traffic increases past a threshold level associated with the time marker;

sampling the received raw data at a higher sampling rate when the raw data is received at a rate above the threshold level and sampling at a lower sampling rate when the raw data is received at a rate below the threshold level; and

storing a first sampled data portion in a high speed access data store when the first sampled data portion is sampled at the higher sampling rate and storing a second sampled data portion that is sampled at the lower sampling rate in a data store having lower speed access than the high speed access data store.

12. The computer-readable storage medium of claim 11, further comprising:

receiving a query;

creating a cube for a data warehouse;

loading the cube with information from one of the high speed access data store and the data store having lower speed access; and

responding to the query using the cube.