Separating uploads into aggregate and raw data storage

• US 20070027843A1
• Filed: 07/28/2005
• Published: 02/01/2007
• Est. Priority Date: 07/28/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Murphy, Shawn, Auerbach, Alan
• Granted Patent: US 7,587,388 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system for sampling raw data from a reporting application and segmenting portions of the sampled data into at least one of a set of raw data stores. The system enables ad hoc querying to be done against a data warehouse using the set of raw data stores and in conjunction with an aggregate store. A reporting aspect of the system is responsible for locating the appropriate store when responding to a query. The system also segments data based on anticipated usage of the raw data and appropriately places them into a raw data store having an access speed that corresponds to the anticipated usage level of the data.

Claims

1. A method of data warehousing incoming data from a reporting application, the method comprising:

receiving raw data from a reporting application;

selecting a time marker, wherein the time marker relates to a product cycle milestone;

sampling the received data at a higher rate when the data is received close to the time marker;

storing a first sampled data portion in a first memory medium; and

storing a second sampled data portion in a second memory medium when the second sampled portion is received further in time from the time marker than the first sampled data portion, and wherein the first memory medium is a high speed access medium and the second memory medium is a low speed access medium.

2. The method of claim 1, wherein selecting a time marker comprises selecting a time marker based on a usage characteristic of the reporting application.

3. The method of claim 2, wherein the first sampled data portion is stored in a high speed access medium when the anticipated usage characteristic is high and a low speed access medium when the anticipated usage characteristic is low.

4. The method of claim 2, wherein the usage characteristic is an incoming data traffic pattern.

5. The method of claim 1, further comprising monitoring the received data for a usage pattern and selecting a time marker based on the usage pattern.

6. The method of claim 1, further comprising monitoring the received data for a set of data characteristics relating to a metric and selecting the content of the first and second sample portions based on the set of characteristics.

7. The method of claim 6, wherein selecting the content of the first and second sample portions comprises selecting a control set of reporting application data and randomly selecting a variable set of reporting application data.

8. The method of claim 7, further comprising monitoring the control set and the variable set and selecting a time marker based on a statistically significant deviation between the control set and variable set.

9. The method of claim 1, further comprising:

receiving a query;

checking a directory table to determine whether the query can be answered based on one of an aggregate data store and a set of raw data stores;

determining a set of raw data stores to retrieve information to answer the query;

estimating a duration for retrieving the information based on a characteristic of the set of raw data stores, wherein the characteristic is an access speed of a data store; and

retrieving the information from one of an aggregate data store and a raw data store.

10. The method of claim 9, further comprising creating a cube if query data is stored in a raw data store and loading the cube with data from the raw data store, and wherein the query is answered through the data warehouse using the cube.

11. A computer system comprising:

a reporting application;

a data warehouse;

a set of raw data stores, wherein the raw data stores have different access speeds;

a configuration server adapted to alter the configuration of the data warehouse; and

a process server that accepts data from the reporting application and determines whether to load the data into one of the data warehouse and the set of raw data stores and wherein the process server is further configured to load the data into one of the raw data stores of the set of raw data stores based on an anticipated usage probability of the data.

12. The system of claim 11, further comprising a reporting server and a lookup directory, wherein the reporting server is configured to accept a user query and determine the location of data necessary to answer the query based on the lookup directory.

13. The system of claim 12, wherein the reporting server is configured to estimate the duration of a query based on the location of the data and the media in which the data is stored.

14. The system of claim 13, wherein the reporting server retrieves data from a location based on the estimated duration.

15. The system of claim 13, wherein the reporting server answers the query by performing one of initiating the generation of a cube for the data warehouse and loading the cube with information from a raw data store, and creating a database table and populating the table using a SQL command.

16. The system of claim 11, wherein the process server determines whether to load the data into one of the data warehouse and the set of raw data stores based on a set of time markers.

17. The system of claim 11, wherein the process server monitors the usage pattern of the received data and determines whether to load the data into one of the data warehouse and the set of raw data stores based on the usage pattern.

18. The system of claim 17, wherein the process server determines whether to load the data into one of the set of raw data stores based on the access speed of the raw data store.

19. A computer-readable medium having computer-executable instructions for performing operations comprising:

receiving raw data from a reporting application;

selecting a time marker;

sampling the received data based on the time marker;

storing a sampled data portion in a high speed access data store when the data portion is close to the time marker and storing the sampled data portion in a low speed access data store when the data portion is far from the time marker.

20. The computer-readable medium of claim 19, further comprising:

receiving a query;

creating a cube for a data warehouse;

loading the cube with information from one of the high speed access data store and low speed access data store; and

responding to the query using the cube.