DATA PROCESSING OVER VERY LARGE DATABASES

• US 20080027920A1
• Filed: 07/26/2006
• Published: 01/31/2008
• Est. Priority Date: 07/26/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Prasad, Rajeev, Hargrove, Thomas H., Schipunov, Vladimir
• Granted Patent: US 7,624,118 B2
• Time in Patent Office: Days
• US Class Current: 707/4
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system that facilitates data processing includes a receiver component that receives an SQL query. A partitioning component partitions the SQL query into multiple tasks and provides the tasks to multiple cluster nodes for processing. The system enables very large amounts of data (e.g., multiple terabytes) to be quickly prepared for analytical processing, such as for use in connection with a search engine, an advertisement provision system, etc.

Claims

1. A system that facilitates data processing, comprising:

a receiver component that receives an SQL query; and

a partitioning component that partitions the SQL query into multiple tasks and provides the tasks to multiple cluster nodes for processing.

2. The system of claim 1, further comprising an aggregation component that aggregates processed data received from the clusters.

3. The system of claim 1, further comprising a rollback component that initiates a rollback of data to a known good state if a load of data into the system fails.

4. The system of claim 1, a subset of the cluster nodes act as reader nodes that read web logs from a web server and provide a subset of the web logs to particular cluster nodes that act as writer nodes and format the data in a suitable form for querying.

5. The system of claim 1, further comprising a monitoring component that monitors the multiple cluster nodes to ensure that the multiple tasks are being performed.

6. The system of claim 1, the multiple cluster nodes include a hierarchical arrangement of sub-clusters of nodes.

7. The system of claim 1, unreliable communications are undertaken between the partitioning component and the multiple cluster nodes.

8. The system of claim 1, the cluster nodes communicate with one another by way of unreliable messaging.

9. The system of claim 1, a cluster node includes a second partitioning component that partitions the received tasks into multiple sub-tasks and provides multiple sub-clusters with the sub-tasks for execution.

10. The system of claim 9, the cluster node includes an aggregation component that aggregates data resultant from execution of the sub-tasks at sub-clusters associated with the cluster node.

11. The system of claim 1, the plurality of clusters reside within a shared nothing storage architecture.

12. The system of claim 1, further comprising a loading component that loads data into the plurality of cluster nodes from a web server, the loading component employs one or more distributed sort algorithms to assign one or more data partitions to one or more certain clusters.

13. The system of claim 1, further comprising a search engine that utilizes results of the SQL query to selectively provide content to a user.

14. The system of claim 1, further comprising an advertisement server that utilizes results of the SQL query to selectively provide advertisements to a user.

15. A method for preparing large amounts of data for analytical processing, comprising:

receiving a query;

determining multiple tasks based upon the query;

providing the multiple tasks to a plurality of cluster nodes through usage of one-way messaging;

aggregating results provided from the plurality of cluster nodes with respect to the multiple tasks; and

providing the aggregated results to an OLE DB client.

16. The method of claim 15, further comprising:

performing data mining on the aggregated results; and

providing at least one of search content and an advertisement based at least in part upon the data mining.

17. The method of claim 15, the received query is an SQL query.

18. The method of claim 17, further comprising:

partitioning the tasks into a plurality of sub-tasks at one or more of the plurality of cluster nodes; and

providing the sub-tasks to multiple cluster nodes that are subservient to the cluster node that is providing the sub-tasks.

19. The method of claim 15, further comprising:

receiving an identity of a user; and

generating the query based at least in part upon the received identity.

20. A data processing system, comprising:

means for receiving an SQL query that is to be executed over multiple terabytes of data; and

means for determining multiple tasks associated with the received SQL query and providing the multiple tasks to a plurality of cluster nodes for processing.