PROCESSING DATA ACROSS A DISTRIBUTED NETWORK

• US 20080126552A1
• Filed: 09/08/2006
• Published: 05/29/2008
• Est. Priority Date: 09/08/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: BERGER, ALEXANDER, Pasumansky, Mosha, Hargrove, Thomas
• Granted Patent: US 7,844,976 B2
• Time in Patent Office: Days
• US Class Current: 709/230
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups

Abstract

A system, method, and related techniques are disclosed for processing data across a distributed network to a plurality of machines. The method may include receiving a first user-supplied transform and generating a first package based on the first user-supplied transform. The method may further include receiving a designated key and generating a second package based on the key. Furthermore, the method may include receiving a second user-supplied transform and generating a third package based on the second user-supplied transform. Moreover, the method may include distributing the first, second, and third packages to a plurality of machines using a cluster API.

Claims

1. A method for processing data across a distributed network of a plurality of machines, comprising:

receiving one or more data items;

filtering the one or more data items at a first package;

sorting the filtered one or more data items at a second package; and

summing frequencies related to the sorted one or more data items at a third package;

wherein the first, second, and third packages are distributed across the plurality of machines.

2. The method according to claim 1, wherein the one or more data items are filtered based on a first user-supplied transform.

3. The method according to claim 1, wherein the filtered one or more data items are sorted based on a designated key.

4. The method according to claim 1, further comprising distributing the one or more data items to the plurality of machines using a cluster API.

5. The method according to claim 4, wherein the cluster API uses a source and destination transform in order to distribute the one or more data items to the plurality of machines.

6. The method according to claim 5, wherein the source transform reads from one or more streams of data, and the destination writes to one or more streams of data.

7. The method according to claim 1, wherein the frequencies are summed based on a second user-supplied transform.

8. A method for processing data across a distributed network to a plurality of machines, comprising:

receiving a first user-supplied transform;

generating a first package based on the first user-supplied transform;

receiving a designated key;

generating a second package based on the designated key;

receiving a second user-supplied transform;

generating a third package based on the second user-supplied transform; and

distributing the first, second, and third packages to the plurality of machines using a cluster API.

9. The method according to claim 8, wherein the first user-supplied transform defines filtering rules.

10. The method according to claim 8, wherein the designated key defines sorting rules.

11. The method according to claim 8, wherein the second user-supplied transform defines frequency summing rules.

12. The method according to claim 8, wherein the cluster API is a Dryad API.

13. The method according to claim 8, wherein the cluster API uses a source and a destination transform in order to communicate with the first, second, and third packages.

14. One or more computer-readable media having computer-usable instructions stored thereon for performing a method for processing data across a distributed network to a plurality of machines, comprising:

identifying one or more data items;

transmitting a first user-supplied transform to generate a first package;

transmitting a designated key to generate a second package;

transmitting a second user-supplied transform to generate a third package;

receiving output data that has been processed within the first, second, and third packages, wherein the first, second, and third packages are distributed across the plurality of machines.

15. The media according to claim 14, wherein the first user-supplied transform defines filtering rules.

16. The media according to claim 14, wherein the designated key defines sorting rules.

17. The media according to claim 14, wherein the second user-supplied transform defines frequency summing rules.

18. The media according to claim 14, wherein the output data is processed in an order of the first package, second package, and third package.

19. The media according to claim 14, wherein the one or more data items are identified from a log file of a search engine.

20. The media according to claim 14, wherein the designated key is at least one of an alphabetized order, a numeric order, a user ID, a geographic location, a URL address, and an IP address.