Formal Language and Translator for Parallel Processing of Data

• US 20080098370A1
• Filed: 10/20/2006
• Published: 04/24/2008
• Est. Priority Date: 10/20/2006
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Reed, Benjamin Clay, Olston, Christopher, Ravikumar, Shanmugasundaram, Tomkins, Andrew, Josifovski, Vanja, Fontoura, Marcus Felipe
• Granted Patent: US 7,921,416 B2
• Time in Patent Office: Days
• US Class Current: 717/139
• CPC Class Codes: G06F 16/2438 Embedded query languages
• CPC Class Codes: G06F 16/2452 Query translation

Abstract

The present invention, in an example embodiment, provides a special-purpose formal language and translator for the parallel processing of large databases in a distributed system. The special-purpose language has features of both a declarative programming language and a procedural programming language and supports the co-grouping of tables, each with an arbitrary alignment function, and the specification of procedural operations to be performed on the resulting co-groups. The language'"'"'s translator translates a program in the language into optimized structured calls to an application programming interface for implementations of functionality related to the parallel processing of tasks over a distributed system. In an example embodiment, the application programming interface includes interfaces for MapReduce functionality, whose implementations are supplemented by the embodiment.

Claims

1. A method, comprising:

accepting as input a program written in a formal language, wherein the formal language allows a declarative co-grouping of one or more tables, each with an alignment function, and a specification of zero or more procedural operations to be performed on each resulting co-group;

translating the program into one or more jobs, wherein each job comprises one or more structured calls to an application programming interface for encoded logic that is operable to generate a plurality of tasks for the parallel processing of the job on one or more data processing devices in a distributed system.

2. The method of claim 1, further comprising the steps of:

assigning the tasks to the data processing devices; and

processing two or more of the tasks in parallel.

3. The method of claim 1, wherein the formal language comprises language constructs that are both declarative and procedural.

4. The method of claim 1, wherein the distributed system utilizes a master-servant communication protocol.

5. The method of claim 1, wherein the tables are flat-file databases.

6. The method of claim 1, wherein the application programming interface comprises Map and Reduce interfaces.

7. The method of claim 1, wherein the number of structured calls is minimized through query optimization.

8. Logic encoded in one or more tangible media for execution and when executed is operable to:

accept as input a program written in a formal language, wherein the formal language allows a declarative co-grouping of one or more tables, each with an alignment function, and a specification of zero or more procedural operations to be performed on each resulting co-group;

translate the program into one or more jobs, wherein each job comprises one or more structured calls to an application programming interface for encoded logic that is operable to generate a plurality of tasks for the parallel processing of the job on one or more data processing devices in a distributed system.

9. The logic of claim 8, wherein the logic when executed is further operable to:

assign the tasks to data-processing devices in a distributed system; and

process the tasks in parallel.

10. The logic of claim 8, wherein the formal language comprises language constructs that are both declarative and procedural.

11. The logic of claim 8, wherein the distributed system utilizes a master-servant communication protocol.

12. The logic of claim 8, wherein the tables are flat-file databases.

13. The logic of claim 8, wherein the application programming interface comprises Map and Reduce interfaces.

14. The logic of claim 8, wherein the number of structured calls is minimized through query optimization.

15. An apparatus comprising:

means to accept as input a program written in a formal language, wherein the formal language allows a declarative co-grouping of one or more tables, each with an alignment function, and a specification of zero or more procedural operations to be performed on each resulting co-group; and

means to translate the program into one or more jobs, wherein each job comprises one or more structured calls to an application programming interface for encoded logic that is operable to generate a plurality of tasks for the parallel processing of the job on one or more data processing devices in a distributed system;

means to assign the tasks to data-processing devices in a distributed system; and

means to process the tasks in parallel.

16. The apparatus of claim 15, wherein the formal language comprises language constructs that are both declarative and procedural.

17. The apparatus of claim 15, wherein the distributed system utilizes a master-servant communication protocol.

18. The apparatus of claim 15, wherein the tables are flat-file databases.

19. The apparatus of claim 15, wherein the application programming interface comprises Map and Reduce interfaces.

20. The apparatus of claim 15, wherein the number of structured calls is minimized through query optimization.