Discoscript: a simplified distributed computing scripting language

• US 7,861,222 B2
• Filed: 06/13/2007
• Issued: 12/28/2010
• Est. Priority Date: 06/13/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ramsey, William D, Chaiken, Ronnie I
• Primary Examiner(s): Zhen; Wei Y
• Assistant Examiner(s): Wu; Junchun
• Application Number:
  US11/762,333
• Publication Number:
  US 20080313610A1
• Time in Patent Office: 1,294 Days
• Field of Search: None
• US Class Current: 717/115
• CPC Class Codes: G06F 8/314 Parallel programming languages G06F8/313 takes precedence
• CPC Class Codes: G06F 9/45512 Command shells

Abstract

Scripting core commands and aggregations of such commands are provided to script authors to enable them to generate scripts that can be parallel-processed without requiring the author to be aware of parallel-processing techniques. The scripting core commands and aggregations abstract mechanisms that can be executed in parallel, enabling the script author to focus on higher-level concepts. The scripting core commands provided include commands for applying a function in parallel and distributing and joining data in parallel. For added flexibility, one or more scripting core commands can utilize functions written in a different programming language and referenced appropriately in code blocks.

Claims

1. One or more computer storage media comprising computer-executable instructions for providing parallel-processing-capable scripting commands, the computer-executable instructions directed to steps comprising:

interpreting a process scripting command, specifying a process input data and either a process executable file comprising one or more process functions or a process code block comprising the one or more process functions, to generate computer-executable instructions for applying, in parallel across one or more processes, the one or more process functions to the process input data to generate a process output data;

interpreting a distribute scripting command, specifying at least a distribute input data, to generate computer-executable instructions for dividing, in parallel across one or more processes, the distribute input data into two or more subdivisions representing a distribute output data;

interpreting an aggregate scripting command, specifying multiple aggregate input data, to generate computer-executable instructions for combining, in parallel across one or more processes, the multiple aggregate input data into an aggregate output data;

interpreting a join scripting command, specifying a first and second join input data, both having an equivalent number of segments, to generate computer-executable instructions for combining, in parallel across one or more processes, each segment of the first join input data with a corresponding segment of the second join input data to form a join output data; and

interpreting a cross-product scripting command, specifying a first and second cross-product input data, to generate computer-executable instructions for combining, in parallel across one or more processes, each segment of the first cross-product input data with each segment of the second cross-product input data to form a cross-product output data.

2. The computer storage media of claim 1, wherein the generated computer-executable instructions for dividing the distribute input data are generated in accordance with one or more distribute functions if the distribute scripting command specifies either a distribute executable program comprising the one or more distribute functions or a distribute code block comprising the one or more distribute functions;

wherein the generated computer-executable instructions for combining the multiple aggregate input data are generated in accordance with one or more aggregate functions if the aggregate scripting command specifies either an aggregate executable program comprising the one or more aggregate functions or an aggregate code block comprising the one or more aggregate functions; and

wherein the generated computer-executable instructions for combining each segment of the first join input data with the corresponding segment of the second join input data are generated in accordance with one or more join functions if the join command specifies either a join executable program comprising the one or more join functions or a join code block comprising the one or more join functions.

3. The computer storage media of claim 1 comprising further computer-executable instructions directed to interpreting abstractions of the process scripting command, the distribute scripting command, the aggregate scripting command, the join command, or the cross-product command, which perform specific, commonly used operations.

4. The computer storage media of claim 1, wherein the generated computer-executable instructions for combining the multiple aggregate input data comprise computer-executable instructions for combining each subdivision of the multiple aggregate input data in one process with a corresponding subdivision of the multiple aggregate input data in at least one other process if the aggregate scripting command follows the distribute scripting command such that the two or more subdivisions output by the distribute scripting command comprise the multiple aggregate input data.

5. The computer storage media of claim 1, wherein the interpreting of at least one of the process, distribute, aggregate, join or cross-product commands comprises identifying at least one of the process input data, the distribute input data, the multiple aggregate input data, the first and second join input data, and the first and second cross-product input data as output data of a preceding command if the at least one of the process input data, the distribute input data, the multiple aggregate input data, the first and second join input data, and the first and second cross-product input data is not explicitly specified.

6. The computer storage media of claim 1, wherein the process code block comprises a language identification of a computer programming language in which the one or more process functions was written.

7. The computer storage media of claim 6 comprising further computer-executable instructions directed to invoking a compiler associated with the computer programming language identified by the language identification to compile the one or more process functions.

8. The computer storage media of claim 6 comprising further computer-executable instructions directed to wrapping the process code block with computer-executable instructions conforming to the computer programming language identified by the language identification and enabling compilation and linking of the process code block with a script comprising the process scripting command.

9. One or more computer storage media comprising computer-executable instructions for interpreting a script, the computer-executable instructions directed to steps comprising:

interpreting the script with reference to a library of computer-executable commands for generating a program that can be executed across multiple processes, the library comprising;

a process command for applying a process function to a process input data in parallel across one or more processes, a distribute command for dividing a distribute input data into two or more subdivisions in parallel across one or more processes, an aggregate command for combining multiple aggregate input data in parallel across one or more processes, a join command for combining a first join input data and a second join input data, both having an equivalent number of segments, such that each segment of the first join input data is combined with a corresponding segment of the second join input data in parallel across one or more processes, and a cross-product command for combining each segment of a first cross-product input data, that was specified as part of the cross-product command, with each segment of a second cross-product input data, that was also specified as part of the cross-product command, in parallel across one or more processes; and

invoking a compiler to compile a process code block comprising the process function, if the script comprises a process scripting command identifying the process code block.

10. The computer storage media of claim 9, wherein the process code block comprises a language identification of a computer programming language in which the one or more process functions was written;

and wherein the invoked compiler is appropriate for the computer programming language identified by the language identifier.

11. The computer storage media of claim 9 comprising further computer-executable instructions directed to wrapping the process code block with computer-executable instructions conforming to the computer programming language identified by the language identification and enabling compilation and linking of the process code block with the script, if the script comprises the process scripting command identifying the process code block.

12. The computer storage media of claim 9, wherein the interpreting the script comprises executing a process executable file if the script comprises a process scripting command identifying the process executable file.

13. A method for providing parallel-processing-capable commands, the method comprising the steps of:

interpreting a process scripting command, specifying a process input data and either a process executable file comprising one or more process functions or a process code block comprising the one or more process functions, to generate computer-executable instructions for applying, in parallel across one or more processes, the one or more process functions to the process input data to generate a process output data;

interpreting a distribute scripting command, specifying at least a distribute input data, to generate computer-executable instructions for dividing, in parallel across one or more processes, the distribute input data into two or more subdivisions representing a distribute output data;

interpreting an aggregate scripting command, specifying multiple aggregate input data, to generate computer-executable instructions for combining, in parallel across one or more processes, the multiple aggregate input data into an aggregate output data;

interpreting a join scripting command, specifying a first and second join input data, both having an equivalent number of segments, to generate computer-executable instructions for combining, in parallel across one or more processes, each segment of the first join input data with a corresponding segment of the second join input data to form a join output data; and

interpreting a cross-product scripting command, specifying a first and second cross-product input data, to generate computer-executable instructions for combining, in parallel across one or more processes, each segment of the first cross-product input data with each segment of the second cross-product input data to form a cross-product output data.

14. The method of claim 13, wherein the generated computer-executable instructions for dividing the distribute input data are generated in accordance with one or more distribute functions if the distribute scripting command specifies either a distribute executable program comprising the one or more distribute functions or a distribute code block comprising the one or more distribute functions;

wherein the generated computer-executable instructions for combining the multiple aggregate input data are generated in accordance with one or more aggregate functions if the aggregate scripting command specifies either an aggregate executable program comprising the one or more aggregate functions or an aggregate code block comprising the one or more aggregate functions; and

wherein the generated computer-executable instructions for combining each segment of the first join input data with the corresponding segment of the second join input data are generated in accordance with one or more join functions if the join command specifies either a join executable program comprising the one or more join functions or a join code block comprising the one or more join functions.

15. The method of claim 13 further comprising the steps of interpreting abstractions of the process scripting command, the distribute scripting command, the aggregate scripting command, the join command, or the cross-product command, which perform specific, commonly used operations.

16. The method of claim 13, wherein the generated computer-executable instructions for combining the multiple aggregate input data comprise computer-executable instructions for combining each subdivision of the multiple aggregate input data in one process with a corresponding subdivision of the multiple aggregate input data in at least one other process if the aggregate scripting command follows the distribute scripting command such that the two or more subdivisions output by the distribute scripting command comprise the multiple aggregate input data.

17. The method of claim 13, wherein the interpreting of at least one of the process, distribute, aggregate, join or cross-product commands comprises identifying at least one of the process input data, the distribute input data, the multiple aggregate input data, the first and second join input data, and the first and second cross-product input data as output data of a preceding command if the at least one of the process input data, the distribute input data, the multiple aggregate input data, the first and second join input data, and the first and second cross-product input data is not explicitly specified.

18. The method of claim 13, wherein the process code block comprises a language identification of a computer programming language in which the one or more process functions was written.

19. The method of claim 18 further comprising the steps of invoking a compiler associated with the computer programming language identified by the language identification to compile the one or more process functions.

20. The method of claim 18 further comprising the steps of wrapping the process code block with computer-executable instructions conforming to the computer programming language identified by the language identification and enabling compilation and linking of the process code block with a script comprising the process scripting command.