SIMULATION ENVIRONMENT FOR DISTRIBUTED PROGRAMS

• US 20140201717A1
• Filed: 03/18/2014
• Published: 07/17/2014
• Est. Priority Date: 09/14/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mascaro, Massimo, Ostrovsky, Igor, Omara, Emad A.
• Granted Patent: US 9,442,827 B2
• Time in Patent Office: Days
• US Class Current: 717/125
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 8/20 Software design
• CPC Class Codes: G06F 8/443 Optimisation
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06Q 10/06315 Needs-based resource requirements planning or analysis

Abstract

A dataflow of a distributed application is visualized in a locally simulated execution environment. A scheduler receives a job graph which includes a graph of computational vertices that are designed to be executed on multiple distributed computer systems. The scheduler queries a graph manager to determine which computational vertices of the job graph are ready for execution in a local execution environment. The scheduler queries a cluster manager to determine the organizational topology of the distributed computer systems to simulate the determined topology in the local execution environment. The scheduler queries a data manager to determine data storage locations for each of the computational vertices indicated as being ready for execution in the local execution environment. The scheduler also indicates an instance of each computational vertex to be spawned and executed in the local execution environment based on the organizational topology and indicated data storage locations.

Claims

1. A computing system for visualizing a dataflow of a distributed application in a locally simulated execution environment, the computing system comprising:

one or more processors; and

one or more storage media having stored computer-executable instructions which, when executed by the one or more processors, cause the computing system to perform a method for visualizing a dataflow of a distributed application in a locally simulated execution environment, wherein the method includes;

an act of receiving a subscription to receive dataflow process data from a dataflow visualizer, the dataflow visualizer being configured to display various aspects of a simulated distributed application execution;

an act of monitoring dataflow during execution of the distributed application, wherein the distributed application comprises a plurality of computational vertices that correspond to a job graph of the distributed application; and

an act of sending the monitored dataflow data to the dataflow visualizer for display, wherein the dataflow data is displayed as the application is being executed.

2. The computing system of claim 1, wherein the dataflow visualizer includes a stage visualizer which visualizes the job graph as initially received.

3. The computing system of claim 1, wherein the dataflow visualizer includes an executing graph visualizer that displays the following dataflow data:

the job graph at a particular stage, one or more computational vertices that have been executed, one or more computational vertices that are ready to run and one or more computational vertices not ready to run.

4. The computing system of claim 3, wherein the dataflow visualizer displays dataflow data for each computational vertex of the plurality of computational vertices on the node link to the corresponding vertex.

5. The computing system of claim 4, wherein the method further comprises:

an act of estimating the amount of processing time necessary to process a computational vertex;

an act of estimating the amount of memory necessary to process the computational vertex; and

an act of displaying the estimated processing time and memory amount for the computational vertex on the computational vertex node.

6. The computing system of claim 1, wherein the dataflow visualizer includes a code visualizer that displays what is happening in software code of at least one of the plurality of computational vertices at each step in the software code.

7. The computing system of claim 1, wherein the method further comprises an act of predicting an overall processing resource cost for processing the distributed application in a cloud.

8. One or more computer-readable hardware storage device for implementing a method for visualizing dataflow of a distributed application in a locally simulated execution environment, the computer-readable hardware storage device having stored computer-executable instructions which, when executed by one or more processors of the computing system, cause the computing system to implement the method, wherein the method includes:

an act of receiving a subscription to receive dataflow process data from a dataflow visualizer, the dataflow visualizer being configured to display various aspects of a simulated distributed application execution;

an act of monitoring dataflow during execution of the distributed application, wherein the distributed application comprises a plurality of computational vertices that correspond to a job graph of the distributed application; and

an act of sending the monitored dataflow data to the dataflow visualizer for display, wherein the dataflow data is displayed as the application is being executed.

9. The one or more computer program product of claim 8, wherein the dataflow visualizer includes an executing graph visualizer that displays the following dataflow data:

the job graph at a particular stage, one or more computational vertices that have been executed, one or more computational vertices that are ready to run and one or more computational vertices not ready to run.

10. The one or more computer-readable hardware storage device of claim 9, wherein the dataflow visualizer displays dataflow data for each computational vertex of the plurality of computational vertices on the node link to the corresponding vertex.

11. The one or more computer-readable hardware storage device of claim 10, wherein the method further comprises:

an act of estimating the amount of processing time necessary to process a computational vertex;

an act of estimating the amount of memory necessary to process the computational vertex; and

an act of displaying the estimated processing time and memory amount for the computational vertex on the computational vertex node.

12. The one or more computer-readable hardware storage device of claim 8, wherein the dataflow visualizer includes a code visualizer that displays what is happening in software code of at least one of the plurality of computational vertices at each step in the software code.

13. The one or more computer-readable hardware storage device of claim 8, wherein the method further comprises an act of predicting an overall processing resource cost for processing the distributed application in a cloud.

14. A method implemented by a computing system for visualizing a dataflow of a distributed application in a locally simulated execution environment, wherein the computing system includes one or more processors and one or more storage media having stored computer-executable instructions which, when executed by the one or more processors, cause the computing system to perform the method, the method comprising:

an act of the computing system receiving a subscription to receive dataflow process data from a dataflow visualizer, the dataflow visualizer being configured to display various aspects of a simulated distributed application execution;

an act of the computing system monitoring dataflow during execution of the distributed application, wherein the distributed application comprises a plurality of computational vertices that correspond to a job graph of the distributed application; and

an act of the computing system sending the monitored dataflow data to the dataflow visualizer for display, wherein the dataflow data is displayed as the application is being executed.

15. The method of claim 14, wherein the dataflow visualizer includes a stage visualizer which visualizes the job graph as initially received.

16. The method of claim 14, wherein the dataflow visualizer includes an executing graph visualizer that displays the following dataflow data:

the job graph at a particular stage, one or more computational vertices that have been executed, one or more computational vertices that are ready to run and one or more computational vertices not ready to run.

17. The method of claim 16, wherein the dataflow visualizer displays dataflow data for each computational vertex of the plurality of computational vertices on the node link to the corresponding vertex.

18. The method of claim 17, the method further comprising the following:

an act of estimating the amount of processing time necessary to process a computational vertex;

an act of estimating the amount of memory necessary to process the computational vertex; and

an act of displaying the estimated processing time and memory amount for the computational vertex on the computational vertex node.

19. The method of claim 14, wherein the dataflow visualizer includes a code visualizer that displays what is happening in software code of at least one of the plurality of computational vertices at each step in the software code.

20. The method of claim 14, the method further comprising an act of predicting an overall processing resource cost for processing the distributed application in a cloud.