Execution of dataflow jobs

• US 8,539,192 B2
• Filed: 01/08/2010
• Issued: 09/17/2013
• Est. Priority Date: 01/08/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Castillo, Claris, Spreitzer, Michael J., Steinder, Malgorzata, Whalley, Ian N.
• Primary Examiner(s): BATAILLE, PIERRE MICHE
• Application Number:
  US12/684,343
• Publication Number:
  US 20110173410A1
• Time in Patent Office: 1,348 Days
• Field of Search: 711/112, 711/170, 711/173, 711/129, 711/E12.046
• US Class Current: 711/170
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring

Abstract

A method, system and computer program product for storing data in memory. An example system includes at least one multistage application configured to generate intermediate data in a generating stage of the application and consume the intermediate data in a subsequent consuming stage of the application. A runtime profiler is configured to monitor the application'"'"'s execution and dynamically allocate memory to the application from an in-memory data grid.

Claims

1. A system for storing data in memory:

at least one multistage application configured to generate intermediate data in a generating stage of the application and consume the intermediate data in a subsequent consuming stage of the application;

a runtime profiler configured to monitor the application'"'"'s execution and dynamically allocate memory to the application from an in-memory data grid; and

a pre-profiler configured to generate, prior to execution of the multistage application, an initial profile received by the runtime profiler, the initial profile providing an estimated allocation of memory needed from the in-memory data grid to store intermediate data for the multistage application.

2. The system of claim 1, further comprising:

a plurality of computers, each computer including local volatile memory; and

a grid manager coupling the local volatile memory of each computer to form the in-memory data grid.

3. The system of claim 1, wherein the initial profile is based, at least in part, on historical data about a size of the intermediate data generated by the multistage application.

4. The system of claim 1, wherein the initial profile is based, at least in part, on input data to the multistage application.

5. The system of claim 1, wherein the initial profile is based, at least in part, on user input indicating an approximate size of the intermediate data generated by the multistage application.

6. A method for storing data in memory:

receiving an estimation of memory required to store intermediate data of a multistage application, the multistage application configured to generate the intermediate data in a generating stage of the application and consume the intermediate data in a subsequent consuming stage of the application;

allocating memory from an in-memory data grid before execution of the multistage application based on the estimation of memory required to store the intermediate data of the multistage application;

executing the multistage application;

monitoring a change in memory required to store the intermediate data during execution of the multistage application; and

adjusting the allocated memory at the in-memory data grid based on the change in memory required to store the intermediate data during execution of the multistage application.

7. The method of claim 6, wherein the estimation of memory required to store the intermediate data of the multistage application is based, at least in part, on historical data about a size of the intermediate data generated by the multistage application.

8. The method of claim 6, wherein the estimation of memory required to store the intermediate data of the multistage application is based, at least in part, on input data to the multistage application.

9. The method of claim 6, wherein the estimation of memory required to store the intermediate data of the multistage application is based, at least in part, on user input indicating an approximate size of the intermediate data generated by the multistage application.

10. The method of claim 6, wherein allocated memory at the in-memory data grid is adjusted to ensure the multistage application has sufficient memory to store the intermediate data at each stage of its execution.

11. The method of claim 6, wherein the in-memory data grid includes a plurality of computers, each computer including local volatile memory, and a grid manager coupling the local volatile memory of each computer to form the in-memory data grid.

12. The method of claim 11, wherein monitoring the change in the memory required to store the intermediate data includes receiving from the grid manager indication of execution progress of the multistage application.

13. A computer program product for storing data in memory, the computer program product comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code configured to;

receive an estimation of memory required to store intermediate data of a multistage application, the multistage application configured to generate the intermediate data in a generating stage of the application and consume the intermediate data in a subsequent consuming stage of the application;

allocate memory from an in-memory data grid before execution of the multistage application based on the estimation of memory required to store the intermediate data of the multistage application;

monitor a change in memory required to store the intermediate data during execution of the multistage application; and

adjust the allocated memory at the in-memory data grid based on the change in memory required to store the intermediate data during execution of the multistage application.

14. The computer program product of claim 13, wherein the estimation of memory required to store the intermediate data of the multistage application is based, at least in part, on historical data about a size of the intermediate data generated by the multistage application.

15. The computer program product of claim 13, wherein the estimation of memory required to store the intermediate data of the multistage application is based, at least in part, on input data to the multistage application.

16. The computer program product of claim 13, wherein the estimation of memory required to store the intermediate data of the multistage application is based, at least in part, on user input indicating an approximate size of the intermediate data generated by the multistage application.

17. The computer program product of claim 13, wherein allocated memory at the in-memory data grid is adjusted to ensure the multistage application has sufficient memory to store the intermediate data at each stage of its execution.

18. The computer program product of claim 13, wherein the in-memory data grid includes a plurality of computers, each computer including local volatile memory, and a grid manager coupling the local volatile memory of each computer to form the in-memory data grid.

19. The computer program product of claim 18, wherein the computer readable program code for monitoring the change in the memory required to store the intermediate data includes the computer readable program code for receiving from the grid manager indication of execution progress of the multistage application.