Collective Operations in a File System Based Execution Model

• US 20120060018A1
• Filed: 09/02/2010
• Published: 03/08/2012
• Est. Priority Date: 09/02/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hensbergen, Eric Van, Shinde, Pravin
• Granted Patent: US 8,375,070 B2
• Time in Patent Office: Days
• US Class Current: 712/220
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes

Abstract

A mechanism is provided for group communications using a MULTI-PIPE synthetic file system. A master application creates a multi-pipe synthetic file in the MULTI-PIPE synthetic file system, the master application indicating a multi-pipe operation to be performed. The master application then writes a header-control block of the multi-pipe synthetic file specifying at least one of a multi-pipe synthetic file system name, a message type, a message size, a specific destination, or a specification of the multi-pipe operation. Any other application participating in the group communications then opens the same multi-pipe synthetic file. A MULTI-PIPE file system module then implements the multi-pipe operation as identified by the master application. The master application and the other applications then either read or write operation messages to the multi-pipe synthetic file and the MULTI-PIPE synthetic file system module performs appropriate actions.

Claims

1. A method, in a data processing system, for group communications using a MULTI-PIPE synthetic file system, the method comprising:

creating, by a master application in the data processing system, a multi-pipe synthetic file in a plurality of multi-pipe synthetic files in the MULTI-PIPE synthetic file system, wherein the master application indicates a multi-pipe operation to be performed;

writing, by the master application, a header-control block to the multi-pipe synthetic file specifying at least one of a multi-pipe synthetic file system name, a message type, a message size, a specific destination, or a specification of the multi-pipe operation by writing to a specific offset;

opening, by at least one other application in the plurality of applications participating in the group communications, the multi-pipe synthetic file in the MULTI-PIPE synthetic file system;

signaling, by the MULTI-PIPE synthetic file system, a MULTI-PIPE file system module in an operating system of the multi-pipe operation; and

implementing, by the MULTI-PIPE file system module, the multi-pipe operation as identified by the master application, wherein the master application and the at least one other application in the group communication either read or write operation messages to the multi-pipe synthetic file, wherein the messages are signaled to the MULTI-PIPE files system module, and wherein the MULTI-PIPE synthetic file system module performs an appropriate action based on the mode identified by the master application.

2. The method of claim 1, wherein the multi-pipe synthetic file opened by the at least one other application is limited to only being used in the mode directed by the master application and wherein the at least one other application is not able to send any control messages.

3. The method of claim 1, wherein the mode that the at least one other applications open the multi-pipe synthetic file in is dependent on the multi-pipe operation requested by the master application.

4. The method of claim 1, wherein the MULTI-PIPE file system module is implemented as at least one of user space file servers, embedded within the operating system itself as built-in file systems, or a set of dynamically loaded modules.

5. The method of claim 1, where the MULTI-PIPE synthetic file system is constructed to allow allocation of new instances of the plurality of multi-pipe synthetic files via user action, either by leveraging existing file-system interfaces or by using a synthetic control file which the plurality of applications may send commands to in order to allocate new instances of the plurality of multi-pipe synthetic files.

6. The method of claim 1, wherein any application writing a message with a header-control block is directed to a specific enumerated reader queue where it will be received by a specific application when read from the multi-pipe file.

7. The method of claim 1, wherein the master application writing a broadcast message to the multi-pipe synthetic file triggers a copy of the broadcast message to be delivered to all applications in the group communications reading from the multi-pipe synthetic file by encoding a broadcast flag in the header-control block, wherein the master application writing the broadcast message which is longer than an atomic I/O buffer size of the multi-pipe synthetic file is delivered using the header-control block with the message size specifying the total size of the broadcast message, and wherein the master application writes the broadcast message which is longer than the atomic I/O buffer size of the multi-pipe synthetic file as a contiguous unit to at least one other application.

8. The method of claim 1, where different ones of the at least one other application are isolated from one another by the MULTI-PIPE synthetic file system enabling application appropriate multiplexing without the limitations of native file system I/O multiplexing of the operating system and wherein, in non-enumerated multi-pipe synthetic files, individual ones of the at least one other application reading from the multi-pipe synthetic file receive complete messages in a round robin fashion from the master application without fragment message intermixing.

9. The method of claim 1, where the master application changes the multi-pipe synthetic file system name of the multi-pipe synthetic file by writing to the header-control block of the multi-pipe synthetic file and wherein the master application clears the specification of the multi-pipe operation of the multi-pipe synthetic file by writing to the header-control block of the multi-pipe synthetic file.

10. The method of claim 1, wherein:

all of the at least one other application opening the multi-pipe synthetic file are blocked until a first predetermined number of the at least one other application are blocked as specified by the master application in the header-control block written to the multi-pipe synthetic file by the master application,

all of the at least one other application reading the multi-pipe synthetic file are blocked until a second predetermined number of the at least one other application reading the multi-pipe synthetic file are blocked as specified by the master application in the header-control block written to the multi-pipe synthetic file by the master application,

all of the at least one other application writing the multi-pipe synthetic file are blocked until a third predetermined number of the at least one other application writing the multi-pipe synthetic file are blocked as specified by the master application in the header-control block written to the multi-pipe synthetic file by the master application,

the master application writing a control message to the multi-pipe file synthetic file is blocked until a fourth predetermined number of the at least one other application have closed the multi-pipe synthetic file as specified by the master application in the header-control block written to the multi-pipe synthetic file by the master application, and

by the master application writes a reduction control message to the multi-pipe synthetic file, a subsequent read on the multi-pipe synthetic file by the master application will block until all of the at least one other applications which have opened the multi-pipe synthetic file for writing to the multi-pipe synthetic file write a value to the multi-pipe synthetic file forming a set of values and the MULTI-PIPE file system module reduces the set of values and returns a result of the reduction to the master application and wherein all of the at least one other applications receive the result of the reduction from the MULTI-PIPE file system module.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

create, using a master application in the computing device, a multi-pipe synthetic file in a plurality of multi-pipe synthetic files in the MULTI-PIPE synthetic file system, wherein the master application indicates a multi-pipe operation to be performed;

write, using the master application, a header-control block to the multi-pipe synthetic file specifying at least one of a multi-pipe synthetic file system name, a message type, a message size, a specific destination, or a specification of the multi-pipe operation by writing to a specific offset;

open, using at least one other application in the plurality of applications participating in the group communications, the multi-pipe synthetic file in the MULTI-PIPE synthetic file system;

signal, using the MULTI-PIPE synthetic file system, a MULTI-PIPE file system module in an operating system of the multi-pipe operation; and

implement, using the MULTI-PIPE file system module, the multi-pipe operation as identified by the master application, wherein the master application and the at least one other application in the group communication either read or write operation messages to the multi-pipe synthetic file, wherein the messages are signaled to the MULTI-PIPE files system module, and wherein the MULTI-PIPE synthetic file system module performs an appropriate action based on the mode identified by the master application.

12. The computer program product of claim 11, wherein the multi-pipe synthetic file opened by the at least one other application is limited to only being used in the mode directed by the master application and wherein the at least one other application is not able to send any control messages.

13. The computer program product of claim 11, wherein the mode that the at least one other applications open the multi-pipe synthetic file in is dependent on the multi-pipe operation requested by the master application.

14. The computer program product of claim 11, wherein the MULTI-PIPE file system module is implemented as at least one of user space file servers, embedded within the operating system itself as built-in file systems, or a set of dynamically loaded modules.

15. The computer program product of claim 11, where the MULTI-PIPE synthetic file system is constructed to allow allocation of new instances of the plurality of multi-pipe synthetic files via user action, either by leveraging existing file-system interfaces or by using a synthetic control file which the plurality of applications may send commands to in order to allocate new instances of the plurality of multi-pipe synthetic files.

16. An apparatus, comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

create, using a master application in the apparatus, a multi-pipe synthetic file in a plurality of multi-pipe synthetic files in the MULTI-PIPE synthetic file system, wherein the master application indicates a multi-pipe operation to be performed;

write, using the master application, a header-control block to the multi-pipe synthetic file specifying at least one of a multi-pipe synthetic file system name, a message type, a message size, a specific destination, or a specification of the multi-pipe operation by writing to a specific offset;

open, using at least one other application in the plurality of applications participating in the group communications, the multi-pipe synthetic file in the MULTI-PIPE synthetic file system;

signal, using the MULTI-PIPE synthetic file system, a MULTI-PIPE file system module in an operating system of the multi-pipe operation; and

implement, using the MULTI-PIPE file system module, the multi-pipe operation as identified by the master application, wherein the master application and the at least one other application in the group communication either read or write operation messages to the multi-pipe synthetic file, wherein the messages are signaled to the MULTI-PIPE files system module, and wherein the MULTI-PIPE synthetic file system module performs an appropriate action based on the mode identified by the master application.

17. The apparatus of claim 16, wherein the multi-pipe synthetic file opened by the at least one other application is limited to only being used in the mode directed by the master application and wherein the at least one other application is not able to send any control messages.

18. The apparatus of claim 16, wherein the mode that the at least one other applications open the multi-pipe synthetic file in is dependent on the multi-pipe operation requested by the master application.

19. The apparatus of claim 16, wherein the MULTI-PIPE file system module is implemented as at least one of user space file servers, embedded within the operating system itself as built-in file systems, or a set of dynamically loaded modules.

20. The apparatus of claim 16, where the MULTI-PIPE synthetic file system is constructed to allow allocation of new instances of the plurality of multi-pipe synthetic files via user action, either by leveraging existing file-system interfaces or by using a synthetic control file which the plurality of applications may send commands to in order to allocate new instances of the plurality of multi-pipe synthetic files.