Remote Asynchronous Data Mover

• US 20100268788A1
• Filed: 04/16/2009
• Published: 10/21/2010
• Est. Priority Date: 04/16/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Arimilli, Ravi K., Rajamony, Ramakrishnan, Starke, William J., Arimilli, Lakshminarayana B., Sinharoy, Balaram, Speight, William E., Kalla, Ronald N.
• Granted Patent: US 7,996,564 B2
• Time in Patent Office: Days
• US Class Current: 709/213
• CPC Class Codes: G06F 12/10 Address translation
• CPC Class Codes: G06F 12/1081 for peripheral access to main memory, e.g. direct memory access [DMA]
• CPC Class Codes: G06F 9/54 Interprogram communication

Abstract

A distributed data processing system executes multiple tasks within a parallel job, including a first local task on a local node and at least one task executing on a remote node, with a remote memory having real address (RA) locations mapped to one or more of the source effective addresses (EA) and destination EA of a data move operation initiated by a task executing on the local node. On initiation of the data move operation, remote asynchronous data move (RADM) logic identifies that the operation moves data to/from a first EA that is memory mapped to an RA of the remote memory. The local processor/RADM logic initiates a RADM operation that moves a copy of the data directly from/to the first remote memory by completing the RADM operation using the network interface cards (NICs) of the source and destination processing nodes, determined by accessing a data center for the node IDs of remote memory.

Claims

1. A data processing system having a first processing node comprising:

at least one processor which executes processes of a first task of multiple tasks within a parallel job, including at least one second task executing on a first remote processing node;

a memory communicatively coupled to the at least one processor and having a plurality of physical storage locations having associated real addresses (RAs) for storing data;

wherein one or more effective addresses of a process of the first task executing on the at least one processor is mapped to a RA of at least one physical storage location in a first remote memory of the first remote processing node;

a remote asynchronous data mover (RADM Mover) associated with the at least one processor and which, when triggered by processor execution of a remote asynchronous data move (RADM) instruction/request, which moves data to/from a first effective address (EA) that is memory mapped to a RA of a physical storage location in the remote memory, performs the following function;

initiate a remote asynchronous data move (RADM) operation that moves a copy of the data from/to the first remote memory by;

(a) retrieving from a global resource manager (GRM) identifying information indicating which remote processing node among multiple remote processing nodes has the first remote memory in which the EA of the RADM instruction is mapped to a RA;

(b) completing a user-level virtual move of data at the at least one processor by utilizing a source EA and a destination EA within the RADM instruction; and

(c) triggering a completion of a physical move of the data at the first remote memory utilizing network interface controllers of the first remote processing node and a second node involved with the RADM operation, wherein the first remote node and second node are identified by respective node IDs (NIDs) retrieved from the GRM, and wherein a physical move of the copy of the data occurs concurrent with other ongoing processing on the at least one processor.

2. The data processing system of claim 1, wherein:

the second node is a local processing node; and

when the data is being moved from the first remote node to the local processing node, the NIC of the local processing node receives the data from a remote NIC of the first remote node and places the data within a RA location of memory to which the destination EA of the RADM instruction is mapped, wherein the RADM operation moves a copy of data from the first remote node to the local processing node without direct processor or OS involvement at the local processing node.

3. The data processing system of claim 1, wherein:

the second node is a local processing node; and

when the data is being moved from the local processing node to the first remote node, the NIC of the local processing node retrieves the copy of the data directly from the RA mapped to the source EA of the RADM instruction and forwards the data towards a remote NIC of the first remote node, which remote NIC places the data within a RA location of first remote memory to which the destination EA of the RADM instruction is mapped, wherein the RADM operation moves a copy of data from the local processing node to the first remote node with little direct processor or OS involvement at the local processing node.

4. The data processing system of claim 3, wherein the NIC is a host fabric interface (HFI).

5. The data processing system of claim 1, wherein both the first remote node and the second node are remotely connected, via a network fabric to a local processing node.

6. The data processing system of claim 1, further comprising:

a data center having the indentifying information about the node-level EA-to-RA mappings for a parallel job with multiple tasks executing across multiple processing nodes, wherein the data center maintains task-distribution and address mapping information generated during initialization of the parallel job by the GRM;

wherein the RADM Mover accesses the data center to retrieve the identifying information during said retrieving.

7. The data processing system of claim 1, further comprising:

processing logic for recognizing the RADM instruction/request within an instruction set architecture, which RADM instructions initiate a remote asynchronous data move operation, said RADM instruction including a source effective address (EA) and a destination effective address for the RADM operation, wherein said processor retrieves the source EA and the destination EA from the RADM instruction and places the addresses within one or more registers of the RADM Mover;

wherein the asynchronous data mover includes one or more registers for holding the source effective address (EA) and the destination EA of the RADM operation.

8. The data processing system of claim 7, wherein:

the RADM instruction comprises a set of fields, including a first field with the source EA, a second field with the destination EA, and a third field indicating a move operation to be performed on the data at the source EA; and

the RADM Mover processes information from a received RADM instruction to provide a fourth field with node identifying information, indicating a physical or network location of one or more processing nodes with memory storage locations having RAs to which the source EA and destination EA are mapped; and

wherein the asynchronous data mover forwards the information from the RADM instruction along with the fourth field to the network interface controller (NIC), which forwards an RADM operation request to NICs of a corresponding source processing node and a corresponding destination processing node for completion of the RADM operation.

9. In a data processing system having at least one processor which executes processes of a first task of multiple tasks within a parallel job, including at least one second task executing on a first remote processing node and a memory communicatively coupled to the at least one processor and having a plurality of physical storage locations having associated real addresses (RAs) for storing data, wherein one or more effective addresses of a process of the first task executing on the at least one processor is mapped to a RA of at least one physical storage location in a first remote memory of the first remote processing node, a method comprising:

receiving information related to a remote asynchronous data move (RADM) instruction/request, which moves data to/from a first effective address (EA) that is memory mapped to a RA of a physical storage location in the remote memory; and

initiate a remote asynchronous data move (RADM) operation that moves a copy of the data from/to the first remote memory by;

(a) retrieving from a global resource manager (GRM) identifying information indicating which remote processing node among multiple remote processing nodes has the first remote memory in which the EA of the RADM instruction is mapped to a RA;

(b) completing a user-level virtual move of data at the at least one processor by utilizing a source EA and a destination EA within the RADM instruction; and

(c) triggering a completion of a physical move of the data at the first remote memory utilizing network interface controllers of the first remote processing node and a second node involved with the RADM operation, wherein the first remote node and second node are identified by respective node IDs (NIDs) retrieved from the GRM, and wherein a physical move of the copy of the data occurs concurrent with other ongoing processing on the at least one processor.

10. The method of claim 9, wherein the second node is a local processing node, and the method further comprises:

when the data is being moved from the first remote node to the local processing node, the NIC of the local processing node receiving the data from a remote NIC of the first remote node and places the data within a RA location of memory to which the destination EA of the RADM instruction is mapped, wherein the RADM operation moves a copy of data from the first remote node to the local processing node without direct processor or OS involvement at the local processing node; and

when the data is being moved from the local processing node to the first remote node, the NIC of the local processing node retrieving the copy of the data directly from the RA mapped to the source EA of the RADM instruction and forwards the data towards a remote NIC of the first remote node, which remote NIC places the data within a RA location of first remote memory to which the destination EA of the RADM instruction is mapped, wherein the RADM operation moves a copy of data from the local processing node to the first remote node with little direct processor or OS involvement at the local processing node.

11. The method of claim 10, wherein the NIC is a host fabric interface (HFI).

12. The method of claim 9, wherein both the first remote node and the second node are remotely connected, via a network fabric to a local processing node.

13. The method of claim 9, further comprising:

the RADM Mover accessing a data center to retrieve the identifying information during said retrieving, wherein the data center includes the indentifying information about the node-level EA-to-RA mappings for a parallel job with multiple tasks executing across multiple processing nodes, wherein the data center maintains task-distribution and address mapping information generated during initialization of the parallel job by the GRM.

14. The method of claim 9, further comprising:

recognizing the RADM instruction/request within an instruction set architecture, which RADM instructions initiate a remote asynchronous data move operation, said RADM instruction including a source effective address (EA) and a destination effective address for the RADM operation; and

said processor retrieving the source EA and the destination EA from the RADM instruction and placing the addresses within one or more registers of the RADM Mover;

wherein the asynchronous data mover includes one or more registers for holding the source effective address (EA) and the destination EA of the RADM operation.

15. The method of claim 14, wherein the RADM instruction comprises a set of fields, including a first field with the source EA, a second field with the destination EA, and a third field indicating a move operation to be performed on the data at the source EA, and the method further comprising:

the RADM Mover processing information from a received RADM instruction to provide a fourth field with node identifying information, indicating a physical or network location of one or more processing nodes with memory storage locations having RAs to which the source EA and destination EA are mapped; and

the RADM Mover forwarding the information from the RADM instruction along with the fourth field to the network interface controller (NIC), which forwards an RADM operation request to NICs of a corresponding source processing node and a corresponding destination processing node for completion of the RADM operation.

18. The article of manufacture of claim 10, wherein the NIC is a host fabric interface (HFI) and wherein both the first remote node and the second node are remotely connected, via a network fabric to a local processing node.

20. The method of claim 9, further comprising program code for:

recognizing the RADM instruction/request within an instruction set architecture, which RADM instructions initiate a remote asynchronous data move operation, said RADM instruction including a source effective address (EA) and a destination effective address for the RADM operation;

said processor retrieving the source EA and the destination EA from the RADM instruction and placing the addresses within one or more registers of the RADM Mover;

wherein the asynchronous data mover includes one or more registers for holding the source effective address (EA) and the destination EA of the RADM operation;

wherein the RADM instruction comprises a set of fields, including a first field with the source EA, a second field with the destination EA, and a third field indicating a move operation to be performed on the data at the source EA, and the program code further comprising code for;

the RADM Mover processing information from a received RADM instruction to provide a fourth field with node identifying information, indicating a physical or network location of one or more processing nodes with memory storage locations having RAs to which the source EA and destination EA are mapped; and

the RADM Mover forwarding the information from the RADM instruction along with the fourth field to the network interface controller (NIC), which forwards an RADM operation request to NICs of a corresponding source processing node and a corresponding destination processing node for completion of the RADM operation.

16. An article of manufacture embodied as a computer program product with program code configured for execution within a data processing system having at least one processor which executes processes of a first task of multiple tasks within a parallel job, including at least one second task executing on a first remote processing node and a memory communicatively coupled to the at least one processor and having a plurality of physical storage locations having associated real addresses (RAs) for storing data, wherein one or more effective addresses of a process of the first task executing on the at least one processor is mapped to a RA of at least one physical storage location in a first remote memory of the first remote processing node, the program code comprising code for:

receiving at a remote asynchronous data mover (RADM Mover) associated with the at least one processor, information related to a remote asynchronous data move (RADM) instruction/request, which moves data to/from a first effective address (EA) that is memory mapped to a RA of a physical storage location in the remote memory; and

initiate a remote asynchronous data move (RADM) operation that moves a copy of the data from/to the first remote memory by;

(a) retrieving from a global resource manager (GRM) identifying information indicating which remote processing node among multiple remote processing nodes has the first remote memory in which the EA of the RADM instruction is mapped to a RA;

(b) completing a user-level virtual move of data at the at least one processor by utilizing a source EA and a destination EA within the RADM instruction; and

(c) triggering a completion of a physical move of the data at the first remote memory utilizing network interface controllers of the first remote processing node and a second node involved with the RADM operation, wherein the first remote node and second node are identified by respective node IDs (NIDs) retrieved from the GRM, and wherein a physical move of the copy of the data occurs concurrent with other ongoing processing on the at least one processor.

17. The article of manufacture of claim 16, wherein the second node is a local processing node, and the program code further comprises code for:

when the data is being moved from the first remote node to the local processing node, the NIC of the local processing node receiving the data from a remote NIC of the first remote node and places the data within a RA location of memory to which the destination EA of the RADM instruction is mapped, wherein the RADM operation moves a copy of data from the first remote node to the local processing node without direct processor or OS involvement at the local processing node; and

when the data is being moved from the local processing node to the first remote node, the NIC of the local processing node retrieving the copy of the data directly from the RA mapped to the source EA of the RADM instruction and forwards the data towards a remote NIC of the first remote node, which remote NIC places the data within a RA location of first remote memory to which the destination EA of the RADM instruction is mapped, wherein the RADM operation moves a copy of data from the local processing node to the first remote node with little direct processor or OS involvement at the local processing node.

19. The article of manufacture of claim 16, further comprising program code for:

the RADM Mover accessing a data center to retrieve the identifying information during said retrieving, wherein the data center includes the indentifying information about the node-level EA-to-RA mappings for a parallel job with multiple tasks executing across multiple processing nodes, wherein the data center maintains task-distribution and address mapping information generated during initialization of the parallel job by the GRM.