Memory Bus Protocol To Enable Clustering Between Nodes Of Distinct Physical Domain Address Spaces

• US 20140095651A1
• Filed: 03/15/2013
• Published: 04/03/2014
• Est. Priority Date: 10/02/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Kapil, Sanjiv, Radovic, Zoran
• Granted Patent: US 9,400,821 B2
• Time in Patent Office: Days
• US Class Current: 709/216
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/082 Associative directories G06F12/0822 takes precedence
• CPC Class Codes: G06F 12/12 Replacement control
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 12/1475 in a virtual system, e.g. with translation means
• CPC Class Codes: G06F 13/102 where the programme performs an interfacing function, e.g. device driver G06F13/105 takes precedence; scheduling within device drivers G06F9/52; contention policies within device drivers G06F9/4881
• CPC Class Codes: G06F 13/362 with centralised access control
• CPC Class Codes: G06F 13/4068 Electrical coupling
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 15/7889 Reconfigurable logic implemented as a co-processor instruction execution using a coprocessor G06F9/3877
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/24569 Query processing with adaptation to specific hardware, e.g. adapted for using GPUs or SSDs
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 21/6209 : to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 9/30065 : Loop control instructions; iterative instructions, e.g. LOOP, REPEAT
• CPC Class Codes: G06F 9/381 : Loop buffering
• CPC Class Codes: G06F 9/3851 : from multiple instruction streams, e.g. multistreaming
• CPC Class Codes: G06F 9/3867 : using instruction pipelines
• CPC Class Codes: G06F 9/546 : Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 69/14 : Multichannel or multilink protocols
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A system and method for transferring data and messages between nodes in a cluster is disclosed. Each node in the cluster is a separate physical domain but is connected to other nodes in the cluster through point-to-point high speed links. Each side of a link is coupled to a coprocessor which facilitates the movement of data between and among the nodes. Because each physical domain is separate from any other domain, the coprocessor in a physical domain uses a certificate, called and RKey, to obtain permission to transfer data to another physical domain. When an RKey is received from another physical domain, the coprocessor in the receiving domain validates the key and obtains the physical addresses associated with the key so that it can provide or accept the remote data. Data transfers between pairs of remote nodes in the cluster are permitted as well.

Claims

1. A computer system comprising:

a plurality of computing nodes,

wherein each of the computing nodes of the plurality of computing nodes includes a coprocessor and a memory coupled to the coprocessor, and

wherein each memory resides in a separate and distinct physical domain; and

a plurality of point-to-point physical communications links, wherein one or more communications links of the plurality of links is coupled between each pair of nodes in the plurality of nodes, wherein each coprocessor in a node is coupled to the one or more communications links to transfer data over the at least one communications link, and

wherein each coprocessor is configured;

to transfer data between the memory coupled to the coprocessor and the memory of another node to which the coprocessor is coupled by the one or more communications link using a certificate that grants access to a portion of the memory in the other node, or

to transfer data between two other nodes in the cluster to which the coprocessor is coupled by the one or more communications links using a first certificate that grants access rights to a portion of memory in the first of the two other nodes and a second certificate that grants access rights to a portion of memory in the second of the two other nodes.

2. The system of claim 1,

wherein the coprocessor is a destination node for the data transfer, the destination node including a persistent address space;

and

wherein the destination node transmits an acknowledgment indicating that the transferred data resides in the persistent address space.

3. The system of claim 1, wherein each coprocessor in a node further includes a data transfer machine that moves data from one memory location to another memory location.

4. The system of claim 1, wherein each certificate is an RKey owned by each of the other nodes.

5. The system of claim 4,

wherein each node has a software hypervisor running on the node;

and

wherein the coprocessor of each node operates under control of the software hypervisor.

6. The system of claim 5, wherein each software hypervisor maintains a key table in the memory of the node that maps RKeys to physical addresses of the memory.

7. The system of claim 4,

wherein each RKey includes a hash field for validating the RKey;

and

wherein the coprocessors in the other nodes validate any RKey that is received in the other nodes using the hash field in the RKey.

8. The system of claim 4, wherein each RKey expires after a specified time, an expired RKey revoking access rights that the RKey granted.

9. The system of claim 1,

wherein each node has a software hypervisor running on the node;

wherein the coprocessor in each node transfers data in response to a command issued by the hypervisor; and

wherein the command is part of a control block that includes

a control word that includes the size of the data transfer, and

a completion address that specifies a location at which completion data of a transfer is to be posted in the memory coupled to the coprocessor.

10. The system of claim 9, wherein the command is selected from the group consisting of:

a copy command, a copy and signal command, a fill command, a store command,

a compare and swap command, an atomic add command, an atomic OR command, and an interrupt and sync command

11. A method of transferring data between nodes in a cluster, the method comprising:

receiving and decoding by a local node a command for a data transfer;

determining from the decoded command that the data transfer has a source location in the local node and a destination location in a remote node distinct from the local node, the local and remote nodes being coupled to each other by one or more point-to-point communication links;

obtaining a certificate granting rights to transfer data to the remote node,

transferring the certificate from the local node to the remote node,

wherein the certificate is successfully validated in the remote node;

transferring data from the local node to the remote node using the certificate;

receiving an acknowledgement from the remote node that the transfer is complete; and

updating a completion status indicating the command was successful.

12. The method of claim 11,

wherein the remote node is a destination node that includes a persistent address space;

and

wherein the acknowledgment guarantees that the transferred data resides in the persistent address space.

13. The method of claim 11, wherein the certificate is an RKey owned by the remote node.

14. The method of claim 13,

wherein the RKey includes a hash field to facilitate validation of the RKey;

and

wherein the remote nodes validate the RKey that is received using the hash field in the RKey.

15. The method of claim 13,

wherein the remote node includes a memory;

and

wherein the remote node maintains a key table that maps any received RKey to physical addresses in the memory.

16. A method of transferring data between nodes in a cluster, the method comprising:

receiving and decoding by a local node a command for a data transfer;

determining from the decoded command that the data transfer has a source location in a first remote node and a destination location in a second remote node, each remote node being distinct from each other and the local node and being coupled to the local node by one or more point-to-point communication links;

obtaining a first and a second certificate granting rights to transfer data between the first and second remote nodes,

transferring the first and second certificates from the local node to the first and second remote nodes, wherein the first and second certificates are successfully validated, respectively, in the first and second remote nodes;

transferring data from the first remote node to the second remote node using the certificates;

receiving an acknowledgement from the second remote node that the transfer is complete; and

updating a completion status indicating the command was successful.

17. The system of claim 16,

wherein the second remote node is a destination node containing a persistent address space;

and

wherein the acknowledgment from the second node indicates that the transferred data resides in the persistent address space.

18. The system of claim 16, wherein each certificate is an RKey owned by the remote node;

19. The system of claim 18,

wherein the RKey includes a hash field to facilitate validation of the RKey;

and

wherein the remote nodes validate the RKey that is received using the hash field in the RKey.

20. The method of claim 18,

wherein each of the remote nodes includes a memory;

and

wherein each of the remote nodes maintains a key table that maps any received RKey to physical addresses in the memories of the respective remote nodes.