Memory bus protocol to enable clustering between nodes of distinct physical domain address spaces

• US 9,400,821 B2
• Filed: 03/15/2013
• Issued: 07/26/2016
• Est. Priority Date: 10/02/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Kapil, Sanjiv, Radovic, Zoran
• Primary Examiner(s): Homayounmehr, Farid
• Assistant Examiner(s): Sarker, Sanchit
• Application Number:
  US13/838,542
• Publication Number:
  US 20140095651A1
• Time in Patent Office: 1,229 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/082 Associative directories G06F12/0822 takes precedence
• CPC Class Codes: G06F 12/12 Replacement control
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 12/1475 in a virtual system, e.g. with translation means
• CPC Class Codes: G06F 13/102 where the programme performs an interfacing function, e.g. device driver G06F13/105 takes precedence; scheduling within device drivers G06F9/52; contention policies within device drivers G06F9/4881
• CPC Class Codes: G06F 13/362 with centralised access control
• CPC Class Codes: G06F 13/4068 Electrical coupling
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 15/7889 Reconfigurable logic implemented as a co-processor instruction execution using a coprocessor G06F9/3877
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/24569 Query processing with adaptation to specific hardware, e.g. adapted for using GPUs or SSDs
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 21/6209 : to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 9/30065 : Loop control instructions; iterative instructions, e.g. LOOP, REPEAT
• CPC Class Codes: G06F 9/381 : Loop buffering
• CPC Class Codes: G06F 9/3851 : from multiple instruction streams, e.g. multistreaming
• CPC Class Codes: G06F 9/3867 : using instruction pipelines
• CPC Class Codes: G06F 9/546 : Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 69/14 : Multichannel or multilink protocols
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A system and method for transferring data and messages between nodes in a cluster is disclosed. Each node in the cluster is a separate physical domain but is connected to other nodes in the cluster through point-to-point high speed links. Each side of a link is coupled to a coprocessor which facilitates the movement of data between and among the nodes. Because each physical domain is separate from any other domain, the coprocessor in a physical domain uses a certificate, called and RKey, to obtain permission to transfer data to another physical domain. When an RKey is received from another physical domain, the coprocessor in the receiving domain validates the key and obtains the physical addresses associated with the key so that it can provide or accept the remote data. Data transfers between pairs of remote nodes in the cluster are permitted as well.

Claims

1. A computer system comprising:

a plurality of computing nodes, wherein each computing node of the plurality of computing nodes includes one or more coprocessors, one or more central processing units (CPUs), and a memory directly coupled to the one or more coprocessors,

wherein each memory of said each computing node resides in a separate and distinct physical domain;

a plurality of point-to-point physical communications links coupling the plurality of computing nodes,

wherein the one or more coprocessors of said each computing node interface directly with the one or more communications links to transfer data over the plurality of point-to-point physical communications link without involving the one or more CPUs of said each computing node,

wherein a particular coprocessor of a particular node is configured to transfer data between the memory coupled to the particular coprocessor at the particular node and a first remote memory of a first remote node over the one or more communications links, without involving the one or more CPUs, using a first certificate that grants access to a portion of the memory in the first remote node,

wherein the particular node is a destination node for the data transfer, the destination node including a persistent address space,

wherein the destination node transmits an acknowledgment indicating that the transferred data resides in the persistent address space.

2. The system of claim 1, wherein the particular coprocessor further includes a data transfer machine that moves data from one memory location to another memory location.

3. The system of claim 1, wherein each certificate is a remote key (RKey) owned by a node of the plurality of computing nodes.

4. The system of claim 3,

wherein each node of the plurality of computing nodes has a software hypervisor running on said each node;

and

wherein the one or more coprocessors of said each node operates under control of the software hypervisor.

5. The system of claim 4, wherein each software hypervisor of the one or more coprocessors of said each node maintains a key table in the memory of said each node that maps RKeys to physical addresses of the memory of said each node.

6. The system of claim 3,

wherein each RKey includes a hash field for validating the RKey;

and

wherein the coprocessors of the plurality of computing nodes validate any RKey that is received using the hash field in the RKey.

7. The system of claim 3, wherein each RKey expires after a specified time, an expired RKey revoking access rights that the RKey granted.

8. The system of claim 1,

wherein each node of the plurality of computing nodes has a software hypervisor running on said each node;

wherein the one or more coprocessors in each node transfers data in response to a command issued by the software hypervisor; and

wherein the command is part of a control block that includes

a control word that includes a size of the data transfer, and

a completion address that specifies a location at which completion data of a transfer is to be posted in the memory coupled to the one or more coprocessors.

9. The system of claim 3, wherein the first remote node is further configured to invalidate its own RKey table, causing validation of the first certificate to fail.

10. A method of transferring data between nodes in a cluster, the method comprising:

receiving and decoding by a coprocessor of a local node a command for a data transfer without involving a CPU of the local node;

determining from the decoded command that the data transfer has a source location in the local node and a destination location in a remote node distinct from the local node, the local node and the remote node being coupled to each other by one or more point-to-point communication links, wherein the coprocessor and a remote coprocessor of the remote node each interface directly with the one or more point-to-point communication links, wherein the remote node is a destination node that includes a persistent address space;

obtaining a certificate granting rights to transfer data to the remote node,

transferring the certificate from the local node to the remote node,

wherein the certificate is successfully validated in the remote node by the remote coprocessor without involving a CPU of the remote node;

transferring data from the local node to the remote node, after successful hardware authentication of the certificate by the remote coprocessor, without involving a CPU of the remote node or a CPU or the local node;

receiving an acknowledgement from the remote node that indicates that the transferred data resides in the persistent address space; and

updating a completion status indicating the command was successful.

11. The method of claim 10, wherein the certificate is an RKey owned by the remote node.

12. The method of claim 11,

wherein the RKey includes a hash field to facilitate validation of the RKey;

and

wherein the remote nodes validate the RKey that is received using the hash field in the RKey.

13. The method of claim 11,

wherein the remote node includes a memory;

and

wherein the remote node maintains a key table that maps any received RKey to physical addresses in the memory.

14. A method of transferring data between nodes in a cluster, the method comprising:

receiving and decoding by coprocessor of a local node a command for a data transfer without involving a corresponding CPU of the local node;

determining from the decoded command that the data transfer has a source location in a first remote node and a destination location in a second remote node that is destination node containing a persistent address space, each remote node being distinct from each other and the local node and being coupled to the local node by one or more point-to-point communication links;

wherein the coprocessor of the local node and a coprocessor of the first remote node interface directly with the one or more point-to-point communication links, wherein the command for the data transfer is received over the one or more point-to-point communication links from the coprocessor of the first remote node;

obtaining a first certificate and a second certificate granting rights to transfer data between the first remote node and second remote node,

transferring the first and second certificates from the local node to the first and second remote nodes, wherein hardware authentication of the first certificate is performed by a first coprocessor of the first remote node without involving a corresponding CPU of the first remote node, wherein hardware authentication of the second certificate is performed by a second coprocessor of the second remote node without involving a corresponding CPU of the second remote node;

after successful hardware authentication, transferring data from the first remote node to the second remote node without involving a corresponding CPU of the local node;

receiving an acknowledgement from the second remote node that indicates that the transferred data resides in the persistent address space; and

updating a completion status indicating the command was successful.

15. The method of claim 14, wherein the first certificate is a first RKey owned by the first remote node and the second certificate is a second RKey owned by the second remote node.

16. The method of claim 15,

wherein the first RKey and the second RKey each includes a hash field to facilitate validation of the RKey;

and

wherein the first RKey and the second RKey are validated using the hash field in the RKey.

17. The method of claim 15,

wherein each of the remote nodes includes a memory;

and

wherein each of the remote nodes maintains a key table that maps any received RKey to physical addresses in the memories of the respective remote nodes.