Memory bus protocol to enable clustering between nodes of distinct physical domain address spaces
First Claim
1. A computer system comprising:
- a plurality of computing nodes, wherein each computing node of the plurality of computing nodes includes one or more coprocessors, one or more central processing units (CPUs), and a memory directly coupled to the one or more coprocessors,wherein each memory of said each computing node resides in a separate and distinct physical domain;
a plurality of point-to-point physical communications links coupling the plurality of computing nodes,wherein the one or more coprocessors of said each computing node interface directly with the one or more communications links to transfer data over the plurality of point-to-point physical communications link without involving the one or more CPUs of said each computing node,wherein a particular coprocessor of a particular node is configured to transfer data between the memory coupled to the particular coprocessor at the particular node and a first remote memory of a first remote node over the one or more communications links, without involving the one or more CPUs, using a first certificate that grants access to a portion of the memory in the first remote node,wherein the particular node is a destination node for the data transfer, the destination node including a persistent address space,wherein the destination node transmits an acknowledgment indicating that the transferred data resides in the persistent address space.
1 Assignment
0 Petitions
Accused Products
Abstract
A system and method for transferring data and messages between nodes in a cluster is disclosed. Each node in the cluster is a separate physical domain but is connected to other nodes in the cluster through point-to-point high speed links. Each side of a link is coupled to a coprocessor which facilitates the movement of data between and among the nodes. Because each physical domain is separate from any other domain, the coprocessor in a physical domain uses a certificate, called and RKey, to obtain permission to transfer data to another physical domain. When an RKey is received from another physical domain, the coprocessor in the receiving domain validates the key and obtains the physical addresses associated with the key so that it can provide or accept the remote data. Data transfers between pairs of remote nodes in the cluster are permitted as well.
-
Citations
17 Claims
-
1. A computer system comprising:
-
a plurality of computing nodes, wherein each computing node of the plurality of computing nodes includes one or more coprocessors, one or more central processing units (CPUs), and a memory directly coupled to the one or more coprocessors, wherein each memory of said each computing node resides in a separate and distinct physical domain; a plurality of point-to-point physical communications links coupling the plurality of computing nodes, wherein the one or more coprocessors of said each computing node interface directly with the one or more communications links to transfer data over the plurality of point-to-point physical communications link without involving the one or more CPUs of said each computing node, wherein a particular coprocessor of a particular node is configured to transfer data between the memory coupled to the particular coprocessor at the particular node and a first remote memory of a first remote node over the one or more communications links, without involving the one or more CPUs, using a first certificate that grants access to a portion of the memory in the first remote node, wherein the particular node is a destination node for the data transfer, the destination node including a persistent address space, wherein the destination node transmits an acknowledgment indicating that the transferred data resides in the persistent address space. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A method of transferring data between nodes in a cluster, the method comprising:
-
receiving and decoding by a coprocessor of a local node a command for a data transfer without involving a CPU of the local node; determining from the decoded command that the data transfer has a source location in the local node and a destination location in a remote node distinct from the local node, the local node and the remote node being coupled to each other by one or more point-to-point communication links, wherein the coprocessor and a remote coprocessor of the remote node each interface directly with the one or more point-to-point communication links, wherein the remote node is a destination node that includes a persistent address space; obtaining a certificate granting rights to transfer data to the remote node, transferring the certificate from the local node to the remote node, wherein the certificate is successfully validated in the remote node by the remote coprocessor without involving a CPU of the remote node; transferring data from the local node to the remote node, after successful hardware authentication of the certificate by the remote coprocessor, without involving a CPU of the remote node or a CPU or the local node; receiving an acknowledgement from the remote node that indicates that the transferred data resides in the persistent address space; and updating a completion status indicating the command was successful. - View Dependent Claims (11, 12, 13)
-
-
14. A method of transferring data between nodes in a cluster, the method comprising:
-
receiving and decoding by coprocessor of a local node a command for a data transfer without involving a corresponding CPU of the local node; determining from the decoded command that the data transfer has a source location in a first remote node and a destination location in a second remote node that is destination node containing a persistent address space, each remote node being distinct from each other and the local node and being coupled to the local node by one or more point-to-point communication links; wherein the coprocessor of the local node and a coprocessor of the first remote node interface directly with the one or more point-to-point communication links, wherein the command for the data transfer is received over the one or more point-to-point communication links from the coprocessor of the first remote node; obtaining a first certificate and a second certificate granting rights to transfer data between the first remote node and second remote node, transferring the first and second certificates from the local node to the first and second remote nodes, wherein hardware authentication of the first certificate is performed by a first coprocessor of the first remote node without involving a corresponding CPU of the first remote node, wherein hardware authentication of the second certificate is performed by a second coprocessor of the second remote node without involving a corresponding CPU of the second remote node; after successful hardware authentication, transferring data from the first remote node to the second remote node without involving a corresponding CPU of the local node; receiving an acknowledgement from the second remote node that indicates that the transferred data resides in the persistent address space; and updating a completion status indicating the command was successful. - View Dependent Claims (15, 16, 17)
-
Specification