System and method for performing a cluster topology self-healing process in a distributed data system cluster
First Claim
1. A method, comprising:
- detecting a failed node within a cluster comprising a first node, the failed node, and a second node, wherein the failed node stores a data set;
in response to said detecting, performing a cluster topology self-healing process to copy the data set from the first node to the second node, wherein the data set is dividable into a plurality of domains, and wherein the cluster topology self-healing process includes;
locking one of the plurality of domains on the second node, wherein said locking does not lock any other one of the plurality of domains on the second node;
storing data included in the one of the plurality of domains sent from the first node to the second node;
releasing the one of the plurality of domains; and
repeating said locking, said copying, and said releasing for each other one of the plurality of domains.
2 Assignments
0 Petitions
Accused Products
Abstract
A cluster topology self-healing process is performed in order to replicate a data set stored on a failed node from a first node storing another copy of the data set to a second non-failed node. The self-healing process is performed by: locking one of several domains included in the data set, where locking that domain does not lock any of the other domains in the data set; storing data sent from the first node to the second node in the domain; and releasing the domain. This process of locking, storing, and releasing is repeated for each other domain in the data set. Each domain may be locked for significantly less time than it takes to copy the entire data set. Accordingly, client access requests targeting a locked domain will be delayed for less time than if the entire data set is locked during the self-healing process.
197 Citations
25 Claims
-
1. A method, comprising:
-
detecting a failed node within a cluster comprising a first node, the failed node, and a second node, wherein the failed node stores a data set;
in response to said detecting, performing a cluster topology self-healing process to copy the data set from the first node to the second node, wherein the data set is dividable into a plurality of domains, and wherein the cluster topology self-healing process includes;
locking one of the plurality of domains on the second node, wherein said locking does not lock any other one of the plurality of domains on the second node;
storing data included in the one of the plurality of domains sent from the first node to the second node;
releasing the one of the plurality of domains; and
repeating said locking, said copying, and said releasing for each other one of the plurality of domains. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A distributed data cluster comprising:
-
a plurality of nodes;
an interconnect coupling the plurality of nodes;
wherein a first node included in the plurality of nodes is configured to detect a failure of a second node included in the plurality of nodes, wherein the first node stores a copy of a data set stored by the second node;
wherein in response to detecting the failure of the second node, the first node is configured to perform a copy operation for the data set with a third node included in the plurality of nodes over the interconnect;
wherein the third node is configured to perform the copy operation by repeatedly;
locking a subset of the data set on the third node, storing a copy of the locked subset received from the first node, and releasing the locked subset for each subset of a plurality of subsets included in the data set;
wherein the third node is configured to lock the subset of the data without locking any other one of the plurality of subsets. - View Dependent Claims (11, 12, 13, 14, 15, 16)
-
-
17. A distributed data system, comprising:
-
a distributed data system cluster comprising a plurality of nodes, wherein the a distributed data system cluster stores a plurality of data sets, and wherein each data set is replicated on at least two of the plurality of nodes; and
a client node coupled to the plurality of nodes by a network, wherein the client node is configured to send a write access request targeting a first data set of the plurality of data sets to the cluster via the network;
wherein in response to detecting a failure of a first node of the plurality of nodes storing the first data set of the plurality of data sets, the distributed data system cluster is configured to perform a cluster topology self-healing process to copy the first data set from a second node to a third node;
wherein the distributed data system cluster is configured to perform the cluster topology self-healing process by;
locking a subset of a plurality of subsets included in the first data set on the third node, wherein the distributed data system cluster is configured to perform said locking without locking any other subset of the plurality of subsets;
copying data included in the subset of the first one of the data sets from the second node to the third node;
releasing the subset of the first one of the data sets; and
repeating said locking, said copying, and said releasing for each other subset included in the first data set;
wherein if the distributed data system cluster receives the client write access request during performance of the cluster topology self-healing process, the distributed data system cluster is configured to respond to the client write access request by modifying a first subset of the first data set targeted by the client write access request if the first subset of the first data set is not locked for performance of the cluster topology self-healing process. - View Dependent Claims (18, 19, 20, 21, 22, 23)
-
-
24. A device for use in a distributed data system cluster, the device comprising:
-
a communication interface configured to send and receive communications from one or more other nodes, wherein the communication interface is configured to detect a failed node within the distributed data system cluster;
a data store coupled to the communication interface and configured to store data; and
a replication topology manager coupled to the communication interface and configured to participate in a copy operation involving a data set with another node in response to the communication interface detecting the failed node, wherein a copy of the data set is stored on the failed node;
wherein the replication topology manager is configured to participate in the copy operation by;
locking a first subset of the data set in the data store, wherein the replication topology manager is configured to acquire a lock on the first subset of the data set without acquiring a lock on any other subset of the data set;
performing a copy operation for the first subset of the data set;
releasing the first subset of the data set; and
repeating said locking, said performing, and said releasing for each other subset of the data set.
-
-
25. A system, comprising:
-
means for detecting a failed node within a cluster comprising a first node, the failed node, and a second node, wherein the failed node stores a data set;
means for performing a cluster topology self-healing process to copy the data set from the first node to the second node in response to detecting the failed node, wherein the data set is dividable into a plurality of domains, and wherein the cluster topology self-healing process includes;
locking one of the plurality of domains, wherein said locking does not lock any other one of the plurality of domains; and
storing data included in the one of the plurality of domains sent from the first node to the second node; and
releasing the one of the plurality of domains; and
repeating said locking, said copying, and said releasing for each other one of the plurality of domains.
-
Specification