System and method for reforming a distributed data system cluster after temporary node failures or restarts

• US 7,206,836 B2
• Filed: 09/23/2002
• Issued: 04/17/2007
• Est. Priority Date: 09/23/2002
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Kannan, Mahesh, Dinker, Darpan, Gopinath, Pramod
• Primary Examiner(s): Pwu; Jeffrey
• Assistant Examiner(s): Nguyen; Phuoc H
• Application Number:
  US10/252,232
• Publication Number:
  US 20040059805A1
• Time in Patent Office: 1,667 Days
• Field of Search: 709/214,220,224,226
• US Class Current: 709/224
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover

Abstract

Data stored within a cluster may be distributed among nodes each storing a portion of the data. The data may be replicated wherein different nodes store copies of the same portion of the data. In response to detecting the failure of a node, the cluster may initiate a timeout period. If the node remains failed throughout the timeout period, the cluster may copy the portion of the data stored on the failed node onto one or more other nodes of the cluster. If the node returns to the cluster during the timeout period, the cluster may maintain the copy of the data on the previously failed node without copying the portion of the data stored on the failed node onto any other nodes. By delaying self-healing of the cluster for the timeout period, an unbalanced data distribution may be avoided in cases where a failed node quickly rejoins the cluster.

Claims

1. A distributed data system cluster, comprising:

a plurality of nodes configured to store data as a distributed data cluster, wherein data stored on a first portion of the nodes is replicated on a second portion of the nodes;

an interconnect coupling the plurality of nodes;

wherein in response to detecting a failure of a first node included in the plurality of nodes, a second node included in the plurality of nodes that stores a copy of a data set stored on the first node is configured to;

determine whether the first node returns to the cluster during a timeout period initiated after the failure is detected; and

perform a copy operation for the data set over the interconnect with an other node included in the plurality of nodes if the first node fails to return to the cluster during a timeout period; and

refrain from performing the copy operation for the data set if the first node returns to the cluster during the timeout period.

2. The distributed data system cluster of claim 1, wherein the second node is configured to detect the failure of the first node if the first node fails to respond to a communication within a predetermined time period.

3. The distributed data system cluster of claim 1, wherein the second node is configured to verify the failure of the first node during the timeout period.

4. The distributed data system cluster of claim 1, wherein the timeout period has a duration at least equal to a duration of time required to perform a warm restart of the first node.

5. The distributed data system cluster of claim 1, wherein if the second node is a primary node for the data set, the second node is configured to perform the copy operation by writing the data set to the other node.

6. A method, comprising:

detecting a failure of a first node of a plurality of nodes included in a distributed data system cluster;

in response to said detecting, initiating a timeout period;

if the timeout period expires before the first node returns to the distributed data system cluster, replicating a portion of the data stored on the first node onto a second node included in the plurality of nodes; and

if the first node returns to the cluster during the timeout period, maintaining the portion of the data on the first node instead of replicating the portion of the data onto the second node.

7. The method of claim 6, wherein said detecting the failure comprises sending a communication to the first node and not receiving a response to the communication from the first node.

8. The method of claim 6, further comprising verifying the failure of the first node during the timeout period, wherein said verifying comprises sending one or more communications to the first node during the timeout period.

9. The method of claim 6, wherein the timeout period has a duration at least equal to a duration of time required to perform a warm restart of the first node.

10. The method of claim 6, wherein said replicating comprises copying the portion of the data from one or more other nodes storing the portion of the data onto the second node.

11. The method of claim 6, wherein the first node stores a plurality of data sets, and wherein the method comprises copying the plurality of data sets onto one or more of the plurality of nodes.

12. The method of claim 6, further comprising configuring the second node as a new backup node for the portion of the data if the first node is configured as a backup node for the portion of the data;

wherein said replicating comprises copying the data set from a master node for the portion of the data to the new backup node.

13. The method of claim 6, further comprising a third node included in the plurality of nodes responding to client requests targeting the portion of the data during the timeout period;

wherein prior to said detecting the failure, the first node was responsible for responding to client requests targeting the portion of the data.

14. The method of claim 13, further comprising the first node regaining a responsibility to respond to client requests targeting the data if the first node returns to the distributed data system cluster before expiration of the timeout period.

15. The method of claim 13, further comprising the third node losing a responsibility to respond to client requests targeting the data set if the first node returns to the distributed data system cluster before expiration of the timeout period.

16. The method of claim 6, further comprising verifying the data stored on the first node if the first node returns to the distributed data system cluster before expiration of the timeout period.

17. The method of claim 6, further comprising updating the data stored on the first node if the first node returns to the distributed data system cluster before expiration of the timeout period if another node'"'"'s copy of the data was modified after said detecting the failure and before the first node returns to the distributed data system cluster.

18. A distributed data system, comprising:

a cluster comprising a plurality of nodes, wherein the cluster is configured to store data, and wherein the data is replicated within the cluster;

a client node configured to request access to the data stored on the cluster; and

an interconnect coupling the client node to the cluster;

wherein the cluster is configured to detect a failure of a first node included in the plurality of nodes and to initiate a timeout period in response to detecting the failure of the first node;

wherein if the first node remains failed throughout the timeout period, the cluster is configured to copy a portion of the data stored on the failed node onto one or more other nodes included in the cluster; and

wherein if the first node returns to the cluster during the timeout period, the cluster is configured to maintain the portion of the data on the first node instead of copying the portion of the data onto the one or more other nodes.

19. The distributed data system of claim 18, wherein the cluster is configured to detect the failure of the first node if the first node fails to respond to a communication sent to the first node.

20. The distributed data system of claim 18, wherein the cluster is configured to verify the failure of the first node during the timeout period by sending one or more communications to the first node during the timeout period.

21. The distributed data system of claim 18, wherein the timeout period has a duration at least equal to a duration of time required to perform a warm restart of the first node.

22. The distributed data system of claim 18, wherein the cluster is configured to copy the portion of the data stored on the first node by copying the portion of the data from a plurality of nodes storing the portion of the data onto another plurality of nodes included in the cluster.

23. The distributed data system of claim 18, wherein the portion of the data stored on the first node comprises a plurality of data sets, and wherein the cluster is configured to copy each of the plurality of data sets onto a different one of the one or more other nodes included in the cluster.

24. The distributed data system of claim 18, wherein if the first node is configured as a backup node for the portion of the data stored on the first node, the cluster is configured to:

configure another node included in the cluster as a new backup node for the portion of the data; and

copy the portion of the data from a master node to the new backup node.

25. The distributed data system of claim 18, wherein the cluster is configured to assign a responsibility for responding to client access requests targeting the portion of the data stored on the first node to an other node during the timeout period;

wherein prior to the failure, the first node was responsible for responding to client access requests targeting the portion of the data.

26. The distributed data system of claim 18, wherein the cluster is configured to verify the portion of the data stored on the first node if the first node returns to the cluster before expiration of the timeout period.

27. A device for use in a distributed data system cluster, the device comprising:

a communication interface configured to send and receive communications from one or more other nodes, wherein the communication interface is configured to detect a failed node within the distributed data system cluster and to detect whether the failed node returns to the distributed data system cluster;

a data store coupled to the communication interface and configured to store data; and

a replication topology manager coupled to the communication interface and configured to initiate a timeout period in response to the communication interface detecting the failed node;

wherein if the timeout period expires before the failed node returns to the distributed data system cluster, the replication topology manager is configured to initiate a copy operation for a portion of the data stored in the data store;

wherein if the communication interface detects that the failed node has returned to the distributed data system cluster before expiration of the timeout period, the replication topology manager is configured to not initiate the copy operation.

28. The device of claim 27, wherein in response to the replication topology manager initiating the copy operation, the communication interface is configured to send a read request for the portion of the data to another node within the distributed data system cluster, wherein the communication interface is configured to provide a copy of the portion of the data received in response to the read request to the data store, and wherein the data store is configured to store the copy of the portion of the data.

29. The device of claim 27, wherein in response to the replication topology manager initiating the copy operation, the data store is configured to provide a copy of the portion of the data to the communication interface and the communication interface is configured to send a write request and the copy of the portion of the data to another node within the distributed data system cluster.

30. The device of claim 27, wherein the replication topology manager includes a countdown timer, wherein the replication topology manager is configured to initialize the countdown timer to a timeout value corresponding to a length of the timeout period in response to the communication interface detecting the failed node.

31. A system, comprising:

means for detecting a failure of a first node of a plurality of nodes included in a distributed data system cluster, wherein data is distributed among the plurality of nodes;

means for initiating a timeout period;

means for replicating a portion of the data stored on the first node onto a second node included in the plurality of nodes if the timeout period expires after the failure of the first node is detected and before the first node returns to the distributed data system cluster;

wherein if the first node returns to the cluster during the timeout period, the means for replicating the portion of the data maintain the portion of the data on the first node instead of replicating the portion of the data onto the second node.