Handling failed transaction peers in a distributed hash table

• US 9,141,480 B2
• Filed: 09/24/2013
• Issued: 09/22/2015
• Est. Priority Date: 10/17/2012
• Status: Active Grant

• Current Assignee: DataDirect Networks Incorporated
• Original Assignee: DataDirect Networks Incorporated
• Inventors: Nowoczynski, Paul, Cope, Jason Micah, Uppu, Pavan, Molaro, Don, Piszczek, Michael, Manning, Gordon
• Primary Examiner(s): BRYAN, JASON B
• Application Number:
  US14/035,913
• Publication Number:
  US 20140108863A1
• Time in Patent Office: 728 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/141 for bus or memory accesses
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31

Abstract

Data storage systems and methods for storing data are described herein. The storage system includes at least two data storage nodes for storing portions of a distributed hash table and related data. After a first node attempts to complete a write request at a second node and is unable to complete the request, the first node ceases responses to interactions from other nodes. Once the first node'"'"'s failure to respond has caused a sufficient number of nodes to cease responding, the nodes enter a service mode to resolve the live lock. While in live lock, the nodes determine the oldest, unfulfilled request using a system-wide logical timestamp associated with write requests. Once the oldest request is determined, a removal vote to remove the non-responsive node from the group is initiated and, if other nodes agree, the non-responsive node is removed from the group of nodes.

Claims

1. A data storage method comprising:

a first storage node issuing a first request to a second storage node, the first request including a logical timestamp indicating the time, relative to other requests, that the first request was issued;

the first storage node exhausting a first pool of non-committed requests before the second storage node acknowledges the first request;

the first storage node halting any responses to requests issued by other storage nodes until the second storage node acknowledges the first request;

at least one other storage node issuing a second request to the first storage node and receiving no response;

the at least one other storage node exhausting a second pool of non-committed requests before the first storage node acknowledges the second request;

the at least one other storage node halting any responses to requests issued by other storage nodes until the first storage node acknowledges the second request;

the first storage node and the at least one other storage node, together, using the logical timestamp to determine that the second storage node failure to acknowledge the first request is a basis for the first storage node and the at least one other storage node halting responses to requests; and

the first storage node and the at least one other storage node entering a service mode to resolve the failure of the second storage node to acknowledge the first request.

2. The method of claim 1 wherein the service mode comprises:

the at least one other storage node evaluating the second pool of non-committed requests to determine that a second logical timestamp associated with the second request is oldest;

the first storage node evaluating the first pool of non-committed requests to determine that the logical timestamp associated with the first request is oldest; and

the first storage node and the at least one other storage node together determining that the second storage node is not responding and removing the second storage node from a group of available storage nodes.

3. The method of claim 2 wherein the together determining further comprises:

the first storage node requesting a removal vote from the at least one other storage node with regard to the second storage node, the removal vote being a vote to remove the second storage node from the group of available storage nodes;

the at least one other storage node determining that the second logical timestamp associated with the second request is oldest; and

the at least one other storage node agreeing with the first storage node to the removal vote.

4. The method of claim 3 wherein a total number of storage nodes casting the removal vote must be greater in number than a number of nodes necessary to continue operation of the group of available storage nodes.

5. The method of claim 1 further comprising:

the first storage node identifying a data object stored on the second storage node and identifying a third storage node, the third storage node ranked after the second storage node in a ranking associating the data object with at least the second storage node and the third storage node; and

the first storage node duplicating the data object to the third storage node.

6. The method of claim 5 wherein a ranking of the first storage node and a ranking of the at least one other storage node, along with the rankings of the second storage node and the third storage node do not change when the second storage node is removed from a group of available storage nodes.

7. The method of claim 1 wherein the logical timestamp uses a monotonic clock to stamp all requests across a group of available storage nodes.

8. The method of claim 1 wherein the logical timestamp may be used to resolve conflicts in which two or more of a group of available storage nodes attempt to access to a data object by relying upon a most recent logical timestamp to resolve any potential conflict.

9. The method of claim 1 wherein the first and second requests are re-requested to one of a group of available storage nodes after the first storage node and the at least one other storage node exit the service mode.

10. A distributed data storage system comprising:

a first storage node having a processor, a memory, and a non-volatile storage, the first storage node configured to perform actions including;

issuing a first request to a second storage node, the first request including a logical timestamp indicating the time, relative to other requests, that the first request was issued,

exhausting a first pool of non-committed requests before the second storage node acknowledges the first request, and

halting any responses to requests issued by other storage nodes until the second storage node acknowledges the first request;

at least one other storage node having a processor, a memory, and a non-volatile storage, the at least on other storage node configured to perform actions including;

issuing a second request to the first storage node and receiving no response,

exhausting a second pool of non-committed requests before the first storage node acknowledges the second request,

halting any responses to requests issued by other storage nodes until the first storage node acknowledges the second request; and

wherein the first storage node and the at least one other storage node, together

use the logical timestamp to determine that the second storage node failure to acknowledge the first request is the basis for the first storage node and the at least one other storage node halting responses to requests, and

enter a service mode to resolve the failure of the second storage node to acknowledge the request.

11. The distributed data storage system of claim 10 wherein:

the at least one other storage node evaluating the second pool of non-committed requests to determine that a second logical timestamp associated with the second request is oldest;

the first storage node evaluating the first pool of non-committed requests to determine that the logical timestamp associated with the first request is oldest; and

the first storage node and the at least one other storage node together determining that the second storage node is not responding and removing the second storage node from a group of available storage nodes.

12. The distributed data storage system of claim 11 wherein the first storage node and the at least one other storage node are further configured to perform actions including:

requesting a removal vote with regard to the second storage node, the removal vote being a vote to remove the second storage node from the group of available storage nodes;

determining that the second logical timestamp associated with the second request is oldest; and

agreeing to the removal vote.

13. The distributed data storage system of claim 12 wherein a total number of storage nodes casting the removal vote must be greater in number than a number of nodes necessary to continue operation of the group of available storage nodes.

14. The distributed data storage system of claim 10 wherein the first storage node is further configured to perform actions including:

identifying a data set stored on the second storage node and identifying a third storage node, the third storage node ranked after the second storage node in a ranking associating the data set with at least the second storage node and the third storage node; and

duplicating the data object to the third storage node.

15. The distributed data storage system of claim 14 wherein a ranking of the first storage node and a ranking of the at least one other storage node, along with the rankings of the second storage node and the third storage node do not change when the second storage node is removed from a group of available storage nodes.

16. The distributed data storage system of claim 10 wherein the logical timestamp uses a monotonic clock to stamp all requests across a group of available storage nodes.

17. The distributed data storage system of claim 10 wherein the logical timestamp may be used to resolve conflicts in which two or more of a group of available storage nodes attempt to access a data object by relying upon a most recent logical timestamp to resolve any potential conflict.

18. The distributed data storage system of claim 10 wherein the first and second requests are re-requested to one of a group of available storage nodes after the first storage node and the at least one other storage node exit the service mode.