System and method for detecting problematic data storage nodes

• US 9,329,955 B2
• Filed: 06/28/2011
• Issued: 05/03/2016
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: Compuverde AB (Pure Storage, Inc.)
• Original Assignee: Compuverde AB (Pure Storage, Inc.)
• Inventors: Bernbo, Stefan, Melander, Christian, Persson, Roger, Petersson, Gustav
• Primary Examiner(s): Chbouki, Tarek
• Application Number:
  US13/170,672
• Publication Number:
  US 20120023179A1
• Time in Patent Office: 1,771 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A method for maintaining a data storage system is disclosed. The method may include monitoring for receipt of a first broadcast message from a first data storage node, where the first broadcast message may indicate that the first data storage node is operating correctly. The method may also include detecting that the first data storage node is malfunctioning based on not receiving the first broadcast message for a predetermined period of time. The method may also include initiating a data replication procedure based on detecting that the first data storage node is malfunctioning. The data replication procedure may include sending a first multicast message to a plurality of data storage nodes requesting identification of a second data storage node that maintains a copy of a file stored on the first data storage node.

Claims

1. A method implemented on a first data storage node for maintaining a data storage system, the method comprising:

the first data storage node monitoring for receipt of instances of a broadcast message from each of a plurality of data storage nodes included in a storage group, wherein a local node list stored at the first data storage node identifies the plurality of data storage nodes as being included in the storage group, the plurality of data storage nodes identified in the local node list includes at least a second data storage node, and instances of the broadcast message include an indication of a node list version being used by the data storage node that transmitted the instance of the broadcast message such that receipt of a given instance of the broadcast message by the first storage node indicates to the first storage node both that the data storage node that sent the given instance of the broadcast message is operating correctly and whether the storage group has been updated to add or remove a data storage node;

the first data storage node receiving a first instance of the broadcast message from the second data storage node;

the first data storage node updating the local node list based on the first instance of the broadcast message indicating that the second data storage node is using a different node list version than the version of the local node list stored at the first data storage node, wherein the first data storage adds at least a third data storage node to the local node list and begins monitoring for receipt of instances of the broadcast message from the third data storage node in response to the first instance of the broadcast message indicating the second data storage node is using the different node list version;

the first data storage node detecting that the second data storage node is malfunctioning based on failing to receive a subsequent instance of the broadcast message from the second data storage node for a predetermined period of time; and

the first data storage node initiating a data replication procedure based on detecting that the second data storage node is malfunctioning, wherein a file stored on the first data storage node is to be replicated in the data replication procedure, the file is associated with a host list stored on the first data storage node, the host list indicates a subset of data storage nodes in the storage group that also store the file, and the host list indicates that the second data storage node is in the subset of data storage nodes.

2. The method of claim 1, wherein the first data storage removes at least a fourth data storage node from the local node list and stops monitoring for receipt of instances of the broadcast message from the fourth data storage node in response to the first instance of the broadcast message indicating the second data storage node is using the different node list version.

3. The method of claim 1, wherein the data replication procedure further comprises:

the first data storage node maintaining a local copy of the file;

the first data storage node sending a first multicast message to a plurality of data storage nodes indicated in the host list requesting identification of a fourth data storage node that maintains a copy of the file; and

the first data storage node sending a second multicast message to the plurality of data storage nodes requesting to become a master node for the file, wherein the master node controls the replication and maintenance of the file within the data storage system.

4. The method of claim 3, wherein the data replication procedure further comprises:

the first data storage node receiving a response to the first multicast message, wherein the response includes at least one of a version identification for the copy of the file or a list of data storage nodes that maintain a copy of the file; and

the first data storage node updating a master host list for the file based on the response.

5. The method of claim 4, wherein the data replication procedure further comprises the first data storage node sending a message to the third data storage node instructing the fourth data storage node to save a copy of the file.

6. The method of claim 1, wherein the first data storage node updating its node list comprises:

the first data storage node sending a message to the second data storage node that requests the second data storage node sends a version of the node list stored by the second data storage node to the first data storage node; and

receiving the version of the node list stored by the second data storage node from the second data storage node, wherein the version of the node list stored by the second data storage node includes the third data storage node.

7. The method of claim 1, wherein the first instance of the broadcast message from the second data storage node comprises an address for the second data storage node, and the broadcast message is sent as a multicast message.

8. A system for maintaining data in a network, the system comprising a plurality of data storage nodes included in a storage group, the plurality of data storage nodes comprising:

a first data storage node configured to monitor for receipt of instances of a broadcast message from each of the plurality of data storage nodes included in a storage group, wherein a local node list stored at the first data storage node identifies the plurality of data storage nodes as being included in the storage group, the plurality of data storage nodes identified in the local node list includes at least a second data storage node, and instances of the broadcast message include an indication of a node list version being used by the data storage node that transmitted the instance of the broadcast message such that receipt of a given instance of the broadcast message by the first storage node indicates to the first storage node both that the data storage node that sent the given instance of the broadcast message is operating correctly and whether the storage group has been updated to add or remove a data storage node;

the second data storage node configured to send a first instance of the broadcast message, the first instance of the broadcast message comprising an indication of a node list version used by the second data storage node; and

the first data storage node further configured to;

receive the first instance of the broadcast message from the second data storage node,

update the local node list based on the first instance of the broadcast message indicating that the second data storage node is using a different node list version than the version of the local node list stored at the first data storage node,

add at least a third data storage node to the local node list and begin monitoring for receipt of instances of the broadcast message from the third data storage node in response to the first instance of the broadcast message indicating the second data storage node is using the different node list version,

detect that the second data storage node is malfunctioning based on failing to receive a subsequent instance of the broadcast message from the second data storage node for a predetermined period of time, and

initiate a data replication procedure based on detecting that the second data storage node is malfunctioning, wherein at least one file stored on the first data storage node is to be replicated in the data replication procedure, the at least one file is associated with a host list stored on the first data storage node, the host list indicates a subset of data storage nodes in the storage group that also store the at least one file, and the host list indicates that the second data storage node is in the subset of data storage nodes.

9. The system of claim 8, wherein each of the plurality of data storage nodes are configured to periodically broadcast instances of the broadcast message in order to indicate that the broadcasting node is functioning correctly.

10. The system of claim 9, wherein each of the plurality of data storage nodes are configured to monitor for receipt of instances of the broadcast message.

11. The system of claim 8, wherein the data replication procedure comprises:

the first data storage node maintaining a local copy of the at least one file at the first data storage node; and

the first data storage node sending a multicast message to the plurality of data storage nodes requesting that the first data storage node becomes a master node for the at least one file, wherein the master node controls the replication and maintenance of the at least one file within the data storage system.

12. The system of claim 11, wherein the data replication procedure further comprises:

the first data storage node receiving a response to the multicast message from a fourth data storage node, wherein the response to the multicast message includes at least one of a version identification for the copy of the at least one file stored on the fourth data storage node or a list of data storage nodes that maintain a copy of the at least one file; and

the first data storage node updating a master host list for the at least one file based on the response to the multicast message.

13. The system of claim 12, wherein the data replication procedure further comprises the first data storage node requesting an updated version of the at least one file based on the response indicating that the local copy of the at least one file is not the most current version of the file.

14. The system of claim 12, wherein the data replication procedure further comprises the first data storage node querying the plurality of data storage nodes or a subset of the plurality of network storage nodes, the query comprising a data identifier corresponding to the at least one file and a request that data storage nodes receiving the query indicate whether they can save a copy of the at least one file.

15. The system of claim 14, wherein the data replication procedure further comprises:

the first data storage node receiving at least two responses to the query;

the first data storage node identifying at least two of the plurality of data storage nodes to store a copy of the at least one file based on the responses;

the first data storage node sending an instruction to store the at least one file to the at least two of the plurality of data storage nodes; and

the first data storage node updating the master host list for the at least one file based on the instruction.

16. The system of claim 8, wherein the first instance of the broadcast message from the second data storage node comprises an address for the second data storage node.

17. A first data storage node configured to:

monitor for receipt of instances of a broadcast message from each of a plurality of data storage nodes included in a storage group, wherein a local node list stored at the first data storage node identifies the plurality of data storage nodes as being included in the storage group, the plurality of data storage nodes identified in the local node list includes at least a second data storage node, and instances of the broadcast message include an indication of a node list version being used by the data storage node that transmitted the instance of the broadcast message such that receipt of a given instance of the broadcast message by the first storage node indicates to the first storage node both that the data storage node that sent the given instance of the broadcast message is operating correctly and whether the storage group has been updated to add or remove a data storage node;

receive a first instance of the broadcast message from the second data storage node;

update the local node list based on the first instance of the broadcast message indicating that the second data storage node is using a different node list version than the version of the local node list stored at the first data storage node;

add at least a third data storage node to the local node list and begin monitoring for receipt of instances of the broadcast message from the third data storage node in response to the first instance of the broadcast message indicating the second data storage node is using the different node list version;

detect that the second data storage node is malfunctioning based on failing to receive a subsequent instance of the broadcast message from the second data storage node for a predetermined period of time; and

initiate a data replication procedure based on detecting that the second data storage node is malfunctioning, wherein a file stored on the first data storage node is to be replicated in the data replication procedure, the file is associated with a host list stored on the first data storage node, and the host list indicates a subset of data storage nodes in the storage group that also store the file, the host list indicates that the second data storage node is in the subset of data storage nodes.

18. The first data storage node of claim 17, wherein the first data storage node is further configured to remove at least a fourth data storage node from the local node list and stop monitoring for receipt of instances of the broadcast message from the fourth data storage node in response to the first instance of the broadcast message indicating the second data storage node is using the different node list version.

19. The first data storage node of claim 17, wherein the first data storage node is further configured to:

send a message requesting a copy of the node list maintained by the second storage node to the second data storage node in response to the first instance of the broadcast message indicating the second data storage node is using the different node list version;

receive the copy of the node list maintained by the second storage node; and

update the local host list based on the copy of the node list maintained by the second storage node.

20. The first data storage node of claim 17, wherein the first data storage node is further configured to periodically send instances of the broadcast message to the plurality of data storage nodes in the storage group, the instance of the broadcast message sent from the first data storage node indicating a version number for the local host list.