Detecting and reconciling system resource metadata anomolies in a distributed storage system

• US 9,244,958 B1
• Filed: 06/13/2013
• Issued: 01/26/2016
• Est. Priority Date: 06/13/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Vasanth, Jai, Dhawan, Medhavi, MacCanti, Maximiliano
• Primary Examiner(s): Lewis, Cheryl
• Application Number:
  US13/917,320
• Time in Patent Office: 957 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: Y10S 707/956 Hierarchical
• CPC Class Codes: Y10S 707/966 Distributed
• CPC Class Codes: Y10S 707/9994 Distributed or remote access
• CPC Class Codes: Y10S 707/99941 Database schema or data structure

Abstract

A system that implements detection and reconciliation of system resource metadata for a distributed storage system is described. A node may obtain resource metadata specific to the node from another node that maintains system resource metadata for a distributed storage system. Based on the resource metadata specific to the node, a determination may be made that the node is not reconciled with the system resource metadata. A corrective operation may be performed to reconcile the node with the system resource metadata. A corrective operation may include terminating a resource, making unavailable a resource, modifying resource attributes, or sending a resource metadata update to system resource metadata for correction.

Claims

1. A system, comprising:

a plurality of compute nodes implemented on one or more computing devices and configured to implement a distributed storage service;

wherein a control plane node of the plurality of compute nodes is configured to maintain system resource metadata for the distributed storage service;

wherein at least one storage node of the plurality of compute nodes is configured to detect one or more resource metadata anomalies;

wherein, to detect one or more resource metadata anomalies, the at least one storage node is configured to;

request resource metadata specific to the at least one storage node from the control plane node;

receive the resource metadata specific to the at least one storage node;

compare the received resource metadata with local resource information maintained at the least one storage node;

based, at least in part, on the comparison, identify the one or more resource metadata anomalies; and

wherein, the at least one storage node is further configured to;

for each of the detected one or more resource metadata anomalies, perform a corrective operation to reconcile the at least one storage node with the system resource metadata.

2. The system of claim 1, wherein the resource metadata specific to the at least one storage node indicates that a replica of data maintained by the distributed storage service is not stored at the at least one storage node, wherein the local resource information maintained at the at least one storage node indicates that the replica is stored at the at least one storage node, and wherein to perform the corrective operation to reconcile the at least one storage node with the system resource metadata, the at least one storage node is configured to terminate a process servicing the replica at the at least one storage node.

3. The system of claim 1,

wherein the resource metadata specific to the at least one storage node indicates that a particular resource is maintained by the at least one storage node, wherein the local resource information maintained at the at least one storage node indicates that the particular resource is not maintained at the at least one storage node;

wherein to perform the corrective operation to reconcile the at least one storage node with the system resource metadata, the at least one storage node is configured to send a resource metadata update to the control plane node indicating that the particular resource is not maintained at the at least one storage node;

wherein the control plane node is further configured to update the system resource metadata for the distributed storage service in response to receiving the resource metadata update from the at least one storage node.

4. The system of claim 1, wherein the detection of one or more resource anomalies is performed by the at least one storage node periodically.

5. A method, comprising:

performing, by one or more computing devices implementing a node for a distributed system;

obtaining resource metadata specific to the node from another node of the distributed system, wherein the other node maintains system resource metadata for the distributed system;

based, at least in part, on the resource metadata specific to the node, determining that the node is not reconciled with the system resource metadata; and

in response to determining that the node is not reconciled with the system resource metadata, performing a corrective operation to reconcile the node with the system resource metadata.

6. The method of claim 5, wherein said determining that the node is not reconciled with the system resource metadata, comprises:

comparing the resource metadata specific to the node with local resource information maintained at the node;

based, at least in part, on the comparison, determining that a resource maintained at the node is not indicated as maintained at the node by the resource metadata.

7. The method of claim 6, wherein the node is a storage node for a distributed storage service implemented by the distributed system, wherein the resource not indicated as maintained at the storage node by the resource metadata is a replica of data maintained by the distributed storage service, and wherein said performing the corrective operation to reconcile the node with the system resource metadata comprises terminating a process for servicing the replica at the storage node.

8. The method of claim 5, wherein said determining that the node is not reconciled with the system resource metadata, comprises:

comparing the resource metadata specific to the node with local resource information maintained at the node; and

based, at least in part, on the comparison, determining that a resource is not maintained at the node when the resource metadata indicates that the resource is maintained at the node.

9. The method of claim 8, wherein said performing the corrective operation to reconcile the node with the system resource metadata comprises sending a notification indicating that the resource is not maintained at the node.

10. The method of claim 5, wherein said determining that the node is not reconciled with the system resource metadata, comprises:

comparing the resource metadata specific to the node with local resource information maintained at the node; and

based, at least in part, on the comparison, determining that one or more resource attributes of the node include different values than indicated values for the one or more resource attributes of the node by the resource specific metadata.

11. The method of claim 10, wherein said performing the corrective operation to reconcile the node with the system resource metadata comprises modifying the one or more resource attributes of the node to include the indicated values for the one or more resource attributes of the node indicated by the resource specific metadata.

12. The method of claim 5, wherein said obtaining, said determining, and said performing, are performed in response to detecting a possible metadata anomaly event.

13. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices implementing a storage node of distributed storage service cause the storage node to implement:

obtaining resource metadata specific to the storage node from another node of a plurality of nodes including the storage node together implementing the distributed storage service, wherein the other node maintains system resource metadata for the distributed storage service;

based, at least in part, on the resource metadata specific to the storage node, determining that the storage node is not reconciled with the system resource metadata; and

in response to determining that the storage node is not reconciled with the system resource metadata, performing a corrective operation to reconcile the storage node with the system resource metadata.

14. The non-transitory, computer-readable storage medium of claim 13, wherein, in said determining that the storage node is not reconciled with the system resource metadata, the storage node implements:

comparing the resource metadata specific to the storage node with local resource information maintained at the storage node;

based, at least in part, on the comparison, determining that a resource maintained at the storage node is not indicated as maintained at the storage node by the resource metadata.

15. The non-transitory, computer-readable storage medium of claim 13, wherein, in said performing the corrective operation to reconcile the storage node with the system resource metadata, the storage node further implements identifying the resource maintained at the storage node as unavailable for further processing.

16. The non-transitory, computer-readable storage medium of claim 13,

wherein, in said determining that the storage node is not reconciled with the system resource metadata, the storage node implements:

comparing the resource metadata specific to the node with local resource information maintained at the node;

based, at least in part, on the comparison, determining that a resource is not maintained at the node when the resource metadata indicates that the resource is maintained at the node;

wherein, in said performing the corrective operation to reconcile the storage node with the system resource metadata, the storage node implements;

sending a notification indicating that the resource is not maintained at the storage node.

17. The non-transitory, computer-readable storage medium of claim 13, wherein, in said determining that the storage node is not reconciled with the system resource metadata, the storage node implements:

comparing the resource metadata specific to the node with local resource information maintained at the node;

based, at least in part, on the comparison, determining that one or more resource attributes of the node include different values than indicated values for the one or more resource attributes of the node by the resource specific metadata;

wherein, in said performing a corrective operation to reconcile the storage node with the system resource metadata, the storage node further implements;

delaying for a specific period of time to allow for additional updates to the local resource information;

determining that the storage node has a current version of the local resource information; and

performing a local corrective operation at the storage node.

18. The non-transitory, computer-readable storage medium of claim 17, wherein one of the one or more resource attributes is a throughput provision value for a resource maintained at the storage node, and wherein, in said performing the local corrective operation, the storage node further implements configuring the storage node to set the throughput provision value to the indicated value of the resource metadata for the throughput provision value.

19. The non-transitory, computer-readable storage medium of claim 13, wherein the storage node further implements:

detecting a possible metadata anomaly event; and

in response to detecting the possible metadata anomaly event, performing said obtaining, said determining, and said performing.

20. The non-transitory, computer-readable storage medium of claim 13, wherein said obtaining, said determining, and said performing are performed periodically.