Object loss reporting in a data storage system

• US 8,935,221 B1
• Filed: 07/25/2012
• Issued: 01/13/2015
• Est. Priority Date: 07/25/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Lazier, Colin L., Deen, Brian J.
• Primary Examiner(s): LIAO, JASON G
• Application Number:
  US13/558,070
• Time in Patent Office: 902 Days
• Field of Search: None
• US Class Current: 707/690
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/24 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

In response to receiving a request from a client to store an object, a key-durable storage system may assign the object to a volume in its data store, generate a key for the object (e.g., an opaque identifier that encodes information for locating the object in the data store), store the object on one disk in the assigned volume, store the key redundantly in the assigned volume (e.g., using a replication or erasure coding technique), and may return the key to the client. To retrieve the object, the client may send a request including the key, and the system may return the object to the client. If a disk fails, the system may determine which objects were lost, and may return the corresponding keys to the appropriate clients in a notification. The system may be used to back up a more expensive object-redundant storage system.

Claims

1. A system, comprising:

a data store; and

one or more computing devices, each comprising at least one processor and memory, wherein the one or more computing devices implement a key-durable storage service, and wherein the key-durable storage service is configured to;

receive a request from a client to store a data object in the data store, wherein the request comprises the data object;

generate a key for the data object, wherein the key comprises information usable to determine the location of the data object in the data store;

store the data object and the key in the data store, wherein the data store stores a plurality of data objects and respective keys generated therefor, and wherein the key-durable storage service does not maintain a data structure that maps the plurality of data objects or the respective keys to the locations of the data objects in the data store;

return the key to the client; and

in response to a loss of data in the data store;

determine that the lost data comprises the data object; and

return a notification of lost data to the client, wherein the notification comprises the key.

2. The system of claim 1, wherein, to store the data object and the key in the data store, the key-durable storage service is configured to:

store a single instance of the data object in the data store; and

store multiple instances of the key in the data store.

3. The system of claim 1,

wherein, to store the data object and the key in the data store, the key-durable storage service is configured to:

assign the data object to a particular one of a plurality of volumes defined in the data store; and

store the data object and the key on the particular volume; and

wherein the key comprises an encoding of an identifier of the particular volume.

4. The system of claim 1, wherein the key-durable storage service is further configured to:

receive a request from a client to retrieve the data object, wherein the request to retrieve the data object comprises the key;

locate the data object in the data store dependent on the information in the key; and

return the data object to the client.

5. The system of claim 1, wherein, to determine that the lost data comprises the data object, the key-durable storage service is configured to:

determine the data objects that remain stored in the data store following the loss of data;

identify the respective keys stored in the memory for each of the data objects that remain stored in the data store; and

determine that the key is a respective key for a data object that no longer remains stored in the data store.

6. The system of claim 1,

wherein the client is an object-redundant storage system;

and

wherein the data object is a replica of an object stored in the object-redundant storage system or a shard of an object stored in the object-redundant storage system that was created through erasure coding.

7. A method, comprising:

performing by one or more computers;

storing a plurality of data objects in a data store on behalf of one or more clients with a given durability, wherein the data store comprises a plurality of storage nodes;

generating, for each of the plurality of data objects, a respective key comprising information usable to determine the location at which the data object is stored in the data store;

storing, in the data store, the respective key for each of the plurality of data objects with a higher durability than the given durability;

detecting a failure or loss of one of the plurality of storage nodes;

determining that one or more of the plurality of data objects were stored on the one of the plurality of storage nodes; and

generating a notification that the one or more data objects are no longer stored in the data store, wherein the notification comprises the respective key for each of the one or more data objects.

8. The method of claim 7,

wherein the plurality of storage nodes comprises a plurality of volumes, each of which comprises multiple disks;

and

wherein, for each of the plurality of data objects, storing the data object comprises;

assigning the data object to one of the plurality of volumes for storage; and

storing the data object on one of the disks of the assigned volume.

9. The method of claim 8, wherein, for each of the plurality of data objects, storing the respective key comprises:

storing the respective key on two or more of the disks of the assigned volume.

10. The method of claim 9, wherein the two or more disks on which the respective key is stored comprises the one disk on which the data object is stored.

11. The method of claim 9,

wherein said detecting comprises detecting that a given disk of one of the plurality of volumes has failed or has been taken out of service;

and

wherein said determining comprises;

determining the data objects that remain stored on the disks of the one of the plurality of volumes other than the given disk following said detecting;

identifying the respective keys stored on the disks of the one of the plurality of volumes other than the given disk following said detecting; and

determining that each of one or more keys stored on the disks of the one of the plurality of volumes other than the given disk is a respective key for a data object that no longer remains stored on the disks of the one of the plurality of volumes other than the given disk.

12. The method of claim 8, wherein, for each of the plurality of data objects, storing the respective key comprises:

erasure coding the respective key to create two or more shards; and

storing each of the two or more shards on a different disk of the assigned volume.

13. The method of claim 7, wherein said storing the respective key for each of the plurality of data objects with a higher durability comprises storing multiple instances of the respective key in the data store, and wherein the number of instances is dependent on one or more of:

a desired durability or an expected failure rate for the plurality of storage nodes.

14. The method of claim 7, further comprising:

sending the notification to a client on whose behalf the one or more data objects were stored in the data store.

15. A non-transitory, computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to:

implement a key-durable storage service on one or more devices on a service provider network, wherein the key-durable storage service is configured to;

store a plurality of data objects in a data store of the service provider network on behalf of one or more storage service clients in accordance with an object durability policy; and

store a respective key that was generated by the key-durable storage service for each of the plurality of data objects in the data store, wherein storing the respective keys comprises storing the respective keys in the data store in accordance with a key durability policy that specifies a higher durability than a durability specified by the object durability policy; and

expose application programming interfaces (APIs) of the key-durable storage service to one or more clients on a client network of the service provider network, wherein the APIs of the key-durable storage service provide a standard interface between storage operations of the key-durable storage service and the one or more clients, and wherein the APIs of the key-durable storage service include;

a put object API for uploading data objects from the client network to the data store;

a get object API for downloading stored data objects from the data store to the client network; and

a notify API for sending a notification of a lost data from the key-durable storage service to a client that includes a respective key for a data object that is no longer stored in the data store due to a loss of data in the data store.

16. The non-transitory, computer-readable storage medium of claim 15,

wherein the key-durable storage service does not maintain a data structure that maps the plurality of data objects or the respective keys to the locations of the data objects in the data store;

and

wherein the respective key that was generated by the key-durable storage service for each of the plurality of data objects in the data store comprises an encoding usable to determine the location of the data object in the data store.

17. The non-transitory, computer-readable storage medium of claim 16,

wherein the encoding is encrypted;

wherein the encoding represents an identifier of a storage volume on which the data object is stored in the data store or an identifier of a disk on which the data object is stored in the data store.

18. The non-transitory, computer-readable storage medium of claim 15,

wherein a request received in accordance with the put object API comprises a particular data object as an input parameter, but does not include a key as an input parameter;

and

wherein a response returned in accordance with the put object API returns a respective key that was generated by the key-durable storage service for the particular data object.

19. The non-transitory, computer-readable storage medium of claim 15,

wherein a request received in accordance with the get object API comprises a key that was generated by the key-durable storage service for a particular data object as an input parameter;

and

wherein a response returned in accordance with the get object API returns the particular data object.

20. The non-transitory, computer-readable storage medium of claim 15,

wherein the key-durable storage service is configured to:

detect a loss of data in the data store;

determine the respective key for a given data object that is no longer stored in the data store due to the loss of data in the data store; and

send a notification to a client on whose behalf the given data object was stored in the data store in accordance with the notify API.

21. The non-transitory, computer-readable storage medium of claim 15,

wherein the data store comprises a plurality of volumes, each of which comprises multiple disks;

wherein in response to receiving a request to store a given data object in accordance with the put API, the key-durable storage service is configured to;

assign the given data object to one of the plurality of volumes for storage;

store the data object on one of the disks of the assigned volume;

generate a key for the given data object; and

store the respective key on two or more of the disks of the assigned volume.

22. The non-transitory, computer-readable storage medium of claim 21,

wherein the key-durable storage service is further configured to:

detect that a given disk of one of the plurality of volumes has failed or has been taken out of service;

determine the data objects that remain stored on the disks of the one of the plurality of volumes following said detecting;

identify the respective keys stored on the disks of the one of the plurality of volumes following said detecting; and

determine that each of one or more keys stored on the disks of the one of the plurality of volumes is a respective key for a particular data object that no longer remains stored on the disks of the one of the plurality of volumes; and

send a notification to a client on whose behalf the particular data object was stored in the data store in accordance with the notify API.

23. A system, comprising:

a plurality of storage devices configured as a volume-based data store that stores a plurality of data objects and respective keys generated therefor by a key-durable storage service, wherein the key-durable storage service does not maintain a data structure that maps the plurality of data objects or the respective keys to locations at which the data objects or respective keys are stored in the data store; and

one or more computing devices, each comprising at least one processor and memory, wherein the one or more computing devices implement the key-durable storage service, and wherein the key-durable storage service is configured to;

receive, from one or more storage service clients, a plurality of data objects to be stored in the data store; and

for each of the received data objects;

assign one of a plurality of volumes of the data store for storing the received data object;

store, without redundancy, the received data object on the assigned volume;

generate a key for the received data object; and

store, with redundancy, the key for the received data object on the assigned volume; and

in response to a loss of data on a particular volume in the data store;

determine that the lost data comprises a given data object; and

generate a notification of the lost data, wherein the notification comprises a respective key that was generated for the given data object by the key-durable storage service when the given data object was stored in the data store.

24. The system of claim 23,

wherein each of the plurality of volumes comprises multiple disks;

wherein, for each of the received data objects, storing the received data object comprises storing the received data object on one of the disks of the assigned volume.

25. The system of claim 24,

wherein, for each of the received data objects, storing the key for the received object comprises storing the key for the received data object on the one of the disks of the assigned volume and on one or more other disks of the assigned volume.

26. The system of claim 24,

wherein, for each of the received data objects, storing the key for the received object comprises storing the key for the received data object on two or more disks of the assigned volume other than the one disk of the assigned volume.

27. The system of claim 24, wherein, for each of the received data objects, the amount of redundancy with which the key for the received data object is stored is dependent on one or more of:

a desired durability or an expected disk failure rate.

28. The system of claim 23,

wherein each of the plurality of volumes comprises multiple disks;

wherein, in at least one of the plurality of volumes, each disk in a subset of the multiple disks is configured to store data objects that are assigned to the volume for storage; and

wherein, in the at least one of the plurality of volumes, each disk in a different, non-overlapping subset of the multiple disks is configured to store keys for data objects that are assigned to the volume for storage.

29. The system of claim 23, wherein at least some of received data objects are replicas of an object stored in an object-redundant storage system or shards of an object stored in an object-redundant storage system that were created through erasure coding.

30. The system of claim 23,

wherein, for each of the received data objects, the key comprises an encoding of an identifier of the assigned volume;

and

wherein the key-durable storage service is further configured to

receive, from a storage service client, a request to retrieve a particular one of the received data objects from the data store, wherein the request to retrieve the particular data object comprises the key that was generated for the particular data object by the key-durable storage service;

locate the data object in the data store dependent on the encoding in the key; and

return the particular data object to the storage service client.