Write horizon data management

• US 9,465,551 B1
• Filed: 05/13/2015
• Issued: 10/11/2016
• Est. Priority Date: 11/20/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Donlan, Bryan James, Kumar, Sandeep
• Primary Examiner(s): SONG, HUA JASMINE
• Application Number:
  US14/711,525
• Time in Patent Office: 517 Days
• Field of Search: 711/166
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0673 Single storage device
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

Conditions are enforced to prevent unintended deletion of data stored by a data storage system. For example, to delete a collection of data, a condition on the collection of data'"'"'s size may be enforced. The collection may be required to be empty, for example. In addition, a condition that there not exist a pending data processing operation that can affect fulfillment of the condition on the collection of data'"'"'s size is also enforced.

Claims

1. A computer-implemented method comprising:

receiving a request to upload a data object to a logical data container;

determining a counter associated with processing a plurality of requests to upload one or more data objects to one or more logical data containers, the plurality of requests including the request;

accessing a value recorded for the logical data container, the value indicative of a potential pending request affecting the logical data container; and

processing the request based at least in part on a comparison of the counter and the value, the processing comprising;

if the counter is greater than the value, updating the value to be equal to or greater than the counter and uploading the data object to the logical data container based at least in part on the updated value, and

if the value is equal to or greater than the counter, uploading the data object to the logical data container without an update to the value.

2. The computer-implemented method of claim 1, wherein the plurality of requests are processed in a batch and result in a generation of a completed batch, and wherein the counter identifies the generation.

3. The computer-implemented method of claim 1, further comprising:

receiving a second request to delete the logical data container;

evaluating the second request based at least in part on the updated value and an update to the counter; and

denying the request based at least in part on the updated value indicating that the potential pending request affects the logical data container.

4. The computer-implemented method of claim 3, wherein denying the request comprises determining that the update to the counter comprises an increase to the counter such that the increased counter is greater than the updated value and indicates the potential pending request.

5. The computer-implemented method of claim 1, wherein the logical data container is provided in a data storage system, wherein the data storage system comprises a front end and a metadata plane, wherein the front end is configured to maintain the value, and wherein the metadata plane is configured to maintain the counter.

6. The computer-implemented method of claim 1, wherein processing the request comprises:

causing data of the data object to be received;

causing the received data to be stored in the logical data container; and

providing a notification in response to the request indicating that uploading the data object is complete.

7. The computer-implemented method of claim 1, wherein the logical data container is provided in a data storage system, wherein processing the request comprises:

staging the data object in transient storage of the data storage system;

generating a scheduled job to move the data object from the transient storage to the logical data container, wherein the scheduled job is associated with a batch processing of a plurality of upload requests; and

moving the data object to the logical data container based at least in part on the scheduled job.

8. One or more non-transitory computer-readable media bearing instructions that, upon execution with one or more processors, cause a computer system to, at least:

receive a request to add a set of data to a volume;

determine a first value associated with one or more previous requests to add data to the volume;

compare the first value and a second value associated with the volume, the second value indicative of a potential pending request affecting stored data in the volume; and

process the request based at least in part on the comparison of the first value and the second value, the processing comprising;

if the first value is greater than the second value, updating the second value to be equal to or greater than the first value and adding the set of data to the volume based at least in part on the updated second value, and

if the second value is equal to or greater than the first value, adding the set of data to the volume without an update to the second value.

9. The one or more non-transitory computer-readable media of claim 8, wherein the first value indicates a generation associated with processing the one or more previous requests.

10. The one or more non-transitory computer-readable media of claim 8, wherein the first value is maintained in a first table, wherein the second value is maintained in a second table, and wherein the first table and the second table are different.

11. The one or more non-transitory computer-readable media of claim 10, wherein the first table is associated with a generation of the one or more previous requests, and wherein the first table stores information about a plurality of operations applicable to respective data objects based at least in part on the one or more previous requests.

12. The one or more non-transitory computer-readable media of claim 10, wherein the second table identifies one or more logical data containers of the volume, respective number of data objects stored in the one or more logical data containers, and respective values of the one or more logical data containers, and wherein the respective values indicate potential pending requests affecting the one or more logical data containers.

13. The one or more non-transitory computer-readable media of claim 8, wherein the volume is provided by a data storage system, wherein the data storage system comprises a front end and a metadata plane, wherein the front end is configured to maintain the second value upon receiving the request, and wherein the metadata plane is configured to maintain the first value upon processing the request.

14. The one or more non-transitory computer-readable media of claim 8, wherein the second value relative to the first value is indicative of the potential pending request affecting the stored data in the volume.

15. A system comprising:

one or more processors;

one or more non-transitory computer-readable media bearing instructions that, upon execution with the one or more processors, cause the system to, at least;

receive a request to store a collection of data in a storage space;

determine a first value associated with a previous request to store data in the storage space;

compare the first value and a second value maintained for the storage space, the second value indicative of a potential pending request affecting stored data in the storage space; and

process the request based at least in part on the comparison of the first value and the second value, the processing comprising;

if the first value is greater than the second value, updating the second value to be equal to or greater than the first value and storing the collection of data in the storage space based at least in part on the updated second value, and

if the second value is equal to or greater than the first value, storing the collection of data in the storage space without an update to the second value.

16. The system of claim 15, wherein the first value indicates a generation associated with processing the previous request.

17. The system of claim 15, wherein the second value is increased based at least in part on the first value being greater than the second value, wherein the instructions further cause the system to:

increase the first value based at least in part on processing the potential pending request;

receive a second request to store a second collection of data in the storage space;

compare the increased second value to the increased first value; and

process the second request without changing the increased second value based at least in part on determining that the increased second value exceeds the increased first value.

18. The system of claim 15, wherein updating the second value comprises increasing the second value, wherein the second value is increased based at least in part on being within a predetermined amount of time of an anticipated update to the first value.

19. The system of claim 15, wherein updating the second value comprises increasing the second value, wherein the second value is increased based at least in part on a probability distribution.

20. The system of claim 15, wherein the instructions further cause the system to increase the first value based at least in part on processing the request.