Consistent object renaming in distributed systems

• US 9,449,008 B1
• Filed: 03/31/2014
• Issued: 09/20/2016
• Est. Priority Date: 03/31/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Oikarinen, Matti Juhani, Vincent, Pradeep, Frigo, Matteo
• Primary Examiner(s): Rostami, Mohammad S
• Application Number:
  US14/231,070
• Time in Patent Office: 904 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/166 File name conversion
• CPC Class Codes: G06F 16/1774 Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
• CPC Class Codes: G06F 16/211 Schema design and management

Abstract

In response to a rename request to change a file name at a storage service from a first name to a second name, a workflow comprising at least two atomic operations is initiated. In the first atomic operation, a lock is obtained on a first directory entry for the first name, and an intent record for the rename workflow is stored. In a second atomic operation, a pointer of a second directory entry for the second name is modified, and an indication of the pointer modification is stored. In a third set of operations, the intent record is deleted, the lock is released, and the first directory entry is deleted.

Claims

1. A system, comprising:

one or more computing devices configured to;

receive, at a metadata subsystem of a distributed multi-tenant file store service comprising a plurality of metadata nodes, a rename request to change a name of a particular file store object from a first name to a second name, wherein, corresponding to each of a plurality of file store objects, the metadata subsystem stores a node entry and a directory entry, wherein the node entry comprises a plurality of metadata fields associated with the corresponding file store object and the directory entry includes (a) a name of the file store object and (b) a pointer to the node entry;

determine, based on deadlock avoidance analysis, that a first directory entry corresponding to the first name is to be locked as part of an initial atomic operation of a workflow sequence corresponding to the rename request, wherein the first directory entry comprises a pointer to a first node entry;

attempt the initial atomic operation at a first metadata node of a plurality of metadata nodes of the metadata subsystem, wherein the first atomic operation comprises;

(a) obtaining a lock on the first directory entry;

(b) generating a workflow identifier corresponding to the workflow sequence, and (c) storing a record of an intent to rename the particular file store object, wherein the record indicates the workflow identifier;

in response to a determination that the initial atomic operation succeeded, attempt a second atomic operation of the workflow sequence at a second metadata node of the plurality of metadata nodes responsible for storing a second directory entry associated with the second name, wherein the second atomic operation comprises;

(a) verifying that the second directory entry is unlocked;

(b) setting the second directory'"'"'s pointer to point to the first node entry; and

(c) storing an indication that a pointer modification corresponding to the workflow identifier succeeded;

in response to a determination that the second atomic operation succeeded, attempt a third set of operations of the workflow sequence, wherein the third set of operations comprises;

(a) deleting the record of the intent;

(b) deleting the lock on the first directory entry; and

(c) deleting the first directory entry.

2. The system as recited in claim 1, wherein the one or more computing devices are further configured to:

delete, asynchronously with respect to the third set of operations, the indication that the pointer modification succeeded.

3. The system as recited in claim 1, wherein the one or more computing devices are further configured to:

receive a different rename request to change a name of a different file store object from a third name to a fourth name;

determine, based on deadlock avoidance analysis, that a fourth directory entry corresponding to the fourth name is to be locked as part of an initial atomic operation of a second workflow sequence;

attempt the initial atomic operation of the second workflow sequence comprising;

(a) obtaining a lock on the fourth directory entry;

(b) generating a second workflow identifier corresponding to the second workflow sequence; and

(c) storing a record of an intent to rename the different file store object, wherein the record of the intent to rename the different file store object comprises the second workflow identifier;

in response to a determination that the initial atomic operation of the second workflow sequence succeeded, attempt a second atomic operation of the second workflow sequence comprising;

(a) verifying that a third directory entry corresponding to the third name is unlocked, wherein the third directory entry includes a pointer to a different node entry of the different file store object;

(b) deleting the third directory entry; and

(c) storing an indication that a deletion corresponding to the second workflow identifier succeeded;

in response to a determination that the second atomic operation of the second workflow sequence succeeded, attempting a third set of operations of the second workflow sequence comprising;

(a) storing a pointer to the different node entry of the different file storage object in the fourth directory entry (b) deleting the record of the intent to rename the different file store object; and

(c) deleting the lock on the fourth directory entry.

4. The system as recited in claim 1, wherein the metadata subsystem comprises a replacement node configured to assume responsibilities of the first metadata node in the event of a failure, wherein the record of the intent and the workflow identifier are accessible from the replacement node, and wherein, subsequent to a failure of the first metadata node before a result of the attempt to perform the second atomic operation is determined, the replacement node is configured to access the record of the intent and the workflow identifier to query the second metadata node to determine whether the pointer modification corresponding to the workflow identifier succeeded.

5. The system as recited in claim 1, wherein the said deadlock avoidance analysis comprises a comparison of the first name to the second name to determine whether the first directory entry is to be locked as part of the initial atomic operation.

6. A method, comprising:

performing, by one or more computing devices;

receiving, at a storage service, a rename request to change a name of a particular file store object from a first name to a second name;

determining that a first directory entry corresponding to the first name is to be locked as part of an initial atomic operation of a workflow sequence corresponding to the rename request, wherein the first directory entry comprises a pointer to a first node entry storing a plurality of metadata fields of the particular file store object;

attempting the initial atomic operation comprising;

(a) obtaining a lock on the first directory entry;

(b) storing a workflow identifier corresponding to the workflow sequence, and (c) storing a record of an intent to rename the particular file store object;

in response to determining that the initial atomic operation succeeded, attempting a second atomic operation of the workflow sequence comprising;

(a) verifying that a second directory entry corresponding to the second name is unlocked;

(b) setting the second directory'"'"'s pointer to point to the first node entry; and

(c) storing an indication that a pointer modification corresponding to the workflow identifier succeeded;

in response to determining that the second atomic operation succeeded, attempting a third set of operations of the workflow sequence comprising;

(a) deleting the record of the intent;

(b) deleting the lock on the first directory entry; and

(c) deleting the first directory entry.

7. The method as recited in claim 6, wherein said determining that a first directory entry corresponding to the first name is to be locked as part of an initial atomic operation comprises performing a deadlock avoidance analysis.

8. The method as recited in claim 7, wherein the deadlock avoidance analysis includes a comparison of the first name to the second name to determine whether the first directory entry is to be locked as part of the initial atomic operation.

9. The method as recited in claim 8, further comprising performing, by the one or more computing devices to implement the comparison:

combining, with the first name, an identifier of a node entry associated with the first name; and

combining, with the second name, an identifier of a node entry associated with the second name.

10. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

receiving, at the storage service, a different rename request to change a name of a different file store object from a third name to a fourth name;

determining that a fourth directory entry corresponding to the fourth name is to be locked as part of an initial atomic operation of a second workflow sequence;

attempting the initial atomic operation of the second workflow sequence comprising;

(a) obtaining a lock on the fourth directory entry;

(b) storing a second workflow identifier corresponding to the second workflow sequence; and

(c) storing a record of an intent to rename the different file store object;

in response to determining that the initial atomic operation of the second workflow sequence succeeded, attempting a second atomic operation of the second workflow sequence comprising;

(a) verifying that a third directory entry corresponding to the third name is unlocked, wherein the third directory entry includes a pointer to a different node entry of the different file store object;

(b) deleting the third directory entry; and

(c) storing an indication that a deletion corresponding to the second workflow identifier succeeded;

in response to determining that the second atomic operation succeeded, attempting a third set of operations of the second workflow sequence comprising;

(a) storing a pointer to the different node entry of the different file storage object in the fourth directory entry (b) deleting the record of the intent to rename the different file store object; and

(c) deleting the lock on the fourth directory entry.

11. The method as recited in claim 6, wherein the initial atomic operation is performed at a first node of a metadata subsystem of the storage service, and wherein the second atomic operation is performed at a second node of the metadata subsystem.

12. The method as recited in claim 11, wherein the record of the intent is stored at a persistent storage device associated with the first node, and wherein the indication that the pointer modification succeeded is stored at a persistent storage device associated with the second node.

13. The method as recited in claim 12, wherein a replacement node of the metadata subsystem is configured to assume responsibilities of the first node in the event of a failure at the first node, further comprising performing, by the one or more computing devices:

receiving, at the storage service, a different rename request to change a name of a different file store object from a third name to a fourth name;

determining that a third directory entry corresponding to the third name is to be locked as part of an initial atomic operation of a second workflow sequence, wherein the third directory entry comprises a pointer to a different node entry storing a plurality of metadata fields of the different file store object;

attempting, at the first node, the initial atomic operation of the second workflow sequence comprising;

(a) obtaining a lock on the third directory entry;

(b) storing a second workflow identifier corresponding to the second workflow sequence, and (c) storing a record of an intent to rename the different file store object;

in response to determining that the initial atomic operation of the second workflow sequence succeeded, submitting a request from the first node to the second node to attempt a second atomic operation of the second workflow sequence comprising;

(a) verifying that a fourth directory entry corresponding to the fourth name is unlocked;

(b) setting the fourth directory entry'"'"'s pointer to point to the different node entry; and

(c) storing an indication that a pointer modification corresponding to the second workflow identifier succeeded;

in response to determining that the first node failed prior to determining a result of the request to attempt the second atomic operation,

reading, at the replacement node, the record of the intent to rename the different workflow sequence, and the second workflow identifier;

submitting a query from the replacement node to the second node to determine whether the fourth directory entry'"'"'s pointer modification succeeded, wherein said query includes the second workflow identifier as a parameter.

14. The method as recited in claim 13, further comprising performing, by the one or more computing devices:

in response to determining the fourth directory entry'"'"'s pointer modification succeeded, attempting a third set of operations of the second workflow sequence at the replacement node, wherein said third set of operations comprises;

(a) deleting the record of the intent to rename the different file store object;

(b) deleting the lock on the third directory entry; and

(c) deleting the third directory entry.

15. The method as recited in claim 13, further comprising performing, by the one or more computing devices:

in response to determining that (a) the fourth directory entry'"'"'s pointer modification has not succeeded and (b) the second workflow sequence is to be abandoned,

submitting a request from the replacement node to the second node to store an indication that the second workflow sequence has been cancelled; and

in response to determining, by the replacement node, that the second node has stored the indication that the second workflow sequence has been cancelled, (a) deleting the record of the intent to rename the different file store object; and

(b) deleting the lock on the third directory entry.

16. The method as recited in claim 6, further comprising performing, by the one or more computing devices:

deleting, asynchronously with respect to said third set of operations, the indication that the pointer modification succeeded.

17. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:

in response to a particular rename request to change a name of a particular file store object at a storage service from a first name to a second name,

attempt an initial atomic operation including;

(a) obtaining a lock on a first directory entry corresponding to the first name, wherein the first directory entry includes a pointer to a first node entry comprising a plurality of metadata fields pertaining to the particular file storage object; and

(b) storing a record of an intent to rename the particular file store object;

in response to a determination that the initial atomic operation succeeded, request a second atomic operation comprising;

(a) setting a second directory'"'"'s pointer to point to the first node entry; and

(b) storing an indication that a pointer modification corresponding to the particular rename request succeeded;

in response to a determination that the second atomic operation succeeded, attempt a third set of operations of the workflow sequence comprising;

(a) deleting the record of the intent;

(b) deleting the lock on the first directory entry; and

(c) deleting the first directory entry.

18. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the instructions when executed at the one or more processors:

prior to the attempt to perform the initial atomic operation, determine, using a deadlock avoidance analysis, that the initial atomic operation is to comprise obtaining the lock on the first directory entry.

19. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the instructions when executed at the one or more processors:

in response to a different rename request to change a name of a different file store object from a third name to a fourth name,

attempt another atomic operation comprising;

(a) obtaining a lock on a fourth directory entry corresponding to the fourth name; and

(b) storing a record of an intent to rename the different file store object;

in response to determining that the other atomic operation succeeded, attempt a subsequent atomic operation comprising;

(a) deleting a third directory entry corresponding to the third name, wherein the third directory entry includes a pointer to a different node entry of the different file storage object; and

(c) storing an indication that a deletion corresponding to the different rename request succeeded; and

in response to determining that the subsequent atomic operation succeeded, (a) storing a pointer to the different node entry of the different file storage object in the fourth directory entry, (b) deleting the record of the intent to rename the different file store object; and

(c) deleting the lock on the fourth directory entry.

20. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the initial atomic operation is performed at a first node of a metadata subsystem of the storage service, and wherein the second atomic operation is performed at a second node of the metadata subsystem.

21. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the instructions, when executed at the one or more processors:

initiate a deletion, asynchronously with respect to said third set of operations, of the indication that the pointer modification succeeded.