Storing directory metadata in a dispersed storage network

• US 9,037,904 B2
• Filed: 09/08/2014
• Issued: 05/19/2015
• Est. Priority Date: 12/22/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Volvovski, Ilya, Baptist, Andrew, Leggette, Wesley
• Primary Examiner(s): PATEL, KAMINI B
• Application Number:
  US14/479,905
• Publication Number:
  US 20150006996A1
• Time in Patent Office: 253 Days
• Field of Search: 714/6.24
• US Class Current: 714/6.24
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 12/1009 using page tables, e.g. page table structures
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a processing module dispersed storage error encoding data to produce encoded data slices and updating directory metadata regarding storing the data in a dispersed storage network (DSN) memory to produce updated directory metadata. The method continues with the processing module dispersed storage error encoding the updated directory metadata to produce encoded directory metadata slices and transmitting one or more data slice write requests to the DSN that each include a first transaction number. The method continues with the processing module transmitting one or more directory metadata write requests to the DSN memory, wherein the one or more directory metadata write requests includes a second transaction number, and when a favorable write response condition exists, transmitting a commit request to the DSN memory to commit storage of at least one of: the encoded data slices and the encoded directory metadata slices.

Claims

1. A method for updating directory information in a distributed storage network (DSN) memory, comprises:

dispersed storage error encoding data to produce a plurality of sets of encoded data slices;

transmitting the plurality of sets of encoded data slices to the DSN memory for storing the plurality of sets of encoded data slices;

determining to update directory metadata stored in a local cache memory in response to storing the plurality of sets of encoded data slices in the DSN memory;

updating the directory metadata stored in the local cache memory;

determining to synchronize the directory metadata stored in the local cache memory with directory information stored in the DSN memory;

in response to determining to synchronize the directory metadata, dispersed storage error encoding the directory metadata to produce a plurality of sets of encoded directory metadata slices; and

transmitting the plurality of sets of encoded directory metadata slices to the DSN memory for storing the plurality of sets of encoded directory metadata slices with the directory information stored in the DSN memory.

2. The method of claim 1, wherein transmitting the plurality of sets of encoded data slices to the DSN memory for storing the plurality of sets of encoded data slices includes:

transmitting one or more data slice write requests to the DSN memory for storing the plurality of sets of encoded data slices in the DSN memory, wherein the one or more data slice write requests includes a first transaction number.

3. The method of claim 2, wherein transmitting the plurality of sets of encoded directory metadata slices to the DSN memory for storing the plurality of sets of encoded directory metadata slices includes:

selecting a portion of the directory metadata to produce the plurality of sets of encoded directory metadata slices, wherein the portion of the directory metadata is selected based on a change indicator indicating a last update to the directory information in the DSN memory; and

transmitting one or more directory metadata write requests to the DSN memory for storing the plurality of sets of encoded directory metadata slices in the DSN memory, wherein the one or more directory metadata write requests includes a second transaction number.

4. The method of claim 3, wherein the one or more directory metadata write requests further includes a revision number corresponding to the portion of the directory metadata being stored and an expected slice revision number corresponding to a last revision number of the directory metadata stored in the local cache memory.

5. The method of claim 4, further comprising:

when a favorable write response condition exists for the one or more data slice write requests and the one or more directory metadata write requests, transmitting one or more common commit requests to the DSN memory to commit storage of the plurality of sets of encoded data slices and the plurality of sets of encoded directory metadata slices, wherein the one or more common commit requests includes the first transaction number and the second transaction number.

6. The method of claim 1, wherein updating directory metadata stored in the local cache memory to produce updated directory metadata comprises:

retrieving the directory metadata from at least one of the DSN memory and a local memory;

determining a DSN memory storage location information for the plurality of sets of encoded data slices, wherein the DSN memory storage location information includes one or more of;

a source name corresponding to a DSN memory storage location, a data size indicator, a data type indicator, snapshot information, a timestamp; and

a segment allocation table; and

modifying the directory metadata to include a filename associated with the data and the DSN memory storage location to produce the updated directory metadata.

7. The method of claim 1, wherein determining to synchronize the directory metadata stored in the local cache memory with directory information stored in the DSN memory is based on one or more of the following:

a time interval between a last update and a time threshold;

an indicator requiring synchronization after a store operation;

a data type indicator;

a frequency of change indicator;

a system activity indicator;

a timestamp of a previous synchronization message;

or

a command.

8. A dispersed storage (DS) device, comprises:

at least one local memory, wherein the local memory stores directory metadata; and

one or more processing modules that include at least one processing device, operable to;

retrieve directory metadata from the local memory and directory information from a dispersed storage network (DSN) memory;

determine directory metadata updates to the directory metadata in the local memory based on the directory information from the DSN memory;

update the directory metadata in the local memory using the directory metadata updates;

determine to synchronize the directory metadata in the local memory with the directory information in the DSN memory;

dispersed storage error encode a portion of the directory metadata to produce a plurality of sets of encoded directory metadata slices; and

transmit the plurality of sets of encoded directory metadata slices to the DSN memory for storing the plurality of sets of encoded directory metadata slices with the directory information stored in the DSN memory.

9. The DS device of claim 8, wherein the one or more processing modules are operable to:

determine synchronization status information; and

determine to synchronize the directory metadata in the local memory with the directory information in the DSN memory based on the synchronization status information.

10. The DS device of claim 9, wherein the synchronization status information is based on at least one of:

a timer;

a number of previous failed synchronization attempts;

time elapsed since a last attempt to synchronize;

one or more historical synchronization attempts;

an update frequency indicator;

a number of users authorized to access a common stored vault;

or a service level agreement.

11. The DS device of claim 8, wherein the one or more processing modules are operable to select a portion of the directory metadata to produce the plurality of sets of encoded directory metadata slices based on a change indicator indicating a last update to the directory information in the DSN memory.

12. The DS device of claim 8, wherein the one or more processing modules are operable to transmit the plurality of sets of encoded directory metadata slices to the DSN memory for storing the plurality of sets of encoded directory metadata slices with the directory information stored in the DSN memory by:

transmitting one or more directory metadata write requests to the DSN memory for storing the plurality of sets of encoded directory metadata slices in the DSN memory, wherein the one or more directory metadata write requests includes a transaction number.

13. The DS device of claim 12, wherein the one or more directory metadata write requests further include a revision number corresponding to the portion of the directory metadata being stored and an expected slice revision number corresponding to a last revision number of the directory metadata stored in the local memory.

14. A dispersed storage (DS) device, comprises:

at least one memory;

one or more processing modules that include at least one processing device, operable to;

receive an add snapshot request, wherein the add snapshot request includes information to identify a child snapshot, an associated parent snapshot and a snapshot file;

retrieve a plurality of encoded snapshot file slices from dispersed a storage network (DSN) memory;

dispersed storage error decode the plurality of encoded snapshot file slices to recover the snapshot file, wherein the snapshot file indicates files authorized for access; and

modify the snapshot file to associate the child snapshot with the associated parent snapshot and create a timestamp indicating a time of modification.

15. The DS device of claim 14, wherein the one or more processing modules are further operable to:

recover a parent snapshot directory file associated with the parent snapshot from the DSN memory, wherein the parent snapshot directory file includes a plurality of parent file names;

create a child snapshot directory file associated with the child snapshot to include the plurality of parent file names included in the parent snapshot directory file at the time of modification indicated by the timestamp;

dispersed error encode the child snapshot directory file to produce a plurality of sets of encoded child snapshot directory file slices; and

transmit the plurality of sets of child snapshot directory file slices to the DSN memory for storing in the DSN memory.

16. The DS device of claim 15, wherein the one or more processing modules are further operable to:

receive an add directory entry request, wherein the add directory entry request includes information to identify the child snapshot directory file and a new file name;

recover the child snapshot directory file from the DSN memory;

modify the child snapshot directory file to include the new file name;

disperse error encode the modified child snapshot directory file to produce a plurality of sets of encoded modified child snapshot directory file slices; and

transmit the plurality of sets of modified child snapshot directory file slices to the DSN memory for storing in the DSN memory.

17. The DS device of claim 16, wherein the one or more processing modules do not update the parent snapshot directory file with the new file name included in the child snapshot directory file.

18. The DS device of claim 15, wherein the one or more processing modules are further operable to:

receive a delete directory entry request, wherein the delete directory entry request includes information to identify the child snapshot directory file and a current file name for deletion;

recover the child snapshot directory file from the DSN memory;

modify the child snapshot directory file to disassociate the current file name for deletion from the child snapshot directory file;

disperse error encoding the modified child snapshot directory file to produce a plurality of sets of modified encoded child snapshot directory file slices; and

transmit the plurality of sets of modified child snapshot directory file slices to the DSN memory for storing in the DSN memory.

19. The DS device of claim 18, wherein the current file name for deletion from the child snapshot directory file is included in the plurality of parent file names and wherein the current file name from deletion remains in the parent snapshot directory file after deletion from the child snapshot directory file.