Dispersed storage network file system directory

• US 20140281815A1
• Filed: 05/30/2014
• Published: 09/18/2014
• Est. Priority Date: 03/12/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Baptist, Andrew, Motwani, Manish
• Granted Patent: US 10,387,247 B2
• Time in Patent Office: Days
• US Class Current: 714/766
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 12/0646 Configuration or reconfiguration
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/125 characterised by the use of retention policies retention policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A dispersed storage device manages a file system directory of a dispersed storage network by receiving a data object to be stored and a user file name of the data object, calculating a data compression function of the data object, creating a file identifier based on a result of the data compression function, creating a source name for the data object using the file identifier and linking the user file name to the source name in the file system directory.

Claims

1. A method for execution by a processing module in a distributed storage (DS) unit, the method comprising:

determining virtual memory use of a vault based on virtual memory use of one DS unit of a set of DS units that services the vault within a dispersed storage network (DSN), wherein each DS unit of the set of DS units includes allocated physical memory to the virtual memory of the vault, wherein each DS unit of the set of DS units is configured to store one or more error coded (EC) data slices for each set of a plurality of sets of EC data slices, and wherein a data segment is dispersed error encoded to produce a set of EC data slices of the plurality of sets of EC data slices;

when the virtual memory use of the vault is determined to exceed a vault threshold, determining whether the virtual memory use of the vault can be expanded based on one or more dispersed data storage parameters; and

when the virtual memory use of the vault can be expanded;

determining available physical memory for each DS unit of the set of DS units; and

when at least a decode threshold number of DS units of the set of DS units has available physical memory above an availability threshold, allocating at least a portion of available physical memory within the at least a decode threshold number of DS units of the set of DS units for the vault to expand the virtual memory use of the vault, wherein the decode threshold number corresponds to a minimum number of EC data slices required to reconstruct the data segment.

2. The method of claim 1, wherein the determining virtual memory use of the vault comprising:

requesting, from the one DS unit, the virtual memory use of the one DS unit that includes a list of EC data slices stored within the one DS unit for the vault, data sizes for the listed EC data slices, and DSN addresses for the listed EC data slices, and wherein the DSN addresses at least partially include EC data slice names for the listed EC data slices; and

interpreting the virtual memory use of the one DS unit based on a predetermined logical size of the vault to determine when the virtual memory use of the vault exceeds the vault threshold.

3. The method of claim 1, wherein the one or more dispersed data storage parameters comprises one or more of:

valid user identifiers;

a predetermined logical size of the vault;

the vault threshold;

an indication of whether the predetermined logical size of the vault is fixed or can be expanded; and

a vault identifier.

4. The method of claim 1 further comprising:

when the vault cannot be expanded based on the one or more dispersed data storage parameters, performing one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault.

5. The method of claim 4, wherein the one or more mitigation operations comprising one or more of:

restricting write access to the vault;

requesting expansion of the vault;

transferring one or more sets of the plurality of sets of EC data slices to another vault; and

establishing overwriting access to the vault, wherein older EC data slices are deleted to make available memory space for new EC data slices.

6. The method of claim 1 further comprising:

when less than the decode threshold number of DS units of the set of DS units has available physical memory above the availability threshold, performing one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault.

7. The method of claim 1 further comprising:

when the at least a decode threshold number of DS units of the set of DS units, but fewer than all of the DS units of the set of DS units, has available physical memory above the availability threshold, requesting expansion of physical memory allocation for each DS unit of the set of DS units that does not have available physical memory above the availability threshold; and

storing, in a rebuild list, identify of EC data slices associated with write requests to the vault that cannot be currently processes by each of the DS units of the set of DS units that does not have available physical memory above the availability threshold.

8. The method of claim 1 further comprising:

when the vault cannot be expanded based on the one or more dispersed data storage parameters, prohibiting additional EC data slices to be written to the vault and allowing read only access to the plurality of sets of EC data slices.

9. A distributed storage (DS) processing unit comprises:

a network interface configured to support communications within a dispersed storage network (DSN); and

a processing module configured to;

determine virtual memory use of a vault based on virtual memory use of one DS unit of a set of DS units that services the vault within the DSN, wherein each DS unit of the set of DS units includes allocated physical memory to the virtual memory of the vault, wherein each DS unit of the set of DS units is configured to store one or more error coded (EC) data slices for each set of a plurality of sets of EC data slices, and wherein a data segment is dispersed error encoded to produce a set of EC data slices of the plurality of sets of EC data slices;

when the virtual memory use of the vault is determined to exceed a vault threshold, determine whether the virtual memory use of the vault can be expanded based on one or more dispersed data storage parameters; and

when the virtual memory use of the vault can be expanded;

determine available physical memory for each DS unit of the set of DS units; and

when at least a decode threshold number of DS units of the set of DS units has available physical memory above an availability threshold, allocate at least a portion of available physical memory within the at least a decode threshold number of DS units of the set of DS units for the vault to expand the virtual memory use of the vault, wherein the decode threshold number corresponds to a minimum number of EC data slices required to reconstruct the data segment.

10. The DS processing unit of claim 9, wherein the processing module is further configured to:

receive information, from the one DS unit, related to the virtual memory use of the one DS unit that includes a list of EC data slices stored within the one DS unit for the vault, data sizes for the listed EC data slices, and DSN addresses for the listed EC data slices, and wherein the DSN addresses at least partially include EC data slice names for the listed EC data slices; and

interpret the virtual memory use of the one DS unit based on a predetermined logical size of the vault to determine when the virtual memory use of the vault exceeds the vault threshold.

11. The DS processing unit of claim 9, wherein the one or more dispersed data storage parameters comprises one or more of:

valid user identifiers;

a predetermined logical size of the vault;

the vault threshold;

an indication of whether the predetermined logical size of the vault is fixed or can be expanded; and

a vault identifier.

12. The DS processing unit of claim 9, wherein the processing module is further configured to:

perform one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault when the vault cannot be expanded based on the one or more dispersed data storage parameters.

13. The DS processing unit of claim 12, wherein the processing module is further configured to perform the one or more mitigation operations by performing one or more of:

restrict write access to the vault;

request expansion of the vault;

transfer one or more sets of the plurality of sets of EC data slices to another vault; and

establish overwriting access to the vault, wherein older EC data slices are deleted to make available memory space for new EC data slices.

14. The DS processing unit of claim 9, wherein the processing module is further configured to:

perform one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault when less than the decode threshold number of DS units of the set of DS units has available physical memory above the availability threshold.

15. The DS processing unit of claim 9, wherein the processing module is further configured to:

request expansion of physical memory allocation for each DS unit of the set of DS units that does not have available physical memory above the availability threshold when the at least a decode threshold number of DS units of the set of DS units, but fewer than all of the DS units of the set of DS units, has available physical memory above the availability threshold; and

store, in a rebuild list, identify of EC data slices associated with write requests to the vault that cannot be currently processes by each of the DS units of the set of DS units that does not have available physical memory above the availability threshold.

16. A computer readable medium having instructions causing a processing module in a distributed storage (DS) unit to execute a method comprising:

determining virtual memory use of a vault based on virtual memory use of one DS unit of a set of DS units that services the vault within a dispersed storage network (DSN), wherein each DS unit of the set of DS units includes allocated physical memory to the virtual memory of the vault, wherein each DS unit of the set of DS units is configured to store one or more error coded (EC) data slices for each set of a plurality of sets of EC data slices, and wherein a data segment is dispersed error encoded to produce a set of EC data slices of the plurality of sets of EC data slices;

when the virtual memory use of the vault is determined to exceed a vault threshold, determining whether the virtual memory use of the vault can be expanded based on one or more dispersed data storage parameters; and

when the virtual memory use of the vault can be expanded;

determining available physical memory for each DS unit of the set of DS units; and

when at least a decode threshold number of DS units of the set of DS units has available physical memory above an availability threshold, allocating at least a portion of available physical memory within the at least a decode threshold number of DS units of the set of DS units for the vault to expand the virtual memory use of the vault, wherein the decode threshold number corresponds to a minimum number of EC data slices required to reconstruct the data segment.

17. The computer readable medium having instructions causing the processing module to execute the method of claim 16, wherein the determining virtual memory use of the vault comprising:

requesting, from the one DS unit, the virtual memory use of the one DS unit that includes a list of EC data slices stored within the one DS unit for the vault, data sizes for the listed EC data slices, and DSN addresses for the listed EC data slices, and wherein the DSN addresses at least partially include EC data slice names for the listed EC data slices; and

interpreting the virtual memory use of the one DS unit based on a predetermined logical size of the vault to determine when the virtual memory use of the vault exceeds the vault threshold.

18. The computer readable medium having instructions causing the processing module to execute the method of claim 16, wherein the one or more dispersed data storage parameters comprises one or more of:

valid user identifiers;

a predetermined logical size of the vault;

the vault threshold;

an indication of whether the predetermined logical size of the vault is fixed or can be expanded; and

a vault identifier.

19. The computer readable medium having instructions causing the processing module to execute the method of claim 16 further comprising:

when the vault cannot be expanded based on the one or more dispersed data storage parameters, performing one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault.

20. The computer readable medium having instructions causing the processing module to execute the method of claim 19, wherein the one or more mitigation operations comprising one or more of:

restricting write access to the vault;

requesting expansion of the vault;

transferring one or more sets of the plurality of sets of EC data slices to another vault; and

establishing overwriting access to the vault, wherein older EC data slices are deleted to make available memory space for new EC data slices.