Dispersed storage network file system directory

• US 10,387,247 B2
• Filed: 05/30/2014
• Issued: 08/20/2019
• Est. Priority Date: 03/12/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Baptist, Andrew, Motwani, Manish
• Primary Examiner(s): Burke, Jeff A
• Assistant Examiner(s): Hoffler, Raheem
• Application Number:
  US14/291,199
• Publication Number:
  US 20140281815A1
• Time in Patent Office: 1,908 Days
• Field of Search: 707693
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 12/0646 Configuration or reconfiguration
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/125 characterised by the use of retention policies retention policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A dispersed storage device manages a file system directory of a dispersed storage network by receiving a data object to be stored and a user file name of the data object, calculating a data compression function of the data object, creating a file identifier based on a result of the data compression function, creating a source name for the data object using the file identifier and linking the user file name to the source name in the file system directory.

Claims

1. A method for execution by a computing device of a dispersed storage network (DSN), the method comprising:

determining, by the computing device, virtual memory use of a vault based on virtual memory use of one dispersed storage (DS) unit of a set of DS units that services the vault within the DSN, wherein data is disperse storage error encoded in accordance with an error coding scheme to produce a plurality of sets of error coded (EC) data slices that are stored in the vault, wherein the error coding scheme includes a pillar width number and a read threshold number, wherein the pillar width number is a total number of EC data slices within a set of EC data slices of the plurality of sets of EC data slices, wherein the read threshold number is a number of EC data slices of the set of EC data slices that is required to reconstruct a corresponding data segment of the data, wherein the read threshold number is less than the pillar width number, and wherein the virtual memory use of a DS unit of the set of DS units is representative of the virtual memory use of other DS units of the set of DS units for the vault due to the error coding scheme;

when the virtual memory use of the vault exceeds a vault threshold, determining, by the computing device, whether the virtual memory use of the vault can be expanded based on one or more dispersed data storage parameters; and

when the virtual memory use of the vault can be expanded;

determining, by the computing device, available physical memory for each DS unit of the set of DS units; and

when at least the read threshold number of DS units of the set of DS units has available physical memory above an availability threshold, allocating, by the computing device, at least a portion of available physical memory within the at least the read threshold number of DS units of the set of DS units for the vault to expand the virtual memory use of the vault.

2. The method of claim 1, wherein the determining virtual memory use of the vault comprising:

requesting, from the one DS unit, the virtual memory use of the one DS unit that includes a list of EC data slices stored within the one DS unit for the vault, data sizes for the listed EC data slices, and DSN addresses for the listed EC data slices, and wherein the DSN addresses at least partially include EC data slice names for the listed EC data slices; and

interpreting the virtual memory use of the one DS unit based on a predetermined logical size of the vault to determine when the virtual memory use of the vault exceeds the vault threshold.

3. The method of claim 1, wherein the one or more dispersed data storage parameters comprises one or more of:

valid user identifiers;

a predetermined logical size of the vault;

the vault threshold;

an indication of whether the predetermined logical size of the vault is fixed or can be expanded; and

a vault identifier.

4. The method of claim 1 further comprising:

when the vault cannot be expanded based on the one or more dispersed data storage parameters, performing one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault.

5. The method of claim 4, wherein the one or more mitigation operations comprising one or more of:

restricting write access to the vault;

requesting expansion of the vault;

transferring one or more sets of the plurality of sets of EC data slices to another vault; and

establishing overwriting access to the vault, wherein older EC data slices are deleted to make available memory space for new EC data slices.

6. The method of claim 1 further comprising:

when less than the read threshold number of DS units of the set of DS units has available physical memory above the availability threshold, performing one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault.

7. The method of claim 1 further comprising:

when the at least the read threshold number of DS units of the set of DS units, but fewer than all of the DS units of the set of DS units, has available physical memory above the availability threshold, requesting expansion of physical memory allocation for each DS unit of the set of DS units that does not have available physical memory above the availability threshold; and

storing, in a rebuild list, identities of EC data slices associated with write requests to the vault that cannot be currently processed by each of the DS units of the set of DS units that does not have available physical memory above the availability threshold.

8. The method of claim 1 further comprising:

when the vault cannot be expanded based on the one or more dispersed data storage parameters, prohibiting additional EC data slices to be written to the vault and allowing read only access to the plurality of sets of EC data slices.

9. A distributed storage (DS) processing unit of a dispersed storage network (DSN) comprises:

memory;

a network interface configured to support communications within the DSN; and

a processing module operably coupled to the memory and the network interface, wherein the processing module is operable to;

determine virtual memory use of a vault based on virtual memory use of one dispersed storage (DS) unit of a set of DS units that services the vault within the DSN, wherein data is disperse storage error encoded in accordance with an error coding scheme to produce a plurality of sets of error coded (EC) data slices that are stored in the vault, wherein the error coding scheme includes a pillar width number and a read threshold number, wherein the pillar width number is a total number of EC data slices within a set of EC data slices of the plurality of sets of EC data slices, wherein the read threshold number is a number of EC data slices of the set of EC data slices that is required to reconstruct a corresponding data segment of the data, wherein the read threshold number is less than the pillar width number, and wherein the virtual memory use of a DS unit of the set of DS units is representative of the virtual memory use of other DS units of the set of DS units for the vault due to the error coding scheme;

when the virtual memory use of the vault exceeds a vault threshold, determine whether the virtual memory use of the vault can be expanded based on one or more dispersed data storage parameters; and

when the virtual memory use of the vault can be expanded;

determine available physical memory for each DS unit of the set of DS units; and

when at least the read threshold number of DS units of the set of DS units has available physical memory above an availability threshold, allocate at least a portion of available physical memory within the at least the read threshold number of DS units of the set of DS units for the vault to expand the virtual memory use of the vault.

10. The DS processing unit of claim 9, wherein the processing module is further operable to:

receive information, from the one DS unit, related to the virtual memory use of the one DS unit that includes a list of EC data slices stored within the one DS unit for the vault, data sizes for the listed EC data slices, and DSN addresses for the listed EC data slices, and wherein the DSN addresses at least partially include EC data slice names for the listed EC data slices; and

interpret the virtual memory use of the one DS unit based on a predetermined logical size of the vault to determine when the virtual memory use of the vault exceeds the vault threshold.

11. The DS processing unit of claim 9, wherein the one or more dispersed data storage parameters comprises one or more of:

valid user identifiers;

a predetermined logical size of the vault;

the vault threshold;

an indication of whether the predetermined logical size of the vault is fixed or can be expanded; and

a vault identifier.

12. The DS processing unit of claim 9, wherein the processing module is further operable to:

perform one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault when the vault cannot be expanded based on the one or more vault parameters.

13. The DS processing unit of claim 12, wherein the processing module is further operable to perform the one or more mitigation operations by performing one or more of:

restrict write access to the vault;

request expansion of the vault;

transfer one or more sets of the plurality of sets of EC data slices to another vault; and

establish overwriting access to the vault, wherein older EC data slices are deleted to make available memory space for new EC data slices.

14. The DS processing unit of claim 9, wherein the processing module is further operable to:

perform one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault when less than the read threshold number of DS units of the set of DS units has available physical memory above the availability threshold.

15. The DS processing unit of claim 9, wherein the processing module is further operable to:

request expansion of physical memory allocation for each DS unit of the set of DS units that does not have available physical memory above the availability threshold when the at least a read threshold number of DS units of the set of DS units, but fewer than all of the DS units of the set of DS units, has available physical memory above the availability threshold; and

store, in a rebuild list, identities of EC data slices associated with write requests to the vault that cannot be currently processed by each of the DS units of the set of DS units that does not have available physical memory above the availability threshold.

16. A non-transitory computer readable medium having instructions causing a processing module in a distributed storage (DS) unit of a dispersed storage network (DSN) to execute a method comprising:

determining virtual memory use of a vault based on virtual memory use of one dispersed storage (DS) unit of a set of DS units that services the vault within the DSN, wherein data is disperse storage error encoded in accordance with an error coding scheme to produce a plurality of sets of error coded (EC) data slices that are stored in the vault, wherein the error coding scheme includes a pillar width number and a read threshold number, wherein the pillar width number is a total number of EC data slices within a set of EC data slices of the plurality of sets of EC data slices, wherein the read threshold number is a number of EC data slices of the set of EC data slices that is required to reconstruct a corresponding data segment of the data, wherein the read threshold number is less than the pillar width number, and wherein the virtual memory use of a DS unit of the set of DS units is representative of the virtual memory use of other DS units of the set of DS units for the vault due to the error coding scheme;

when the virtual memory use of the vault exceeds a vault threshold, determining whether the virtual memory use of the vault can be expanded based on one or more dispersed data storage parameters; and

when the virtual memory use of the vault can be expanded;

determining available physical memory for each DS unit of the set of DS units; and

when at least the read threshold number of DS units of the set of DS units has available physical memory above an availability threshold, allocating at least a portion of available physical memory within the at least the read threshold number of DS units of the set of DS units for the vault to expand the virtual memory use of the vault.

17. The non-transitory computer readable medium having instructions causing the processing module to execute the method of claim 16, wherein the determining virtual memory use of the vault comprising:

requesting, from the one DS unit, the virtual memory use of the one DS unit that includes a list of EC data slices stored within the one DS unit for the vault, data sizes for the listed EC data slices, and DSN addresses for the listed EC data slices, and wherein the DSN addresses at least partially include EC data slice names for the listed EC data slices; and

interpreting the virtual memory use of the one DS unit based on a predetermined logical size of the vault to determine when the virtual memory use of the vault exceeds the vault threshold.

18. The non-transitory computer readable medium having instructions causing the processing module to execute the method of claim 16, wherein the one or more dispersed data storage parameters comprises one or more of:

valid user identifiers;

a predetermined logical size of the vault;

the vault threshold;

an indication of whether the predetermined logical size of the vault is fixed or can be expanded; and

a vault identifier.

19. The non-transitory computer readable medium having instructions causing the processing module to execute the method of claim 16 further comprising:

when the vault cannot be expanded based on the one or more dispersed data storage parameters, performing one or more mitigation operations to the vault so that the virtual memory use of the vault does not exceed a predetermined logical size of the vault.

20. The non-transitory computer readable medium having instructions causing the processing module to execute the method of claim 19, wherein the one or more mitigation operations comprising one or more of:

restricting write access to the vault;

requesting expansion of the vault;

transferring one or more sets of the plurality of sets of EC data slices to another vault; and

establishing overwriting access to the vault, wherein older EC data slices are deleted to make available memory space for new EC data slices.