Storage pools for a dispersed storage network

• US 9,774,679 B2
• Filed: 07/11/2014
• Issued: 09/26/2017
• Est. Priority Date: 08/29/2013
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Storm, Michael Colin, Motwani, Manish, Leggette, Wesley
• Primary Examiner(s): Chery, Mardochee
• Application Number:
  US14/328,919
• Publication Number:
  US 20150067295A1
• Time in Patent Office: 1,173 Days
• Field of Search: 711171
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module selecting storage pools within the DSN with available capacity for storing data of a storage group. The method continues by selecting one or more dispersed storage (DS) units within each of the selected storage pools based on a selection criteria and mapping the one or more DS units to the storage group. The method continues by receiving a write request to store a data object to the storage group and by storing the data object in at least one of the mapped one or more DS units. The method continues with the DS processing module issuing an indication unutilized storage space calculated on a proportionate basis based on storage utilized for the storage group as a percentage of total storage utilized and updating a write proportion value based on received storage utilization responses.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

querying a plurality of storage pools within the DSN regarding available storage capacity, wherein a storage pool is a logical collection of distributed storage units (SUs);

receiving, from at least one SU from each of the storage pools of the plurality of storage pools, information regarding available storage capacity;

determining a selection criteria for one or more storage groups based on the information regarding available storage capacity;

selecting one or more dispersed storage (DS) units within each of the storage pools of the plurality of storage pools based on a selection criteria;

mapping the one or more DS units of each selected storage pool to a storage group;

receiving a write request to store a data object;

identifying a set of storage units of the storage group for storing the data object; and

storing the data object as a plurality of sets of encoded data slices in the set of storage units.

2. The method of claim 1, wherein the selection criteria includes one or more of:

metadata for the one or more DS units;

information associated with the storage group including one or more of;

data type;

capacity requirements;

access frequency;

data storage retrieval formats;

data retrieval sequencing;

required speed; and

number of users within storage group; and

information associated with the one or more DS units including one or more of;

type of memory;

error rates;

age of memory; and

maintenance history of memory.

3. The method of claim 1 further comprises:

selecting the storage pools in proportion to an amount of free space for each storage group.

4. The method of claim 1 further comprises:

issuing storage utilization queries to two or more of the storage pools;

issuing, by each of the one or more DS units, a storage utilization response, where the response includes an indication of one or more of unutilized storage space for the storage group, utilized storage for the storage group, where unutilized storage space is calculated on a proportionate basis based on storage utilized for the storage group as a percentage of total storage utilized;

updating a write proportion value based on received storage utilization responses; and

utilizing the write proportion value to proportionally allocate storage to the storage pools for future write requests for the storage group.

5. The method of claim 1 further comprises:

for storage group n, determining, by the one or more DS units, unutilized storage in accordance with formula;

unutilized for the storage group=unutilized for all storage groups*(utilized for the storage group/utilized for all storage groups).

6. The method of claim 1, wherein the storage group includes one or more of:

a vault, a portion of a vault and a vault region.

7. A method for selecting a storage pool by a dispersed storage (DS) processing module of a dispersed storage network (DSN), the method comprises:

issuing storage utilization queries to two or more DSN storage pools, wherein each storage pool is a logical collection of distributed storage units (SUs);

issuing, by a DS unit of each of the two or more DSN storage pools, a storage utilization response;

updating a write proportion value for one or more storage groups based on received storage utilization responses;

mapping the one or more DS units of each of the two or more DSN storage pools to a storage group;

receiving a write request to store a data object in the DSN;

identifying a storage group of the one or more storage groups, where the storage group is associated with the write request;

selecting a first DSN storage pool of the two or more DSN storage pools based on the write proportion value and a recent write pattern; and

facilitating storing a portion of the data object as a plurality of sets of encoded data slices in the first DSN storage pool, wherein the remainder of the data object will be stored in a second DSN storage pool.

8. The method of claim 7, wherein the storage group includes one or more of:

a vault, a portion of a vault and a vault region.

9. The method of claim 7 further comprises:

for storage group n, determining, by the DS unit, unutilized storage in accordance with formula;

unutilized for a storage group=unutilized for all storage groups*(utilized for the storage group/utilized for all storage groups).

10. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

select storage pools within a dispersed storage network (DSN) having available data capacity, wherein a storage pool is a logical collection of distributed storage units (SUs);

a second module, when operable within a computing device, causes the computing device to;

select one or more dispersed storage (DS) units within each of the selected storage pools based on a selection criteria;

a third module, when operable within a computing device, causes the computing device to;

map the selected one or more DS units of each selected storage pool to a storage group;

a fourth module, when operable within a computing device, causes the computing device to;

receive a write request to store a data object;

a fifth module, when operable within a computing device, causes the computing device to;

identify a set of storage units of the storage group for storing the data object; and

a sixth module, when operable within a computing device, causes the computing device to;

store the data object as a plurality of sets of encoded data slices in the set of storage units.

11. The dispersed storage (DS) module of claim 10, wherein the selection criteria includes metadata for the one or more DS units.

12. The dispersed storage (DS) module of claim 10 further comprises a sixth module, when operable within a computing device, causes the computing device to:

select storage to the storage pools in proportion to an amount of free space for each storage group.

13. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

issue storage utilization queries to two or more DSN storage pools, wherein a storage pool is a logical collection of distributed storage units (SUs);

a second module, when operable within the computing device, causes the computing device to;

issue, by each DS unit, a storage utilization response, where the response includes an indication of one or more of unutilized storage space for storage group, utilize storage for storage group, where unutilized storage space is calculated on a proportionate basis based on storage utilized for an associated storage group as a percentage of total storage utilized; and

a third module, when operable within the computing device, causes the computing device to;

update a write proportion value based on received storage utilization responses; and

a fourth module, when operable within the computing device, causes the computing device to;

utilize the write proportion value to proportionally allocate storage to the two or more DSN storage pools for future write requests for the storage group.

14. The dispersed storage (DS) module of claim 13, wherein the second module calculates the proportionate basis by selecting a given storage pool in proportion to an amount of free space for each storage group.

15. The dispersed storage (DS) module of claim 13 further comprises a fifth module, when operable within the computing device, causes the computing device to receive a write request to store a data object in the DSN.

16. The dispersed storage (DS) module of claim 15 further comprises a sixth module, when operable within the computing device, causes the computing device to identify a storage group based on the write request.

17. The dispersed storage (DS) module of claim 16, wherein the sixth module identifies a storage group based on one or more of:

data type, requesting entity and vault ID.

18. The dispersed storage (DS) module of claim 17 further comprises a seventh module, when operable within the computing device, causes the computing device to:

select a DSN storage pool based on the write proportion value and a recent write pattern for the storage pool.

19. The dispersed storage (DS) module of claim 18, wherein the seventh module is further configured to:

identify the recent write pattern;

select the storage in accordance with the write pattern so that the storage pool is selected at the write proportion value for the storage group overall writes; and

verify unutilized space is greater than space required to facilitate storing the data object.

20. The dispersed storage (DS) module of claim 13 further comprises an eighth module, when operable within the computing device, causes the computing device to:

facilitate storing a data object as a plurality of sets of encoded data slices in a selected DSN storage pool.