Optimized disk load distribution

• US 9,959,076 B2
• Filed: 05/30/2014
• Issued: 05/01/2018
• Est. Priority Date: 10/09/2007
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K.
• Primary Examiner(s): MIRZA, ADNAN M
• Application Number:
  US14/292,206
• Publication Number:
  US 20140281223A1
• Time in Patent Office: 1,432 Days
• Field of Search: 709203
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 21/6272 by registering files or documents with a third party
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 2221/2107 File encryption
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/085 Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/0894 Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage

Abstract

A method in a distributed storage (DS) unit begins by the unit receiving one or more data slices to store. The method continues by determining a memory utilization method for the one or more data slices. The method continues when a multiple memory utilization method is determined. The method continues by determining a data slice distribution method. The method continues by storing a plurality of the one or more data slices in the multiple memories using the data slice distribution method and updating a virtual address to physical location table to reflect the storing.

Claims

1. A method performed in a distributed storage (DS) unit, the method comprises:

receiving one or more dispersed error encoded data slices, wherein the one or more dispersed error encoded data slices represent a first data object, and further wherein a data object is segmented into a plurality of data segments, and wherein each data segment is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of dispersed error encoded data slices;

determining a memory utilization method for the one or more dispersed error encoded data slices, wherein the memory utilization method is based on attributes of the first data object and wherein the memory utilization method is configured to facilitate coordination by the DS unit with other DS units storing data objects similar to the first data object in a distributed storage network (DSN);

determining whether a plurality of memory devices in the DS unit will be used to store the one or more dispersed error encoded data slices;

when a plurality of memory devices in the DS unit are to be used to store the one or more dispersed error encoded data slices, determining a dispersed error encoded data slice distribution method;

storing a plurality of the one or more dispersed error encoded data slices in the plurality of memory devices in the DS unit using the dispersed error encoded data slice distribution method; and

updating a virtual address to physical location table to reflect the storing.

2. The method of claim 1, wherein the determining a memory utilization method further comprises:

memory selection criteria optimizing retrieval of the one or more dispersed error encoded data slices, the memory selection criteria including one or more of;

number of dispersed error encoded data slices;

slice sizes;

metadata associated with the dispersed error encoded data slices;

type of data indicator;

priority indicator;

available memory;

memory performance data; and

memory cost data.

3. The method of claim 1 further comprises:

the dispersed error encoded data slice distribution method including one or more of;

evenly distributing the one or more dispersed error encoded data slices across the plurality of memory devices in the DS unit; and

varying the distribution of the one or more dispersed error encoded data slices across the plurality of memory devices in the DS unit; and

mapping data slices associated with contiguous dispersed error encoded data slice name ranges to each of the plurality of memory devices in the DS unit.

4. The method of claim 1 further comprises:

the determining a memory utilization method for the one or more dispersed error encoded data slices including one or more of;

when a number of the one or more dispersed error encoded data slices is below a threshold, selecting a specific memory to store all of the received one or more dispersed error encoded data slices; and

when a number of the one or more dispersed error encoded data slices is above a threshold, selecting multiple memories to distribute the received one or more dispersed error encoded data slices; and

selecting other DS unit memory utilization methods for similar dispersed error encoded data slices.

5. The method of claim 1 further comprises:

the virtual address to physical location table maintaining a local virtual distributed storage network (DSN) address to physical location table to keep track of locations of a stored one or more dispersed error encoded data slices of the plurality of the one or more dispersed error encoded data slices.

6. The method of claim 1, wherein the determining a memory utilization method further comprises:

when the memory utilization method determines only one memory is needed, the dispersed error encoded data slices are stored in the one memory and the virtual address to a physical location table is updated to reflect the storing.

7. A distributed storage (DS) unit comprises:

an interface operable to receive a request to store one or more dispersed error encoded data slices from a distributed storage network (DSN), wherein a stored one or more slices of the plurality of the one or more dispersed error encoded data slices are encoded using a dispersed error coding function to produce dispersed error encoded data slices, wherein a threshold number of the dispersed error encoded data slices is sufficient to reconstruct the data segment;

a plurality of physical memories; and

a storage unit control module operable to;

receive one or more dispersed error encoded data slices, wherein the one or more dispersed error encoded data slices represent a first data object, and further wherein a data object is segmented into a plurality of data segments, and wherein each data segment is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of dispersed error encoded data slices;

determine a memory utilization method for the one or more dispersed error encoded data slices, wherein the memory utilization method is based on attributes of the first data object and wherein the memory utilization method is configured to facilitate coordination by the DS unit with other DS units storing data objects similar to the first data object in a distributed storage network (DSN);

determine whether a plurality of memory devices in the DS unit will be used to store the one or more dispersed error encoded data slices;

when a plurality of memory devices in the DS unit will be used to store the one or more dispersed error encoded data slices, determine a dispersed error encoded data slice distribution method;

store a plurality of the one or more dispersed error encoded data slices in the plurality of memory devices in the DS unit using the dispersed error encoded data slice distribution method; and

update a virtual DSN address to physical location table.

8. The distributed storage unit of claim 7, wherein one or more dispersed error encoded data slices are part of a batch of dispersed error encoded data slices including one or more of:

dispersed error encoded data slices of a same pillar;

dispersed error encoded data slices of a same data segment;

dispersed error encoded data slices of the same pillar and data segment.

9. The distributed storage unit of claim 8, wherein the storage unit control module is further operable to:

count a number of dispersed error encoded data slices to determine the number of dispersed error encoded data slices in a batch and select one memory of the plurality of memory devices in the DS unit or more than one memory of the plurality of memory devices in the DS unit based in part on the number of dispersed error encoded data slices in the batch.

10. The storage unit control module of claim 7, wherein the storage unit control module is further operable to:

coordinate with other DS units to facilitate a common storage approach including one or more of a similar;

memory utilization method and a dispersed error encoded data slice distribution method.

11. The distributed storage unit of claim 7, wherein the one or more dispersed error encoded data slices include metadata indicating one or more of:

a priority;

data type;

user ID;

security level;

speed of retrieval requirement;

a performance requirement;

a reliability requirement; and

a cost requirement.

12. The distributed storage unit of claim 7, wherein the storage unit control module is further configured to determine the memory utilization method using memory selection criteria optimizing retrieval of the one or more dispersed error encoded data slices, the memory selection criteria including one or more of:

number of dispersed error encoded data slices;

dispersed error encoded data slice sizes;

metadata associated with the dispersed error encoded data slices;

type of data indicator;

priority indicator;

available memory;

memory performance data; and

memory cost data.

13. The storage unit control module of claim 7, wherein the storage unit control module is further configured to determine the memory utilization method by selecting one memory device of the plurality of memory devices when a number of dispersed error encoded data slices in a batch is below a threshold and more than one memory device of the plurality of memory devices when the number of dispersed error encoded data slices in the batch is above a threshold.

14. The distributed storage unit of claim 7, wherein the storage unit control module is further configured to determine the dispersed error encoded data slice distribution method by selecting a number of memory devices of the plurality of memory devices based on one or more of:

a number of dispersed error encoded data slices in a slice batch;

a dispersed error encoded data slice size;

a priority;

performance requirements; and

a memory performance data.

15. The distributed storage unit of claim 7, wherein the dispersed error encoded data slice distribution method selects a higher number of memories when performance requirements are more demanding.

16. The distributed storage unit of claim 15, wherein the performance requirements include one or more of:

faster retrieval time as compared to an average required retrieval time.

17. The distributed storage unit of claim 7, wherein the dispersed error encoded data slice distribution method selects from either an uneven distribution or even distribution of the dispersed error encoded data slices between the plurality of memory devices in the DS unit based on performance data of each the plurality of memory devices in the DS unit.

18. The distributed storage unit of claim 7, wherein dispersed error encoded data slices are received from one or more of:

a distributed storage (DS) processing unit;

a storage integrity processing unit;

a user device; and

a DS managing unit.

19. A method performed in a distributed storage (DS) system, the method comprises:

receiving a request for storage of a first data object in one or more memories of a first DS unit, wherein a data object is segmented into a plurality of data segments, and wherein each data segment is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of dispersed error encoded data slices;

determining a memory utilization method used for storing a similar second data object within a second DS unit, wherein the memory utilization method is based on attributes of the first data object and wherein the memory utilization method is configured to facilitate coordination by the DS unit with other DS units storing data objects similar to the first data object in a distributed storage network (DSN);

storing a plurality of dispersed error encoded data slices of the first data object in memories within the first DS unit using the memory utilization method; and

updating a virtual address to physical location table to reflect the storing.

20. The method of claim 19, wherein the determining a memory utilization method includes criteria for selecting common memory utilization strategies including one or more of:

response times;

retrieval frequencies;

R/W error rates;

historical performance;

maintenance histories; and

memory longevity.