Method for managing throughput in a distributed storage network

• US 10,104,168 B2
• Filed: 09/30/2016
• Issued: 10/16/2018
• Est. Priority Date: 12/12/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Dhuse, Greg R., Leggette, Wesley B., Resch, Jason K.
• Primary Examiner(s): Blair, April Y
• Assistant Examiner(s): GANDHI, DIPAKKUMAR B
• Application Number:
  US15/281,430
• Publication Number:
  US 20170019466A1
• Time in Patent Office: 746 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 2463/061 applying further key derivation, e.g. deriving traffic keys from a pair-wise master key cryptographic mechanisms or cryptographic arrangements for generation of secret information including derivation or calculation of cryptographic keys or passwords H04L9/0861
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/0825 using asymmetric-key encryption or public key infrastructure [PKI], e.g. key signature or public key certificates
• CPC Class Codes: H04L 9/085 Secret sharing or secret splitting, e.g. threshold schemes

Abstract

A method for managing throughput in a distributed storage network includes encoding data to produce a plurality of sets of encoded data slices. According to the method, one or more write slice requests are generated corresponding to one or more sets of encoded data slices, and write slice requests are then output to a set of distributed storage and task execution units. For each distributed storage and task execution unit, a data ingest rate is generated and a write threshold number of distributed storage and task execution units is determined. A transmit data rate is determined and write slice requests are determined, followed by the generation write slice requests to the distributed storage and task execution units. A write threshold number of write slice requests is then input to the distributed storage and task execution units.

Claims

1. A method for execution by one or more processing modules of one or more computing devices, the method comprises:

encoding a data object to produce a plurality of sets of encoded data slices, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of encoded data slices of the plurality of sets of encoded data slices, wherein a write threshold number of encoded data slices provide for successful storage of the set of encoded data slices;

generating one or more write slice requests, wherein each of the write slice requests correspond to one or more sets of encoded data slices of the plurality of sets of encoded data slices;

outputting the one or more write slice requests to a set of distributed storage and task execution units;

for each distributed storage and task execution unit of the set of distributed storage and task execution units, determining a data ingest rate of a set of data ingest rates;

determining a write threshold number of distributed storage and task execution units of the set of distributed storage and task execution units based on the set of data ingest rates;

determining a transmit data rate;

generating a write threshold number of write slice requests that is based on the write threshold number of encoded data slices; and

outputting, in accordance with the transmit data rate, the write threshold number of write slice requests to the write threshold number of distributed storage and task execution units of the set of distributed storage and task execution units.

2. The method of claim 1 further comprises:

facilitating rebuilding the encoded data slices of the one or more sets of encoded data slices not included in the write threshold number of write slice requests to the distributed storage and task execution units not selected as part of the write threshold number of distributed storage and task execution units.

3. The method of claim 1, wherein the determining a data ingest rate of a set of data ingest rates is based on at least one of an estimated reliability level for data storage, a reliability level threshold, an estimated access speed, a lowest access speed of the write threshold number of distributed storage and task execution units, a speed threshold, an access capability estimator, a predetermination, an estimated rebuilding impact, or a rebuilding impact threshold.

4. The method of claim 1, wherein the determining a data ingest rate of a set of data ingest rates is based on at least one of a query, a test, monitoring loading levels, a list, or receiving loading information as part of an access request.

5. The method of claim 1, wherein the transmit data rate is determined, such that a data ingress rate plus an aggregate of a plurality of data loading levels is less than the data ingress rate of the set of distributed storage and task execution units.

6. The method of claim 1, wherein the write threshold number of distributed storage and task execution units of the set of distributed storage and task execution units equals at least the minimum number of encoded data slices required for rebuilding encoded data slices not written to corresponding distributed storage and task execution units.

7. A dispersed storage module comprises:

a first module, when operable within a computing device, causes the computing device to;

encode a data object to produce a plurality of sets of encoded data slices, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of encoded data slices of the plurality of sets of encoded data slices, wherein a write threshold number of encoded data slices provide for successful storage of the set of encoded data slices;

generate one or more write slice requests, wherein each of the write slice requests correspond to one or more sets of encoded data slices of the plurality of sets of encoded data slices; and

output the one or more write slice requests to a set of distributed storage and task execution units;

a second module, when operable within the computing device, causes the computing device to;

determine a data ingest rate of a set of data ingest rates for each distributed storage and task execution unit of the set of distributed storage and task execution units; and

determine a write threshold number of distributed storage and task execution units of the set of distributed storage and task execution units based on the set of data ingest rates;

a third module, when operable within the computing device, causes the computing device to;

determine a transmit data rate;

and

a fourth module, when operable within the computing device, causes the computing device to;

generate a write threshold number of write slice requests that is based on the write threshold number of encoded data slices; and

output, in accordance with the transmit data rate, the write threshold number of write slice requests to the write threshold number of distributed storage and task execution units of the set of distributed storage and task execution units.

8. The dispersed storage module of claim 7 further comprises:

a fifth module, when operable within the computing device, causes the computing device to;

facilitate rebuilding the encoded data slices of the one or more sets of encoded data slices not included in the write threshold number of write slice requests.

9. The dispersed storage module of claim 7, wherein the second module, when operable within the computing device, causes the computing device to:

determine a data ingest rate of a set of data ingest rates based on at least one of an estimated reliability level for data storage, a reliability level threshold, an estimated access speed, a lowest access speed of the write threshold number of distributed storage and task execution units, a speed threshold, an access capability estimator, a predetermination, an estimated rebuilding impact, or a rebuilding impact threshold.

10. The dispersed storage module of claim 7, wherein the second module, when operable within the computing device, causes the computing device to:

determine a data ingest rate of a set of data ingest rates based on at least one of a query, a test, monitoring loading levels, a list, or receiving loading information as part of an access request.

11. A non-transient computer readable medium containing program instructions for causing a computer to perform a method of:

encoding a data object to produce a plurality of sets of encoded data slices, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of encoded data slices of the plurality of sets of encoded data slices, wherein a write threshold number of encoded data slices provide for successful storage of the set of encoded data slices;

generating one or more write slice requests, wherein each of the write slice requests correspond to one or more sets of encoded data slices of the plurality of sets of encoded data slices;

outputting the one or more write slice requests to a set of distributed storage and task execution units;

for each distributed storage and task execution unit of the set of distributed storage and task execution units, determining a data ingest rate of a set of data ingest rates;

determining a write threshold number of distributed storage and task execution units of the set of distributed storage and task execution units based on the set of data ingest rates;

determining a transmit data rate;

generating a write threshold number of write slice requests that is based on the write threshold number of encoded data slices; and

outputting, in accordance with the transmit data rate, the write threshold number of write slice requests to the write threshold number of distributed storage and task execution units of the set of distributed storage and task execution units.

12. The non-transient computer readable medium of claim 11, further containing program instructions for causing the computer to perform a method of:

facilitating rebuilding the encoded data slices of the one or more sets of encoded data slices not included in the write threshold number of write slice requests to the distributed storage and task execution units not selected as part of the write threshold number of distributed storage and task execution units.

13. The non-transient computer readable medium of claim 11, wherein the determining a data ingest rate of a set of data ingest rates is based on at least one of an estimated reliability level for data storage, a reliability level threshold, an estimated access speed, a lowest access speed of the write threshold number of distributed storage and task execution units, a speed threshold, an access capability estimator, a predetermination, an estimated rebuilding impact, or a rebuilding impact threshold.

14. The non-transient computer readable medium of claim 11, wherein the determining a data ingest rate of a set of data ingest rates is based on at least one of a query, a test, monitoring loading levels, a list, or receiving loading information as part of an access request.

15. The non-transient computer readable medium of claim 11, wherein the transmit data rate is determined, such that a data ingress rate plus an aggregate of a plurality of data loading levels is less than the data ingress rate of the set of distributed storage and task execution units.

16. The non-transient computer readable medium of claim 11, wherein the write threshold number of distributed storage and task execution units of the set of distributed storage and task execution units equals at least the minimum number of encoded data slices required for rebuilding encoded data slices not written to corresponding distributed storage and task execution units.