Maintaining storage thresholds in a distributed storage network

• US 10,001,950 B2
• Filed: 01/06/2017
• Issued: 06/19/2018
• Est. Priority Date: 03/29/2016
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Dhuse, Greg R., Khadiwala, Ravi V., Resch, Jason K., Volvovski, Ilya
• Primary Examiner(s): Chaudry, Mujtaba M
• Application Number:
  US15/399,871
• Publication Number:
  US 20170286226A1
• Time in Patent Office: 529 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 2221/2133 Verifying human interaction, e.g., Captcha
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57
• CPC Class Codes: G06F 9/485 Task life-cycle, e.g. stopping, restarting, resuming execution G06F9/4881 takes precedence
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/373 : with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes
• CPC Class Codes: H03M 13/616 : Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 61/10 : of different types
• CPC Class Codes: H04L 61/50 : Address allocation
• CPC Class Codes: H04L 63/08 : for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/104 : Grouping of entities
• CPC Class Codes: H04L 63/108 : when the policy decisions are valid for a limited amount of time
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54

Abstract

A method begins by a processing module of a dispersed storage network (DSN), in response to issuing a write slice request for a send threshold number of encoded data slices (EDSs) to a set of storage units, receiving access responses from the set of storage units. The method continues with the processing module receiving less than a send threshold number of access responses within a first predetermined amount of time and issuing one or more additional write slice requests to one or more storage units of the set of storage units. When a send threshold number of EDSs have been stored in storage units of the set of storage units the method continues by determining a preservation threshold number of EDSs, where the preservation threshold number of EDSs is a number of EDSs of the set of EDSs that will be stored for at least a second predetermined amount of time, and the preservation threshold number of EDSs is less than the send threshold number of EDSs.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

encoding a data segment to produce a set of encoded data slices (EDSs), wherein a decode threshold number of EDSs of the set of EDSs is required to recover the data segment;

issuing a write slice request for a send threshold number of EDSs to storage units of a set of storage units, wherein the send threshold number of EDSs is the number of EDSs slices that are transmitted by the one or more computing devices of the DSN in an initial write slice request;

determining a preservation threshold number of EDSs, wherein the preservation threshold number of EDSs is a number of EDSs of the set of EDSs that will be stored for at least a first predetermined amount of time, wherein the preservation threshold number of EDSs is less than the send threshold number of EDSs;

receiving a plurality of access responses from the storage units of a set of storage units; and

when receiving less than a preservation threshold number of access responses within a second predetermined amount of time, issuing one or more additional write slice requests to one or more storage units of the set of storage units, wherein the preservation threshold number of access responses is equal to the preservation threshold number of EDSs.

2. The method of claim 1, wherein the send threshold number of EDSs is less than an Information Dispersal Algorithm (IDA) width.

3. The method of claim 1, further comprising:

based on a change to the DSN, changing the number of EDSs of the preservation threshold number of EDSs.

4. The method of claim 3, wherein the change to the DSN is a change in a DSN traffic rate.

5. The method of claim 3, wherein the change to the DSN is a change in system performance of the DSN.

6. The method of claim 1 further comprises:

generating, by the one or more processing modules of one or more computing devices of the DSN, one or more additional sets of encoded data slices and sending the additional sets of encoded data slices to a second distributed storage unit of the DSN when a first distributed storage unit has a performance below a predetermined performance level.

7. The method of claim 1, wherein the first predetermined amount of time is based on a long-term storage of the preservation threshold number of EDSs.

8. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

encoding a data segment to produce a set of encoded data slices (EDSs), wherein a decode threshold number of EDSs of the set of EDSs is required to recover the data segment;

determining a preservation threshold number of EDSs, wherein the preservation threshold number of EDSs is a number of EDSs of the set of EDSs that will be stored for at least a predetermined amount of time;

storing a preservation threshold number of EDSs in storage units of a set of storage units;

detecting that a number of EDSs stored in storage units of the set of storage units is less than or equal to a delayed rebuild threshold number of EDSs, wherein the delayed rebuild threshold number of EDSs is equal to or greater than the decode threshold number EDS; and

when a number of EDSs stored in storage units of the set of storage units is less than or equal to the delayed rebuild threshold number of EDSs;

initiating the rebuild of one or more EDSs, such that the number of EDSs stored in storage units of the set of storage units is equal to or greater than the preservation threshold number of EDSs.

9. The method of claim 8 further comprises:

rebuilding, by the one or more processing modules of one or more computing devices of the DSN, one or more missing encoded data slices of the set of encoded data slices and sending one or more write slice requests to one or more storage units of the set of storage units.

10. The method of claim 9, wherein each of the one or more write slice requests includes the rebuilt one or more missing encoded data slices of the set of encoded data slices.

11. The method of claim 10 further comprises:

receiving one or more access responses from the one or more storage units of the set of storage units;

based on the access responses, determining a number of successfully stored encoded data slices of the set of EDSs; and

continuing to store the rebuilt one or more missing encoded data slices until the preservation threshold number of EDSs is successfully stored.

12. The method of claim 9, wherein one or more of the rebuilt one or more missing encoded data slices of the set of encoded data slices is stored in a second premises, wherein the second premises is remotely located from a first premises, and the first premises is a location of the EDSs transmitted by the one or more computing devices of the DSN in an initial write slice request.

13. The method of claim 12 further comprises:

when detecting that the number of EDSs stored in the first location is less than or equal to a delayed rebuild threshold number of EDSs, transmitting one or more EDSs from the second premises to the first premises.

14. The method of claim 8, wherein a difference between the preservation threshold number of EDSs and the rebuild threshold number of EDSs comprises a Rebuild Traffic Reduction Factor (RTRF) and the RTRF is used by the one or more processing modules of one or more computing devices of the DSN to manage DNS traffic.