Cycling out dispersed storage processing units from access pools to perform expensive operations

• US 10,140,061 B2
• Filed: 03/23/2017
• Issued: 11/27/2018
• Est. Priority Date: 03/29/2016
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Khadiwala, Ravi V.
• Primary Examiner(s): Sun, Charlie
• Application Number:
  US15/467,121
• Publication Number:
  US 20170286154A1
• Time in Patent Office: 614 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 2221/2133 Verifying human interaction, e.g., Captcha
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57
• CPC Class Codes: G06F 9/485 Task life-cycle, e.g. stopping, restarting, resuming execution G06F9/4881 takes precedence
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/373 : with erasure correction and erasure determination, e.g. for packet loss recovery or setting of erasures for the decoding of Reed-Solomon codes
• CPC Class Codes: H03M 13/616 : Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 61/10 : of different types
• CPC Class Codes: H04L 61/50 : Address allocation
• CPC Class Codes: H04L 63/08 : for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/104 : Grouping of entities
• CPC Class Codes: H04L 63/108 : when the policy decisions are valid for a limited amount of time
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54

Abstract

A method includes identifying, by a dispersed storage (DS) processing unit of a plurality of DS processing units of a dispersed storage network (DSN), a DSN operation to be performed that will adversely interfere with processing of DSN access requests. The method further includes sending, by the DS processing unit, a suspension request to a process balancing module of the DSN where the suspension request is for temporarily suspending processing of DSN access requests. The method further includes determining, by the process balancing module, whether sufficient processing resources of active DS processing units of the plurality of DS processing units are available for processing DSN access requests on behalf of the DS processing unit. When determined that sufficient processing resources are available, the method further includes granting the suspension request, and allocating DSN access requests of the DS processing unit to one ore of the active DS processing units.

Claims

1. A method comprises:

identifying, by a dispersed storage (DS) processing unit of a plurality of DS processing units of a dispersed storage network (DSN), a DSN operation to be performed that will adversely interfere with processing of DSN access requests;

sending, by the DS processing unit, a suspension request to a process balancing module of the DSN, wherein the suspension request is for temporarily suspending processing of DSN access requests;

determining, by the process balancing module, whether sufficient processing resources of active DS processing units of the plurality of DS processing units are available for processing DSN access requests on behalf of the DS processing unit; and

when determined that sufficient processing resources are available;

granting, by the process balancing module, the suspension request; and

allocating, by the process balancing module, DSN access requests of the DS processing unit to one or more of the active DS processing units.

2. The method of claim 1 further comprises:

executing, by the DS processing unit, the DSN operation;

upon completion of execution of the DSN operation, sending, by the DS processing unit, a return notification to the process balancing module; and

in response to the return notification, resuming, by the process balancing module, allocation of the DSN access requests of the DS processing unit to the DS processing unit.

3. The method of claim 1 further comprises:

the DSN operation including one or more of garbage collection, software update, a system administration operation, and a diagnostic operation; and

the DSN access requests including two or more of a read request, a write request, a rebuild request, a list request, a delete request, and a modify request.

4. The method of claim 1 further comprises:

when determined that sufficient processing resources are not available, temporarily denying, by the process balancing module, the suspension request until sufficient processing resources become available.

5. The method of claim 1, wherein the determining that sufficient processing resources are available comprises:

determining, by the process balancing module, a duration for which the DS processing unit will be unavailable for processing the DSN access requests of the DS processing unit;

estimating, by the process balancing module, processing requirements for processing the DSN access requests of the DS processing unit;

determining, by the process balancing module, a number of other DS processing units of the plurality of DS processing units that are currently in a temporary suspension mode;

estimating, by the process balancing module, processing requirements for processing other DSN access requests by the active DS processing units during the duration; and

determining, by the process balancing module, processing capabilities of the active DS processing units.

6. The method of claim 1, wherein the allocating the DSN access requests of the DS processing unit further comprises:

allocating, by the process balancing module, the DSN access requests of the DS processing unit to the active DS processing units in accordance with a load balancing function.

7. The method of claim 1, wherein the allocating the DSN access requests of the DS processing unit further comprises:

allocating, by the process balancing module, read requests to a first active DS processing unit of the one or more active DS processing units;

allocating, by the process balancing module, write requests to a second active DS processing unit of the one or more active DS processing units;

allocating, by the process balancing module, list requests to a third active DS processing unit of the one or more active DS processing units; and

allocating, by the process balancing module, rebuild support requests to a fourth active DS processing unit of the one or more active DS processing units.

8. The method of claim 1, wherein the allocating the DSN access requests of the DS processing unit further comprises:

for a write request of the DSN access requests;

allocating, by the process balancing module, a unity matrix multiplication to a first active DS processing unit of the one or more active DS processing units, wherein the first active DS processing unit generates a first subset of encoded data slices of a set of encoded data slices by matrix multiplying a unity matrix of an encoding matrix with a data matrix, wherein a data segment of a data object is divided into data blocks that are arranged into the data matrix;

allocating, by the process balancing module, first redundancy matrix multiplication to a second active DS processing unit of the one or more active DS processing units, wherein the second active DS processing unit generates a first redundancy encoded data slice of the set of encoded data slices by matrix multiplying a first redundancy row of the encoding matrix with the data matrix; and

allocating, by the process balancing module, second redundancy matrix multiplication to a third active DS processing unit of the one or more active DS processing units, wherein the third active DS processing unit generates a second redundancy encoded data slice of the set of encoded data slices by matrix multiplying a second redundancy row of the encoding matrix with the data matrix.

9. The method of claim 1, wherein the allocating the DSN access requests of the DS processing unit further comprises:

allocating, by the process balancing module, the DSN access requests of the DS processing unit to one of the active DS processing units when the one of the active DS processing units is part of a common local area network with the DS processing unit.

10. A computer readable memory comprises:

a first memory that stores operational instructions that, when executed by a dispersed storage (DS) processing unit of a plurality of DS processing units of a dispersed storage network (DSN), causes the DS processing unit to;

identify a DSN operation to be performed that will adversely interfere with processing of DSN access requests;

send a suspension request to a process balancing module of the DSN, wherein the suspension request is for temporarily suspending processing of DSN access requests; and

a second memory that stores operational instructions that, when executed by the process balancing module, causes the process balancing module to;

determine whether sufficient processing resources of active DS processing units of the plurality of DS processing units are available for processing DSN access requests on behalf of the DS processing unit; and

when determined that sufficient processing resources are available;

grant the suspension request; and

allocate DSN access requests of the DS processing unit to one of more of the active DS processing units.

11. The computer readable memory of claim 10 further comprises:

a third memory that stores operational instructions that, when executed by the DS processing unit, causes the DS processing unit to;

execute the DSN operation; and

upon completion of execution of the DSN operation, send a return notification to the process balancing module; and

a fourth memory that stores operational instructions that, when executed by the process balancing module, causes the process balancing module to;

in response to the return notification, resume allocation of the DSN access requests of the DS processing unit to the DS processing unit.

12. The computer readable memory of claim 10 further comprises:

the DSN operation including one or more of garbage collection, software update, a system administration operation, and a diagnostic operation; and

the DSN access requests including two or more of a read request, a write request, a rebuild request, a list request, a delete request, and a modify request.

13. The computer readable memory of claim 10, wherein the second memory further stores operational instructions that, when executed by the process balancing module, causes the process balancing module to:

when determined that sufficient processing resources are not available, temporarily deny the suspension request until sufficient processing resources become available.

14. The computer readable memory of claim 10, wherein the second memory further stores operational instructions that, when executed by the process balancing module, causes the process balancing module to determine that sufficient processing resources are available by:

determining a duration for which the DS processing unit will be unavailable for processing the DSN access requests of the DS processing unit;

estimating processing requirements for processing the DSN access requests of the DS processing unit;

determining a number of other DS processing units of the plurality of DS processing units that are currently in a temporary suspension mode;

estimating processing requirements for processing other DSN access requests by the active DS processing units during the duration; and

determining processing capabilities of the active DS processing units.

15. The computer readable memory of claim 10, wherein the second memory further stores operational instructions that, when executed by the process balancing module, causes the process balancing module to allocate the DSN access requests of the DS processing unit by:

allocate the DSN access requests of the DS processing unit to the active DS processing units in accordance with a load balancing function.

16. The computer readable memory of claim 10, wherein the second memory further stores operational instructions that, when executed by the process balancing module, causes the process balancing module to allocate the DSN access requests of the DS processing unit by:

allocating read requests to a first active DS processing unit of the one or more active DS processing units;

allocating write requests to a second active DS processing unit of the one or more active DS processing units;

allocating list requests to a third active DS processing unit of the one or more active DS processing units; and

allocating rebuild support requests to a fourth active DS processing unit of the one or more active DS processing units.

17. The computer readable memory of claim 10, wherein the second memory further stores operational instructions that, when executed by the process balancing module, causes the process balancing module to allocate the DSN access requests of the DS processing unit by:

for a write request of the DSN access requests;

allocating a unity matrix multiplication to a first active DS processing unit of the one or more active DS processing units, wherein the first active DS processing unit generates a first subset of encoded data slices of a set of encoded data slices by matrix multiplying a unity matrix of an encoding matrix with a data matrix, wherein a data segment of a data object is divided into data blocks that are arranged into the data matrix;

allocating first redundancy matrix multiplication to a second active DS processing unit of the one or more active DS processing units, wherein the second active DS processing unit generates a first redundancy encoded data slice of the set of encoded data slices by matrix multiplying a first redundancy row of the encoding matrix with the data matrix; and

allocating second redundancy matrix multiplication to a third active DS processing unit of the one or more active DS processing units, wherein the third active DS processing unit generates a second redundancy encoded data slice of the set of encoded data slices by matrix multiplying a second redundancy row of the encoding matrix with the data matrix.

18. The computer readable memory of claim 10, wherein the second memory further stores operational instructions that, when executed by the process balancing module, causes the process balancing module to allocate the DSN access requests of the DS processing unit by:

allocating the DSN access requests of the DS processing unit to one of the active DS processing units when the one of the active DS processing units is part of a common local area network with the DS processing unit.