Adjusting dispersed storage network traffic due to rebuilding

• US 9,424,132 B2
• Filed: 04/18/2014
• Issued: 08/23/2016
• Est. Priority Date: 05/30/2013
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Volvovski, Ilya, Gladwin, S. Christopher, Grube, Gary W., Markison, Timothy W., Resch, Jason K., Shirley, Thomas Franklin Jr., Dhuse, Greg, Motwani, Manish, Baptist, Andrew, Leggette, Wesley
• Primary Examiner(s): Ehne, Charles
• Application Number:
  US14/256,205
• Publication Number:
  US 20140359348A1
• Time in Patent Office: 858 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3419 by assessing time
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 11/349 for interfaces, buses
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A method begins by a dispersed storage (DS) processing module of a dispersed storage network (DSN) monitoring network traffic within the DSN. When the network traffic compares unfavorably to a desired network traffic function, the method continues with the DS processing module reducing a portion of the network traffic due to rebuilding flagged encoded data slices by at least one of changing rebuilding criteria for the flagged encoded data slices and changing rebuilding network traffic protocols.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

monitoring network traffic within the DSN; and

when the network traffic compares unfavorably to a desired network traffic function, reducing a portion of the network traffic due to rebuilding flagged encoded data slices by at least one of changing rebuilding criteria for the flagged encoded data slices and changing rebuilding network traffic protocols.

2. The method of claim 1, wherein the monitoring the network traffic comprises one or more of:

monitoring latency in responding to data access requests;

monitoring volume of the data access requests;

monitoring latency in responding to system level access requests; and

monitoring volume of the system level access requests.

3. The method of claim 1 further comprises:

the desired network traffic function including a curve; and

increasingly changing the at least one of the rebuilding criteria and the rebuilding network traffic protocols as the network traffic increases the unfavorable comparison to the curve of the desired network traffic function.

4. The method of claim 1, wherein the changing the rebuilding criteria comprises one or more of:

decreasing a rebuilding threshold which triggers a rebuilding process for one of the flagged encoded data slices;

increasing a wait time for a storage unit storing a flagged encoded data slice of the flagged encoded data slices to come back on line; and

adjusting rebuild queue priorities of the flagged encoded data slices.

5. The method of claim 1, wherein the changing the rebuilding network traffic protocols comprises one or more of:

bundling rebuilding requests to a storage unit for multiple ones of the flagged encoded data slices;

serializing transmission of a set of rebuilding requests to storage units regarding one of the flagged encoded data slices; and

assigning an individual response timeslot to each of multiple storage units for individually responding to a rebuild request regarding one or more the flagged encoded data slices.

6. The method of claim 1 further comprises:

after the reducing the portion of the network traffic due to the rebuilding flagged encoded data slices;

continue the monitoring the network traffic within the DSN;

when the network traffic compares favorably to the desired network traffic function, undoing the changing of the at least one of the rebuilding criteria and the rebuilding network traffic protocols; and

when the network traffic continues to compare unfavorably to the desired network traffic function, further reducing the portion of the network traffic due to the rebuilding of the flagged encoded data slices by at least one of further changing the rebuilding criteria and further changing the rebuilding network traffic protocols.

7. A dispersed storage (DS) module of a dispersed storage network (DSN), the DS module comprises:

a first module, when operable within a computing device, causes the computing device to;

monitor network traffic within the DSN; and

a second module, when operable within the computing device, causes the computing device to;

when the network traffic compares unfavorably to a desired network traffic function, reduce a portion of the network traffic due to rebuilding flagged encoded data slices by at least one of changing rebuilding criteria for the flagged encoded data slices and changing rebuilding network traffic protocols.

8. The DS module of claim 7, wherein the first module functions to cause the computing device to monitor the network traffic by one or more of:

monitoring latency in responding to data access requests;

monitoring volume of the data access requests;

monitoring latency in responding to system level access requests; and

monitoring volume of the system level access requests.

9. The DS module of claim 7 further comprises:

the desired network traffic function includes a curve; and

the second module further functions to cause the computing device to;

increasingly change the at least one of the rebuilding criteria and the rebuilding network traffic protocols as the network traffic increases the unfavorable comparison to the curve of the desired network traffic function.

10. The DS module of claim 7, wherein the second module functions to cause the computing device to change the rebuilding criteria by one or more of:

decreasing a rebuilding threshold which triggers a rebuilding process for one of the flagged encoded data slices;

increasing a wait time for a storage unit storing a flagged encoded data slice of the flagged encoded data slices to come back on line; and

adjusting rebuild queue priorities of the flagged encoded data slices.

11. The DS module of claim 7, wherein the second module functions to cause the computing device to change the rebuilding network traffic protocols by one or more of:

bundling rebuilding requests to a storage unit for multiple ones of the flagged encoded data slices;

serializing transmission of a set of rebuilding requests to storage units regarding one of the flagged encoded data slices; and

assigning an individual response timeslot to each of multiple storage units for individually responding to a rebuild request regarding one or more the flagged encoded data slices.

12. The DS module of claim 7 further comprises:

the second module further functions to cause the computing device to;

after reducing the portion of the network traffic due to the rebuilding flagged encoded data slices;

continue to monitor the network traffic within the DSN;

when the network traffic compares favorably to the desired network traffic function, undo the changing of the at least one of the rebuilding criteria and the rebuilding network traffic protocols; and

when the network traffic continues to compare unfavorably to the desired network traffic function, further reduce the portion of the network traffic due to the rebuilding of the flagged encoded data slices by at least one of further changing the rebuilding criteria and further changing the rebuilding network traffic protocols.

13. A computer readable storage medium comprises:

a first memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

monitor network traffic within the DSN; and

a second memory section that stores operational instructions that, when executed by the one or more processing modules of the one or more computing devices of the DSN, causes the one or more computing devices to;

when the network traffic compares unfavorably to a desired network traffic function, reduce a portion of the network traffic due to rebuilding flagged encoded data slices by at least one of changing rebuilding criteria for the flagged encoded data slices and changing rebuilding network traffic protocols.

14. The computer readable storage medium of claim 13, wherein the one or more processing modules functions to execute the operational instructions stored by the first memory section to cause the one or more computing devices of the DSN to monitor the network traffic by one or more of:

monitoring latency in responding to data access requests;

monitoring volume of the data access requests;

monitoring latency in responding to system level access requests; and

monitoring volume of the system level access requests.

15. The computer readable storage medium of claim 13 further comprises:

the desired network traffic function includes a curve; and

the second memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

increasingly change the at least one of the rebuilding criteria and the rebuilding network traffic protocols as the network traffic increases the unfavorable comparison to the curve of the desired network traffic function.

16. The computer readable storage medium of claim 13, wherein the one or more processing modules functions to execute the operational instructions stored by the second memory section to cause the one or more computing devices of the DSN to change the rebuilding criteria by one or more of:

decreasing a rebuilding threshold which triggers a rebuilding process for one of the flagged encoded data slices;

increasing a wait time for a storage unit storing a flagged encoded data slice of the flagged encoded data slices to come back on line; and

adjusting rebuild queue priorities of the flagged encoded data slices.

17. The computer readable storage medium of claim 13, wherein the one or more processing modules functions to execute the operational instructions stored by the second memory section to cause the one or more computing devices of the DSN to change the rebuilding network traffic protocols by one or more of:

bundling rebuilding requests to a storage unit for multiple ones of the flagged encoded data slices;

serializing transmission of a set of rebuilding requests to storage units regarding one of the flagged encoded data slices; and

assigning an individual response timeslot to each of multiple storage units for individually responding to a rebuild request regarding one or more the flagged encoded data slices.

18. The computer readable storage medium of claim 13 further comprises:

the second memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

after reducing the portion of the network traffic due to the rebuilding flagged encoded data slices;

continue to monitor the network traffic within the DSN;

when the network traffic compares favorably to the desired network traffic function, undo the changing of the at least one of the rebuilding criteria and the rebuilding network traffic protocols; and

when the network traffic continues to compare unfavorably to the desired network traffic function, further reduce the portion of the network traffic due to the rebuilding of the flagged encoded data slices by at least one of further changing the rebuilding criteria and further changing the rebuilding network traffic protocols.