Prioritizing rebuilding of stored data in a dispersed storage network

• US 9,898,373 B2
• Filed: 03/17/2014
• Issued: 02/20/2018
• Est. Priority Date: 04/26/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K.
• Primary Examiner(s): Mehrmanesh, Elmira
• Application Number:
  US14/215,335
• Publication Number:
  US 20140331086A1
• Time in Patent Office: 1,436 Days
• Field of Search: 714 62, 714 622, 714 624
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module of a dispersed storage network (DSN), when in a rebuilding mode, determining a level of rebuilding urgency, where the level of rebuilding urgency is based on an inversely proportional function of a level of error encoded redundancy for error encoded data giving rise to the rebuilding mode. When the level of rebuilding urgency is at or above a critical level, the method continues with the DS processing module placing a set of storage units of the DSN in a critical rebuilding mode, where, when in the critical rebuilding mode, the set of storage units prioritize rebuilding of the error encoded data giving rise to the rebuilding mode over other error encoded data operations.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

when in a rebuilding mode, determining a data reconstruction margin for a set of encoded slices of error encoded data to determine a level of rebuilding urgency, wherein the set of encoded slices is associated with one or more detected storage errors, wherein a data segment of data is encoded to produce the set of encoded data slices that includes a total of “

n”

encoded data slices, wherein “

k”

is a minimum number of encoded data slices that are needed to reconstruct the data segment, wherein “

a”

represents a number of available and error-free encoded data slices of the “

n”

encoded data slices, and wherein the data reconstruction margin is a difference between the “

a”

available and error-free encoded data slices and the “

k”

encoded data slices that are needed to reconstruct the data segment; and

when the level of rebuilding urgency is at or above a critical level, placing a set of storage units of the DSN in a critical rebuilding mode, wherein, when in the critical rebuilding mode, the set of storage units prioritize rebuilding of the error encoded data associated with the one or more detected storage errors over other error encoded data operations, and wherein the level of rebuilding urgency is determined to be at or above the critical level when the data reconstruction margin is at or below a critical threshold.

2. The method of claim 1, wherein the determining the level of rebuilding urgency further comprises:

for the set of encoded data slices of the error encoded data;

determining “

f”

number of encoded data slices of the set of encoded data slices that are inaccessible;

for each of the “

f”

number of inaccessible encoded data slices, determining a nature for the inaccessibility; and

based on a collection of the nature of the inaccessibility and on quantity of the “

f”

number of inaccessible encoded data slices, determining the level of rebuilding urgency.

3. The method of claim 1 further comprises:

the determining the level of rebuilding urgency further including;

for the set of encoded data slices of the error encoded data;

determining “

f”

number of encoded data slices of the set of encoded data slices that are inaccessible due to memory device failures;

when the “

f”

number of inaccessible encoded data slices is at or above a critical number for the set of encoded data slices, determining that the level of rebuilding urgency is the critical level; and

the placing the set of storage units of the DSN in the critical rebuilding mode including;

for the “

f”

number of inaccessible encoded data slices, initiating a rebuilding process to produce “

f”

number of rebuilt encoded data slices; and

identifying new storage locations for the “

f”

number of rebuilt encoded data slices.

4. The method of claim 1 further comprises:

when the level of rebuilding urgency is at or below a normal rebuilding level, placing the set of storage units of the DSN in a normal rebuilding mode.

5. The method of claim 4 further comprises:

when the level of rebuilding urgency is above the normal rebuilding level and below the critical level, placing the set of storage units of the DSN in a prioritized rebuilding mode, wherein, when in the prioritized rebuilding mode, the set of storage units prioritize the rebuilding of the error encoded data over at least one of the other error encoded data operations.

6. The method of claim 1, wherein the placing the set of storage units of the DSN in the critical rebuilding mode comprises:

rejecting read requests regarding other error encoded data stored in the set of storage units;

rejecting write requests regarding the other error encoded data stored in the set of storage units; and

rejecting inquiry requests regarding the other error encoded data stored in the set of storage units.

7. The method of claim 1, wherein the placing the set of storage units of the DSN in the critical rebuilding mode comprises:

identifying an address range in which the error encoded data lies to produce an affected address range;

rejecting access requests to other error encoded data stored in the affected address range; and

processing normally access requests to other data stored in other address ranges.

8. A dispersed storage (DS) module of a dispersed storage network (DSN), the DS module comprises:

a first module, when operable within a computing device, causes the computing device to;

when in a rebuilding mode, determining a data reconstruction margin for a set of encoded slices of error encoded data to determine a level of rebuilding urgency, wherein the set of encoded slices is associated with one or more detected storage errors, wherein the set of encoded slices is associated with one or more detected storage errors, wherein a data segment of data is encoded to produce the set of encoded data slices that includes a total of “

n”

encoded data slices, wherein “

k”

is a minimum number of encoded data slices that are needed to reconstruct the data segment, wherein “

a”

represents a number of available and error-free encoded data slices of the “

n”

encoded data slices, wherein the data reconstruction margin is a difference between the “

a”

available and error-free encoded data slices and the “

k”

encoded data slices that are needed to reconstruct the data segment; and

a second module, when operable within the computing device, causes the computing device to;

when the level of rebuilding urgency is at or above a critical level, place a set of storage units of the DSN in a critical rebuilding mode, wherein, when in the critical rebuilding mode, the set of storage units prioritize rebuilding of the error encoded data associated with the one or more detected storage errors over other error encoded data operations, and wherein the level of rebuilding urgency is determined to be at or above the critical level when the data reconstruction margin is at or below a critical threshold.

9. The DS module of claim 8, wherein the first module functions to cause the computing device to further determine the level of rebuilding urgency by:

for the set of encoded data slices of the error encoded data;

determining “

f”

number of encoded data slices of the set of encoded data slices that are inaccessible;

for each of the “

f”

number of inaccessible encoded data slices, determining a nature for the inaccessibility; and

based on a collection of the nature of the inaccessibility and on quantity of the “

f”

number of inaccessible encoded data slices, determining the level of rebuilding urgency.

10. The DS module of claim 8 further comprises:

the first module functions to cause the computing device to further determine the level of rebuilding urgency by;

for the set of encoded data slices of the error encoded data;

determining “

f”

number of encoded data slices of the set of encoded data slices that are inaccessible due to memory device failures;

when the “

f”

number of inaccessible encoded data slices is at or above a critical number for the set of encoded data slices, determining that the level of rebuilding urgency is the critical level; and

the second module further functions to cause the computing device to place the set of storage units of the DSN in the critical rebuilding mode by;

for the “

f”

number of inaccessible encoded data slices, initiating a rebuilding process to produce “

f”

number of rebuilt encoded data slices; and

identifying new storage locations for the “

f”

number of rebuilt encoded data slices.

11. The DS module of claim 8 further comprises:

the second module further functions to cause the computing device to;

when the level of rebuilding urgency is at or below a normal rebuilding level, place the set of storage units of the DSN in a normal rebuilding mode.

12. The DS module of claim 11 further comprises:

the second module further functions to cause the computing device to;

when the level of rebuilding urgency is above the normal rebuilding level and below the critical level, place the set of storage units of the DSN in a prioritized rebuilding mode, wherein, when in the prioritized rebuilding mode, the set of storage units prioritize the rebuilding of the error encoded data over at least one of the other error encoded data operations.

13. The DS module of claim 8, wherein the second module functions to cause the computing device to place the set of storage units of the DSN in the critical rebuilding mode by:

rejecting read requests regarding other error encoded data stored in the set of storage units;

rejecting write requests regarding the other error encoded data stored in the set of storage units; and

rejecting inquiry requests regarding the other error encoded data stored in the set of storage units.

14. The DS module of claim 8, wherein the second module functions to cause the computing device to place the set of storage units of the DSN in the critical rebuilding mode by:

identifying an address range in which the error encoded data lies to produce an affected address range;

rejecting access requests to other error encoded data stored in the affected address range; and

processing normally access requests to other data stored in other address ranges.