Dispersed storage encoded data slice rebuild

• US 10,359,935 B2
• Filed: 01/25/2017
• Issued: 07/23/2019
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K.
• Primary Examiner(s): McCarthy, Christopher S
• Application Number:
  US15/415,236
• Publication Number:
  US 20170132082A1
• Time in Patent Office: 909 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/306 User profiles

Abstract

A method begins by a dispersed storage (DS) processing module identifying a plurality of encoded data slices requiring rebuilding. The method continues with the DS processing module determining an amount of reserve memory required for storage of rebuilt slices for the identified plurality of encoded data slices requiring rebuilding. The method continues with the DS processing module updating memory utilization information to include the amount of reserve memory required. The method continues with the DS processing module indicating the memory utilization. The method continues with the DS processing module obtaining rebuilt slices. The method continues with the DS processing module storing the rebuilt slices in the memory and updating the memory utilization information.

Claims

1. A dispersed storage (DS) processing module comprises:

memory to store a plurality of encoded data slices;

a first processing module, when operable within a computing device, causes the computing device to;

identify a first plurality of encoded data slices that require rebuilding, the identifying including at least attempting to retrieve a second plurality of encoded data slices from a DS memory to perform an integrity check of the second plurality of encoded data slices and for encoded data slices that are retrieved with an error and that are not received, flagging these encoded data slices as the first plurality of encoded data slices that require rebuilding;

a second processing module, when operable within the computing device, causes the computing device to;

determine an amount of memory required for storage of rebuilt encoded data slices for the first plurality of encoded data slices;

a third processing module, when operable within the computing device, causes the computing device to;

access memory utilization information regarding memory, wherein the memory utilization information includes encoded data slice storage information regarding amount of memory used to store valid encoded data slices, available storage information regarding amount of memory that is unused, and rebuilt storage information regarding an actual amount of memory used to store rebuilt encoded data slices;

update the memory utilization information by allocating a portion of available memory to the amount of memory required for storage of the rebuilt encoded data slices;

a fifth processing module, when operable within the computing device, causes the computing device to;

obtain rebuilt encoded data slices;

a sixth processing module, when operable within the computing device, causes the computing device to;

store the rebuilt encoded data slices in the memory associated with the rebuilt storage information; and

a seventh processing module, when operable within the computing device, causes the computing device to;

modify the rebuilt storage information and the available storage information to reflect the stored rebuilt encoded data slices.

2. The dispersed storage (DS) processing module of claim 1 further comprises:

the first processing module, when operable within the computing device, further causes the computing device to;

periodically identify the first plurality of encoded data slices by retrieving one or more of;

a list of the second plurality of encoded data slices;

names of the second plurality of encoded data slices;

address ranges of the second plurality of encoded data slices;

or

related vault lists including the second plurality of encoded data slices.

3. The dispersed storage (DS) processing module of claim 1 further comprises:

the first processing module, when operable within the computing device, further causes the computing device to;

identify the first plurality of encoded data slices by one or more of;

receiving a request to rebuild one or more encoded data slice;

or

monitoring downloads meeting minimum read/write (R/W) width thresholds but less than a full pillar width.

4. The dispersed storage (DS) processing module of claim 1 further comprises:

the third processing module, when operable within the computing device, further causes the computing device to;

update the memory utilization information with the amount of memory;

utilized for stored encoded data slices;

utilized for rebuilt encoded data slices;

reserved for rebuilt encoded data slices;

or

un-utilized.

5. The dispersed storage (DS) processing module of claim 1 further comprises:

the fifth processing module, when operable within the computing device, further causes the computing device to;

obtain rebuilt encoded data slices including one or more of;

receiving the rebuilt encoded data slices;

or

generating the rebuilt encoded data slices.

6. The dispersed storage (DS) processing module of claim 1 further comprises:

the seventh processing module, when operable within the computing device, further causes the computing device to;

update the memory utilization information including;

lowering reserved memory for the rebuilt encoded data slices; and

raising utilized memory for the rebuilt encoded data slices.

7. A dispersed storage (DS) processing module comprises:

memory to store a plurality of encoded data slices;

a first processing module, when operable within a computing device, causes the computing device to;

identify a plurality of encoded data slices that require rebuilding, wherein rebuilding of the plurality of encoded data slices is queued for at least one of individual, group, or batch processing and the batch processing will be performed at a time delay from the queuing;

a second processing module, when operable within the computing device, causes the computing device to;

determine an amount of memory required for storage of rebuilt encoded data slices for the plurality of encoded data slices;

a third processing module, when operable within the computing device, causes the computing device to;

access memory utilization information regarding memory, wherein the memory utilization information includes slice storage information regarding amount of memory used to store valid encoded data slices, available storage information regarding amount of memory that is unused, and rebuilt storage information regarding an actual amount of memory used to store rebuilt encoded data slices;

update the memory utilization information by allocating a portion of available memory to the amount of memory required for storage of the rebuilt encoded data slices;

a fifth processing module, when operable within the computing device, causes the computing device to;

obtain rebuilt encoded data slices;

a sixth processing module, when operable within the computing device, causes the computing device to;

store the rebuilt encoded data slices in the memory associated with the rebuilt storage information; and

a seventh processing module, when operable within the computing device, causes the computing device to;

modify the rebuilt storage information and the available storage information to reflect the stored rebuilt encoded data slices.

8. The dispersed storage (DS) processing module of claim 7 further comprises:

the first processing module, when operable within the computing device, further causes the computing device to;

periodically identify the plurality of encoded data slices by retrieving one or more of;

a list of the plurality of encoded data slices;

names of the plurality of encoded data slices;

address ranges of the plurality of encoded data slices;

or

related vault lists including the plurality of encoded data slices.

9. The dispersed storage (DS) processing module of claim 7 further comprises:

the first processing module, when operable within the computing device, further causes the computing device to;

identify the plurality of encoded data slices by one or more of;

receiving a request to rebuild one or more encoded data slice;

or

monitoring downloads meeting minimum read/write (R/W) width thresholds but less than a full pillar width.

10. The dispersed storage (DS) processing module of claim 7 further comprises:

the third processing module, when operable within the computing device, further causes the computing device to;

update the memory utilization information with the amount of memory;

utilized for stored encoded data slices;

utilized for rebuilt encoded data slices;

reserved for rebuilt encoded data slices;

or

un-utilized.

11. The dispersed storage (DS) processing module of claim 7 further comprises:

the fifth processing module, when operable within the computing device, further causes the computing device to;

obtain rebuilt encoded data slices including one or more of;

receiving the rebuilt encoded data slices;

or

generating the rebuilt encoded data slices.

12. The dispersed storage (DS) processing module of claim 7 further comprises:

the seventh processing module, when operable within the computing device, further causes the computing device to;

update the memory utilization information including;

lowering reserved memory for the rebuilt encoded data slices; and

raising utilized memory for the rebuilt encoded data slices.

13. A computing device comprises:

memory to store a plurality of encoded data slices;

a first processing module, when operable within the computing device, causes the computing device to;

identify a first plurality of encoded data slices that require rebuilding, the identifying including at least attempting to retrieve a second plurality of encoded data slices from a dispersed storage (DS) memory to perform an integrity check of the second plurality of encoded data slices and for encoded data slices that are retrieved with an error and that are not received, flagging these encoded data slices as the first plurality of encoded data slices that require rebuilding, wherein the identifying further includes monitoring downloads meeting minimum read/write (R/W) width thresholds, but that are less than a full pillar width;

a second processing module, when operable within the computing device, causes the computing device to;

determine an amount of memory required for storage of rebuilt encoded data slices for the first plurality of encoded data slices;

a third processing module, when operable within the computing device, causes the computing device to;

access memory utilization information regarding memory, wherein the memory utilization information includes encoded data slice storage information regarding amount of memory used to store valid encoded data slices, available storage information regarding amount of memory that is unused, and rebuilt storage information regarding an actual amount of memory used to store rebuilt encoded data slices;

update the memory utilization information by allocating a portion of available memory to the amount of memory required for storage of the rebuilt encoded data slices;

a fifth processing module, when operable within the computing device, causes the computing device to;

obtain rebuilt encoded data slices;

a sixth processing module, when operable within the computing device, causes the computing device to;

store the rebuilt encoded data slices in the memory associated with the rebuilt storage information; and

a seventh processing module, when operable within the computing device, causes the computing device to;

modify the rebuilt storage information and the available storage information to reflect the stored rebuilt encoded data slices.

14. The computing device of claim 13 further comprises:

the first processing module, when operable within the computing device, further causes the computing device to;

periodically identify the first plurality of encoded data slices by retrieving one or more of;

a list of the second plurality of encoded data slices;

names of the second plurality of encoded data slices;

address ranges of the second plurality of encoded data slices;

or

related vault lists including the second plurality of encoded data slices.

15. The computing device of claim 13 further comprises:

the third processing module, when operable within the computing device, further causes the computing device to;

update the memory utilization information with the amount of memory;

utilized for stored encoded data slices;

utilized for rebuilt encoded data slices;

reserved for rebuilt encoded data slices;

or

un-utilized.

16. The computing device of claim 13 further comprises:

the fifth processing module, when operable within the computing device, further causes the computing device to;

obtain rebuilt encoded data slices including one or more of;

receiving the rebuilt encoded data slices;

or

generating the rebuilt encoded data slices.

17. The computing device of claim 13 further comprises:

the seventh processing module, when operable within the computing device, further causes the computing device to;

update the memory utilization information including;

lowering reserved memory for the rebuilt encoded data slices; and

raising utilized memory for the rebuilt encoded data slices.