ACCOUNTING FOR DATA THAT NEEDS TO BE REBUILT OR DELETED

• US 20150039934A1
• Filed: 05/30/2014
• Published: 02/05/2015
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Resch, Jason K.
• Granted Patent: US 9,626,125 B2
• Time in Patent Office: Days
• US Class Current: 714/6.22
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/306 User profiles

Abstract

A method begins by a dispersed storage (DS) processing module identifying a plurality of encoded data slices requiring rebuilding. The method continues with the DS processing module determining an amount of reserve memory required for storage of rebuilt slices for the identified plurality of encoded data slices requiring rebuilding. The method continues with the DS processing module updating memory utilization information to include the amount of reserve memory required. The method continues with the DS processing module indicating the memory utilization. The method continues with the DS processing module obtaining rebuilt slices. The method continues with the DS processing module storing the rebuilt slices in the memory and updating the memory utilization information.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

identifying a plurality of encoded data slices that require rebuilding, wherein rebuilding of the plurality of encoded data slices is queued for at least one of individual, group, or batch processing and the processing will be performed at a significant time delay from the queuing;

determining an amount of memory required for storage of rebuilt encoded data slices for the plurality of encoded data slices;

updating utilization information of the memory by allocating a portion of available memory to the amount of memory required;

obtaining rebuilt encoded data slices;

storing the rebuilt encoded data slices in the reserve memory; and

modifying the utilization information to reflect the stored rebuilt encoded data slices.

2. The method of claim 1, wherein the memory utilization information includes:

the memory utilized for the stored encoded data slices;

the memory utilized for the rebuilt encoded data slices;

the memory reserved for the rebuilt encoded data slices; and

un-utilized memory.

3. The method of claim 2 further comprising designating all utilized and the reserved memory as utilized.

4. The method of claim 1, wherein identifying the plurality of encoded data slices includes one or more of:

receiving a request to rebuild one or more encoded data slices;

detecting a missing encoded data slice; and

monitoring downloads to the memory meeting minimum read/write (R/W) width thresholds but less than a full pillar width.

5. The method of claim 1 further comprising sending the updated memory utilization to one or more of:

a distributed storage and/or task (DST) processing unit;

a managing unit; and

a user device.

6. The method of claim 1, wherein updating the memory utilization information includes lowering the reserved memory for the rebuilt encoded data slices and raising utilized for the rebuilt encoded data slices.

7. The method of claim 1, wherein identifying the plurality of encoded data slices further comprises determining when DSN read/write (R/W) requests occur for the plurality of encoded data slices and comparing to known times of inaccessibility for the memory storing the plurality of encoded data slices.

8. The method of claim 1, wherein identifying the plurality of encoded data slices further comprises querying vaults related to the plurality of encoded data slices to determine one or more missing or corrupted encoded data slices.

9. The method of claim 1, wherein obtaining rebuilt encoded data slices includes one or more of:

receiving the rebuilt encoded data slices; and

generating the rebuilt encoded data slices.

10. A dispersed storage (DS) module comprises:

memory to store a plurality of data slices;

a first module, when operable within a computing device, causes the computing device to;

identify a plurality of encoded data slices that require rebuilding, wherein rebuilding of the plurality of encoded data slices is queued for at least one of individual, group, or batch processing and the processing will be performed at a significant time delay from the queuing;

a second module, when operable within the computing device, causes the computing device to;

determine an amount of memory required for storage of rebuilt encoded data slices for the plurality of encoded data slices;

a third module, when operable within the computing device, causes the computing device to;

update utilization information of the memory by allocating a portion of available memory to the amount of memory required;

a fourth module, when operable within the computing device, causes the computing device to;

indicate the memory utilization;

a fifth module, when operable within the computing device, causes the computing device to;

obtain rebuilt data slices;

a sixth module, when operable within the computing device, causes the computing device to;

store the rebuilt encoded data slices in the reserve memory; and

a seventh module, when operable within the computing device, causes the computing device to;

modifying the utilization information to reflect the stored rebuilt encoded data slices.

11. The dispersed storage (DS) module of claim 10 further comprises:

the first module, when operable within the computing device, further causes the computing device to;

periodically identify the plurality of data slices by retrieving one or more of;

a list of the data slices;

names of the plurality of data slices;

address ranges of the plurality of data slices; and

related vault lists.

12. The dispersed storage (DS) module of claim 10 further comprises:

the first module, when operable within the computing device, further causes the computing device to;

identify the plurality of data slices by one or more of;

receiving a request to rebuild one or more data slice;

detecting one or more missing or corrupted data slices; and

monitoring downloads meeting minimum read/write (R/W) width thresholds but less than a full pillar width.

13. The dispersed storage (DS) module of claim 10 further comprises:

the third module, when operable within the computing device, further causes the computing device to;

update memory utilization information with the amount of memory;

utilized for stored data slices;

utilized for rebuilt data slices;

reserved for rebuilt data slices; and

un-utilized.

14. The dispersed storage (DS) module of claim 10 further comprises:

the fifth module, when operable within the computing device, further causes the computing device to;

obtain rebuilt data slices including one or more of;

receiving the rebuilt data slices; and

generating the rebuilt data slices.

15. The dispersed storage (DS) module of claim 10 further comprises:

the seventh module, when operable within the computing device, further causes the computing device to;

update the memory utilization information including;

lowering reserved memory for rebuilt data slices; and

raising utilized for rebuilt data slices.

16. A dispersed storage (DS) method comprises:

attempting to retrieve a plurality data slices from a DS memory to perform an integrity check of the plurality of data slices;

for data slices that are retrieved, flagging data slices that include at least one error;

for data slices that are not retrieved, flagging these data slices as missing;

determining an amount of reserve DS memory required to rebuild flagged data slices;

first updating DS memory utilization information to include the amount of reserve memory required;

rebuilding the flagged data slices;

storing the rebuilt data slices in memory; and

second updating the memory utilization information to include at least the amount of reserve memory to reflect the stored rebuilt data slices.

17. The method as per claim 16, wherein the attempting to retrieve a plurality data slices includes periodically attempting to retrieve one or more of:

a list of data slices;

a list of slice names;

a range of data slice addresses; and

a range of data slice names.

18. The method as per claim 16 further comprising querying related vaults within the dispersed storage system to determine the missing data slices.

19. The method as per claim 16, wherein one or more steps of the method are performed during lowered dispersed storage activity.

20. The method as per claim 16 further comprising updating one or more of:

managing units handling one or more vaults storing the plurality of data slices;

peers within a common storage group of the plurality of flagged data slices; and

peers within a common storage group of the plurality of data slices rebuilt.