Accounting for data that needs to be rebuilt or deleted

• US 9,626,125 B2
• Filed: 05/30/2014
• Issued: 04/18/2017
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K.
• Primary Examiner(s): McCarthy, Christopher
• Application Number:
  US14/292,009
• Publication Number:
  US 20150039934A1
• Time in Patent Office: 1,054 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/306 User profiles

Abstract

A method begins by a dispersed storage (DS) processing module identifying a plurality of encoded data slices requiring rebuilding. The method continues with the DS processing module determining an amount of reserve memory required for storage of rebuilt slices for the identified plurality of encoded data slices requiring rebuilding. The method continues with the DS processing module updating memory utilization information to include the amount of reserve memory required. The method continues with the DS processing module indicating the memory utilization. The method continues with the DS processing module obtaining rebuilt slices. The method continues with the DS processing module storing the rebuilt slices in the memory and updating the memory utilization information.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

identifying a plurality of encoded data slices that require rebuilding, wherein rebuilding of the plurality of encoded data slices is queued for at least one of individual, group, or batch processing and the processing will be performed at a time delay from the queuing;

accessing memory utilization information regarding memory, wherein the memory utilization information includes slice storage information regarding amount of memory used to store valid encoded data slices, available storage information regarding amount of memory that is unused, and rebuilt storage information regarding an actual amount of memory used to store rebuilt encoded data slices;

determining an amount of memory required for storage of rebuilt encoded data slices for the plurality of encoded data slices;

updating the memory utilization information by allocating a portion of available memory to the amount of memory required for storage of the rebuilt encoded data slices;

obtaining rebuilt encoded data slices;

storing the rebuilt encoded data slices in the memory associated with the rebuilt storage information; and

modifying the rebuilt storage information and the available storage information to reflect the stored rebuilt encoded data slices.

2. The method of claim 1, wherein the memory utilization information further includes:

reserved rebuilding storage information regarding amount of memory reserved for storing rebuilt encoded data slices.

3. The method of claim 2 further comprising designating all utilized and the reserved memory as utilized.

4. The method of claim 1, wherein identifying the plurality of encoded data slices includes one or more of:

receiving a request to rebuild one or more encoded data slices;

detecting a missing encoded data slice; and

monitoring downloads to the memory meeting minimum read/write (R/W) width thresholds but less than a full pillar width.

5. The method of claim 1 further comprising sending the updated memory utilization information to one or more of:

a distributed storage and/or task (DST) processing unit;

a managing unit; and

a user device.

6. The method of claim 1, wherein updating the memory utilization information includes lowering reserved memory for the rebuilt encoded data slices and raising utilized memory for the rebuilt encoded data slices.

7. The method of claim 1, wherein identifying the plurality of encoded data slices further comprises determining when DSN read/write (R/W) requests occur for the plurality of encoded data slices and comparing to known times of inaccessibility for the memory storing the plurality of encoded data slices.

8. The method of claim 1, wherein identifying the plurality of encoded data slices further comprises querying vaults related to the plurality of encoded data slices to determine one or more missing or corrupted encoded data slices.

9. The method of claim 1, wherein obtaining rebuilt encoded data slices includes one or more of:

receiving the rebuilt encoded data slices; and

generating the rebuilt encoded data slices.

10. A dispersed storage (DS) method comprises:

attempting to retrieve a plurality of data slices from a DS memory to perform an integrity check of the plurality of data slices;

for data slices that are retrieved with an error and that are not received, flagging the data slices for rebuilding to produced flagged data slices

accessing memory utilization information regarding memory of a storage unit, wherein the memory utilization information includes slice storage information regarding amount of memory used to store valid encoded data slices, reserved rebuilding storage information regarding amount of memory reserved for storing rebuilt encoded data slices, and rebuilt storage information regarding an actual amount of memory used to store rebuilt encoded data slices;

determining an amount of memory required to rebuild the flagged data slices;

determining whether the amount of memory required to rebuild the flagged data slices is less than the amount of memory reserved for storing rebuilt encoded data slices;

when the amount of memory required to rebuild the flagged data slices is less than the amount of memory reserved for storing rebuilt encoded data slices;

updating the reserved rebuilding storage information to include the amount of memory required to rebuild the flagged data slices;

rebuilding the flagged data slices to produce rebuilt data slices;

storing the rebuilt data slices in the memory reserved for storing the rebuilt data slices; and

updating the rebuilt storage information and the reserved rebuilding storage information to reflect storage of the rebuilt data slices.

11. The method as per claim 10, wherein the attempting to retrieve a plurality data slices includes periodically attempting to retrieve one or more of:

a list of data slices;

a list of slice names;

a range of data slice addresses; and

a range of data slice names.

12. The method as per claim 10 further comprising querying related vaults within a dispersed storage system to determine missing data slices.

13. The method as per claim 10, wherein one or more steps of the method are performed during lowered dispersed storage activity.

14. The method as per claim 10 further comprising updating one or more of:

managing units handling one or more vaults storing the plurality of data slices;

peers within a common storage group of the flagged data slices; and

peers within a common storage group storing the rebuilt data slices.