Initiating rebuild actions from DS processing unit errors

• US 10,073,645 B2
• Filed: 11/29/2016
• Issued: 09/11/2018
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: McShane, Niall J., Resch, Jason K., Volvovski, Ilya
• Primary Examiner(s): Tabone, Jr., John J
• Application Number:
  US15/363,622
• Publication Number:
  US 20170153943A1
• Time in Patent Office: 651 Days
• Field of Search: 714764, 714763, 714770, 714773
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/616 Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561

Abstract

A method begins by detecting a recovery error when decoding a seemingly valid threshold number of existing encoded data slice. The method continues by sending a notice of the recovery error and a known integrity check value for the data segment to a rebuild module. The method continues by the rebuild module retrieving the set of existing encoded data slices and selectively decoding a different combination of a decode threshold number of existing encoded data slices of the set of existing encoded data slices until the data segment is successfully recovered. The method continues by dispersed storage error encoding the successfully recovered data segment to produce a set of new encoded data slices. The method continues by comparing the seemingly valid encoded data slices with corresponding new encoded data slices on an encoded data slice by encoded data slice basis to identify a corrupted encoded data slice.

Claims

1. A method comprises:

detecting, by a computing device of a dispersed storage network (DSN), a recovery error when decoding a seemingly valid threshold number of existing encoded data slices of a set of existing encoded data slices, wherein a data segment of a data object was dispersed storage error encoded to produce the set of existing encoded data slices;

sending, by the computing device, a notice of the recovery error and a known integrity check value for the data segment to a rebuild module of the DSN;

retrieving, by the rebuild module, the set of existing encoded data slices;

selectively decoding, by the rebuild module, a different combination of a decode threshold number of existing encoded data slices of the set of existing encoded data slices until the data segment is successfully recovered;

dispersed storage error encoding, by the rebuild module, the successfully recovered data segment to produce a set of new encoded data slices; and

comparing, by the rebuild module, the seemingly valid threshold number of existing encoded data slices with corresponding new encoded data slices of the set of new encoded data slices on an encoded data slice by encoded data slice basis to identify a corrupted encoded data slice of the seemingly valid threshold number of existing encoded data slices.

2. The method of claim 1, wherein the sending further comprises:

sending pillar numbers of the seemingly valid threshold number of existing encoded data slices.

3. The method of claim 1, wherein the detecting the recovery error comprises:

comparing an integrity check value of the successfully recovered data segment with a locally stored integrity check value of the data segment; and

when the comparing yields an unfavorable result, indicating the recovery error.

4. The method of claim 1, wherein the selectively decoding further comprises:

for a first combination of the set of existing encoded data slices, recovering a first data segment;

generating an integrity check value for the first data segment;

comparing the integrity check value for the first data segment with the known integrity check value; and

when the comparing is favorable, indicating the first combination produces the successfully recovered data segment.

5. The method of claim 4 further comprises:

when the comparing is unfavorable, selecting a second combination of the set of existing encoded data slices to produce a second recovered data segment;

generating a second integrity check value for the second recovered data segment;

comparing the second integrity check value with the known integrity check value; and

when the comparing is favorable, indicating the second combination produces the successfully recovered data segment.

6. The method of claim 1, wherein the comparing comprises:

for a first existing encoded data slice of the seemingly valid threshold number of existing encoded data slices;

comparing a first existing slice name of the first existing encoded data slice with a first new slice name of a first new encoded data slice of the set of new encoded data slices;

comparing a first existing integrity check value of the first existing encoded data slice with a first new integrity check value of the first new encoded data slice;

comparing a first existing encoded portion of the first existing encoded data slice with a first new encoded portion of the first new encoded data slice; and

when one or more of the comparisons is unfavorable, identifying the first existing encoded data slice as the corrupted encoded data slice.

7. The method of claim 1 further comprises:

sending a new encoded data slice (EDS) to a storage unit to replace the corrupted encoded data slice.

8. A rebuild module comprises:

an interface;

memory; and

a processing module operably coupled to the interface and the memory, wherein the processing module is operable to;

receive, via the interface and from a computing device of a dispersed storage network (DSN), a recovery error regarding a seemingly valid threshold number of existing encoded data slices of a set of existing encoded data slices and a known integrity check of a data segment, wherein the data segment of a data object was dispersed storage error encoded to produce the set of existing encoded data slices;

retrieve, via the interface the set of existing encoded data slices from storage units of the DSN;

selectively decode a different combination of a decode threshold number of existing encoded data slices of the set of existing encoded data slices until the data segment is successfully recovered;

dispersed storage error encode the successfully recovered data segment to produce a set of new encoded data slices; and

compare the seemingly valid threshold number of existing encoded data slices with corresponding new encoded data slices of the set of new encoded data slices on an encoded data slice by encoded data slice to identified a corrupted encoded data slice of the seemingly valid threshold number of existing encoded data slices.

9. The rebuild module of claim 8, wherein the processing module is further operable to:

receive, via the interface, pillar numbers of the seemingly valid threshold number of existing encoded data slices.

10. The rebuild module of claim 8, wherein the processing module is further operable to detect the recovery error by:

comparing an integrity check value of the successfully recovered data segment with a locally stored integrity check value of the data segment; and

when the comparing yields an unfavorable result, indicate the recovery error.

11. The rebuild module of claim 8, wherein the processing module is further operable to perform the selectively decoding by:

for a first combination of the set of existing encoded data slices, recovering a first data segment;

generating an integrity check value for the first data segment;

comparing the integrity check value for the first data segment with the known integrity check value; and

when the comparing is favorable, indicating the first combination produces the successfully recovered data segment.

12. The rebuild module of claim 11, wherein the processing module is further operable to:

when the comparing is unfavorable, select a second combination of the set of existing encoded data slices to produce a second recovered data segment;

generate a second integrity check value for the second recovered data segment;

compare the second integrity check value with the known integrity check value; and

when the comparing is favorable, indicate the second combination produces the successfully recovered data segment.

13. The rebuild module of claim 8, wherein the processing module is further operable to:

for a first existing encoded data slice of the seemingly valid threshold number of existing encoded data slices;

compare a first existing slice name of the first existing encoded data slice with a first new slice name of a first new encoded data slice of the set of new encoded data slices;

compare a first existing integrity check value of the first existing encoded data slice with a first new integrity check value of the first new encoded data slice;

compare a first existing encoded portion of the first existing encoded data slice with a first new encoded portion of the first new encoded data slice; and

when one or more of the comparisons is unfavorable, identify the first existing encoded data slice as the corrupted encoded data slice.

14. The rebuild module of claim 8, wherein the processing module is further operable to:

send, via the interface, a new encoded data slice (EDS) to a storage unit to replace the corrupted encoded data slice.