Re-encoding data in a dispersed storage network

• US 10,169,125 B2
• Filed: 03/21/2016
• Issued: 01/01/2019
• Est. Priority Date: 05/29/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K., Dhuse, Greg
• Primary Examiner(s): Schell, Joseph O
• Application Number:
  US15/075,946
• Publication Number:
  US 20160350171A1
• Time in Patent Office: 1,016 Days
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/073 in a memory management context, e.g. virtual memory or cache management memory management G06F12/00; testing of static memory units G11C29/00
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1612 where the redundant component is persistent storage
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 : in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 : Management of blocks
• CPC Class Codes: G06F 3/0644 : Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0649 : Lifecycle management
• CPC Class Codes: G06F 3/065 : Replication mechanisms
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a storage unit partially decoding a first encoded data slice of a set of encoded data slices in accordance with previous dispersed storage error encoding parameters to produce a partially decoded first encoded data slice that is stored by another storage unit. The method continues with the storage unit partially re-encoding the partially decoded first encoded data slice in accordance with updated dispersed storage error encoding parameters to produce a first partially re-encoded data slice that is used to create a new first encoded data slice of a new set of encoded data slices.

Claims

1. A method comprises:

determining to create a new set of encoded data slices based on an unfavorable storage performance level associated with one or more storage units (SUs) within a dispersed storage network (DSN);

partially decoding, by a storage unit (SU) of the DSN, a first encoded data slice of a set of encoded data slices in accordance with previous dispersed storage error encoding parameters having a previous threshold number to produce a partially decoded first encoded data slice, wherein the first encoded data slice is stored by another SU of the DSN and is transmitted from the another SU via the DSN and received via an interface of the SU that is configured to interface and communicate with the DSN, and wherein a data segment of a data object is encoded into the set of encoded data slices in accordance with the previous dispersed storage error encoding parameters;

partially re-encoding, by the SU, the partially decoded first encoded data slice in accordance with updated dispersed storage error encoding parameters having an updated threshold number to produce a first partially re-encoded data slice, wherein the first partially re-encoded data slice is used to create a new first encoded data slice of the new set of encoded data slices that corresponds to the data segment being dispersed storage error encoded in accordance with the updated dispersed storage error encoding parameters, wherein

the partially re-encoding comprises;

obtaining a new encoding matrix corresponding to the updated dispersed storage error encoding parameters;

reducing the new encoding matrix based on a matrix position corresponding to the new first encoded data slice of the new set of encoded data slices that corresponds to the data segment being dispersed storage error encoded in accordance with the updated dispersed storage error encoding parameters; and

matrix multiplying the reduced new encoding matrix with the partially decoded first encoded data slice to produce the first partially re-encoded data slice;

receiving, by the SU via the DSN and via the interface of the SU, a plurality of second partially re-encoded data slices from a sub-set of other SUs of the DSN, wherein the plurality of second partially re-encoded data slices is created in accordance with the updated dispersed storage error encoding parameters based on partially re-encoding by the sub-set of other SUs of the DSN; and

generating, by the SU, a new second encoded data slice of the new set of encoded data slices from the plurality of second partially re-encoded data slices.

2. The method of claim 1, wherein the partially decoding comprises:

obtaining a decoding matrix corresponding to the previous dispersed storage error encoding parameters;

generating a reduced data matrix based on a second encoded data slice of the set of encoded data slices that correspond to the data segment being encoded in accordance with the previous dispersed storage error encoding parameters; and

matrix multiplying the decoding matrix and the reduced data matrix to produce the partially decoded first encoded data slice.

3. The method of claim 1, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

4. The method of claim 1, wherein the receiving the plurality of second partially re-encoded data slices comprises:

receiving one of the plurality of second partially re-encoded data slices from a first other SU of the sub-set of other SUs, wherein the first other SU generated the one of the plurality of second partially re-encoded data slices by;

partially decoding a second encoded data slice of the set of encoded data slices in accordance with the previous dispersed storage error encoding parameters and an encoded data slice of the set of encoded data slices stored by the first other SU to produce a first partially decoded second encoded data slice; and

partially re-encoding the first partially decoded second encoded data slice in accordance with the updated dispersed storage error encoding parameters to produce the one of the plurality of second partially re-encoded data slices; and

receiving another one of the plurality of second partially re-encoded data slices from a second other SU of the sub-set of other SUs, wherein the second other SU generated the another one of the plurality of second partially re-encoded data slices by;

partially decoding the second encoded data slice of the set of encoded data slices in accordance with the previous dispersed storage error encoding parameters and an encoded data slice of the set of encoded data slices stored by the second other SU to produce a second partially decoded second encoded data slice; and

partially re-encoding the second partially decoded second encoded data slice in accordance with the updated dispersed storage error encoding parameters to produce the another one of the plurality of second partially re-encoded data slices.

5. The method of claim 1, wherein the generating the new second encoded data slice comprises:

performing an arithmetic function on the plurality of second partially re-encoded data slices to produce the new second encoded data slice.

6. The method of claim 5, wherein the arithmetic function comprises at least one of:

an exclusive OR function;

an addition function;

a subtraction function;

a multiplication function;

or

a division function.

7. The method of claim 1 further comprises:

storing, by a set of SUs of the DSN, the set of encoded data slices, wherein the another SU is in the set of SUs; and

receiving, by the SU, the plurality of second partially re-encoded data slices from the sub-set of other SUs, wherein the sub-set of other SUs is in the set of SUs.

8. The method of claim 1 further comprises:

determining to create the new set of encoded data slices based on a change request from the previous threshold number to the updated threshold number.

9. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

determine to create a new set of encoded data slices based on an unfavorable storage performance level associated with one or more storage units (SUs) within a dispersed storage network (DSN);

partially decode, by a storage unit (SU) of the DSN, a first encoded data slice of a set of encoded data slices in accordance with previous dispersed storage error encoding parameters having a previous threshold number to produce a partially decoded first encoded data slice, wherein the first encoded data slice is stored by another SU of the DSN and is transmitted from the another SU via the DSN and received via an interface of the SU that is configured to interface and communicate with the DSN, and wherein a data segment of a data object is encoded into the set of encoded data slices in accordance with the previous dispersed storage error encoding parameters;

partially re-encode, by the SU, the partially decoded first encoded data slice in accordance with updated dispersed storage error encoding parameters having an updated threshold number to produce a first partially re-encoded data slice, wherein the first partially re-encoded data slice is used to create a new first encoded data slice of the new set of encoded data slices that corresponds to the data segment being dispersed storage error encoded in accordance with the updated dispersed storage error encoding parameters, wherein the partially re-encode, by the SU, is performed by;

obtaining a new encoding matrix corresponding to the updated dispersed storage error encoding parameters;

reducing the new encoding matrix based on a matrix position corresponding to the new first encoded data slice of the new set of encoded data slices that corresponds to the data segment being dispersed storage error encoded in accordance with the updated dispersed storage error encoding parameters; and

matrix multiplying the reduced new encoding matrix with the partially decoded first encoded data slice to produce the first partially re-encoded data slice;

receive, by the SU via the DSN and via the interface of the SU, a plurality of second partially re-encoded data slices from a sub-set of other SUs of the DSN, wherein the plurality of second partially re-encoded data slices is created in accordance with the updated dispersed storage error encoding parameters based on partially re-encoding by the sub-set of other SUs of the DSN; and

generate, by the SU, a new second encoded data slice of the new set of encoded data slices from the plurality of second partially re-encoded data slices.

10. The non-transitory computer readable storage medium of claim 9, wherein the one or more processing modules functions to execute the operational instructions stored by the at least one memory section to cause the one or more computing devices of the DSN to partially decode by:

obtaining a decoding matrix corresponding to the previous dispersed storage error encoding parameters;

generating a reduced data matrix based on a second encoded data slice of the set of encoded data slices that correspond to the data segment being encoded in accordance with the previous dispersed storage error encoding parameters; and

matrix multiplying the decoding matrix and the reduced data matrix to produce the partially decoded first encoded data slice.

11. The non-transitory computer readable storage medium of claim 9, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

12. The non-transitory computer readable storage medium of claim 9, wherein the one or more processing modules functions to execute the operational instructions stored by the at least one memory section to cause the one or more computing devices of the DSN to receive the plurality of second partially re-encoded data slices by:

receiving one of the plurality of second partially re-encoded data slices from a first other SU of the sub-set of other SUs, wherein the first other SU generated the one of the plurality of second partially re-encoded data slices by;

partially decoding a second encoded data slice of the set of encoded data slices in accordance with the previous dispersed storage error encoding parameters and an encoded data slice of the set of encoded data slices stored by the first other SU to produce a first partially decoded second encoded data slice; and

partially re-encoding the first partially decoded second encoded data slice in accordance with the updated dispersed storage error encoding parameters to produce the one of the plurality of second partially re-encoded data slices; and

receiving another one of the plurality of second partially re-encoded data slices from a second other SU of the sub-set of other SUs, wherein the second other SU generated the another one of the plurality of second partially re-encoded data slices by;

partially decoding the second encoded data slice of the set of encoded data slices in accordance with the previous dispersed storage error encoding parameters and an encoded data slice of the set of encoded data slices stored by the second other SU to produce a second partially decoded second encoded data slice; and

partially re-encoding the second partially decoded second encoded data slice in accordance with the updated dispersed storage error encoding parameters to produce the another one of the plurality of second partially re-encoded data slices.

13. The non-transitory computer readable storage medium of claim 9, wherein the one or more processing modules functions to execute the operational instructions stored by the at least one memory section to cause the one or more computing devices of the DSN to generate the new second encoded data slice by:

performing an arithmetic function on the plurality of second partially re-encoded data slices to produce the new second encoded data slice.

14. The non-transitory computer readable storage medium of claim 13, wherein the arithmetic function comprises at least one of:

an exclusive OR function;

an addition function;

a subtraction function;

a multiplication function;

or

a division function.

15. The non-transitory computer readable storage medium of claim 9 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

store, by a set of SUs of the DSN, the set of encoded data slices, wherein the another SU is in the set of SUs; and

receive, by the SU, the plurality of second partially re-encoded data slices from the sub-set of other SUs, wherein the sub-set of other SUs is in the set of SUs.

16. The non-transitory computer readable storage medium of claim 9 further comprises:

the at least one memory section stores further operational instructions that, when executed by the one or more processing modules, causes the one or more computing devices of the DSN to;

determine to create the new set of encoded data slices based on a change request from the previous threshold number to the updated threshold number.