Storing raid data as encoded data slices in a dispersed storage network

• US 10,049,008 B2
• Filed: 04/11/2016
• Issued: 08/14/2018
• Est. Priority Date: 02/27/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): Bonzo, Bryce P
• Assistant Examiner(s): Arcos, Jeison C
• Application Number:
  US15/095,637
• Publication Number:
  US 20160224423A1
• Time in Patent Office: 855 Days
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1096 Parity calculation or recalculation after configuration or reconfiguration of the system
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/068 Hybrid storage device
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method includes determining whether at least a portion of a data object requires rebuilding, wherein the data object is stored in accordance with a RAID format. The method further includes, when the at least a portion of the data object requires rebuilding, reconstructing stripes from sets of data blocks and parity blocks. The method further includes dividing the recovered data object into data segments. The method further includes dispersed storage error encoding the data segments in accordance with dispersed storage error encoding parameters to produce sets of encoded data slices, wherein a data segment is recoverable from a threshold number of encoded data slices. The method further includes issuing sets of write requests to write the sets of encoded data slices into storage units of a dispersed storage network (DSN).

Claims

1. A method for execution by a computing device, the method comprises:

determining whether at least a portion of a data object requires rebuilding, wherein the data object is divided into a plurality of stripes, wherein a stripe of the plurality of stripes is divided into a set of data blocks, wherein a parity block is generated from the set of data blocks, and wherein the set of data blocks and the parity block are stored in accordance with a redundant array of independent disk (RAID) format;

when the at least a portion of the data object requires rebuilding, reconstructing the plurality of stripes from a plurality of sets of data blocks and a plurality of parity blocks, wherein the set of data blocks is one of the sets of the plurality of sets of data blocks and the parity block is one of the plurality of parity blocks;

dividing the recovered data object into a plurality of data segments;

dispersed storage error encoding the plurality of data segments in accordance with dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein a data segment of the plurality of data segments is recoverable from a threshold number of encoded data slices of a corresponding set of encoded data slices of the plurality of sets of encoded data slices; and

issuing a plurality of sets of write requests to write the plurality of sets of encoded data slices into storage units of a dispersed storage network (DSN).

2. The method of claim 1, wherein the determining whether at least a portion of a data object requires rebuilding comprises one or more of:

detecting a memory device failure, wherein the memory device is storing one or more data blocks of the plurality of sets of data blocks or is storing one or more parity blocks of the plurality of parity blocks;

detecting corruption of one or more data blocks of the plurality of sets of data blocks or corruption of one or more parity blocks of the plurality of parity blocks;

detecting loss of connection to the memory device; and

detecting performance of the memory device is below a performance threshold.

3. The method of claim 1 further comprises:

determining the dispersed storage error encoding parameters based on one or more of;

data type, vault identifier, and user identifier.

4. The method of claim 1 further comprises:

determining storage configuration for storing the plurality of sets of encoded data slices;

when the storage configuration is a combination of local and remote storage;

identifying a first set of the storage units within local DSN memory for storing a first sub-set of encoded data slices, wherein the first sub-set of encoded data slices includes at least a first encoded data slice from each of at least some of the plurality of sets of encoded data slices; and

identifying a second set of the storage units within remote DSN memory for storing a second sub-set of encoded data slices, wherein the second sub-set of encoded data slices includes at least a second encoded data slice from each of the at least some of the plurality of sets of encoded data slices.

5. The method of claim 1, wherein dispersed storage error encoding the data segment comprises:

generating a data matrix of the data segment; and

generating the set of encoded data slices by encoding the data matrix using an encoding matrix in accordance with the dispersal storage error encoding parameters, wherein the set of encoded data slices includes a total number of encoded data slices, and wherein the threshold number is at least two less than the total number.

6. A computer readable memory device comprises:

a first storage section that stores operational instructions that, when executed by a computing device, causes the computing device to;

determine whether at least a portion of a data object requires rebuilding, wherein the data object is divided into a plurality of stripes, wherein a stripe of the plurality of stripes is divided into a set of data blocks, wherein a parity block is generated from the set of data blocks, and wherein the set of data blocks and the parity block are stored in accordance with a redundant array of independent disk (RAID) format;

a second storage section that stores operational instructions that, when executed by the computing device, causes the computing device to;

when the at least a portion of the data object requires rebuilding, reconstruct the plurality of stripes from a plurality of sets of data blocks and a plurality of parity blocks, wherein the set of data blocks is one of the sets of the plurality of sets of data blocks and the parity block is one of the plurality of parity blocks;

a third storage section that stores operational instructions that, when executed by the computing device, causes the computing device to;

divide the recovered data object into a plurality of data segments;

dispersed storage error encode the plurality of data segments in accordance with dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein a data segment of the plurality of data segments is recoverable from a threshold number of encoded data slices of a corresponding set of encoded data slices of the plurality of sets of encoded data slices; and

a fourth storage section that stores operational instructions that, when executed by the computing device, causes the computing device to;

issue a plurality of sets of write requests to write the plurality of sets of encoded data slices into storage units of a dispersed storage network (DSN).

7. The computer readable memory device of claim 6, wherein the determining whether at least a portion of a data object requires rebuilding comprises one or more of:

detecting a memory device failure, wherein the memory device is storing one or more data blocks of the plurality of sets of data blocks or is storing one or more parity blocks of the plurality of parity blocks;

detecting corruption of one or more data blocks of the plurality of sets of data blocks or corruption of one or more parity blocks of the plurality of parity blocks;

detecting loss of connection to the memory device; and

detecting performance of the memory device is below a performance threshold.

8. The computer readable memory device of claim 6, wherein the third storage section further stores operational instructions that, when executed by the computing device, causes the computing device to:

determine the dispersed storage error encoding parameters based on one or more of;

data type, vault identifier, and user identifier.

9. The computer readable memory device of claim 6, wherein the fourth storage section further stores operational instructions that, when executed by the computing device, causes the computing device to:

determine storage configuration for storing the plurality of sets of encoded data slices;

when the storage configuration is a combination of local and remote storage;

identify a first set of the storage units within local DSN memory for storing a first sub-set of encoded data slices, wherein the first sub-set of encoded data slices includes at least a first encoded data slice from each of at least some of the plurality of sets of encoded data slices; and

identify a second set of the storage units within remote DSN memory for storing a second sub-set of encoded data slices, wherein the second sub-set of encoded data slices includes at least a second encoded data slice from each of the at least some of the plurality of sets of encoded data slices.

10. The computer readable memory device of claim 6, wherein the third storage section further stores operational instructions that, when executed by the computing device, causes the computing device to dispersed storage error encode the data segment by:

generating a data matrix of the data segment; and

generating the set of encoded data slices by encoding the data matrix using an encoding matrix in accordance with the dispersal storage error encoding parameters, wherein the set of encoded data slices includes a total number of encoded data slices, and wherein the threshold number is at least two less than the total number.

11. A computer device comprises:

an interface;

memory; and

a processing module operably coupled to the interface and to the memory, wherein the processing module is operable to;

determine whether at least a portion of a data object requires rebuilding, wherein the data object is divided into a plurality of stripes, wherein a stripe of the plurality of stripes is divided into a set of data blocks, wherein a parity block is generated from the set of data blocks, and wherein the set of data blocks and the parity block are stored in accordance with a redundant array of independent disk (RAID) format;

when the at least a portion of the data object requires rebuilding, reconstruct the plurality of stripes from a plurality of sets of data blocks and a plurality of parity blocks, wherein the set of data blocks is one of the sets of the plurality of sets of data blocks and the parity block is one of the plurality of parity blocks;

divide the recovered data object into a plurality of data segments;

dispersed storage error encode the plurality of data segments in accordance with dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein a data segment of the plurality of data segments is recoverable from a threshold number of encoded data slices of a corresponding set of encoded data slices of the plurality of sets of encoded data slices; and

issue a plurality of sets of write requests to write the plurality of sets of encoded data slices into storage units of a dispersed storage network (DSN).

12. The computing device of claim 11, wherein the processing module is further operable to determine whether at least a portion of a data object requires rebuilding by one or more of:

detecting a memory device failure, wherein the memory device is storing one or more data blocks of the plurality of sets of data blocks or is storing one or more parity blocks of the plurality of parity blocks;

detecting corruption of one or more data blocks of the plurality of sets of data blocks or corruption of one or more parity blocks of the plurality of parity blocks;

detecting loss of connection to the memory device; and

detecting performance of the memory device is below a performance threshold.

13. The computing device of claim 11, wherein the processing module is further operable to:

determine the dispersed storage error encoding parameters based on one or more of;

data type, vault identifier, and user identifier.

14. The computing device of claim 11, wherein the processing module is further operable to:

determine storage configuration for storing the plurality of sets of encoded data slices;

when the storage configuration is a combination of local and remote storage;

identify a first set of the storage units within local DSN memory for storing a first sub-set of encoded data slices, wherein the first sub-set of encoded data slices includes at least a first encoded data slice from each of at least some of the plurality of sets of encoded data slices; and

identify a second set of the storage units within remote DSN memory for storing a second sub-set of encoded data slices, wherein the second sub-set of encoded data slices includes at least a second encoded data slice from each of the at least some of the plurality of sets of encoded data slices.

15. The computing device of claim 11, wherein the processing module is further operable to dispersed storage error encode the data segment by:

generating a data matrix of the data segment; and

generating the set of encoded data slices by encoding the data matrix using an encoding matrix in accordance with the dispersal storage error encoding parameters, wherein the set of encoded data slices includes a total number of encoded data slices, and wherein the threshold number is at least two less than the total number.