Storing a data stream in a set of storage devices

• US 9,258,177 B2
• Filed: 06/17/2013
• Issued: 02/09/2016
• Est. Priority Date: 08/02/2012
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Gladwin, S. Christopher, Markison, Timothy W., Dhuse, Greg, Shirley, Thomas Franklin Jr., Leggette, Wesley, Resch, Jason K., Grube, Gary W.
• Primary Examiner(s): Leibovich, Yair
• Application Number:
  US13/919,673
• Publication Number:
  US 20140040660A1
• Time in Patent Office: 967 Days
• Field of Search: 714/6.2
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2053 where persistent mass storage functionality or persistent mass storage control functionality is redundant error detection or correction in information storage based on relative movement between record carrier and transducer G11B20/18
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/611 Specific encoding aspects, e.g. encoding by means of decoding
• CPC Class Codes: H04L 1/0076 Distributed coding, e.g. network coding, involving channel coding coding in both space and time H04L1/0618; cooperative diversity H04B7/022
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A set of storage devices coordinates selection of dispersed storage error encoding parameters for storing a stream of data and coordinates selection of encoded data storage devices and redundancy encoded data storage devices of the set of storage devices. The encoded data storage devices coordinates storage of a decode threshold amount of encoded data of the stream of data in accordance with the dispersed storage error encoding parameters. The encoded data storage devices send information regarding the decode threshold amount of encoded data to the redundancy encoded data storage devices. The redundancy encoded data storage devices generates redundancy encoded data based on the information regarding the decode threshold amount of encoded data and in accordance with the dispersed storage error encoding parameters. The redundancy encoded data storage devices store the redundancy encoded data to provide error encoded reliable storage of the stream of data.

Claims

1. A method comprises:

coordinating, by a set of storage devices, selection of dispersed storage error encoding parameters for storing a stream of data;

coordinating, by the set of storage devices, selection of encoded data storage devices of the set of storage devices and redundancy encoded data storage devices of the set of storage devices based on the selected dispersed storage error encoding parameters;

coordinating, by the encoded data storage devices, storage of a decode threshold amount of encoded data of the stream of data, wherein the encoded data storage devices encode the stream of data in accordance with the dispersed storage error encoding parameters to produce the decode threshold amount of encoded data;

sending, by the encoded data storage devices, information regarding the decode threshold amount of encoded data to the redundancy encoded data storage devices;

generating, by the redundancy encoded data storage devices, redundancy encoded data based on the information regarding the decode threshold amount of encoded data and in accordance with the dispersed storage error encoding parameters, wherein the generating the redundancy encoded data includes;

receiving, by one of the redundancy encoded data storage devices, corresponding portions of the decode threshold amount of encoded data from the encoded data storage devices; and

generating, by the one of the redundancy encoded data storage devices, a corresponding portion of the redundancy encoded data based on corresponding portions of the decode threshold amount of encoded data; and

storing, by the redundancy encoded data storage devices, the redundancy encoded data to provide error encoded reliable storage of the stream of data.

2. The method of claim 1, wherein the coordinating selection of the dispersed storage error encoding parameters comprises one or more of:

determining a redundancy factor;

determining an encoding pillar width;

determining a decode threshold;

determining a data segmenting process;

determining pre-error encoding data processing; and

determining post-error encoding data processing.

3. The method of claim 1, wherein the coordinating selection of the encoded data storage devices and the redundancy encoded data storage devices comprises:

determining a data segment threshold number based on the dispersed storage error encoding parameters and selecting the encoded data storage devices based on the data segment threshold number; and

determining a redundancy factor based on the dispersed storage error encoding parameters and selecting the redundancy encoded data storage devices based on the redundancy factor.

4. The method of claim 1, wherein the coordinating storage of the decode threshold amount of encoded data comprises:

partitioning, by the encoded data storage devices, the stream of data into a stream of data partitions; and

alternatingly encoding, by the encoded data storage devices, a data partition of the stream of data partitions in accordance with the dispersed storage error encoding parameters to produce a decode threshold number of encoded data slices of the decoded threshold amount of encoded data.

5. The method of claim 1, wherein the sending the information regarding the decode threshold amount of encoded data comprises:

generating, by one of the encoded data storage devices, partially error coded data based on a corresponding portion of the decode threshold amount of encoded data; and

sending, by the one of the encoded data storage devices, the partially error coded data to the redundancy encoded data storage devices.

6. The method of claim 1, wherein the sending the information regarding the decode threshold amount of encoded data comprises:

sending, by one of the encoded data storage devices, a corresponding portion of the decode threshold amount of encoded data.

7. The method of claim 1, wherein the generating the redundancy encoded data comprises:

receiving, by one of the redundancy encoded data storage devices, partially error coded data from the encoded data storage devices; and

generating, by the one of the redundancy encoded data storage devices, a corresponding portion of the redundancy encoded data based on the partially error coded data.

8. A dispersed storage system comprises:

one or more first modules that, when operable in one or more storage devices of a set of storage devices, causes the one or more storage devices to;

coordinate selection of dispersed storage error encoding parameters for storing a stream of data; and

coordinate selection of encoded data storage devices of the set of storage devices and redundancy encoded data storage devices of the set of storage devices;

one or more second modules that, when operable in one or more of the encoded data storage devices, causes the one or more encoded data storage devices to;

coordinate storage of a decode threshold amount of encoded data of the stream of data, wherein the one or more second modules encode the stream of data in accordance with the dispersed storage error encoding parameters to produce the decode threshold amount of encoded data; and

send information regarding the decode threshold amount of encoded data to the redundancy encoded data storage devices; and

one or more third modules that, when operable in one or more of the redundancy encoded data storage devices, causes the one or more redundancy encoded data storage devices to;

generate redundancy encoded data based on the information regarding the decode threshold amount of encoded data and in accordance with the dispersed storage error encoding parameters, wherein the redundancy encoded data is generated by;

receiving corresponding portions of the decode threshold amount of encoded data from the encoded data storage devices; and

generating a corresponding portion of the redundancy encoded data based on corresponding portions of the decode threshold amount of encoded data; and

store the redundancy encoded data to provide error encoded reliable storage of the stream of data.

9. The dispersed storage system of claim 8, wherein the one or more first modules further causes the one or more storage devices to coordinate selection of the dispersed storage error encoding parameters by one or more of:

determining a redundancy factor;

determining an encoding pillar width;

determining a decode threshold;

determining a data segmenting process;

determining pre-error encoding data processing; and

determining post-error encoding data processing.

10. The dispersed storage system of claim 8, wherein the one or more first modules further causes the one or more storage devices to coordinate selection of the encoded data storage devices and the redundancy encoded data storage devices by:

determining a data segment threshold number based on the dispersed storage error encoding parameters and selecting the encoded data storage devices based on the data segment threshold number; and

determining a redundancy factor based on the dispersed storage error encoding parameters and selecting the redundancy encoded data storage devices based on the redundancy factor.

11. The dispersed storage system of claim 8, wherein the one or more second modules further causes the one or more encoded data storage devices to coordinate storage of the decode threshold amount of encoded data by:

partitioning the stream of data into a stream of data partitions; and

alternatingly encoding a data partition of the stream of data partitions in accordance with the dispersed storage error encoding parameters to produce a decode threshold number of encoded data slices of the decoded threshold amount of encoded data.

12. The dispersed storage system of claim 8, wherein the one or more second modules further causes the one or more encoded data storage devices to send the information regarding the decode threshold amount of encoded data by:

generating partially error coded data based on a corresponding portion of the decode threshold amount of encoded data; and

sending the partially error coded data to the redundancy encoded data storage devices.

13. The dispersed storage system of claim 8, wherein the one or more second modules further causes the one or more encoded data storage devices to send the information regarding the decode threshold amount of encoded data by:

sending a corresponding portion of the decode threshold amount of encoded data.

14. The dispersed storage system of claim 8, wherein the one or more third modules further causes the one or more redundancy encoded data storage devices to generate the redundancy encoded data by:

receiving, by one third module of the one or more third modules, partially error coded data from the encoded data storage devices; and

generating a corresponding portion of the redundancy encoded data based on the partially error coded data.