Transforming data in a distributed storage and task network

• US 9,015,556 B2
• Filed: 12/06/2012
• Issued: 04/21/2015
• Est. Priority Date: 12/12/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Leggette, Wesley, Baptist, Andrew, Dhuse, Greg, Resch, Jason K., Volvovski, Ilya, Motwani, Manish, Gladwin, S. Christopher, Grube, Gary W., Shirley, Thomas Franklin Jr.
• Primary Examiner(s): Alphonse, Fritz
• Application Number:
  US13/707,542
• Publication Number:
  US 20130151928A1
• Time in Patent Office: 866 Days
• Field of Search: 714/763, 714/752, 714/755, 714/758, 714/6.22, 714/6.24
• US Class Current: 714/763
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/611 Specific encoding aspects, e.g. encoding by means of decoding
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module determining whether at least a portion of temporarily stored data is to be stored long-term, wherein the temporarily stored data is stored in a set of distributed storage and task (DST) units in accordance with a computational-orientated dispersed storage error coding function. When the at least a portion of the temporarily stored data is to be stored long-term, the method continues with the DS processing module identifying one or more DST storing the at least a portion of the temporarily stored data, recovering the at least a portion of the temporarily stored data, dispersed storage error encoding the at least a portion of the temporarily stored data in a pre-dispersed storage error encoded format into a plurality of sets of encoded data slices, and storing the plurality of sets of encoded data slices in the set of DST units.

Claims

1. A method for execution by a device within a distributed storage and task network, the method comprises:

determining whether at least a portion of temporarily stored data is to be stored long-term, wherein the temporarily stored data is stored in a set of distributed storage and task (DST) units in accordance with a computational-orientated dispersed storage error coding function;

when the at least a portion of the temporarily stored data is to be stored long-term;

identifying one or more DST units of the set of DST units storing the at least a portion of the temporarily stored data in accordance with the computational-orientated dispersed storage error coding function;

recovering the at least a portion of the temporarily stored data from the one or more DST units in a pre-dispersed storage error encoded format to produce recovered data;

dispersed storage error encoding the recovered data into a plurality of sets of encoded data slices in accordance with a long-term-storage-orientated dispersed storage error coding function; and

storing the plurality of sets of encoded data slices in the set of DST units.

2. The method of claim 1 further comprises:

the computational-orientated dispersed storage error coding function including first dispersed storage error coding parameters that enables a DST unit of the set of DST units to recover, in the pre-dispersed storage error encoded format, a sub-portion of the portion of the temporarily stored data from encoded data slices the DST unit stores; and

the long-term-storage-orientated dispersed storage error coding function including second dispersed storage error coding parameters that prevents the DST unit from recovering, in the pre-dispersed storage error encoded format, the sub-portion of the portion of the temporarily stored data from the encoded data slices the DST unit stores and requires retrieval of encoded data slices from multiple DST units of the set of DST units to recover, in the pre-dispersed storage error encoded format, the sub-portion of the portion of the temporarily stored data.

3. The method of claim 1 further comprises:

temporarily storing raw data, as the temporarily stored data, in the set of DST units in accordance with the computational-orientated dispersed storage error coding function;

performing, by at least some of the DST units, a task on the temporarily stored data to produce found data; and

dispersed storage error encoding the found data, as the at least a portion of the temporarily stored data, into the plurality of sets of encoded data slices in accordance with the long-term-storage-orientated dispersed storage error coding function.

4. The method of claim 1 further comprises:

temporarily storing found data, as the temporarily stored data, in the set of DST units in accordance with the computational-orientated dispersed storage error coding function, wherein at least some of the DST units performed a task on raw data to produce the found data;

performing, by the at least some of the DST units, another task on the temporarily stored data to produce a sub-set of found data; and

dispersed storage error encoding the sub-set of found data, as the at least a portion of the temporarily stored data, into the plurality of sets of encoded data slices in accordance with the long-term-storage-orientated dispersed storage error coding function.

5. The method of claim 1, wherein the dispersed storage error encoding the at least a portion of the temporarily stored data comprises:

determining, in accordance with the long-term-storage-orientated dispersed storage error coding function, a data mapping of data-based encoded data slices of the at least a portion of the temporarily stored data in a pre-dispersed storage error encoded format;

generating redundancy-based encoded data slices based on the data-based encoded data slices and in accordance with the long-term-storage-orientated dispersed storage error coding function; and

organizing the data-based encoded data slices and the redundancy-based encoded data slices into the plurality of sets of encoded data slices.

6. The method of claim 1, wherein the dispersed storage error encoding the at least a portion of the temporarily stored data comprises:

dispersed storage error encoding the at least a portion of the temporarily stored data in the pre-dispersed storage error encoded format into a plurality of encoded data blocks in accordance with the long-term-storage-orientated dispersed storage error coding function; and

organizing the plurality of encoded data blocks into the plurality of sets of encoded data slices in accordance with the long-term-storage-orientated dispersed storage error coding function.

7. The method of claim 1, wherein the identifying the one or more DST units of the set of DST units comprises at least one of:

receiving a message regarding the at least a portion of the temporarily stored data, wherein the message identifies the one or more DST units;

receiving the message regarding the at least a portion of the temporarily stored data and determining the one or more DST units based storage of the at least a portion of the temporarily stored data; and

determining that no further computations are to be performed on the at least a portion of the temporarily stored data.

8. A method for execution by a device within a distributed storage and task network, the method comprises:

determining that a task is to be performed on data, wherein the data is encoded into a plurality of encoded data blocks in accordance with a long-term-storage-orientated dispersed storage error coding function and the plurality of encoded data blocks are stored in a set of distributed storage and task (DST) units;

requesting reconfiguration of storage of the data from the long-term-storage-orientated dispersed storage error coding function to a computational-orientated dispersed storage error coding function, wherein the data is encoded into groupings of encoded data blocks in accordance with the computational-orientated dispersed storage error coding function;

requesting storage of the groupings of encoded data blocks by the set of DST units, wherein a DST unit of the set of DST units recovers a portion of the data from one of the groupings of encoded data blocks and performs a portion of the task on the recovered portion of the data to produce a partial task resultant;

receiving partial task resultants from the at least some of the set of DST units; and

compiling the partial task resultants to produce a task resultant.

9. The method of claim 8 further comprises:

the computational-orientated dispersed storage error coding function including first dispersed storage error coding parameters that enables the DST unit of the set of DST units to recover, in a pre-dispersed storage error encoded format, a sub-portion of the portion of the temporarily stored data from encoded data slices the DST unit stores; and

the long-term-storage-orientated dispersed storage error coding function including second dispersed storage error coding parameters that prevents the DST unit from recovering, in the pre-dispersed storage error encoded format, the sub-portion of the portion of the temporarily stored data from the encoded data slices the DST unit stores and requires retrieval of encoded data slices from multiple DST units of the set of DST units to recover, in the pre-dispersed storage error encoded format, the sub-portion of the portion of the temporarily stored data.

10. The method of claim 8, wherein the reconfiguring storage of the data comprises:

decoding the plurality of encoded data blocks in accordance with the long-term-storage-orientated dispersed storage error coding function to recover the data;

encoding the recovered data in accordance with the computational-orientated dispersed storage error coding function to produce the groupings of encoded data blocks; and

sending one of the groupings of encoded data blocks to the DST unit.

11. The method of claim 8, wherein the reconfiguring storage of the data comprises:

identifying data-based encoded data slices of the plurality of encoded data blocks in accordance with the long-term-storage-orientated dispersed storage error coding function;

determining a data mapping of the data-based encoded data slices between the set of DST units and the at least some of the DST units; and

facilitating copying of at least some of the data-based encoded data slices to the at least some of the set of DST units in accordance with the data mapping.

12. A dispersed storage (DS) module of within a distributed storage and task network comprises:

a first module, when operable within a computing device, causes the computing device to;

determine whether at least a portion of temporarily stored data is to be stored long-term, wherein the temporarily stored data is stored in a set of distributed storage and task (DST) units in accordance with a computational-orientated dispersed storage error coding function; and

a second module, when operable within the computing device, causes the computing device to;

when the at least a portion of the temporarily stored data is to be stored long-term;

identify one or more DST units of the set of DST units storing the at least a portion of the temporarily stored data in accordance with the computational-orientated dispersed storage error coding function;

recover the at least a portion of the temporarily stored data from the one or more DST units in a pre-dispersed storage error encoded format to produce recovered data;

dispersed storage error encode the recovered data into a plurality of sets of encoded data slices in accordance with a long-term-storage-orientated dispersed storage error coding function; and

store the plurality of sets of encoded data slices in the set of DST units.

13. The DS module of claim 12 further comprises:

the computational-orientated dispersed storage error coding function including first dispersed storage error coding parameters that enables a DST unit of the set of DST units to recover, in the pre-dispersed storage error encoded format, a sub-portion of the portion of the temporarily stored data from encoded data slices the DST unit stores; and

the long-term-storage-orientated dispersed storage error coding function including second dispersed storage error coding parameters that prevents the DST unit from recovering, in the pre-dispersed storage error encoded format, the sub-portion of the portion of the temporarily stored data from the encoded data slices the DST unit stores and requires retrieval of encoded data slices from multiple DST units of the set of DST units to recover, in the pre-dispersed storage error encoded format, the sub-portion of the portion of the temporarily stored data.

14. The DS module of claim 12 further comprises:

a third module, when operable within the computing device, causes the computing device to;

temporarily store raw data, as the temporarily stored data, in the set of DST units in accordance with the computational-orientated dispersed storage error coding function;

perform, by at least some of the DST units, a task on the temporarily stored data to produce found data; and

dispersed storage error encode the found data, as the at least a portion of the temporarily stored data, into the plurality of sets of encoded data slices in accordance with the long-term-storage-orientated dispersed storage error coding function.

15. The DS module of claim 12 further comprises:

a fourth module, when operable within the computing device, causes the computing device to;

temporarily store found data, as the temporarily stored data, in the set of DST units in accordance with the computational-orientated dispersed storage error coding function, wherein at least some of the DST units performed a task on raw data to produce the found data;

perform, by the at least some of the DST units, another task on the temporarily stored data to produce a sub-set of found data; and

dispersed storage error encode the sub-set of found data, as the at least a portion of the temporarily stored data, into the plurality of sets of encoded data slices in accordance with the long-term-storage-orientated dispersed storage error coding function.

16. The DS module of claim 12, wherein the second module functions to dispersed storage error encode the at least a portion of the temporarily stored data by:

determining, in accordance with the long-term-storage-orientated dispersed storage error coding function, a data mapping of data-based encoded data slices of the at least a portion of the temporarily stored data in a pre-dispersed storage error encoded format;

generating redundancy-based encoded data slices based on the data-based encoded data slices and in accordance with the long-term-storage-orientated dispersed storage error coding function; and

organizing the data-based encoded data slices and the redundancy-based encoded data slices into the plurality of sets of encoded data slices.

17. The DS module of claim 12, wherein the second module functions to dispersed storage error encode the at least a portion of the temporarily stored data by:

dispersed storage error encoding the at least a portion of the temporarily stored data in the pre-dispersed storage error encoded format into a plurality of encoded data blocks in accordance with the long-term-storage-orientated dispersed storage error coding function; and

organizing the plurality of encoded data blocks into the plurality of sets of encoded data slices in accordance with the long-term-storage-orientated dispersed storage error coding function.

18. The DS module of claim 12, wherein the second module functions to identify one or more DST units of the set of DST units by at least one of:

receiving a message regarding the at least a portion of the temporarily stored data, wherein the message identifies the one or more DST units;

receiving the message regarding the at least a portion of the temporarily stored data and determining the one or more DST units based storage of the at least a portion of the temporarily stored data; and

determining that no further computations are to be performed on the at least a portion of the temporarily stored data.

19. A dispersed storage (DS) module of within a distributed storage and task network comprises:

a first module, when operable within a computing device, causes the computing device to;

determine that a task is to be performed on data, wherein the data is encoded into a plurality of encoded data blocks in accordance with a long-term-storage-orientated dispersed storage error coding function and the plurality of encoded data blocks are stored in a set of distributed storage and task (DST) units;

a second module, when operable within the computing device, causes the computing device to;

request reconfiguration of storage of the data from the long-term-storage-orientated dispersed storage error coding function to a computational-orientated dispersed storage error coding function, wherein the data is encoded into groupings of encoded data blocks in accordance with the computational-orientated dispersed storage error coding function; and

a third module, when operable within the computing device, causes the computing device to;

request storage of the groupings of encoded data blocks by the set of DST units, wherein a DST unit of the set of DST units recovers a portion of the data from one of the groupings of encoded data blocks and performs a portion of the task on the recovered portion of the data to produce a partial task resultant; and

receive partial task resultants from the at least some of the set of DST units; and

compile the partial task resultants to produce a task resultant.

20. The DS module of claim 19 further comprises:

the computational-orientated dispersed storage error coding function including first dispersed storage error coding parameters that enables the DST unit of the set of DST units to recover, in a pre-dispersed storage error encoded format, a sub-portion of the portion of the temporarily stored data from encoded data slices the DST unit stores; and

the long-term-storage-orientated dispersed storage error coding function including second dispersed storage error coding parameters that prevents the DST unit from recovering, in the pre-dispersed storage error encoded format, the sub-portion of the portion of the temporarily stored data from the encoded data slices the DST unit stores and requires retrieval of encoded data slices from multiple DST units of the set of DST units to recover, in the pre-dispersed storage error encoded format, the sub-portion of the portion of the temporarily stored data.

21. The DS module of claim 19, wherein the second module functions to reconfigure storage of the data by:

decoding the plurality of encoded data blocks in accordance with the long-term-storage-orientated dispersed storage error coding function to recover the data;

encoding the recovered data in accordance with the computational-orientated dispersed storage error coding function to produce the groupings of encoded data blocks; and

sending one of the groupings of encoded data blocks to the DST unit.

22. The DS module of claim 19, wherein the second module functions to reconfigure storage of the data by:

identifying data-based encoded data slices of the plurality of encoded data blocks in accordance with the long-term-storage-orientated dispersed storage error coding function;

determining a data mapping of the data-based encoded data slices between the set of DST units and the at least some of the DST units; and

facilitating copying of at least some of the data-based encoded data slices to the at least some of the set of DST units in accordance with the data mapping.