Executing partial tasks in a distributed storage and task network

• US 8,898,542 B2
• Filed: 12/06/2012
• Issued: 11/25/2014
• Est. Priority Date: 12/12/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Leggette, Wesley, Baptist, Andrew, Dhuse, Greg, Resch, Jason K., Volvovski, Ilya, Motwani, Manish, Gladwin, S. Christopher, Grube, Gary W., Shirley, Thomas Franklin Jr.
• Primary Examiner(s): LAMARRE, GUY J
• Application Number:
  US13/707,502
• Publication Number:
  US 20130151927A1
• Time in Patent Office: 719 Days
• Field of Search: 714/763, 714/766, 714/773, 709/827, 713/193, 718/104
• US Class Current: 714/763
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/611 Specific encoding aspects, e.g. encoding by means of decoding
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module receiving a partial task regarding an encoded data block grouping. The method continues with the DS processing module performing the partial task on the encoded data block grouping to produce a partial task result and determining subsequent treatment of the partial task result. When the subsequent treatment includes storage of the partial task result, the method continues with the DS processing module determining a manner in which the partial task result is to be stored. When the manner in which the partial task result is to be stored is dispersed storage, the method continues with the DS processing module dispersed storage error encoding the partial task result to produce one or more sets of encoded partial task result blocks and outputting the one or more sets of encoded partial task result blocks to a set of DST execution units.

Claims

1. A computer implemented method for execution by a dispersed storage and task (DST) execution unit, the method comprises:

receiving, by the DST execution unit, a partial task regarding an encoded data block grouping of a plurality of encoded data block groupings, wherein data is dispersed storage error encoded in accordance with dispersed storage error coding parameters to produce the plurality of encoded data block groupings;

performing the partial task on the encoded data block grouping to produce a partial task result;

determining subsequent treatment of the partial task result and of the encoded data block grouping; and

when the subsequent treatment includes maintaining storage of the encoded data block grouping and storage of the partial task result;

determining a manner in which the partial task result is to be stored; and

when the manner in which the partial task result is to be stored is dispersed storage;

dispersed storage error encoding the partial task result in accordance with dispersed storage error encoding parameters to produce one or more sets of encoded partial task result blocks; and

outputting the one or more sets of encoded partial task result blocks to a set of DST execution units for storage therein.

2. The method of claim 1, wherein the performing the partial task comprises:

decoding the encoded data block grouping in accordance with the dispersed storage error coding parameters to produce a partition of the data; and

performing the partial task on the partition of the data to produce the partial task result.

3. The method of claim 2 further comprises:

determining whether the encoded data block grouping is encrypted; and

when the encoded data block grouping is encrypted;

determining sensitivity of the partition of the data;

when the sensitivity of the partition of the data is of a first sensitivity;

decrypting the encoded data block grouping to produce the encoded data block grouping;

temporarily storing the partition of the data; and

when the partial task has been performed, deleting the temporary storage of the partition of the data.

4. The method of claim 1, wherein the determining the subsequent treatment comprises at least one of:

extracting subsequent treatment information from the partial task;

analyzing the partial task result for one or more result criteria to determine the subsequent treatment;

sending a query to another device within a distributed computing system to ascertain the subsequent treatment; and

analyzing the encoded data block grouping to identify source data criteria to determine the subsequent treatment.

5. The method of claim 1, wherein the receiving the partial task comprises:

receiving a set of encoded task data blocks;

determining dispersed storage error decoding parameters regarding the partial task; and

decoding the set of encoded task data blocks to recover the partial task.

6. The method of claim 1 further comprises:

when the manner in which the partial task result is to be stored is store locally, storing the partial task result in local memory of the DST execution unit.

7. The method of claim 1 further comprises:

when the manner in which the partial task result is to be stored is group dispersed storage;

coordinating with other DST execution units of a distributed computing system to collect a set of partial task results, wherein the set of partial task results includes the partial task result and partial task results of the other DST execution units that performed a corresponding partial task on other encoded data block groupings of the plurality of encoded data block groupings; and

disperse storage error encoding the set of partial task results to produce an error encoded result data block.

8. The method of claim 1 further comprises:

when the subsequent treatment includes overwriting the encoded data block grouping with the partial task result;

overwriting the encoded data block grouping with the partial task result; and

coordinating with other DST execution units of a distributed computing system to update redundancy data blocks of the plurality of encoded data block groupings based on the partial task results.

9. The method of claim 1, wherein the performing the partial task comprises:

determining DST execution unit'"'"'s ability to fulfill the partial task in a reasonable time frame; and

when the DST execution unit cannot fulfill the partial task in the reasonable time frame;

partitioning the partial task into a set of sub-partial tasks;

portioning the encoded data block grouping into a set of encoded data block sub-groupings;

sending the set of sub-partial tasks and set of encoded data block sub-groupings to another set of DST execution units of a distributed computing system;

receiving sub-partial task results from the other set of DST execution units; and

compiling the sub-partial task results to produce the partial task result.

10. The method of claim 1, wherein the performing the partial task comprises:

configuring an execution unit to perform the partial task based on content of the partial task.

11. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

receive a partial task regarding an encoded data block grouping of a plurality of encoded data block groupings, wherein data is dispersed storage error encoded in accordance with dispersed storage error coding parameters to produce the plurality of encoded data block groupings;

a second module, when operable within the computing device, causes the computing device to;

perform the partial task on the encoded data block grouping to produce a partial task result; and

a third module, when operable within the computing device, causes the computing device to;

determine subsequent treatment of the partial task result and of the encoded data block grouping;

when the subsequent treatment includes maintaining storage of the encoded data block grouping and storage of the partial task result;

determining a manner in which the partial task result is to be stored; and

when the manner in which the partial task result is to be stored is dispersed storage;

dispersed storage error encoding the partial task result in accordance with dispersed storage error encoding parameters to produce one or more sets of encoded partial task result blocks; and

outputting the one or more sets of encoded partial task result blocks to a set of dispersed storage and task (DST) execution units for storage therein.

12. The DS module of claim 11, wherein the second module functions to perform the partial task by:

decoding the encoded data block grouping in accordance with the dispersed storage error coding parameters to produce a partition of the data; and

performing the partial task on the partition of the data to produce the partial task result.

13. The DS module of claim 12 further comprises:

the second module further functions to;

determine whether the encoded data block grouping is encrypted; and

when the encoded data block grouping is encrypted;

determine sensitivity of the partition of the data;

when the sensitivity of the partition of the data is of a first sensitivity;

decrypt the encoded data block grouping to produce the encoded data block grouping;

temporarily store the partition of the data; and

when the partial task has been performed, delete the temporary storage of the partition of the data.

14. The DS module of claim 11, wherein the third module functions to determine the subsequent treatment by at least one of:

extracting subsequent treatment information from the partial task;

analyzing the partial task result for one or more result criteria to determine the subsequent treatment;

sending a query to another device within a distributed computing system to ascertain the subsequent treatment; and

analyzing the encoded data block grouping to identify source data criteria to determine the subsequent treatment.

15. The DS module of claim 11, wherein the first module functions to receive the partial task by:

receiving a set of encoded task data blocks;

determining dispersed storage error decoding parameters regarding the partial task; and

decoding the set of encoded task data blocks to recover the partial task.

16. The DS module of claim 11 further comprises:

when the manner in which the partial task result is to be stored is store locally, the third module further functions to store the partial task result in a local memory.

17. The DS module of claim 11 further comprises:

when the manner in which the partial task result is to be stored is group dispersed storage, the third module further functions to;

coordinate with other DST execution units of a distributed computing system to collect a set of partial task results, wherein the set of partial task results includes the partial task result and partial task results of the other DST execution units that performed a corresponding partial task on other encoded data block groupings of the plurality of encoded data block groupings; and

disperse storage error encode the set of partial task results to produce an error encoded result data block.

18. The DS module of claim 11 further comprises:

when the subsequent treatment includes overwriting the encoded data block grouping with the partial task result, the third module further functions to;

overwrite the encoded data block grouping with the partial task result; and

coordinate with other DST execution units of a distributed computing system to update redundancy data blocks of the plurality of encoded data block groupings based on the partial task results.

19. The DS module of claim 11, wherein the second module functions to perform the partial task by:

determining the DS module'"'"'s ability to fulfill the partial task in a reasonable time frame; and

when the DS module cannot fulfill the partial task in the reasonable time frame;

partitioning the partial task into a set of sub-partial tasks;

portioning the encoded data block grouping into a set of encoded data block sub-groupings;

sending the set of sub-partial tasks and set of encoded data block sub-groupings to another set of DST execution units of a distributed computing system;

receiving sub-partial task results from the other set of DST execution units; and

compiling the sub-partial task results to produce the partial task result.

20. The DS module of claim 11, wherein the second module functions to perform the partial task by:

configuring an execution unit to perform the partial task based on content of the partial task.