Transferring task execution in a distributed storage and task network

• US 9,588,994 B2
• Filed: 01/29/2013
• Issued: 03/07/2017
• Est. Priority Date: 03/02/2012
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): Obisesan, Augustine K
• Assistant Examiner(s): HTAY, LIN LIN M
• Application Number:
  US13/753,418
• Publication Number:
  US 20130232180A1
• Time in Patent Office: 1,498 Days
• Field of Search: 707/827
• US Class Current: 1/1
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module ascertaining processing speeds of distributed storage and task (DST) execution units where the DST execution units receive a set of encoded data slices that includes a sub-set of data-based data slices. The method continues with the DS processing module allocating task performance on the sub-set of data-based data slices to a sub-set of DST execution units, where a first DST execution unit is allocated to perform a first partial task on a first data-based data slice of the sub-set of data-based data slices. When a second DST execution unit has a processing speed that is a threshold speed greater than a processing speed of the first DST execution unit, the method continues with the DS processing module identifying at least one encoded block of the first data-based data slice for transferring processing responsibilities to the second DST execution unit.

Claims

1. A method for execution by a computer to manage distributed computing of a task on data, the method comprises:

ascertaining processing speeds of distributed storage and task (DST) execution units of a set of DST execution units performing tasks similar to the task on other data that is similar to the data, the ascertaining processing speeds including at least a determination that one or more of the distributed storage and task (DST) execution units of the set of DST execution units can execute at least one or more of the tasks at a threshold speed greater than another of the distributed storage and task (DST) execution units of a set of DST execution units;

receiving, by the set of DST execution units, a set of encoded data slices that includes a sub-set of data-based data slices and a sub-set of redundancy-based data slices, wherein a data segment of the data is dispersed storage error encoded to produce the set of encoded data slices;

allocating performance of the task on the sub-set of data-based data slices to a sub-set of the set of DST execution units, wherein a first DST execution unit of the sub-set of DST execution units is allocated to perform a first partial task of the task on a first data-based data slice of the sub-set of data-based data slices on an encoded block by encoded block basis; and

when, based on the ascertained processing speeds, a second DST execution unit of the set of DST execution units has a processing speed that is the threshold speed greater than a processing speed of the first DST execution unit;

identifying at least one encoded block of the first data-based data slice for transferring processing responsibilities from the first DST execution unit to the second DST execution unit;

obtaining the at least one encoded block by the second DST execution unit; and

performing, by the second DST execution unit, a second partial task of the task on the at least one encoded block; and

wherein the second DST execution unit having the processing speed that is the threshold speed greater than the processing speed of the first DST execution unit is based on;

determining that the second DST execution unit can complete performance of the second partial task on the second partial task on encoded blocks of a second data-based data slice of the sub-set of data-based data slices and on the at least one encoded block before the first DST execution can commence performing the first partial task on the at least one encoded block.

2. The method of claim 1, wherein the ascertaining processing speeds comprises at least one of:

determining a number of encoded blocks processed in a given time frame on a per DST execution unit basis; and

determining, on the per DST execution unit basis, a speed at which an encoded block for a given task is processed.

3. The method of claim 1 further comprises:

converting a data segment of data into a data matrix that includes a plurality of data blocks;

encoding the data matrix with an encoding matrix to produce encoded blocks that includes a plurality of data-based data blocks and a plurality of redundancy-based data blocks;

arranging the plurality of data-based data blocks into the sub-set of data-based data slices; and

arranging the plurality of redundancy-based data blocks into the sub-set of redundancy-based data slices.

4. The method of claim 1, wherein the identifying the at least one encoded block comprises:

after the second DST execution unit has completed performance of the second partial task on encoded blocks of a second data-based data slice of the sub-set of data-based data slices, determining how many potentially transferred encoded blocks the second DST execution unit can complete performance of the second partial task on before the first DST execution unit can commence performing the first partial task on the potentially transferred encoded blocks.

5. The method of claim 1 further comprises:

when, based on the ascertained processing speeds, a third DST execution unit of the set of DST execution units has a processing speed that is the threshold speed greater than the processing speed of the first DST execution unit;

identifying at least two encoded blocks of the first data-based data slice for transferring the processing responsibilities from the first DST execution unit to the second DST execution unit and to the third DST execution unit;

obtaining a first one of the at least two encoded blocks by the second DST execution unit;

obtaining a second one of the at least two encoded blocks by the third DST execution unit;

performing, by the second DST execution unit, the second partial task on the first one of the at least two encoded blocks; and

performing, by the third DST execution unit, a third partial task of the task on the second one of the at least two encoded blocks.

6. The method of claim 1 further comprises at least one of:

the second DST execution unit is within the sub-set of the set of DST execution units; and

the second DST execution unit is a DST execution unit of the set of DST execution units storing one of the sub-set of redundancy-based data slices.

7. The method of claim 1, wherein the obtaining the at least one encoded block comprises:

retrieving a threshold number of encoded blocks of an encoded matrix;

rebuilding a grouping of data blocks of a data matrix from the threshold number of encoded blocks;

dispersed storage error encoding the grouping of data blocks of the data matrix to produce a partial rebuilt first data-based data slice; and

selecting the at least one encoded block from the partial rebuild first data-based data slice.

8. The method of claim 1, wherein the obtaining the at least one encoded block comprises:

sending, by the second DST execution unit, a request to the first DST execution unit for the at least one encoded block; and

receiving, in response to the request, the at least one encoded block from the first DST execution unit.

9. A method for execution by a computer to manage distributed computing of a task on data, the method comprises:

ascertaining processing speeds of distributed storage and task (DST) execution units of a set of DST execution units performing tasks similar to the task on other data that is similar to the data;

receiving, by the set of DST execution units, a set of encoded data slices that includes a sub-set of data-based data slices and a sub-set of redundancy-based data slices, wherein a data segment of the data is dispersed storage error encoded to produce the set of encoded data slices;

allocating performance of the task on the sub-set of data-based data slices to a sub-set of the set of DST execution units, wherein a first DST execution unit of the sub-set of DST execution units is allocated to perform a first partial task of the task on a first data-based data slice of the sub-set of data-based data slices on an encoded block by encoded block basis; and

when, based on the ascertained processing speeds, a second DST execution unit of the set of DST execution units has a processing speed that is a threshold speed greater than a processing speed of the first DST execution unit;

identifying at least one encoded block of the first data-based data slice for transferring processing responsibilities from the first DST execution unit to the second DST execution unit;

obtaining the at least one encoded block by the second DST execution unit; and

performing, by the second DST execution unit, a second partial task of the task on the at least one encoded block;

wherein the second DST execution unit having the processing speed that is the threshold speed greater than the processing speed of the first DST execution unit is based on;

determining that the second DST execution unit can complete performance of the second partial task on the second partial task on encoded blocks of a second data-based data slice of the sub-set of data-based data slices and on the at least one encoded block before the first DST execution can commence performing the first partial task on the at least one encoded block; and

when, based on the ascertained processing speeds, a third DST execution unit of the set of DST execution units has a processing speed that is the threshold speed greater than the processing speed of the first DST execution unit;

identifying at least two encoded blocks of the first data-based data slice for transferring the processing responsibilities from the first DST execution unit to the second DST execution unit and to the third DST execution unit;

obtaining a first one of the at least two encoded blocks by the second DST execution unit obtaining a second one of the at least two encoded blocks by the third DST execution unit performing, by the second DST execution unit, the second partial task on the first one of the at least two encoded blocks; and

performing, by the third DST execution unit, a third partial task of the task on the second one of the at least two encoded blocks.

10. The method of claim 9, wherein the ascertaining processing speeds comprises at least one of:

determining a number of encoded blocks processed in a given time frame on a per DST execution unit basis; and

determining, on the per DST execution unit basis, a speed at which an encoded block for a given task is processed.

11. The method of claim 9 further comprises:

converting a data segment of data into a data matrix that includes a plurality of data blocks;

encoding the data matrix with an encoding matrix to produce encoded blocks that includes a plurality of data-based data blocks and a plurality of redundancy-based data blocks;

arranging the plurality of data-based data blocks into the sub-set of data-based data slices; and

arranging the plurality of redundancy-based data blocks into the sub-set of redundancy-based data slices.

12. The method of claim 9, wherein the identifying the at least one encoded block comprises:

after the second DST execution unit has completed performance of the second partial task on encoded blocks of a second data-based data slice of the sub-set of data-based data slices, determining how many potentially transferred encoded blocks the second DST execution unit can complete performance of the second partial task on before the first DST execution unit can commence performing the first partial task on the potentially transferred encoded blocks.

13. The method of claim 9 further comprises at least one of:

the second DST execution unit is within the sub-set of the set of DST execution units; and

the second DST execution unit is a DST execution unit of the set of DST execution units storing one of the sub-set of redundancy-based data slices.

14. The method of claim 9, wherein the obtaining the at least one encoded block comprises:

retrieving a threshold number of encoded blocks of an encoded matrix;

rebuilding a grouping of data blocks of a data matrix from the threshold number of encoded blocks;

dispersed storage error encoding the grouping of data blocks of the data matrix to produce a partial rebuilt first data-based data slice; and

selecting the at least one encoded block from the partial rebuild first data-based data slice.

15. The method of claim 9, wherein the obtaining the at least one encoded block comprises:

sending, by the second DST execution unit, a request to the first DST execution unit for the at least one encoded block; and

receiving, in response to the request, the at least one encoded block from the first DST execution unit.