Selectable parallel processing of dispersed storage error encoding

• US 10,324,791 B2
• Filed: 11/28/2016
• Issued: 06/18/2019
• Est. Priority Date: 11/01/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Dhuse, Greg R., Resch, Jason K., Volvovski, Ilya
• Primary Examiner(s): Shanmugasundaram, Kannan
• Application Number:
  US15/362,460
• Publication Number:
  US 20170075763A1
• Time in Patent Office: 932 Days
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 2212/7205 Cleaning, compaction, garbage collection, erase control
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A method includes a computing device determining dispersed storage error encoded parameters for dispersed storage error encoding a data object. The method further includes the computing device dividing the dispersed storage error encoding into a plurality of encoding tasks. The method further includes allocating, by the computing device, the encoding tasks to a plurality of central processing units (CPUs) available to the computing device. The method further includes dispersed storage error encoding, by the CPUs, the data segments into the plurality of sets of encoded data slices in accordance with the encoding tasks. The method further includes sending, by the computing device, the plurality of sets of encoded data slices to storage units of the DSN for storage therein.

Claims

1. A method comprises:

determining, by a computing device of a dispersed storage network (DSN), dispersed storage error encoded parameters to be utilized in a dispersed storage error encoding process of a data object, wherein the dispersed storage error encoding process includes dividing the data object into a plurality of data segments and dispersed storage error encoding the plurality of the data segments into a plurality of sets of encoded data slices;

dividing, by the computing device, the dispersed storage error encoding process into a plurality of encoding tasks, wherein dividing the dispersed storage error encoding process into the plurality of encoding tasks includes;

creating a first encoding task of the plurality of encoding tasks, wherein the first encoding task is to perform a first matrix multiplication on a first portion of an encoded matrix and a first portion of a data matrix to produce one or more encoded data slices of a set of encoded data slices of the plurality of sets of encoded data slices; and

creating a second encoding task of the plurality of encoding tasks, wherein the second encoding task is to perform a second matrix multiplication on a second portion of the encoded matrix and a second portion of the data matrix to produce another one or more encoded data slices of the set of encoded data slices;

allocating, by the computing device, the plurality of encoding tasks to a plurality of central processing units (CPUs) available to the computing device;

dispersed storage error encoding, by the plurality of CPUs, the plurality of data segments into the plurality of sets of encoded data slices in accordance with the plurality of encoding tasks; and

sending, by the computing device, the plurality of sets of encoded data slices to storage units of the DSN for storage therein.

2. The method of claim 1 further comprises:

determining a number of central processing units available for the dispersed storage error encoding process of the data object; and

selecting the plurality of CPUs from the number of central processing units.

3. The method of claim 1, wherein dividing the dispersed storage error encoding process into a plurality of encoding tasks comprises:

creating an encoding task of the plurality of encoding tasks to be the dividing of the data object into the plurality of data segments.

4. The method of claim 1, wherein dividing the dispersed storage error encoding process into a plurality of encoding tasks comprises:

creating an encoding task of the plurality of encoding tasks to generate the data matrix for a data segment of the plurality of data segments.

5. The method of claim 1, wherein dividing the dispersed storage error encoding process into a plurality of encoding tasks comprises:

creating a first intermediate encoding task of the first encoding task to perform a first intermediate matrix multiplication on a first sub-portion of the first portion of the encoded matrix and a first sub-portion of the first portion of the data matrix to produce one or more encoded data slice portions of an encoded data slice of the one or more encoded data slices; and

creating a second intermediate encoding task of the first encoding task to perform a second intermediate matrix multiplication on a second sub-portion of the first portion of the encoded matrix and a second sub-portion of the first portion of the data matrix to produce another one or more encoded data slice portions of the encoded data slice; and

creating a third intermediate encoding task of the first encoding task to aggregate the one or more encoded data slices portions and the other one or more encoded data slice portions to produce the encoded data slice.

6. The method of claim 1, wherein dividing the dispersed storage error encoding process into a plurality of encoding tasks comprises:

creating a first plurality of encoding sub-task of the plurality of encoding tasks to perform a plurality of component level matrix multiplications on coefficients of the encoding matrix and data blocks of the data matrix to produce a plurality of matrix multiplied components; and

creating a second plurality of encoding sub-task of the plurality of encoding tasks to combine plurality of matrix multiplied components into the set of encoded data slices of the plurality of sets of encoded data slices.

7. A computer readable memory comprises:

a first memory section that stores operational instructions that, when executed by a computing device of a dispersed storage network (DSN), causes the computing device to;

determine dispersed storage error encoded parameters to be utilized in a dispersed storage error encoding process of a data object, wherein the dispersed storage error encoding process includes dividing the data object into a plurality of data segments and dispersed storage error encoding the plurality of the data segments into a plurality of sets of encoded data slices;

divide the dispersed storage error encoding process into a plurality of encoding tasks, wherein dividing the dispersed storage error encoding process into the plurality of encoding tasks includes;

creating a first encoding task of the plurality of encoding tasks, wherein the first encoding task is to perform a first matrix multiplication on a first portion of an encoded matrix and a first portion of a data matrix to produce one or more encoded data slices of a set of encoded data slices of the plurality of sets of encoded data slices; and

creating a second encoding task of the plurality of encoding tasks, wherein the second encoding task is to perform a second matrix multiplication on a second portion of the encoded matrix and a second portion of the data matrix to produce another one or more encoded data slices of the set of encoded data slices; and

allocate the plurality of encoding tasks to a plurality of central processing units (CPUs) available to the computing device;

a second memory section that stores operational instructions that, when executed by the plurality of CPUs, causes the plurality of CPUs to;

dispersed storage error encode the plurality of data segments into the plurality of sets of encoded data slices in accordance with the plurality of encoding tasks; and

a third memory section that stores operational instructions that, when executed by the computing device, causes the computing device to;

send the plurality of sets of encoded data slices to storage units of the DSN for storage therein.

8. The computer readable memory of claim 7, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to:

determine a number of central processing units available for the dispersed storage error encoding process of the data object; and

select the plurality of CPUs from the number of central processing units.

9. The computer readable memory of claim 7, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to divide the dispersed storage error encoding process into a plurality of encoding tasks by:

creating an encoding task of the plurality of encoding tasks to be the dividing of the data object into the plurality of data segments.

10. The computer readable memory of claim 7, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to divide the dispersed storage error encoding process into the plurality of encoding tasks by:

creating an encoding task of the plurality of encoding tasks to generate a data matrix for a data segment of the plurality of data segments.

11. The computer readable memory of claim 7, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to divide the dispersed storage error encoding process into the plurality of encoding tasks by:

creating a first intermediate encoding task of the first encoding task to perform a first intermediate matrix multiplication on a first sub-portion of the first portion of the encoded matrix and a first sub-portion of the first portion of the data matrix to produce one or more encoded data slice portions of an encoded data slice of the one or more encoded data slices; and

creating a second intermediate encoding task of the first encoding task to perform a second intermediate matrix multiplication on a second sub-portion of the first portion of the encoded matrix and a second sub-portion of the first portion of the data matrix to produce another one or more encoded data slice portions of the encoded data slice; and

creating a third intermediate encoding task of the first encoding task to aggregate the one or more encoded data slices portions and the other one or more encoded data slice portions to produce the encoded data slice.

12. The computer readable memory of claim 7, wherein the first memory section further stores operational instructions that, when executed by the computing device, causes the computing device to divide the dispersed storage error encoding process into the plurality of encoding tasks by:

creating a first plurality of encoding sub-task of the plurality of encoding tasks to perform a plurality of component level matrix multiplications on coefficients of the encoding matrix and data blocks of the data matrix to produce a plurality of matrix multiplied components; and

creating a second plurality of encoding sub-task of the plurality of encoding tasks to combine plurality of matrix multiplied components into the set of encoded data slices of the plurality of sets of encoded data slices.