Efficiently storing data in a dispersed storage network

• US 9,203,901 B2
• Filed: 12/06/2012
• Issued: 12/01/2015
• Est. Priority Date: 01/31/2012
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): THAI, TUAN V
• Application Number:
  US13/707,118
• Publication Number:
  US 20130198560A1
• Time in Patent Office: 1,090 Days
• Field of Search: 711/100, 711/111, 711/112, 711/114, 711/154, 711/170
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1096 Parity calculation or recalculation after configuration or reconfiguration of the system
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/2053 where persistent mass storage functionality or persistent mass storage control functionality is redundant error detection or correction in information storage based on relative movement between record carrier and transducer G11B20/18
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module ingesting a plurality of data blocks of a large amount of data. The method continues with the DS processing module temporarily storing the plurality of data blocks and determining whether to not permanently store the plurality of data blocks. When the plurality of data blocks is to be permanently stored, the method continues with the DS processing module storing the plurality of data blocks and generating a group of partial redundancy data. When the plurality of data blocks is not to be permanently stored, the method continues with the DS processing module creating a new plurality of data blocks from other data blocks that are to be permanently stored and generating the group of partial redundancy data based on the new plurality of data blocks.

Claims

1. A method for execution by a dispersed storage (DS) unit of a dispersed storage network (DSN), the method comprises:

ingesting a respective section of data, wherein a large amount of data is divided into sets of partitions, wherein each of the sets of partitions is divided into a set of sections, wherein a set of DS units ingests the set of sections, and wherein the set of DS units includes the DS unit;

for the ingested respective section of data;

dividing the ingested respective section of data into a plurality of data segments;

for a data segment of the plurality of data segments;

dividing the data segment into a plurality of data blocks;

temporarily storing the plurality of data blocks;

determining whether to not permanently store one or more of the plurality of data blocks;

when the one or more of the plurality of data blocks is to be permanently stored;

storing the one or more of the plurality of data blocks; and

generating a group of partial redundancy data based on the one or more of the plurality of data blocks and in accordance with a dispersed storage error coding function; and

when the one or more of the plurality of data blocks is not to be permanently stored;

creating a new plurality of data blocks from data blocks of the plurality of data blocks that are to be permanently stored and data blocks from another data segment that are to be permanently stored; and

generating the group of partial redundancy data based on the new plurality of data blocks in accordance with the dispersed storage error coding function.

2. The method of claim 1, wherein the determining whether to not permanently store the one or more of the plurality of data blocks comprises:

determining that at least one other DS unit of the set of DS units has ingested a respective section that is substantially similar to the ingested respective section; and

in response to the determining that the at least one other DS unit has ingested the respective section that is substantially similar to the ingested respective section;

determining whether a de-duplication function is to be applied to the ingested respective section;

when the de-duplication function is to be applied, identifying the one or more of the plurality of data blocks that are not to be permanently stored; and

when the de-duplication function is not to be applied, indicating that the one or more of the plurality of data blocks is to be permanently stored.

3. The method of claim 1, wherein the determining whether to not permanently store the one or more of the plurality of data blocks comprises:

analyzing the ingested respective section in accordance with data analysis criteria; and

when the analysis of the ingested respective section is unfavorable, identifying the one or more of the plurality of data blocks that are not to be permanently stored.

4. The method of claim 1, wherein the generating the group of partial redundancy data based on the one or more of the plurality of data blocks comprises:

generating a first partial redundancy data for a first redundancy DS unit based on at least some of the plurality of data blocks and a first encoding parameter of the dispersed storage error coding function; and

generating a second partial redundancy data for a second redundancy DS unit based on at least another some of the plurality of data blocks and a second encoding parameter of the dispersed storage error coding function.

5. The method of claim 1, wherein the creating the new plurality of data blocks comprises:

identifying the other data segment as a data segment for which partial redundancy data does not yet exist.

6. The method of claim 1 further comprises:

permanently storing the new plurality of data blocks.

7. A dispersed storage (DS) module of a dispersed storage (DS) unit of a dispersed storage network (DSN), the DS module comprises:

a first module, when operable within a computing device, causes the computing device to;

ingest a respective section of data, wherein a large amount of data is divided into sets of partitions, wherein each of the sets of partitions is divided into a set of sections, wherein a set of DS units ingests the set of sections, and wherein the set of DS units includes the DS unit;

a second module, when operable within the computing device, causes the computing device to;

for the ingested respective section of data;

divide the ingested respective section of data into a plurality of data segments;

for a data segment of the plurality of data segments;

divide the data segment into a plurality of data blocks;

temporarily store the plurality of data blocks; and

determine whether to not permanently store one or more of the plurality of data blocks;

a third module, when operable within the computing device, causes the computing device to;

when the one or more of the plurality of data blocks is to be permanently stored;

store the one or more of the plurality of data blocks; and

generate a group of partial redundancy data based on the one or more of the plurality of data blocks and in accordance with a dispersed storage error coding function; and

a fourth module, when operable within the computing device, causes the computing device to;

when the one or more of the plurality of data blocks is not to be permanently stored;

create a new plurality of data blocks from data blocks of the plurality of data blocks that are to be permanently stored and data blocks from another data segment that are to be permanently stored; and

generate the group of partial redundancy data based on the new plurality of data blocks in accordance with the dispersed storage error coding function.

8. The DS module of claim 7, wherein the second module functions to determine whether to not permanently store the one or more of the plurality of data blocks by:

determining that at least one other DS unit of the set of DS units has ingested a respective section that is substantially similar to the ingested respective section; and

in response to the determining that the at least one other DS unit has ingested the respective section that is substantially similar to the ingested respective section;

determining whether a de-duplication function is to be applied to the ingested respective section;

when the de-duplication function is to be applied, identifying the one or more of the plurality of data blocks that are not to be permanently stored; and

when the de-duplication function is not to be applied, indicating that the one or more of the plurality of data blocks is to be permanently stored.

9. The DS module of claim 7, wherein the second module functions to determine whether to not permanently store the one or more of the plurality of data blocks by:

analyzing the ingested respective section in accordance with data analysis criteria; and

when the analysis of the ingested respective section is unfavorable, identifying the one or more of the plurality of data blocks that are not to be permanently stored.

10. The DS module of claim 7, wherein the third module functions to generate the group of partial redundancy data based on the one or more of the plurality of data blocks by:

generating a first partial redundancy data for a first redundancy DS unit based on at least some of the plurality of data blocks and a first encoding parameter of the dispersed storage error coding function; and

generating a second partial redundancy data for a second redundancy DS unit based on at least another some of the plurality of data blocks and a second encoding parameter of the dispersed storage error coding function.

11. The DS module of claim 7, wherein the fourth module functions to create the new plurality of data blocks by:

identifying the other data segment as a data segment for which partial redundancy data does not yet exist.

12. The DS module of claim 7 further comprises:

the fourth module further functions to permanently store the new plurality of data blocks.