Storing portions of data in a dispersed storage network

• US 8,965,939 B2
• Filed: 02/18/2014
• Issued: 02/24/2015
• Est. Priority Date: 06/06/2011
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Leggette, Wesley, Resch, Jason K.
• Primary Examiner(s): TRIMMINGS, JOHN P
• Application Number:
  US14/182,482
• Publication Number:
  US 20140164877A1
• Time in Patent Office: 371 Days
• Field of Search: 714/763, 714/770, 707/827, 707/826, 707/825, 709/201, 709/203
• US Class Current: 707/827
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/6227 where protection concerns the structure of data, e.g. records, types, queries
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2212/263 Network storage, e.g. SAN or NAS
• CPC Class Codes: G06F 2221/2107 File encryption
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 8/65 Updates security arrangements therefor G06F21/57
• CPC Class Codes: H04L 2209/24 Key scheduling, i.e. generating round keys or sub-keys for block encryption
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 9/085 Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/0861 Generation of secret information including derivation or calculation of cryptographic keys or passwords

Abstract

A method begins, in accordance with a segmentation approach, dividing large data to be stored into regions and dividing a region into segments. The method continues by generating preliminary DSN storage information for one or more regions. The method continues by identifying other large data stored in the DSN that has a relationship with the large data to be stored in the DSN and retrieving DSN storage information for the other large data. The method continues by comparing, at a region level, the preliminary DSN storage information with the retrieved DSN storage information. When a region of the large data to be stored has substantially similar DSN storage information as a region of the other large data, the method continues by utilizing the DSN storage information for the region of the other large data for the DSN storage information of the region of the large data.

Claims

1. A method for execution by a processing module of a computing device, the method comprises:

in accordance with a segmentation approach;

dividing large data to be stored in a dispersed storage network (DSN) into a plurality of regions; and

dividing a region of the plurality of regions into a plurality of segments;

generating preliminary DSN storage information for one or more regions of the plurality of regions;

identifying other large data stored in the DSN that has a relationship with the large data to be stored in the DSN;

retrieving DSN storage information for the other large data;

comparing, at a region level, the preliminary DSN storage information with the retrieved DSN storage information; and

when a region of the large data to be stored has substantially similar DSN storage information as a region of the other large data, utilizing the DSN storage information for the region of the other large data for the DSN storage information of the region of the large data, wherein the DSN storage information for the region of the other large data includes information that identifies where the plurality of data segments of the region of the other large data are stored in the DSN such that duplication of storage of substantial similar regions of data is avoided in the DSN.

2. The method of claim 1 further comprises:

identifying the segmentation approach by;

determining a vault identifier based on one or more of a data identifier of the large data; and

accessing a registry based on the vault identifier to obtain obtaining the segmentation approach.

3. The method of claim 1 further comprises:

when a second region of the large data to be stored does not have substantially similar DSN storage information as a second region of the other large data;

dispersed storage error encoding the plurality of data segments of the second region to produce a plurality of sets of encoded data slices;

generating a plurality of sets of write commands to write the plurality of sets of encoded data slices into the DSN; and

generating DSN storage information for the second region.

4. The method of claim 3 further comprises:

generating a segment allocation table for the large data, wherein the segment allocation table includes the DSN storage information of the region of the other large data for the region of the data and includes the DSN storage information of the second region.

5. The method of claim 1 further comprises:

identifying multiple other large data stored in the DSN that each have a relationship with the large data to be stored in the DSN;

retrieving DSN storage information for each of the multiple other large data;

comparing, at the region level and on other large data by other large data basis, the preliminary DSN storage information with the retrieved DSN storage information of one of the multiple other large data; and

when the region of the large data to be stored has substantially similar DSN storage information as a region of the one of the multiple other large data, utilizing the DSN storage information for the region of the one of the multiple other large data for the DSN storage information of the region of the large data.

6. The method of claim 1, wherein the DSN storage information of the other large data comprises:

a segment allocation table that includes a plurality of regional sections corresponding to a plurality of regions of the other large data, wherein a regional section of the plurality of regional sections includes a start segment vault source name, a segment size, a segmentation approach, a total length, and a region hash.

7. The method of claim 1, wherein the identifying the other large data stored in the DSN that has a relationship with the large data to be stored in the DSN further comprises:

identifying the relationship based on one or more of;

a substantially similar data identifier, a substantially similar requesting entity identifier, a substantially similar user identifier, a substantially similar vault identifier, and a substantially similar group of users identifier.

8. A dispersed storage (DS) module comprises:

a first module, when operable within a computing device, causes the computing device to;

in accordance with a segmentation approach;

divide large data to be stored in a dispersed storage network (DSN) into a plurality of regions; and

divide a region of the plurality of regions into a plurality of segments; and

generate preliminary DSN storage information for one or more regions of the plurality of regions;

a second module, when operable within the computing device, causes the computing device to;

identify other large data stored in the DSN that has a relationship with the large data to be stored in the DSN; and

retrieve DSN storage information for the other large data; and

a third module, when operable within the computing device, causes the computing device to;

compare, at a region level, the preliminary DSN storage information with the retrieved DSN storage information; and

when a region of the large data to be stored has substantially similar DSN storage information as a region of the other large data, utilize the DSN storage information for the region of the other large data for the DSN storage information of the region of the large data, wherein the DSN storage information for the region of the other large data includes information that identifies where the plurality of data segments of the region of the other large data are stored in the DSN such that duplication of storage of substantial similar regions of data is avoided in the DSN.

9. The DS module of claim 8, wherein the first module further causes the computing device to:

identify the segmentation approach by;

determining a vault identifier based on one or more of a data identifier of the large data; and

accessing a registry based on the vault identifier to obtain obtaining the segmentation approach.

10. The DS module of claim 8, wherein the third module further causes the computing device to:

when a second region of the large data to be stored does not have substantially similar DSN storage information as a second region of the other large data;

dispersed storage error encode the plurality of data segments of the second region to produce a plurality of sets of encoded data slices;

generate a plurality of sets of write commands to write the plurality of sets of encoded data slices into the DSN; and

generate DSN storage information for the second region.

11. The DS module of claim 10, wherein the third module further causes the computing device to:

generate a segment allocation table for the large data, wherein the segment allocation table includes the DSN storage information of the region of the other large data for the region of the data and includes the DSN storage information of the second region.

12. The DS module of claim 8, wherein the second module further causes the computing device to:

identify multiple other large data stored in the DSN that each have a relationship with the large data to be stored in the DSN;

retrieve DSN storage information for each of the multiple other large data;

compare, at the region level and on other large data by other large data basis, the preliminary DSN storage information with the retrieved DSN storage information of one of the multiple other large data; and

when the region of the large data to be stored has substantially similar DSN storage information as a region of the one of the multiple other large data, utilize the DSN storage information for the region of the one of the multiple other large data for the DSN storage information of the region of the large data.

13. The DS module of claim 8, wherein the DSN storage information of the other large data comprises:

a segment allocation table that includes a plurality of regional sections corresponding to a plurality of regions of the other large data, wherein a regional section of the plurality of regional sections includes a start segment vault source name, a segment size, a segmentation approach, a total length, and a region hash.

14. The DS module of claim 8, wherein the second module further causes the computing device to identify the other large data stored in the DSN that has a relationship with the large data to be stored in the DSN by:

identifying the relationship based on one or more of;

a substantially similar data identifier, a substantially similar requesting entity identifier, a substantially similar user identifier, a substantially similar vault identifier, and a substantially similar group of users identifier.