Creating a new file for a dispersed storage network

• US 9,584,326 B2
• Filed: 11/21/2012
• Issued: 02/28/2017
• Est. Priority Date: 11/28/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Dhuse, Greg, Leggette, Wesley, Resch, Jason K.
• Primary Examiner(s): To, Baoquoc N
• Application Number:
  US13/683,998
• Publication Number:
  US 20130138706A1
• Time in Patent Office: 1,560 Days
• Field of Search: 707/812, 707/828, 707/737, 711/E12.014
• US Class Current: 1/1
• CPC Class Codes: G06F 11/108 Parity data distribution in semiconductor storages, e.g. in SSD
• CPC Class Codes: G06F 12/0607 Interleaved addressing
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 21/00 Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 65/10 Architectures or entities
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/0825 using asymmetric-key encryption or public key infrastructure [PKI], e.g. key signature or public key certificates
• CPC Class Codes: H04L 9/3236 using cryptographic hash functions

Abstract

A method begins by a dispersed storage (DS) processing module receiving a file creation request to create a new file from portions of a set of existing files. The method continues with the DS processing module identifying data segments associated with the set of existing files corresponding to the portions of the set of existing files to produce identified data segments and identifying sets of encoded data slices of the identified data segments to produce identified sets of encoded data slices. The method continues with the DS processing module determining addressing information for the identified sets of encoded data slices from existing addressing information of the set of existing files and generating a segment allocation table for the new file based on the addressing information of the identified sets of encoded data slices such that the new file is created without duplication of the portions of the existing files.

Claims

1. A method for execution by a computing device, the method comprises:

receiving a file creation request to create a new data file from portions of a set of existing files containing data, wherein an existing file of the set of existing files includes a plurality of data segments that is stored as a plurality of sets of encoded data slices in a dispersed storage network (DSN) memory;

identifying, in accordance with the file creation request, data segments from the pluralities of data segments of the set of existing files corresponding to the portions of the set of existing files to produce identified data segments;

identifying, based on the identified data segments, sets of encoded data slices of the pluralities of sets of encoded data slices of the identified data segments to produce identified sets of encoded data slices;

determining addressing information for the identified sets of encoded data slices from existing addressing information of the set of existing files; and

creating the new data file by generating a segment allocation table for the new data file based on the addressing information of the identified sets of encoded data slices such that the new data file is created without duplication of the portions of the existing files containing the data.

2. The method of claim 1 further comprises:

the existing file of the set of existing files further includes a set of data regions, wherein a data region of the set of data regions includes a consecutive subset of the plurality of data segments;

identifying, in accordance with the file creation request, data regions from the set of data regions of the set of existing files corresponding to the portions of the set of existing files to produce identified data regions;

identifying, based on the identified data regions, the sets of encoded data slices of the plurality of sets of encoded data slices of the identified data regions to produce the identified sets of encoded data slices; and

determining the addressing information for the identified sets of encoded data slices from a set of segment allocation tables associated with the set of existing files, wherein the set of segment allocation tables includes the existing addressing information of the set of existing files at data region level.

3. The method of claim 2, wherein the identifying the sets of encoded data slices comprises:

retrieving a set of segment allocation tables for the set of existing files based on the file creation request, wherein a segment allocation table of the set of segment allocation tables includes a plurality of entries, and wherein an entry of the plurality of entries includes a DSN address regarding storage of a data region of the set of data segments of the set of existing files in the DSN memory and segmentation information regarding the dividing of the data region into the consecutive subset of the plurality of data segments.

4. The method of claim 2, wherein addressing information for an identified set of encoded data slices of the identified sets of encoded data slices comprises:

a start segment vault source name corresponding to a data region associated with the identified set of encoded data slices; and

segmentation information including one or more of;

a segment size of data segments of the data region;

a total length of the data region; and

a segmentation approach of the data region.

5. The method of claim 1, wherein the generating the segment allocation table for the new data file comprises:

identifying consecutive data segments of the identified data segments;

grouping the consecutive data segments into a region; and

generating addressing information for the region, which is stored as an entry in the segment allocation table, wherein individual addressing information of the identified sets of encoded data slices is determinable from the addressing information of the region.

6. The method of claim 1, wherein the identifying data segments comprises at least one of:

receiving, within the file creation request, identification of the identified data segments; and

searching, based on a search criteria, the set of existing files, or addressing information thereof, to identify the data segments.

7. The method of claim 1, wherein the identifying the sets of encoded data slices comprises:

retrieving a set of segment allocation tables for the set of existing files based on the file creation request, wherein a segment allocation table of the set of segment allocation tables includes a plurality of entries, and wherein an entry of the plurality of entries includes a DSN address regarding storage of a data segment of the plurality of data segments in the DSN memory.

8. The method of claim 1 further comprises:

obtaining a segment allocation table vault source name for the segment allocation table;

encoding the segment allocation table using a dispersed storage error coding function to produce encoded table slices; and

outputting the encoded table slices to the DSN memory for storage therein utilizing the segment allocation table vault source name.

9. A dispersed storage (DS) computing device comprises:

an interface;

memory; and

a processing circuit operably coupled to the interface and to the memory, wherein the processing circuit is operable to;

receive a file creation request to create a new data file from portions of a set of existing files containing data, wherein an existing file of the set of existing files includes a plurality of data segments that is stored as a plurality of sets of encoded data slices in a dispersed storage network (DSN) memory;

identify, in accordance with the file creation request, data segments from the plurality of data segments of the set of existing files corresponding to the portions of the set of existing files to produce identified data segments;

identify, based on the identified data segments, sets of encoded data slices of the plurality of sets of encoded data slices of the identified data segments to produce identified sets of encoded data slices;

determine addressing information for the identified sets of encoded data slices from existing addressing information of the set of existing files; and

create the new data file by generating a segment allocation table for the new data file based on the addressing information of the identified sets of encoded data slices such that the new data file is created without duplication of the portions of the existing files containing the data.

10. The DS computing device of claim 9 further comprises:

the existing file of the set of existing files further includes a set of data regions, wherein a data region of the set of data regions includes a consecutive subset of the plurality of data segments;

the processing circuit further functions to;

identify, in accordance with the file creation request, data regions from the set of data regions of the set of existing files corresponding to the portions of the set of existing files to produce identified data regions;

identify, based on the identified data regions, the sets of encoded data slices of the plurality of sets of encoded data slices of the identified data regions to produce the identified sets of encoded data slices; and

determine the addressing information for the identified sets of encoded data slices from a set of segment allocation tables associated with the set of existing files, wherein the set of segment allocation tables includes the existing addressing information of the set of existing files at data region level.

11. The DS computing device of claim 10, wherein the processing circuit functions to identify the sets of encoded data slices by:

retrieving a set of segment allocation tables for the set of existing files based on the file creation request, wherein a segment allocation table of the set of segment allocation tables includes a plurality of entries, and wherein an entry of the plurality of entries includes a DSN address regarding storage of a data region of the set of data segments of the set of existing files in the DSN memory and segmentation information regarding the dividing of the data region into the consecutive subset of the plurality of data segments.

12. The DS computing device of claim 10, wherein addressing information for an identified set of encoded data slices of the identified sets of encoded data slices comprises:

a start segment vault source name corresponding to a data region associated with the identified set of encoded data slices; and

segmentation information including one or more of;

a segment size of data segments of the data region;

a total length of the data region; and

a segmentation approach of the data region.

13. The DS computing device of claim 9, wherein the processing circuit functions to generate the segment allocation table for the new data file by:

identifying consecutive data segments of the identified data segments;

grouping the consecutive data segments into a region; and

generating addressing information for the region, which is stored as an entry in the segment allocation table, wherein individual addressing information of the identified sets of encoded data slices is determinable from the addressing information of the region.

14. The DS computing device of claim 9, wherein the processing circuit functions to identify data segments by at least one of:

receiving, within the file creation request, identification of the identified data segments; and

searching, based on a search criteria, the set of existing files, or addressing information thereof, to identify the data segments.

15. The DS computing device of claim 9, wherein the processing circuit functions to identify the sets of encoded data slices by:

retrieving a set of segment allocation tables for the set of existing files based on the file creation request, wherein a segment allocation table of the set of segment allocation tables includes a plurality of entries, and wherein an entry of the plurality of entries includes a DSN address regarding storage of a data segment of the plurality of data segments in the DSN memory.

16. The DS computing device of claim 9 further comprises:

the processing circuit further functions to;

obtain a segment allocation table vault source name for the segment allocation table;

encode the segment allocation table using a dispersed storage error coding function to produce encoded table slices; and

output the encoded table slices to the DSN memory for storage therein utilizing the segment allocation table vault source name.