Segmenting data for storage in a dispersed storage network

• US 8,892,845 B2
• Filed: 12/01/2011
• Issued: 11/18/2014
• Est. Priority Date: 12/22/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Volvovski, Ilya, Baptist, Andrew, Leggette, Wesley
• Primary Examiner(s): Dudek, Jr., Edward
• Application Number:
  US13/308,889
• Publication Number:
  US 20120166868A1
• Time in Patent Office: 1,083 Days
• Field of Search: 711/206, 711/157, 711/E12.066
• US Class Current: 711/206
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 12/1009 using page tables, e.g. page table structures
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a processing module receiving data of a file for storage in a dispersed storage network (DSN) memory and determining a segmentation scheme for storing the data. The method continues with the processing module determining how to store the data in accordance with the segmentation scheme to produce information for storing the data and generating an entry within a segment allocation table associated with the file, wherein the entry includes the information for storing the data and the segmentation scheme. The method continues with the processing module facilitating storage of the segment allocation table in the DSN memory. The method continues with the processing module segmenting the data in accordance with the segmentation scheme to produce a plurality of data segments and facilitating storage of the plurality of data segments in the DSN memory in accordance with the information for storing the data.

Claims

1. A method comprises:

receiving data of a file for storage in a dispersed storage network (DSN) memory;

determining a segmentation scheme for storing the data;

determining how to store the data in accordance with the segmentation scheme to produce information for storing the data;

generating an entry within a segment allocation table associated with the file, wherein the entry includes the information for storing the data and the segmentation scheme;

facilitating storage of the segment allocation table in the DSN memory by;

obtaining a segment allocation table vault source name;

dispersed storage error encoding the segment allocation table to produce encoded table slices; and

outputting the encoded table slices to the DSN memory for storage therein utilizing the segment allocation table vault source name;

segmenting the data in accordance with the segmentation scheme to produce a plurality of data segments; and

facilitating storage of the plurality of data segments in the DSN memory in accordance with the information for storing the data.

2. The method of claim 1, wherein the determining the segmentation scheme is based on at least one of:

a previous segmentation scheme;

a data size indicator;

a data type indicator;

a storage requirement;

a vault identifier (ID);

a lookup;

a message; and

a query.

3. The method of claim 1 further comprises:

the information for storing the data including;

a start segment vault source name; and

a total length of the data; and

the segmenting scheme including;

a segment size; and

a segmentation approach.

4. The method of claim 3, wherein the segmenting scheme comprises:

a fixed segmentation approach and a corresponding fixed segment size such that each of the plurality of data segments has a size no greater than the fixed segment size.

5. The method of claim 3, wherein the segmenting scheme comprises:

a varying segmentation approach and an initial segment size such that a data segment of the plurality of data segments has a size corresponding to the initial segment size and remaining data segments of the plurality of data segments have a size based on the initial segment size and the varying segmentation approach.

6. The method of claim 1, wherein the facilitating storage of the plurality of data segments in the DSN memory comprises:

for each data segment of the plurality of data segments;

dispersed storage error encoding the data segment to produce a set of encoded data slices;

generating a set of slice names corresponding to the set of encoded data slices based on a start segment vault source name of the segment allocation table, wherein a common segment number of the set of slice names includes a sequentially increasing segment number; and

sending the set of encoded data slices and the set of slice names to the DSN memory.

7. The method of claim 1, wherein the facilitating storage of the plurality of data segments in the DSN memory further comprises:

for each data segment of the plurality of data segments;

dispersed storage error encoding the data segment to produce a set of encoded data slices;

generating a set of slice names corresponding to the set of encoded data slices based on a start segment vault source name of the segment allocation table, wherein an append marker field of the set of slice names includes a sequentially increasing append marker; and

sending the set of encoded data slices and the set of slice names to the DSN memory.

8. A dispersed storage (DS) processing unit comprises:

a first module for receiving data of a file for storage in a dispersed storage network (DSN) memory;

a second module for;

determining a segmentation scheme for storing the data;

determining how to store the data in accordance with the segmentation scheme to produce information for storing the data; and

generating an entry within a segment allocation table associated with the file, wherein the entry includes the information for storing the data and the segmentation scheme;

a third module for facilitating storage of the segment allocation table in the DSN memory, wherein the third module facilitates the storage of the segment allocation table by;

obtaining a segment allocation table vault source name;

dispersed storage error encoding the segment allocation table to produce encoded table slices; and

outputting the encoded table slices to the DSN memory for storage therein utilizing the segment allocation table vault source name;

a fourth module for segmenting the data in accordance with the segmentation scheme to produce a plurality of data segments; and

a fifth module for facilitating storage of the plurality of data segments in the DSN memory in accordance with the information for storing the data.

9. The DS processing unit of claim 8, wherein the second module determines the segmentation scheme based on at least one of:

a previous segmentation scheme;

a data size indicator;

a data type indicator;

a storage requirement;

a vault identifier (ID);

a lookup;

a message; and

a query.

10. The DS processing unit of claim 8 further comprises:

the information for storing the data including;

a start segment vault source name; and

a total length of the data; and

the segmenting scheme including;

a segment size; and

a segmentation approach.

11. The DS processing unit of claim 10, wherein the segmenting scheme comprises:

a fixed segmentation approach and a corresponding fixed segment size such that each of the plurality of data segments has a size no greater than the fixed segment size.

12. The DS processing unit of claim 10, wherein the segmenting scheme comprises:

a varying segmentation approach and an initial segment size such that a data segment of the plurality of data segments has a size corresponding to the initial segment size and remaining data segments of the plurality of data segments have a size based on the initial segment size and the varying segmentation approach.

13. The DS processing unit of claim 8, wherein the facilitating storage of the plurality of data segments in the DSN memory comprises:

for each data segment of the plurality of data segments;

dispersed storage error encoding the data segment to produce a set of encoded data slices;

generating a set of slice names corresponding to the set of encoded data slices based on a start segment vault source name of the segment allocation table, wherein a common segment number of the set of slice names includes a sequentially increasing segment number; and

sending the set of encoded data slices and the set of slice names to the DSN memory.

14. The DS processing unit of claim 8, wherein the fifth module further facilitates storage of the plurality of data segments in the DSN memory by:

for each data segment of the plurality of data segments;

dispersed storage error encoding the data segment to produce a set of encoded data slices;

generating a set of slice names corresponding to the set of encoded data slices based on a start segment vault source name of the segment allocation table, wherein an append marker field of the set of slice names includes a sequentially increasing append marker; and

sending the set of encoded data slices and the set of slice names to the DSN memory.