Dispersed storage with variable slice length and methods for use therewith

• US 9,438,675 B2
• Filed: 06/26/2014
• Issued: 09/06/2016
• Est. Priority Date: 08/29/2013
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew, Volvovski, Ilya, Leggette, Wesley, Dhuse, Greg, Resch, Jason K.
• Primary Examiner(s): Bonzo, Bryce
• Assistant Examiner(s): NGUYEN, THIEN DANG
• Application Number:
  US14/315,981
• Publication Number:
  US 20150067421A1
• Time in Patent Office: 803 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1096 Parity calculation or recalculation after configuration or reconfiguration of the system
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G11C 2029/4402 Internal storage of test result, quality data, chip identification, repair information
• CPC Class Codes: G11C 29/44 Indication or identification of errors, e.g. for repair
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/60 Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources admission control or resource allocation H04L47/70

Abstract

A dispersed storage processing unit selects a slice length for a data segment to be stored in a dispersed storage network (DSN). The data segment is encoded using a dispersed storage error coding function to produce a set of data slices in accordance with the slice length. A storage file is selected based on the slice length. A storage file identifier (ID) is generated that indicates the storage file. A set of DSN addresses are generated corresponding to the set of data slices, wherein the set of DSN addresses each include the storage file ID and a corresponding one of a plurality of offset identifiers (IDs). The set of data slices are written in accordance with the set of DSN addresses. A directory is updated to associate the set of DSN addresses with an identifier of the data segment.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

selecting a first slice length for a first data segment of a data object to be stored in the DSN;

selecting a second slice length for a second data segment of the data object to be stored in the DSN;

encoding the first data segment using a first dispersed storage error coding function to produce a first set of data slices in accordance with the first slice length;

encoding the second data segment using a second dispersed storage error coding function to produce a second set of data slices in accordance with the second slice length;

selecting a first storage file based on the first slice length;

selecting a second storage file based on the second slice length;

generating a first storage file identifier (ID) that indicates the first storage file;

generating a second storage file identifier (ID) that indicates the second storage file;

generating a first set of DSN addresses corresponding to the first set of data slices and a second set of DSN addresses corresponding to the second set of data slices, wherein the first set of DSN addresses each include the first storage file ID and a corresponding one of a first plurality of offset identifiers (IDs) and the second set of DSN addresses each include the second storage file ID and a corresponding one of a second plurality of offset identifiers (IDs);

writing the first set of data slices in accordance with the first set of DSN addresses;

writing the second set of data slices in accordance with the second set of DSN addresses;

updating a directory to associate the first set of DSN addresses with an identifier of the first data segment; and

updating the directory to associate the second set of DSN addresses with an identifier of the second data segment.

2. The method of claim 1 wherein the first storage file is selected when the first slice length has a first value and the second storage file is selected when the second slice length has a second value.

3. The method of claim 1 wherein the first storage file is selected when the first slice length falls within a first range of values and the second storage file is selected when the second slice length falls within a second range of values.

4. The method of claim 1 wherein the first storage file includes a first file header that includes the first plurality of offset identifiers and a first corresponding plurality of slice locations and the second storage file includes a second file header that includes the second plurality of offset identifiers and a second corresponding plurality of slice locations.

5. The method of claim 1 wherein a data slice of the first set of data slices is stored in the first storage file with the first slice length at one of a first plurality of slice locations, and a data slice of the second set of data slices is stored in the second storage file with the second slice length at one of a second plurality of slice locations.

6. The method of claim 1 wherein each of the first and second slice lengths is selected based on one or more of:

a length of the first or second data segment, a predetermination, a vault identifier, a data type of the first or second data segment, or a storage utilization level.

7. The method of claim 1 wherein the selecting of the first and second storage files is further based on one or more of:

a data type of the first or second data segment, a data owner of the data segment, a vault identifier, a data priority level of the first or second data segment, or a preference of a DSN device.

8. A dispersed storage processing unit of a dispersed storage network (DSN) comprises:

at least one module, when operable within a computing device, that causes the computing device to;

select a first slice length for a first data segment of a data object to be stored in the DSN;

select a second slice length for a second data segment of the data object to be stored in the DSN;

encode the first data segment using a first dispersed storage error coding function to produce a first set of data slices in accordance with the first slice length;

encode the second data segment using a second dispersed storage error coding function to produce a second set of data slices in accordance with the second slice length;

select a first storage file based on the first slice length;

select a second storage file based on the second slice length;

generate a first storage file identifier (ID) that indicates the first storage file;

generate a second storage file identifier (ID) that indicates the second storage file;

generate a first set of DSN addresses corresponding to the first set of data slices and a second set of DSN addresses corresponding to the second set of data slices, wherein the first set of DSN addresses each include the first storage file ID and a corresponding one of a first plurality of offset identifiers (IDs) and the second set of DSN addresses each include the second storage file ID and a corresponding one of a second plurality of offset identifiers (IDs);

write the first set of data slices in accordance with the first set of DSN addresses;

write the second set of data slices in accordance with the second set of DSN addresses;

update a directory to associate the first set of DSN addresses with an identifier of the first data segment; and

update the directory to associate the second set of DSN addresses with an identifier of the second data segment.

9. The dispersed storage processing unit of claim 8 wherein the first storage file is selected when the first slice length has a first value and the second storage file is selected when the second slice length has a second value.

10. The dispersed storage processing unit of claim 8 wherein the first storage file is selected when the first slice length falls within a first range of values and the second storage file is selected when the second slice length falls within a second range of values.

11. The dispersed storage processing unit of claim 8 wherein the first storage file includes a first file header that includes the first plurality of offset identifiers and a first corresponding plurality of slice locations and the second storage file includes a second file header that includes the second plurality of offset identifiers and a second corresponding plurality of slice locations.

12. The dispersed storage processing unit of claim 8 wherein a data slice of the first set of data slices is stored in the first storage file with the first slice length at one of a first plurality of slice locations, and a data slice of the second set of data slices is stored in the second storage file with the second slice length at one of a second plurality of slice locations.

13. The dispersed storage processing unit of claim 8 wherein each of the first and second slice lengths is selected based on one or more of:

a length of the first or second data segment, a predetermination, a vault identifier, a data type of the first or second data segment, or a storage utilization level.

14. The dispersed storage processing unit of claim 8 wherein the selection of the first and second storage files is further based on one or more of:

a data type of the first or second data segment, a data owner of the first or second data segment, a vault identifier, a data priority level of the data segment, or a preference of a DSN device.

15. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

select a first slice length for a first data segment of a data object to be stored in the DSN;

select a second slice length for a second data segment of the data object to be stored in the DSN;

encode the first data segment using a first dispersed storage error coding function to produce a first set of data slices in accordance with the first slice length;

encode the second data segment using a second dispersed storage error coding function to produce a second set of data slices in accordance with the second slice length;

select a first storage file based on the first slice length;

select a second storage file based on the second slice length;

generate a first storage file identifier (ID) that indicates the first storage file;

generate a second storage file identifier (ID) that indicates the second storage file;

generate a first set of DSN addresses corresponding to the first set of data slices and a second set of DSN addresses corresponding to the second set of data slices, wherein the first set of DSN addresses each include the first storage file ID and a corresponding one of a first plurality of offset identifiers (IDs) and the second set of DSN addresses each include the second storage file ID and a corresponding one of a second plurality of offset identifiers (IDs);

write the first set of data slices in accordance with the first set of DSN addresses;

write the second set of data slices in accordance with the second set of DSN addresses;

update a directory to associate the first set of DSN addresses with an identifier of the first data segment; and

update the directory to associate the second set of DSN addresses with an identifier of the second data segment.

16. The non-transitory computer readable storage medium of claim 15 wherein the first storage file is selected when the first slice length has a first value and the second storage file is selected when the second slice length has a second value.

17. The non-transitory computer readable storage medium of claim 15 wherein the first storage file includes a first file header that includes the first plurality of offset identifiers and a first corresponding plurality of slice locations and the second storage file includes a second file header that includes the second plurality of offset identifiers and a second corresponding plurality of slice locations.

18. The non-transitory computer readable storage medium of claim 15 wherein a data slice of the first set of data slices is stored in the first storage file with the first slice length at one of a first plurality of slice locations, and a data slice of the second set of data slices is stored in the second storage file with the second slice length at one of a second plurality of slice locations.

19. The non-transitory computer readable storage medium of claim 15 wherein each of the first and second slice lengths is selected based on one or more of:

a length of the first or second data segment, a predetermination, a vault identifier, a data type of the first or second data segment, or a storage utilization level.

20. The non-transitory computer readable storage medium of claim 15 wherein the selection of the first and second storage files is further based on one or more of:

a data type of the first or second data segment, a data owner of the first or second data segment, a vault identifier, a data priority level of the data segment, or a preference of a DSN device.