Storing data copies in a dispersed storage network

• US 10,031,700 B2
• Filed: 10/26/2016
• Issued: 07/24/2018
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Dhuse, Greg R., Horan, Scott M., Khadiwala, Ravi V., Leggette, Wesley B., Motwani, Manish, Resch, Jason K., Viraraghavan, Praveen, Volvovski, Ilya, Vossberg, Trevor J., Wozniak, Ethan S.
• Primary Examiner(s): Chung, Phung M
• Application Number:
  US15/334,830
• Publication Number:
  US 20170153941A1
• Time in Patent Office: 636 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/108 Parity data distribution in semiconductor storages, e.g. in SSD
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 12/0891 using clearing, invalidating or resetting means
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2211/1007 Addressing errors, i.e. silent errors in RAID, e.g. sector slipping and addressing errors
• CPC Class Codes: G06F 2212/1024 Latency reduction
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/263 Network storage, e.g. SAN or NAS
• CPC Class Codes: G06F 2212/403 Error protection encoding, e.g. using parity or ECC codes
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 : Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0665 : at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 : Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G06F 9/4881 : Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H03M 13/1105 : Decoding
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 : using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/616 : Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H03M 13/6502 : Reduction of hardware complexity or efficient processing
• CPC Class Codes: H04L 2101/604 : Address structures or formats
• CPC Class Codes: H04L 61/457 : containing identifiers of data entities on a computer, e.g. file names
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method for use in a dispersed storage network operates to determine first information dispersal algorithm (IDA) parameters; determine second IDA parameters; divide data for storage to produce a plurality of first segments in accordance with the first IDA parameters and a plurality of second segments in accordance with the second IDA parameters; dispersed storage error encode the plurality of first segments utilizing the first IDA parameters to produce sets of first encoded data slices; dispersed storage error encode the plurality of second segments utilizing the second IDA parameters to produce sets of second encoded data slices; and facilitate storage of the sets of first encoded data slices and the sets of second encoded data slices in a plurality of storage units.

Claims

1. A method for execution by a processing system of a dispersed storage and task (DST) processing unit that includes a processor, the method comprises:

determining first information dispersal algorithm (IDA) parameters;

determining second IDA parameters, wherein at least one value of the second IDA parameters is different from a corresponding at least one value of the first IDA parameters;

dividing data for storage to produce a plurality of first segments in accordance with the first IDA parameters and a plurality of second segments in accordance with the second IDA parameters;

dispersed storage error encoding the plurality of first segments utilizing the first IDA parameters to produce sets of first encoded data slices;

dispersed storage error encoding the plurality of second segments utilizing the second IDA parameters to produce sets of second encoded data slices; and

facilitating storage of the sets of first encoded data slices and the sets of second encoded data slices in a plurality of storage units.

2. The method of claim 1 wherein determining the first information dispersal algorithm (IDA) parameters is based on one or more of:

an interpretation of system registry information, a request, interpreting a received message, or a characteristic of data for storage.

3. The method of claim 2 wherein determining the second IDA parameters is based on one or more of:

another interpretation of the system registry information, another request, interpreting the received message, another characteristic of the data for storage, a first byte recovery performance requirement, and a maximum slice size, wherein the maximum slice size is proportional to a second segment size divided by a decode threshold number corresponding to the second IDA.

4. The method of claim 1 wherein the first IDA parameters include a first segment size, wherein the second IDA parameters include a second segment size, and wherein dividing data for storage includes partitioning the data by the first segment size to produce the plurality of first segments and the second segment size to produce the plurality of second segments.

5. The method of claim 1 wherein dispersed storage error encoding the plurality of first segments includes dispersed storage error encoding each first segment of the plurality of first segments utilizing the first IDA parameters to produce the sets of first encoded data slices.

6. The method of claim 1 wherein dispersed storage error encoding the plurality of second segments includes dispersed storage error encoding each second segment of the plurality of second segments utilizing the first IDA parameters to produce the sets of second encoded data slices.

7. The method of claim 1 wherein facilitating the storage includes sending the sets of first encoded data slices and the sets of second encoded data slices to the plurality of storage units for storage.

8. A processing system of a first dispersed storage and task (DST) processing unit comprises:

at least one processor;

a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to;

determine first information dispersal algorithm (IDA) parameters;

determine second IDA parameters, wherein at least one value of the second IDA parameters is different from a corresponding at least one value of the first IDA parameters;

divide data for storage to produce a plurality of first segments in accordance with the first IDA parameters and a plurality of second segments in accordance with the second IDA parameters;

dispersed storage error encode the plurality of first segments utilizing the first IDA parameters to produce sets of first encoded data slices;

dispersed storage error encode the plurality of second segments utilizing the second IDA parameters to produce sets of second encoded data slices; and

facilitate storage of the sets of first encoded data slices and the sets of second encoded data slices in a plurality of storage units.

9. The processing system of claim 8 wherein determining the first information dispersal algorithm (IDA) parameters is based on one or more of:

an interpretation of system registry information, a request, interpreting a received message, or a characteristic of data for storage.

10. The processing system of claim 9 wherein determining the second IDA parameters is based on one or more of:

another interpretation of the system registry information, another request, interpreting the received message, another characteristic of the data for storage, a first byte recovery performance requirement, and a maximum slice size, where wherein the maximum slice size is proportional to a second segment size divided by a decode threshold number corresponding to the second IDA.

11. The processing system of claim 8 wherein the first IDA parameters include a first segment size, wherein the second IDA parameters include a second segment size, and wherein dividing data for storage includes partitioning the data by the first segment size to produce the plurality of first segments and the second segment size to produce the plurality of second segments.

12. The processing system of claim 8 wherein dispersed storage error encoding the plurality of first segments includes dispersed storage error encoding each first segment of the plurality of first segments utilizing the first IDA parameters to produce the sets of first encoded data slices.

13. The processing system of claim 8 wherein dispersed storage error encoding the plurality of second segments includes dispersed storage error encoding each second segment of the plurality of second segments utilizing the first IDA parameters to produce the sets of second encoded data slices.

14. The processing system of claim 8 wherein facilitating the storage includes sending the sets of first encoded data slices and the sets of second encoded data slices to the plurality of storage units for storage.

15. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

determine first information dispersal algorithm (IDA) parameters;

determine second IDA parameters, wherein at least one value of the second IDA parameters is different from a corresponding at least one value of the first IDA parameters;

divide data for storage to produce a plurality of first segments in accordance with the first IDA parameters and a plurality of second segments in accordance with the second IDA parameters;

dispersed storage error encode the plurality of first segments utilizing the first IDA parameters to produce sets of first encoded data slices;

dispersed storage error encode the plurality of second segments utilizing the second IDA parameters to produce sets of second encoded data slices; and

facilitate storage of the sets of first encoded data slices and the sets of second encoded data slices in a plurality of storage units.

16. The non-transitory computer readable storage medium of claim 15 wherein determining the first information dispersal algorithm (IDA) parameters is based on one or more of:

an interpretation of system registry information, a request, interpreting a received message, or a characteristic of data for storage.

17. The non-transitory computer readable storage medium of claim 16 wherein determining the second IDA parameters is based on one or more of:

another interpretation of the system registry information, another request, interpreting the received message, another characteristic of the data for storage, a first byte recovery performance requirement, and a maximum slice size, wherein the maximum slice size is proportional to a second segment size divided by a decode threshold number corresponding to the second IDA.

18. The non-transitory computer readable storage medium of claim 15 wherein the first IDA parameters include a first segment size, wherein the second IDA parameters include a second segment size, and wherein dividing data for storage includes partitioning the data by the first segment size to produce the plurality of first segments and the second segment size to produce the plurality of second segments.

19. The non-transitory computer readable storage medium of claim 15 wherein dispersed storage error encoding the plurality of first segments includes dispersed storage error encoding each first segment of the plurality of first segments utilizing the first IDA parameters to produce the sets of first encoded data slices.

20. The non-transitory computer readable storage medium of claim 15 wherein dispersed storage error encoding the plurality of second segments includes dispersed storage error encoding each second segment of the plurality of second segments utilizing the first IDA parameters to produce the sets of second encoded data slices.