Storage and retrieval of required slices in a dispersed storage network

• US 8,566,354 B2
• Filed: 02/04/2011
• Issued: 10/22/2013
• Est. Priority Date: 04/26/2010
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: Grube, Gary W., Markison, Timothy W.
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): VU, THONG H
• Application Number:
  US13/021,537
• Publication Number:
  US 20110264717A1
• Time in Patent Office: 991 Days
• Field of Search: 375/240.03, 375/2, 375/12, 375/26, 707/741, 707/782, 709/203, 382/199
• US Class Current: 707/782
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0763 by bit configuration check, e.g. of formats or tags
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/1008 in individual solid state devices G06F11/1004 takes precedence
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 21/60 Protecting data
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 21/70 Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: H04L 2209/30 Compression, e.g. Merkle-Damgard construction
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/085 Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/0894 Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage

Abstract

A method begins by a processing module determining whether one or more encoded data slices of a set of encoded data slices of a plurality of sets of encoded data slices are required encoded data slices to produce one or more required encoded data slices. When the one or more of the encoded data slices are required, the method continues with the processing module generating specific dispersed storage network (DSN) storage information for each of the one or more required encoded data slices, generating general DSN storage information for remaining encoded data slices of the set of encoded data slices, and sending the set of encoded data slices to a dispersed storage network in accordance with the specific DSN storage information and the general DSN storage information.

Claims

1. A method comprises:

error encoding a data segment in accordance with a dispersed storage error coding function to produce a set of encoded data slices, wherein the dispersed storage error coding function allows for loss of some encoded data slices of the set of encoded data slices while still allowing the data segment to be recovered from the set of encoded data slices as long as a threshold number of encoded data slices of the set of encoded data slices are retrievable;

determining, for security, whether one or more of the encoded data slices of the set of encoded data slices are to be required encoded data slices to produce one or more required encoded data slices, wherein the one or more required encoded data slices are required to be retrieved when at least the threshold number of encoded data slices of the set of encoded data slices are retrieved for recovering the data segment; and

when the one or more of the encoded data slices are required;

generating specific dispersed storage network (DSN) storage information for each of the one or more required encoded data slices;

generating general DSN storage information for remaining encoded data slices of the set of encoded data slices; and

sending the set of encoded data slices to a dispersed storage network in accordance with the specific DSN storage information and the general DSN storage information.

2. The method of claim 1, wherein the specific DSN storage information comprises:

for a first one of the one or more required encoded data slices, a first dispersed storage (DS) unit identification code, a first slice name, and a first instruction to store the first one of the one or more required encoded data slices based on the first slice name at a first DS unit that corresponds to the first DS unit identification code; and

for a second one of the one or more required encoded data slices, a second DS unit identification code, a second slice name, and a second instruction to store the second one of the one or more required encoded data slices based on the second slice name at a second DS unit that corresponds to the second DS unit identification code.

3. The method of claim 1, wherein the general DSN storage information comprises:

identification codes for a set of dispersed storage (DS) units and an instruction to store the remaining encoded data slices in the set of DS units.

4. The method of claim 1, wherein the determining whether the one or more of the encoded data slices are required comprises at least one of:

determining a security protocol for the plurality of sets of encoded data slices, wherein the security protocol indicates a use of required encoded data slices; and

determining a performance protocol for the plurality of sets of encoded data slices, wherein the performance protocol indicates the use of required encoded data slices.

5. The method of claim 4, wherein the indication of use comprises at least one of:

identifying one or more specific encoded data slices as the one or more required encoded data slices; and

determining the one or more required data slices based on one or more of;

selecting one or more dispersed storage error coding parameters;

analyzing a DS unit operational information;

performing a vault lookup;

interpreting a storage request message; and

interpreting a command.

6. The method of claim 1 further comprises:

appending additional data to at least one of the one or more required encoded data slices.

7. The method of claim 1 further comprises:

determining whether a favorable storage response message was received within a storage time period for the one or more required encoded data slices;

when the favorable storage response message was not received, determining alternate specific DSN storage information for at least one of the one or more required encoded data slices; and

sending the at least one of the one or more required encoded data slices to the dispersed storage network in accordance with the alternate specific DSN storage information.

8. A method comprises:

receiving at least a threshold number of encoded data slices of a set of encoded data slices wherein a data segment was encoded in accordance with a dispersed storage error coding function to produce a set of encoded data slices that is stored in a distributed storage network (DSN) memory, wherein the dispersed storage error coding function allows for loss of some encoded data slices of the set of encoded data slices while still allowing the data segment to be recovered from the set of encoded data slices as long as the threshold number of encoded data slices of the set of encoded data slices are retrievable;

determining whether one or more encoded data slices of the set of encoded data slices are required encoded data slices to produce one or more required encoded data slices; and

when the one or more of the encoded data slices are required;

determining whether the one or more required encoded data slices are included in the at least the threshold number of encoded data slices;

when at least one of the one or more required encoded data slices was not received, determining a subsequent data retrieval approach based on specific DSN retrieval information; and

requesting retrieval of the at least one of the one or more required encoded data slices in accordance with the subsequent data retrieval approach.

9. The method of claim 8, wherein the determining whether the one or more of the encoded data slices are required comprises:

determining a security protocol for the set of encoded data slices, wherein the security protocol indicates a use of required encoded data slices; and

determining a performance protocol for the set of encoded data slices, wherein the performance protocol indicates the use of required encoded data slices.

10. The method of claim 9, wherein the indication of use comprises at least one of:

identifying one or more specific encoded data slices as the one or more required encoded data slices; and

determining the one or more required data slices based on one or more of;

selecting one or more dispersed storage error coding parameters;

performing a vault lookup;

interpreting a storage request message; and

interpreting a command.

11. The method of claim 8, wherein the determining the subsequent data retrieval approach comprises:

determining to re-request retrieval of the at least one of the one or more required encoded data slices;

determining whether the specific DSN retrieval information includes alternate specific DSN retrieval information; and

determining whether a lower quality data retrieval method is allowed.

12. A computer comprises:

an interface;

a memory; and

a processing module operable to;

error encode a data segment in accordance with a dispersed storage error coding function to produce a set of encoded data slices, wherein the dispersed storage error coding function allows for loss of some encoded data slices of the set of encoded data slices while still allowing the data segment to be recovered from the set of encoded data slices as long as a threshold number of encoded data slices of the set of encoded data slices are retrievable;

determine, for security, whether one or more of the encoded data slices of the set of encoded data slices are to be required encoded data slices to produce one or more required encoded data slices, wherein the one or more required encoded data slices are required to be retrieved when at least the threshold number of encoded data slices of the set of encoded data slices are retrieved for recovering the data segment; and

when the one or more of the encoded data slices are required;

generate specific dispersed storage network (DSN) storage information for each of the one or more required encoded data slices;

generate general DSN storage information for remaining encoded data slices of the set of encoded data slices; and

send, via the interface, the set of encoded data slices to a dispersed storage network in accordance with the specific DSN storage information and the general DSN storage information.

13. The computer of claim 12, wherein the specific DSN storage information comprises:

for a first one of the one or more required encoded data slices, a first dispersed storage (DS) unit identification code, a first slice name, and a first instruction to store the first one of the one or more required encoded data slices based on the first slice name at a first DS unit that corresponds to the first DS unit identification code; and

for a second one of the one or more required encoded data slices, a second DS unit identification code, a second slice name, and a second instruction to store the second one of the one or more required encoded data slices based on the second slice name at a second DS unit that corresponds to the second DS unit identification code.

14. The computer of claim 12, wherein the general DSN storage information comprises:

identification codes for a set of dispersed storage (DS) units and an instruction to store the remaining encoded data slices in the set of DS units.

15. The computer of claim 12, wherein the processing module functions to determine whether the one or more of the encoded data slices are required by at least one of:

determining a security protocol for the plurality of sets of encoded data slices, wherein the security protocol indicates a use of required encoded data slices; and

determining a performance protocol for the plurality of sets of encoded data slices, wherein the performance protocol indicates the use of required encoded data slices.

16. The computer of claim 15, wherein the processing module functions to determine the indication of use by at least one of:

identifying one or more specific encoded data slices as the one or more required encoded data slices; and

determining the one or more required data slices based on one or more of;

selecting one or more dispersed storage error coding parameters;

analyzing a DS unit operational information;

performing a vault lookup;

interpreting a storage request message; and

interpreting a command.

17. The computer of claim 12, wherein the processing module further functions to:

append additional data to at least one of the one or more required encoded data slices.

18. The computer of claim 12, wherein the processing module further functions to:

determine whether a favorable storage response message was received within a storage time period for the one or more required encoded data slices;

when the favorable storage response message was not received, determine alternate specific DSN storage information for at least one of the one or more required encoded data slices; and

send, via the interface, the at least one of the one or more required encoded data slices to the dispersed storage network in accordance with the alternate specific DSN storage information.

19. A computer comprises:

an interface;

a memory; and

a processing module operable to;

receive, via the interface, at least a threshold number of encoded data slices of a set of encoded data slices wherein a data segment was encoded in accordance with a dispersed storage error coding function to produce a set of encoded data slices that is stored in a distributed storage network (DSN) memory, wherein the dispersed storage error coding function allows for loss of some encoded data slices of the set of encoded data slices while still allowing the data segment to be recovered from the set of encoded data slices as long as the threshold number of encoded data slices of the set of encoded data slices are retrievable;

determine whether one or more encoded data slices of the set of encoded data slices are required encoded data slices to produce one or more required encoded data slices; and

when the one or more of the encoded data slices are required;

determine whether the one or more required encoded data slices are included in the at least the threshold number of encoded data slices;

when at least one of the one or more required encoded data slices was not received, determine a subsequent data retrieval approach based on the specific DSN retrieval information; and

request retrieval of the at least one of the one or more required encoded data slices in accordance with the subsequent data retrieval approach.

20. The computer of claim 19, wherein the processing module functions to determine whether the one or more of the encoded data slices are required by:

determining a security protocol for the plurality of sets of encoded data slices, wherein the security protocol indicates a use of required encoded data slices; and

determining a performance protocol for the plurality of sets of encoded data slices, wherein the performance protocol indicates the use of required encoded data slices.

21. The computer of claim 20, wherein the processing module functions to determine the indication of use by at least one of:

identifying one or more specific encoded data slices as the one or more required encoded data slices; and

determining the one or more required data slices based on one or more of;

selecting one or more dispersed storage error coding parameters;

performing a vault lookup;

interpreting a storage request message; and

interpreting a command.

22. The computer of claim 19, wherein the processing module functions to determine the subsequent data retrieval approach by:

determining to re-request retrieval of the at least one of the one or more required encoded data slices;

determining whether the specific DSN retrieval information includes alternate specific DSN retrieval information; and

determining whether a lower quality data retrieval method is allowed.