Storage of data with verification in a dispersed storage network

• US 9,891,829 B2
• Filed: 03/06/2017
• Issued: 02/13/2018
• Est. Priority Date: 02/26/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K.
• Primary Examiner(s): Chase, Shelly A
• Application Number:
  US15/450,470
• Publication Number:
  US 20170177229A1
• Time in Patent Office: 344 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 2212/254 Distributed memory
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/616 Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H03M 7/6011 Encoder aspects
• CPC Class Codes: H04L 2209/30 Compression, e.g. Merkle-Damgard construction
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 2209/608 Watermarking
• CPC Class Codes: H04L 63/10 for controlling access to devices or network resources
• CPC Class Codes: H04L 63/101 Access control lists [ACL]
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 63/12 : Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/565 : Conversion or adaptation of application format or content adding application control or application functional data H04L67/561
• CPC Class Codes: H04L 9/085 : Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/0894 : Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage

Abstract

A method begins by a computing device sending a set of redundant dispersed storage error encoding write requests regarding a data object to a set of dispersed storage (DS) processing modules. The method continues with the set of DS processing modules dispersed storage error encoding the data object to produce a group of pluralities of sets of encoded data slices. The method continues with a set of storage units temporarily storing the group of pluralities of sets of encoded data slices. The method continues with the set of storage units permanently storing encoded data slices of the group of pluralities of sets of encoded data slices based on successful execution of a storage verification process to produce a plurality of sets of encoded data slices.

Claims

1. A method comprises:

sending, by a computing device of a dispersed storage network (DSN), a write request to a plurality of dispersed storage (DS) processing units of the DSN, wherein the write request includes a data segment of a data object for storage in DSN memory, wherein the write request further includes a seed for determining dispersed storage error encoding parameters;

executing, by each DS processing unit of the plurality of DS processing units, a non-deterministic function based on the seed to determine the dispersed storage error encoding parameters;

generating, by each DS processing unit of the plurality of DS processing units, a set of encoded data slices in accordance with the determined dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices;

sending, by the plurality of DS processing units, a plurality of sets of write commands regarding the plurality of sets of encoded data slices to a set of storage units of the DSN;

obtaining, by the plurality of DS processing units, a plurality of sets of write responses from the set of storage units regarding the plurality of sets of write commands;

selecting, by the computing device or one of the plurality of DS processing units, a favorable write response for each encoded data slice of the set of encoded data slices from the plurality of sets of write responses to produce a selected set of favorable write responses; and

issuing, by at least some of the DS processing units, write commit commands to the set of storage units in response to the selected set of favorable write responses.

2. The method of claim 1, wherein executing the non-deterministic function further comprises one or more of:

executing a compression function based on the seed;

executing an encryption function based on the seed;

executing an all or nothing transformation based on the seed; and

executing a dispersed storage error encoding function based on the seed.

3. The method of claim 1 further comprises:

generating, by a first DS processing unit of the plurality of DS processing units, a first set of encoded data slices in accordance with the determined dispersed storage error encoding parameters;

generating, by a second DS processing unit of the plurality of DS processing units, a second set of encoded data slices in accordance with the determined dispersed storage error encoding parameters; and

generating, by a third DS processing unit of the plurality of DS processing units, a third set of encoded data slices in accordance with the determined dispersed storage error encoding parameters.

4. The method of claim 3, wherein the sending the plurality of sets of write commands further comprises:

sending, by the first DS processing unit a first set of write commands regarding the first set of encoded data slices to the set of storage units; and

sending, by the second DS processing unit a second set of write commands regarding the second set of encoded data slices to the set of storage units; and

sending, by the third DS processing unit, a third set of write commands regarding the third set of encoded data slices to the set of storage units.

5. The method of claim 1, wherein the obtaining the plurality of sets of write responses comprises:

obtaining, by the first DS processing unit, a first set of write responses regarding storage of the first set of encoded data slices from the set of storage units;

obtaining, by the second DS processing unit, a second set of write responses regarding storage of the second set of encoded data slices from the set of storage units; and

obtaining, by the third DS processing unit, a third set of write responses regarding storage of the third set of encoded data slices from the set of storage units.

6. The method of claim 5, wherein the obtaining, by the first DS processing unit, the first set of write responses comprises at least one of:

receiving, from a first storage unit of the set of storage units, a favorable write response regarding storing a first encoded data slice of the set of encoded data slices;

receiving, from a second storage unit of the set of storage units, an unfavorable write response regarding storing a second encoded data slice of the set of encoded data slices; and

indicating, by the first DS processing unit, the unfavorable write response for a third encoded data slice of the set of encoded data slices when a third storage unit of the set of storage units failed to provide a write response in a given time frame.

7. The method of claim 5 wherein the selecting the favorable write response for each encoded data slice comprises:

for the first set of encoded data slices;

determining which DS processing units of the plurality of DS processing units received the favorable write response for the first set of encoded data slices; and

selecting at least one DS processing unit of the DS processing units receiving the favorable response for the selected favorable response for the first set of encoded data slices;

for the second set of encoded data slices;

determining which DS processing units of the plurality of DS processing units received the favorable write response for the second set of encoded data slices; and

selecting at least one DS processing unit of the DS processing units receiving the favorable response for the selected favorable response for the second set of encoded data slices; and

for the third set of encoded data slices;

determining which DS processing units of the plurality of DS processing units received the favorable write response for the third set of encoded data slices; and

selecting at least one DS processing unit of the DS processing units receiving the favorable response for the selected favorable response for the third set of encoded data slices.

8. A non-transitory computer readable storage medium comprises:

a first memory section that stores operational instructions that, when executed by a computing device of a dispersed storage network (DSN), causes the computing device to;

send a write request to a plurality of dispersed storage (DS) processing units of the DSN, wherein the write request includes a data segment of a data object for storage in DSN memory, wherein the write request further includes a seed for determining dispersed storage error encoding parameters;

a second memory section that stores operational instructions that, when executed by each DS processing unit of the plurality of DS processing units, causes each the DS processing unit to;

execute a non-deterministic function based on the seed to determine the dispersed storage error encoding parameters;

generate a set of encoded data slices in accordance with the determined dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices;

send a set of write commands of a plurality of sets of write commands regarding a set of encoded data slices of the plurality of sets of encoded data slices to a set of storage units of the DSN;

obtain a set of write responses of a plurality of sets of write responses from the set of storage units regarding the set of write commands of the plurality of sets of write commands;

a third memory section that stores operational instructions that, when executed by a DS processing unit or the computing device, causes the DS processing unit or the computing device to;

select a favorable write response for each encoded data slice of the set of encoded data slices from the plurality of sets of write responses to produce a selected set of favorable write responses; and

a fourth memory section that stores operational instructions that, when executed by at least some of the DS processing units, causes the at least some of the DS processing units to;

issue write commit commands to the set of storage units in response to the selected set of favorable write responses.

9. The non-transitory computer readable storage medium of claim 8, wherein the second memory section that stores operational instructions that, when executed by each DS processing unit of the plurality of DS processing units, causes each the DS processing unit to execute the non-deterministic function further by one or more of:

executing a compression function based on the seed;

executing an encryption function based on the seed;

executing an all or nothing transformation based on the seed; and

executing a dispersed storage error encoding function based on the seed.

10. The non-transitory computer readable storage medium of claim 8 wherein the second memory section that stores operational instructions that, when executed by each DS processing unit of the plurality of DS processing units, causes each the DS processing unit to generate the plurality of sets of encoded data slices by:

generating, by a first DS processing unit of the plurality of DS processing units, a first set of encoded data slices in accordance with the determined dispersed storage error encoding parameters;

generating, by a second DS processing unit of the plurality of DS processing units, a second set of encoded data slices in accordance with the determined dispersed storage error encoding parameters; and

generating, by a third DS processing unit of the plurality of DS processing units, a third set of encoded data slices in accordance with the determined dispersed storage error encoding parameters.

11. The non-transitory computer readable storage medium of claim 10 wherein the second memory section that stores operational instructions that, when executed by each DS processing unit of the plurality of DS processing units, causes each the DS processing unit to send the plurality of sets of write commands further by:

sending, by the first DS processing unit, a first set of write commands regarding the first set of encoded data slices to the set of storage units;

sending, by the second DS processing unit, a second set of write commands regarding the second set of encoded data slices to the set of storage units; and

sending, by the third DS processing unit, a third set of write commands regarding the third set of encoded data slices to the set of storage units.

12. The non-transitory computer readable storage medium of claim 11, wherein the second memory section that stores operational instructions that, when executed by each DS processing unit of the plurality of DS processing units, causes each the DS processing unit to obtain the plurality of sets of write responses by:

obtaining, by the first DS processing unit, a first set of write responses regarding storage of the first set of encoded data slices from the set of storage units;

obtaining, by the second DS processing unit, a second set of write responses regarding storage of the second set of encoded data slices from the set of storage units; and

obtaining, by the third DS processing unit, a third set of write responses regarding storage of the third set of encoded data slices from the set of storage units.

13. The non-transitory computer readable storage medium of claim 12, wherein the second memory section that stores operational instructions that, when executed by the first DS processing unit, causes the first DS processing unit to obtain the first set of write responses by at least one of:

receiving, from a first storage unit of the set of storage units, a favorable write response regarding storing a first encoded data slice of the set of encoded data slices;

receiving, from a second storage unit of the set of storage units, an unfavorable write response regarding storing a second encoded data slice of the set of encoded data slices; and

indicating, by the first DS processing unit, the unfavorable write response for a third encoded data slice of the set of encoded data slices when a third storage unit of the set of storage units failed to provide a write response in a given time frame.

14. The non-transitory computer readable storage medium of claim 11 wherein the third memory section further stores operational instructions that, when executed by a DS processing unit or the computing device, causes the DS processing unit or the computing device to select the favorable write response for each encoded data slice by:

for the first set of encoded data slices;

determining which DS processing units of the plurality of DS processing units received the favorable write response for the first set of encoded data slices; and

selecting at least one DS processing unit of the DS processing units receiving the favorable response for the selected favorable response for the first set of encoded data slices;

for the second set of encoded data slices;

determining which DS processing units of the plurality of DS processing units received the favorable write response for the second set of encoded data slices; and

selecting at least one DS processing unit of the DS processing units receiving the favorable response for the selected favorable response for the second set of encoded data slices; and

for the third set of encoded data slices;

determining which DS processing units of the plurality of DS processing units received the favorable write response for the third set of encoded data slices; and

selecting at least one DS processing unit of the DS processing units receiving the favorable response for the selected favorable response for the third set of encoded data slices.