VERIFYING DATA SECURITY IN A DISPERSED STORAGE NETWORK

• US 20100268692A1
• Filed: 04/18/2010
• Published: 10/21/2010
• Est. Priority Date: 04/20/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Cleversafe Incorporated (International Business Machines Corporation)
• Inventors: RESCH, JASON K.
• Granted Patent: US 10,104,045 B2
• Time in Patent Office: Days
• US Class Current: 707/687
• CPC Class Codes: G06F 11/1044 with specific ECC/EDC distribution
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 2211/1059 Parity-single bit-RAID5, i.e. RAID 5 implementations
• CPC Class Codes: H04L 2209/043 of tables, e.g. lookup, substitution or mapping
• CPC Class Codes: H04L 2209/12 Details relating to cryptographic hardware or logic circuitry
• CPC Class Codes: H04L 2209/603 Digital right managament [DRM]
• CPC Class Codes: H04L 2209/608 Watermarking
• CPC Class Codes: H04L 2209/80 Wireless network architectures or network communication protocols for wireless network security H04W12/00
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/085 Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/3236 using cryptographic hash functions

Abstract

An integrity record is appended to data slices prior to being sent to multiple slice storage units. Each of the data slices includes a different encoded version of the same data segment. An integrity indicator of each data slice is computed, and the integrity record is generated based on each of the individual integrity indicators, and may be, for example, list or a hash of the combined integrity indicators. When retrieving data slices from storage, the integrity record can be stripped off, a new integrity indicator of the data slice calculated, and a new integrity record created. The new integrity record can be compared to the original integrity record, and used to verify the integrity of the data slices.

Claims

1. A method comprising:

receiving a plurality of data slices, each of the plurality of data slices including a different encoded version of a data segment encoded to prevent reconstruction of the data segment using a single one of the plurality of data slices, and to permit reconstruction of the data segment using at least a threshold number of the plurality of data slices;

calculating first integrity indicators of each of the plurality of data slices;

generating an integrity record based on the first integrity indicators;

appending the integrity record to each of the plurality of data slices to generate modified data slices; and

transmitting the modified data slices to a plurality of slice storage units.

2. The method of claim 1, further comprising:

caching the first integrity indicators; and

generating the integrity record includes generating a list including the first integrity indicators.

3. The method of claim 1, further comprising:

caching the first integrity indicators; and

generating the integrity record includes generating a hash value using the first integrity indicators.

4. The method of claim 1, wherein the data segment is encoded into a number of pillars, the method further comprising:

receiving a number of data slices equal to the number of pillars.

5. The method of claim 4, wherein the number of pillars is determined, at least in part, by a parameter associated with a vault of a dispersed storage network.

6. A method comprising:

receiving a plurality of data slices from a plurality of storage devices, each of the plurality of data slices including an encoded data slice and a first integrity record;

each encoded data slice including a data segment encoded to prevent reconstruction of the data segment using a single data slice, and to permit reconstruction of the data segment using at least a threshold number of data slices;

each first integrity record including information derived from integrity indicators of multiple different encoded data slices;

separating the first integrity record from the encoded data slice included in each of the plurality of data slices;

calculating a plurality of new integrity indicators, the plurality of new integrity indicators including a new integrity indicator of each encoded data slice;

generating a new integrity record based on the plurality of new integrity indicators;

determining that the new integrity record compares favorably with the first integrity record of at least a threshold number of the plurality of data slices; and

in response to the determining, transmitting the at least a threshold number of the plurality of encoded data slices to a decoder.

7. The method of claim 6, wherein a first data slice of the plurality of data slices received from a known storage device of the plurality of storage devices, the method further comprising:

determining that the new integrity record compares unfavorably with a first integrity record associated with the first data slice; and

generating a flagged data slice by flagging the first data slice in response to the determining that the new integrity record compares unfavorably.

8. The method of claim 7, further comprising:

transmitting a message indicating that the known storage device is to be removed from service.

9. The method of claim 7, further comprising:

discarding multiple data slices of the plurality of data slices in response to the determining that the new integrity record compares unfavorably to the first integrity record associated with the first data slice.

10. The method of claim 6, further comprising:

caching the plurality of new hash values; and

generating the new integrity record by combining the plurality of new hash values into a hash list.

11. The method of claim 6, further comprising:

caching the plurality of new hash values; and

generating the new integrity record by calculating a hash based on the plurality of new hash values.

12. A distributed storage processing unit comprising:

a communications interface to receive a plurality of data slices from a plurality of storage devices, each of the plurality of data slices including an encoded data slice and a first integrity record;

each encoded data slice including a data segment encoded to prevent reconstruction of the data segment using a single data slice, and to permit reconstruction of the data segment using at least a threshold number of data slices;

each first integrity record including information derived from hash values of multiple different encoded data slices;

a processor to;

separate the first integrity record from the encoded data slice included in each of the plurality of data slices;

calculate a plurality of new hash values, the plurality of new hash values including a new hash value of each encoded data slice;

generate a new integrity record based on the plurality of new hash values;

determine that the new integrity record compares favorably with the first integrity record of at least a threshold number of the plurality of data slices; and

provide the at least a threshold number of the plurality of encoded data slices to a decoder.

13. The distributed storage processing unit of claim 12, further comprising:

the communications interface further to receive a first data slice of the plurality of data slices from a known storage device of the plurality of storage devices;

the processor further to;

determine that the new integrity record compares unfavorably with a first integrity record associated with the first data slice; and

generate a flagged data slice by flagging the first data slice to indicate that the new integrity record compares unfavorably.

14. The distributed storage processing unit of claim 13, further comprising:

the communications interface further to transmit a message indicating that the known storage device is to be removed from service.

15. The distributed storage processing unit of claim 13, the processor further configured to:

discard multiple data slices of the plurality of data slices in response to the processor determining that the new integrity record compares unfavorably with the first integrity record associated with the first data slice.

16. The distributed storage processing unit of claim 12, further comprising:

cache memory to store the plurality of new integrity indicators; and

the processor further configured to generate a new integrity record by combining the plurality of new integrity indicators in a list.

17. The distributed storage processing unit of claim 12, further comprising:

cache memory to store the plurality of new integrity indicators; and

the processor further configured to generate the new integrity record by calculating a hash based on the plurality of new integrity indicators.

18. An apparatus comprising:

a processor to receive a plurality of data slices, each of the plurality of data slices including a different encoded version of a data segment encoded to prevent reconstruction of the data segment using a single one of the plurality of data slices, and to permit reconstruction of the data segment using at least a threshold number of the plurality of data slices;

the processor further to;

calculate first integrity indicators of each of the plurality of data slices;

generate an integrity record based on the first integrity indicators;

append the integrity record to each of the plurality of data slices to generate modified data slices; and

a communications output to transmit the modified data slices to a plurality of slice storage units.

19. The distributed storage processor of claim 18, further comprising:

cache memory to store the first integrity indicators; and

the processor further to generate the integrity record by creating a list including the first integrity indicators.

20. The distributed storage processor of claim 18, further comprising:

cache memory to store the first integrity indicators; and

the processor further to generate the integrity record by calculating a hash value using the first integrity indicators.

21. The distributed storage processor of claim 18, wherein the data segment is encoded into a number of pillars, the communications input further configured to:

receive a number of data slices equal to the number of pillars.

22. The distributed storage processor of claim 21, wherein the number of pillars used to encode the data segment is determined, at least in part, by a parameter associated with a vault of the dispersed storage network.