Ensuring data integrity on a dispersed storage grid
First Claim
Patent Images
1. A method of ensuring data integrity in a dispersed data storage network, the method comprising:
- calculating a first checksum for a data segment of data;
appending the first checksum to the data segment to produce an appended data segment;
encoding the appended data segment to produce a set of encoded data slices, wherein each encoded data slice of the set of encoded data slices includes an encoded representation of at least a portion of the first checksum;
generating a set of second checksums for the set of encoded data slices;
appending the set of second checksums to the set of encoded data slices to produce a set of appended encoded data slices; and
transmitting the set of appended encoded data slices to a plurality of storage units for storage therein;
when the data segment is to be reconstructed;
retrieving from the plurality of storage units a threshold number of appended encoded data slices of the set of appended encoded data slices;
extracting the threshold number of second checksums from the threshold number of appended encoded data slices; and
verifying accuracy of the threshold number of appended encoded data slices based on the extracted threshold number of second checksums; and
when verifying the accuracy of the threshold number of appended encoded data slices indicates that the threshold number of appended encoded data slices includes a corrupted appended encoded data slice therein;
retrieving from at least some of the plurality of storage units a sufficient number of appended encoded data slices needed to generate a reconstructed data segment based on the threshold number of appended encoded data slices that includes the corrupted appended encoded data slice;
generating the reconstructed data segment using the sufficient number of appended encoded data slices;
performing an error correcting information dispersal algorithm on the reconstructed data segment thereby forming a reconstructed set of data slices;
calculating a plurality of second checksums for the reconstructed set of data slices;
appending the plurality of second checksums respectively to the reconstructed set of data slices thereby forming a reconstructed set of appended encoded data slices; and
replacing the corrupted appended encoded data slice with a reconstructed appended encoded data slice of the reconstructed set of appended encoded data slices; and
when the accuracy of the threshold number of appended encoded data slices has been verified;
decoding the threshold number of appended encoded data slices to recover the appended data segment;
extracting the first checksum from the recovered appended data segment;
verifying accuracy of the recovered appended data segment based on the first checksum; and
when the accuracy of the recovered appended data segment has been verified, providing the recovered appended data segment as a valid data segment.
7 Assignments
0 Petitions
Accused Products
Abstract
An improved system is disclosed for ensuring the integrity of data stored on a dispersed data storage network. Checksums are used to ensure integrity of both data segments and data slices. Checksums appended to data slices are checked by receiving slice servers to ensure that no errors occurred during transmission. Slice servers also periodically recalculate checksums for stored data slices to ensure that data slices have not been corrupted during storage. Checksums appended to data segments are checked when data segments are read from the storage network.
100 Citations
17 Claims
-
1. A method of ensuring data integrity in a dispersed data storage network, the method comprising:
-
calculating a first checksum for a data segment of data; appending the first checksum to the data segment to produce an appended data segment; encoding the appended data segment to produce a set of encoded data slices, wherein each encoded data slice of the set of encoded data slices includes an encoded representation of at least a portion of the first checksum; generating a set of second checksums for the set of encoded data slices; appending the set of second checksums to the set of encoded data slices to produce a set of appended encoded data slices; and transmitting the set of appended encoded data slices to a plurality of storage units for storage therein; when the data segment is to be reconstructed; retrieving from the plurality of storage units a threshold number of appended encoded data slices of the set of appended encoded data slices; extracting the threshold number of second checksums from the threshold number of appended encoded data slices; and verifying accuracy of the threshold number of appended encoded data slices based on the extracted threshold number of second checksums; and when verifying the accuracy of the threshold number of appended encoded data slices indicates that the threshold number of appended encoded data slices includes a corrupted appended encoded data slice therein; retrieving from at least some of the plurality of storage units a sufficient number of appended encoded data slices needed to generate a reconstructed data segment based on the threshold number of appended encoded data slices that includes the corrupted appended encoded data slice; generating the reconstructed data segment using the sufficient number of appended encoded data slices; performing an error correcting information dispersal algorithm on the reconstructed data segment thereby forming a reconstructed set of data slices; calculating a plurality of second checksums for the reconstructed set of data slices; appending the plurality of second checksums respectively to the reconstructed set of data slices thereby forming a reconstructed set of appended encoded data slices; and replacing the corrupted appended encoded data slice with a reconstructed appended encoded data slice of the reconstructed set of appended encoded data slices; and when the accuracy of the threshold number of appended encoded data slices has been verified; decoding the threshold number of appended encoded data slices to recover the appended data segment; extracting the first checksum from the recovered appended data segment; verifying accuracy of the recovered appended data segment based on the first checksum; and when the accuracy of the recovered appended data segment has been verified, providing the recovered appended data segment as a valid data segment. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A computer comprising:
-
a port configured to support communications with a dispersed data storage network; an application, coupled to the port, that is configured to enable a computer to; calculate a first checksum for a data segment of data; append the first checksum to the data segment to produce an appended data segment; encode the appended data segment to produce a set of encoded data slices, wherein each encoded data slice of the set of encoded data slices includes an encoded representation of at least a portion of the first checksum; generate a set of second checksums for the set of encoded data slices; append the set of second checksums to the set of encoded data slices to produce a set of appended encoded data slices; and transmit the set of appended encoded data slices to a plurality of storage units for storage therein; and when the data segment is to be reconstructed; identify whether a threshold number of appended encoded data slices includes a corrupted appended encoded data slice therein; and when the threshold number of appended encoded data slice includes the corrupted appended encoded data slice; retrieve, from at least some of a plurality of storage units of the dispersed data storage network, a sufficient number of appended encoded data slices needed to generate a reconstructed data segment based on the threshold number of appended encoded data slices that includes the corrupted appended encoded data slice; generate the reconstructed data segment using the sufficient number of appended encoded data slices; perform an error correcting information dispersal algorithm on the reconstructed data segment thereby forming a reconstructed set of data slices; calculating a plurality of second checksums for the reconstructed set of data slices; append the plurality of second checksums respectively to the reconstructed set of data slices thereby forming a reconstructed set of appended encoded data slices; and replace the corrupted appended encoded data slice with one of the reconstructed set of appended encoded data slices. - View Dependent Claims (8, 9, 10, 11, 12)
-
-
13. A computer comprising:
-
a port configured to support communications with a dispersed data storage network; an application, coupled to the port, that is configured to enable a computer to; when a data segment is to be reconstructed; retrieving, from a plurality of storage units of the dispersed data storage network, a threshold number of appended encoded data slices of a set of appended encoded data slices; extracting a threshold number of second checksums from the threshold number of appended encoded data slices; and verifying accuracy of the threshold number of appended encoded data slices based on the extracted threshold number of second checksums; when the threshold number of appended encoded data slices includes a corrupted appended encoded data slice therein; retrieve, from at least some of the plurality of storage units, a sufficient number of appended encoded data slices needed to generate a reconstructed data segment based on the threshold number of appended encoded data slices that includes the corrupted appended encoded data slice; generate the reconstructed data segment using the sufficient number of appended encoded data slices; perform an error correcting information dispersal algorithm on the reconstructed data segment thereby forming a reconstructed set of data slices; calculating a plurality of second checksums for the reconstructed set of data slices; append the plurality of second checksums respectively to the reconstructed set of data slices thereby forming a reconstructed set of appended encoded data slices; and replace the corrupted appended encoded data slice with one of the reconstructed set of appended encoded data slices; and when the accuracy of the threshold number of appended encoded data slices has been verified; decoding the threshold number of appended encoded data slices to recover the appended data segment; extracting the first checksum from the recovered appended data segment; verifying accuracy of the recovered appended data segment based on the first checksum; and when the accuracy of the recovered appended data segment has been verified, providing the recovered appended data segment as a valid data segment. - View Dependent Claims (14, 15, 16, 17)
-
Specification