Asserting integrity with a verifiable codec

• US 10,372,380 B2
• Filed: 03/01/2017
• Issued: 08/06/2019
• Est. Priority Date: 03/01/2017
• Status: Active Grant

• Current Assignee: Beijing Zitiao Network Technology Company Limited (ByteDance Ltd.)
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K., Seaborn, Mark D.
• Primary Examiner(s): Chaudry, Mujtaba M
• Application Number:
  US15/446,866
• Publication Number:
  US 20180254784A1
• Time in Patent Office: 888 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes

Abstract

A method includes retrieving a decode threshold number of encoded data slices, wherein codecs process, in an order, a data segment and the processed data segment is encoded into a set of encoded data slices. The method further includes decoding the decode threshold number of encoded data slices to recover the processed data segment. In a reversed order to the order, applying a first codec on the processed data segment to produce a first partially processed recovered data segment. When the first codec is a verifiable codec, the method further includes separating the first partially processed recovered data segment into an initial integrity value and a processed data segment. The method further includes calculating a new integrity value from the processed data segment. When the new integrity value substantially matches the initial integrity value, the method further includes indicating that the set of encoded data slices is authentic.

Claims

1. A method for execution by a computing device of a dispersed storage network (DSN), the method comprises:

retrieving, by the computing device, a decode threshold number of encoded data slices of a set of encoded data slices from DSN memory, wherein a processed data segment was dispersed storage error encoded to produce the set of encoded data slices and stored in the DSN memory, wherein a data segment of a data object is processed in an order by a plurality of codecs to produce the processed data segment, and wherein the decode threshold number is a minimum number of encoded data slices of the set of encoded data slices needed in order to reconstruct the data segment;

decoding, by the computing device, the decode threshold number of encoded data slices to recover the processed data segment;

determining, by the computing device, reverse order of the plurality of codecs based on the order;

when a first codec of the reverse order of the plurality of codecs is a verifiable codec;

separating, by the computing device, the processed data segment into an integrity value and a partially processed data segment;

calculating, by the computing device, a new integrity value from the partially processed data segment;

determining, by the computing device, whether the new integrity value substantially matches the integrity value; and

when the new integrity value substantially matches the integrity value, performing, by the computing device, another codec of the reverse order of the plurality of codecs on the partially processed data segment to recover the data segment.

2. The method of claim 1 further comprises:

when the first codec is a decompression codec and a second codec of the plurality of codecs is the verifiable codec;

decompressing the processed data segment to produce a decompressed processed data segment;

separating the decompressed processed data segment into the integrity value and a decompressed partially processed data segment;

calculating the new integrity value from the decompressed partially processed data segment; and

when the new integrity value substantially matches the integrity value, indicating that the set of encoded data slices is authentic.

3. The method of claim 1 further comprises:

when the first codec is a decryption codec and a second codec of the plurality of codecs is the verifiable codec;

decrypting the processed data segment to produce a decrypted processed data segment;

separating the decrypted processed data segment into the integrity value and a decrypted partially processed data segment;

calculating the new integrity value from the decrypted partially processed data segment; and

when the new integrity value substantially matches the integrity value, indicating that the set of encoded data slices is authentic.

4. The method of claim 1 further comprises:

when the first codec is a decompression codec, a second codec of the plurality of codecs is a decryption codec, and a third codec of the plurality of codecs is the verifiable codec;

decompressing the processed data segment to produce a decompressed processed data segment;

decrypting the decompressed processed data segment to produce a decrypted processed data segment;

separating the decrypted processed data segment into the integrity value and a decrypted partially processed data segment;

calculating the new integrity value from the decrypted partially processed data segment; and

when the new integrity value substantially matches the integrity value, indicating that the set of encoded data slices is authentic.

5. The method of claim 1 further comprises:

when the new integrity value does not substantially match the integrity value, generating an integrity check error message.

6. The method of claim 1 further comprises:

when the new integrity value does not substantially match the integrity value, indicating that at least one of the decode threshold number of encoded data slices has an integrity error;

requesting another encoded data slice of the set of encoded data slices, wherein the other encoded data slice was not included in the decode threshold number of encoded data slices;

replacing one of the encoded data slices of the decode threshold number of encoded data slices with the other encoded data slice to produce an updated decode threshold number of encoded data slices; and

decoding the updated decode threshold number of encoded data slices to recover the processed data segment.

7. The method of claim 1 further comprises:

encoding the processed data segment to produce a new set of encoded data slices; and

selecting one encoded data slice of the new set of encoded data slices for replacing a missing or corrupt encoded data slice of the set of encoded data slices.

8. A computing device comprises:

an interface;

memory; and

a processing module operably coupled to the interface and the memory, wherein the processing module is operable to;

receive, via the interface, a decode threshold number of encoded data slices of a set of encoded data slices from DSN memory, wherein a processed data segment was dispersed storage error encoded to produce the set of encoded data slices and stored in the DSN memory, wherein a data segment of a data object is processed in an order by a plurality of codecs to produce the processed data segment, and wherein the decode threshold number is a minimum number of encoded data slices of the set of encoded data slices needed in order to reconstruct the data segment;

decode the decode threshold number of encoded data slices to recover the processed data segment; and

determine reverse order of the plurality of codecs based on the order;

when a first codec of the reverse order of the plurality of codecs is a verifiable codec;

separate the processed data segment into an integrity value and a partially processed data segment;

calculate a new integrity value from the partially processed data segment;

determine whether the new integrity value substantially matches the integrity value; and

when the new integrity value substantially matches the integrity value, perform, another codec of the reverse order of the plurality of codecs on the partially processed data segment to recover the data segment.

9. The computing device of claim 8, wherein the processing module is further operable to:

when the first codec is a decompression codec and a second codec of the plurality of codecs is the verifiable codec;

decompress the processed data segment to produce a decompressed processed data segment;

separate the decompressed processed data segment into the integrity value and a decompressed partially processed data segment;

calculate the new integrity value from the decompressed partially processed data segment; and

when the new integrity value substantially matches the integrity value, indicate that the set of encoded data slices is authentic.

10. The computing device of claim 8, wherein the processing module is further operable to:

when the first codec is a decryption codec and a second codec of the plurality of codecs is the verifiable codec;

decrypt the processed data segment to produce a decrypted processed data segment;

separate the decrypted processed data segment into the integrity value and a decrypted partially processed data segment;

calculate the new integrity value from the decrypted partially processed data segment; and

when the new integrity value substantially matches the integrity value, indicating that the set of encoded data slices is authentic.

11. The computing device of claim 8, wherein the processing module is further operable to:

when the first codec is a decompression codec, a second codec of the plurality of codecs is a decryption codec, and a third codec of the plurality of codecs is the verifiable codec;

decompress the processed data segment to produce a decompressed processed data segment;

decrypt the decompressed processed data segment to produce a decrypted processed data segment;

separate the decrypted processed data segment into the integrity value and a decrypted partially processed data segment;

calculate the new integrity value from the decrypted partially processed data segment; and

when the new integrity value substantially matches the integrity value, indicate that the set of encoded data slices is authentic.

12. The computing device of claim 8, wherein the processing module is further operable to:

when the new integrity value does not substantially match the integrity value, generate an integrity check error message.

13. The computing device of claim 8, wherein the processing module is further operable to:

when the new integrity value does not substantially match the integrity value, indicate that at least one of the decode threshold number of encoded data slices has an integrity error;

receive, via the interface, another encoded data slice of the set of encoded data slices, wherein the other encoded data slice was not included in the decode threshold number of encoded data slices;

replace one of the encoded data slices of the decode threshold number of encoded data slices with the other encoded data slice to produce an updated decode threshold number of encoded data slices; and

decode the updated decode threshold number of encoded data slices to recover the processed data segment.

14. The computing device of claim 8, wherein the processing module is further operable to:

encode the processed data segment to produce a new set of encoded data slices; and

select one encoded data slice of the new set of encoded data slices for replacing a missing or corrupt encoded data slice of the set of encoded data slices.