Encoding data in a dispersed storage network

• US 10,331,518 B2
• Filed: 11/20/2017
• Issued: 06/25/2019
• Est. Priority Date: 08/31/2012
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Resch, Jason K., Leggette, Wesley B.
• Primary Examiner(s): Chaudry, Mujtaba M
• Application Number:
  US15/818,641
• Publication Number:
  US 20180074899A1
• Time in Patent Office: 582 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0712 in a virtual computing platform, e.g. logically partitioned systems
• CPC Class Codes: G06F 11/0784 Routing of error reports, e.g. with a specific transmission path or data flow
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/1044 with specific ECC/EDC distribution
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/142 Reconfiguring to eliminate the error group management mechanisms in a peer-to-peer network H04L67/1044
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/60 Protecting data
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/6272 by registering files or documents with a third party
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/611 Specific encoding aspects, e.g. encoding by means of decoding
• CPC Class Codes: H03M 13/616 Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 2209/34 Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/0861 Generation of secret information including derivation or calculation of cryptographic keys or passwords
• CPC Class Codes: H04L 9/0894 : Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage

Abstract

A method for execution by an integrity processing unit includes performing a deterministic function on data for storage to produce an integrity value. The data and the integrity value are combined in accordance with a combining function to produce a data package. The processing system determines an encryption approach in response to determining to encrypt the data package. The data package is encrypted in accordance with the encryption approach to produce a secure package. The secure package is encoded to produce a set of slices. The set of slices is decoded to reproduce the secure package. The secure package is decrypted to reproduce the data package. The data package is de-combined in to generate reproduced data and a received integrity value. The deterministic function is performed on the data to produce a calculated integrity value, and the received integrity value is compared to the calculated integrity value.

Claims

1. A method for execution by an integrity processing unit that includes a processor, the method comprises:

performing a deterministic function on data for storage to produce an integrity value;

combining the data and the integrity value in accordance with a combining function to produce a data package;

determining whether to encrypt the data package;

determining an encryption approach in response to determining to encrypt the data package;

encrypting the data package in accordance with the encryption approach to produce a secure package;

encoding the secure package to produce a set of slices;

decoding the set of slices to reproduce the secure package;

identifying the encryption approach associated with the secure package;

decrypting the secure package in accordance with the encryption approach to reproduce the data package;

de-combining the data package in accordance with the combining function to generate reproduced data and a received integrity value;

performing the deterministic function on the data to produce a calculated integrity value; and

generating a validity indicator of the reproduced data based on a comparison of the received integrity value and the calculated integrity value.

2. The method of claim 1, wherein the secure package is encoded by utilizing a dispersed storage error coding function, further comprising:

transmitting the set of slices to a plurality of storage units for storage in accordance with the dispersed storage error coding function;

receiving a request for the data; and

identifying the set of slices in response to receiving the request and retrieving the set of slices from the plurality of storage units for decoding.

3. The method of claim 1, wherein the deterministic function includes utilizing a hash-based message authentication code (HMAC), and wherein performing the deterministic function includes utilizing a key derived from the data, wherein the key is derived by performing a second deterministic function.

4. The method of claim 1, further comprising:

determining one of a plurality of data security levels associated with the data;

wherein the combining function includes partitioning the integrity value into a plurality of portions and interspersing the plurality of portions evenly across the data, wherein a number of the plurality of portions is a monotonically increasing function of the one of the plurality of data security levels.

5. The method of claim 1, further comprising:

performing a test on the data package to determine whether the data package is already encrypted;

wherein the integrity processing unit determines to encrypt the data package in response to determining that the data package is not already encrypted.

6. The method of claim 1, further comprising encoding the data package to produce the set of slices in response to determining not to encrypt the data.

7. The method of claim 1, wherein the encryption approach includes utilizing an all or nothing transformation function, and wherein decrypting the secure package includes:

combining the secure package to produce encrypted data and a masked key;

performing an exclusive OR function on a hash of the encrypted data and the masked key to produce a key; and

decrypting the encrypted data utilizing the key to produce the data package.

8. The method of claim 1, wherein the validity indicator that indicates that the reproduced data is valid when the comparison indicates that a calculated difference between the received integrity value and the calculated integrity value compares favorably to a difference threshold.

9. The method of claim 1, wherein validity indicator that indicates that the reproduced data is valid when the comparison indicates that the received integrity value is equal to the calculated integrity value.

10. The method of claim 1, further comprising transmitting the validity indicator to a requester of the data.

11. A processing system of an integrity processing unit comprises:

at least one processor;

a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to;

perform a deterministic function on data for storage to produce an integrity value;

combine the data and the integrity value in accordance with a combining function to produce a data package;

determine whether to encrypt the data package;

determine an encryption approach in response to determining to encrypt the data package;

encrypt the data package in accordance with the encryption approach to produce a secure package;

encode the secure package to produce a set of slices;

decode the set of slices to reproduce the secure package;

identify the encryption approach associated with the secure package;

decrypt the secure package in accordance with the encryption approach to reproduce the data package;

de-combine the data package in accordance with the combining function to generate reproduced data and a received integrity value;

perform the deterministic function on the data to produce a calculated integrity value; and

generate a validity indicator of the reproduced data based on a comparison of the received integrity value and the calculated integrity value.

12. The processing system of claim 11, wherein the secure package is encoded by utilizing a dispersed storage error coding function, and wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

transmit the set of slices to a plurality of storage units for storage in accordance with the dispersed storage error coding function;

receive a request for the data; and

identify the set of slices in response to receiving the request and retrieving the set of slices from the plurality of storage units for decoding.

13. The processing system of claim 11, wherein the deterministic function includes utilizing a hash-based message authentication code (HMAC), and wherein performing the deterministic function includes utilizing a key derived from the data, wherein the key is derived by performing a second deterministic function.

14. The processing system of claim 11, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

determine one of a plurality of data security levels associated with the data;

wherein the combining function includes partitioning the integrity value into a plurality of portions and interspersing the plurality of portions evenly across the data, wherein a number of the plurality of portions is a monotonically increasing function of the one of the plurality of data security levels.

15. The processing system of claim 11, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

perform a test on the data package to determine whether the data package is already encrypted;

wherein the integrity processing unit determines to encrypt the data package in response to determining that the data package is not already encrypted.

16. The processing system of claim 11, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to encode the data package to produce the set of slices in response to determining not to encrypt the data.

17. The processing system of claim 11, wherein the encryption approach includes utilizing an all or nothing transformation function, and wherein decrypting the secure package includes:

combining the secure package to produce encrypted data and a masked key;

performing an exclusive OR function on a hash of the encrypted data and the masked key to produce a key; and

decrypting the encrypted data utilizing the key to produce the data package.

18. The processing system of claim 11, wherein the validity indicator that indicates that the reproduced data is valid when the comparison indicates that a calculated difference between the received integrity value and the calculated integrity value compares favorably to a difference threshold.

19. The processing system of claim 11, wherein validity indicator that indicates that the reproduced data is valid when the comparison indicates that the received integrity value is equal to the calculated integrity value.

20. A computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

perform a deterministic function on data for storage to produce an integrity value;

combine the data and the integrity value in accordance with a combining function to produce a data package;

determine whether to encrypt the data package;

determine an encryption approach in response to determining to encrypt the data package;

encrypt the data package in accordance with the encryption approach to produce a secure package;

encode the secure package to produce a set of slices;

decode the set of slices to reproduce the secure package;

identify the encryption approach associated with the secure package;

decrypt the secure package in accordance with the encryption approach to reproduce the data package;

de-combine the data package in accordance with the combining function to generate reproduced data and a received integrity value;

perform the deterministic function on the data to produce a calculated integrity value; and

generate a validity indicator of the reproduced data based on a comparison of the received integrity value and the calculated integrity value.