Securely storing data in an elastically scalable dispersed storage network

• US 10,395,043 B2
• Filed: 07/29/2016
• Issued: 08/27/2019
• Est. Priority Date: 07/29/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Leggette, Wesley B., Motwani, Manish, Ober, Brian F., Resch, Jason K.
• Primary Examiner(s): Wright, Bryan F
• Application Number:
  US15/223,815
• Publication Number:
  US 20180032738A1
• Time in Patent Office: 1,124 Days
• Field of Search: 713193
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 3/061 Improving I/O performance

Abstract

A method for execution by a dispersed storage and task (DST) processing unit includes: generating an encoded data slice from a dispersed storage encoding of a data object and determining when the encoded data slice will not be stored in local dispersed storage. When the encoded data slice will not be stored in the local dispersed storage, the encoded data slice is stored via at least one elastic slice in an elastic dispersed storage, cryptographic material and an elastic storage pointer indicating a location of the elastic slice in the elastic dispersed storage are generated, and the cryptographic material and the elastic storage pointer are stored in the local dispersed storage.

Claims

1. A method for execution by a dispersed storage and task (DST) processing unit that includes a processor, the method comprises:

generating an encoded data slice from a dispersed storage encoding of a data object;

determining when the encoded data slice will not be stored in local dispersed storage;

when the encoded data slice will not be stored in the local dispersed storage;

storing the encoded data slice via at least one elastic slice in an elastic dispersed storage by;

dispersed storage encoding the encoded data slice into a plurality of elastic slices; and

storing the plurality of elastic slices in a plurality of storage units of the elastic dispersed storage;

generating cryptographic material and an elastic storage pointer indicating a location of the elastic slice in the elastic dispersed storage;

generating a read error indicator based on the cryptographic material, wherein the read error indicator is used to verify the encoded slice data or the elastic slices; and

storing the cryptographic material and the elastic storage pointer in the local dispersed storage when the encoded data slice is stored in the at least one elastic slice in the elastic dispersed storage.

2. The method of claim 1 wherein determining when the encoded data slice will not be stored in local dispersed storage is based on at least one of:

an available storage capacity of the local dispersed storage, a comparison of elastic storage cost to local storage cost, a security requirement associated with the data object, an access parameter associated with the data object, or a status parameter of the local dispersed storage.

3. The method of claim 1 wherein the elastic storage pointer includes a DSN memory identifier associated with the elastic dispersed storage, a data object identifier, a namespace location associated with the elastic dispersed storage, a slice name associated with the elastic slice, a slice name associated with the encoded data slice, or a revision identifier associated with the encoded data slice.

4. The method of claim 1 wherein storing the encoded data slice via the elastic slice is performed by:

sending the encoded data slice to another DST processing unit associated with the elastic dispersed storage for dispersed storage encoding the encoded data slice into a plurality of elastic slices and storage of the plurality of elastic slices in a plurality of storage units of the elastic dispersed storage.

5. The method of claim 1 wherein the cryptographic material includes at least one of:

a key encryption key;

encryption key;

hash-message authentication code (HMAC) key;

digital signing key;

or a public/private key pair.

6. The method of claim 1 further comprising:

sending a read request to the local dispersed storage for the encoded data slice;

retrieving the elastic slice from the elastic dispersed storage and verifying the elastic slice;

when the elastic slice is verified, generating a response to the read request from the local dispersed storage that includes the encoded data slice as recovered from the elastic slice; and

when the elastic slice is not verified, generating the response to the read request from the local dispersed storage to includes a read error indication.

7. A processing system of a dispersed storage and task (DST) processing unit comprises:

at least one processor;

a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to;

generate an encoded data slice from a dispersed storage encoding of a data object;

determine when the encoded data slice will not be stored in local dispersed storage;

when the encoded data slice will not be stored in the local dispersed storage;

store the encoded data slice via at least one elastic slice in an elastic dispersed storage by;

dispersed storage encoding the encoded data slice into a plurality of elastic slices; and

storing the plurality of elastic slices in a plurality of storage units of the elastic dispersed storage;

generate cryptographic material and an elastic storage pointer indicating a location of the elastic slice in the elastic dispersed storage;

generate a read error indicator based on the cryptographic material, wherein the read error indicator is used to verify the encoded slice data or the elastic slices; and

store the cryptographic material and the elastic storage pointer in the local dispersed storage when the encoded data slice is stored in the at least one elastic slice in the elastic dispersed storage.

8. The processing system of claim 7 wherein determining when the encoded data slice will not be stored in local dispersed storage is based on at least one of:

an available storage capacity of the local dispersed storage, a comparison of elastic storage cost to local storage cost, a security requirement associated with the data object, an access parameter associated with the data object, or a status parameter of the local dispersed storage.

9. The processing system of claim 7 wherein the elastic storage pointer includes a DSN memory identifier associated with the elastic dispersed storage, a data object identifier, a namespace location associated with the elastic dispersed storage, a slice name associated with the elastic slice, a slice name associated with the encoded data slice, or a revision identifier associated with the encoded data slice.

10. The processing system of claim 7 wherein storing the encoded data slice via the elastic slice is performed by:

sending the encoded data slice to another DST processing unit associated with the elastic dispersed storage for dispersed storage encoding the encoded data slice into a plurality of elastic slices and storage of the plurality of elastic slices in a plurality of storage units of the elastic dispersed storage.

11. The processing system of claim 7 wherein a write request response is received from the local dispersed storage in response to the storing of the elastic storage pointer.

12. The processing system of claim 7 wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

send a read request to the local dispersed storage for the encoded data slice;

retrieve the elastic slice from the elastic dispersed storage and verifying the elastic slice;

when the elastic slice is verified, generate a response to the read request from the local dispersed storage that includes the encoded data slice as recovered from the elastic slice; and

when the elastic slice is not verified, generate the response to the read request from the local dispersed storage to include a read error indication.

13. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

generate an encoded data slice from a dispersed storage encoding of a data object;

determine when the encoded data slice will not be stored in local dispersed storage;

when the encoded data slice will not be stored in the local dispersed storage;

store the encoded data slice via at least one elastic slice in an elastic dispersed storage by;

dispersed storage encoding the encoded data slice into a plurality of elastic slices; and

storing the plurality of elastic slices in a plurality of storage units of the elastic dispersed storage;

generate cryptographic material and an elastic storage pointer indicating a location of the elastic slice in the elastic dispersed storage;

generate a read error indicator based on the cryptographic material, wherein the read error indicator is used to verify the encoded slice data or the elastic slices; and

store the cryptographic material and the elastic storage pointer in the local dispersed storage when the encoded data slice is stored in the at least one elastic slice in the elastic dispersed storage.

14. The non-transitory computer readable storage medium of claim 13 wherein determining when the encoded data slice will not be stored in local dispersed storage is based on at least one of:

an available storage capacity of the local dispersed storage, a comparison of elastic storage cost to local storage cost, a security requirement associated with the data object, an access parameter associated with the data object, or a status parameter of the local dispersed storage.

15. The non-transitory computer readable storage medium of claim 13 wherein the cryptographic material includes at least one of:

a key encryption key;

encryption key;

hash-message authentication code (HMAC) key;

digital signing key;

or a public/private key pair.

16. The non-transitory computer readable storage medium of claim 13 wherein the operational instructions, when executed by the processing system, further cause the processing system to:

send a read request to the local dispersed storage for the encoded data slice; and

retrieve the elastic slice from the elastic dispersed storage and verifying the elastic slice;

when the elastic slice is verified, generate a response to the read request from the local dispersed storage that includes the encoded data slice as recovered from the elastic slice; and

when the elastic slice is not verified, generate the response to the read request from the local dispersed storage to include a read error indication.