Elastic storage in a dispersed storage network

• US 10,459,790 B2
• Filed: 07/26/2016
• Issued: 10/29/2019
• Est. Priority Date: 07/26/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Leggette, Wesley B., Motwani, Manish, Ober, Brian F., Resch, Jason K.
• Primary Examiner(s): Nguyen, Thien
• Application Number:
  US15/219,469
• Publication Number:
  US 20180032269A1
• Time in Patent Office: 1,190 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems

Abstract

A method for execution by a dispersed storage and task (DST) processing unit includes: generating an encoded data slice from a dispersed storage encoding of a data object and determining when the encoded data slice will not be stored in local dispersed storage. When the encoded data slice will not be stored in the local dispersed storage, the encoded data slice is stored via at least one elastic slice in an elastic dispersed storage, an elastic storage pointer is generated indicating a location of the elastic slice in the elastic dispersed storage, and the elastic storage pointer is stored in the local dispersed storage.

Claims

1. A method for execution by a dispersed storage and task (DST) processing unit that includes a processor, the method comprises:

dispersed storage encoding a data object to generate a plurality of encoded data slices;

determining when an encoded data slice of the plurality of encoded data slices will not be stored in local dispersed storage;

when the encoded data slice will not be stored in the local dispersed storage;

dispersed storage encoding the encoded data slice into a plurality of elastic slices;

storing the plurality of elastic slices in a plurality of storage units of an elastic dispersed storage;

generating an elastic storage pointer indicating retrieval information for retrieval of the plurality of elastic slices of the encoded data slice from the plurality of storage units; and

storing the elastic storage pointer in the local dispersed storage.

2. The method of claim 1 wherein determining when the encoded data slice will not be stored in local dispersed storage is based on at least one of:

an available storage capacity of the local dispersed storage, a comparison of elastic storage cost to local storage cost, a security requirement associated with the data object, an access parameter associated with the data object, or a status parameter of the local dispersed storage.

3. The method of claim 1 wherein the elastic storage pointer includes at least one of:

a DSN memory identifier associated with the elastic dispersed storage, a data object identifier, a namespace location associated with the elastic dispersed storage, a slice name associated with the plurality of elastic slices, a slice name associated with the encoded data slice, or a revision identifier associated with the encoded data slice.

4. The method of claim 1, further comprising:

dispersed storage encoding a second data object to generate a second plurality of encoded data slices;

determining when a second encoded data slice of the second plurality of encoded data slices will not be stored in local dispersed storage;

when the second encoded data slice will not be stored in the local dispersed storage;

sending the second encoded data slice to another DST processing unit associated with the elastic dispersed storage for dispersed storage encoding the second encoded data slice into a second plurality of elastic slices and storage of the second plurality of elastic slices in a plurality of storage units of the elastic dispersed storage;

generating a second elastic storage pointer indicating second retrieval information for retrieval of the second plurality of elastic slices of the encoded data slice from the plurality of storage units; and

storing the second elastic storage pointer in the local dispersed storage.

5. The method of claim 1 further comprising:

receiving a write request response from the local dispersed storage in response to the storing of the elastic storage pointer.

6. The method of claim 1 further comprising:

determining the elastic storage pointer for the encoded data slice is stored in the local dispersed storage in response to a read request for the encoded data slice;

extracting the retrieval information from the elastic storage pointer;

retrieving the plurality of elastic slices from the plurality of storage units based on the retrieval information; and

restoring the encoded data slice from the plurality of elastic slices.

7. A processing system of a dispersed storage and task (DST) processing unit comprises:

at least one processor;

a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to;

dispersed storage encode a data object to generate a plurality of encoded data slices;

determine when an encoded data slice of the plurality of encoded data slices will not be stored in local dispersed storage;

when the encoded data slice will not be stored in the local dispersed storage;

dispersed storage encode the encoded data slice into a plurality of elastic slices;

store the plurality of elastic slices in a plurality of storage units of an elastic dispersed storage;

generate an elastic storage pointer indicating retrieval information for retrieval of the plurality of elastic slices of the encoded data slice from the plurality of storage units; and

store the elastic storage pointer in the local dispersed storage.

8. The processing system of claim 7 wherein determining when the encoded data slice will not be stored in local dispersed storage is based on at least one of:

an available storage capacity of the local dispersed storage, a comparison of elastic storage cost to local storage cost, a security requirement associated with the data object, an access parameter associated with the data object, or a status parameter of the local dispersed storage.

9. The processing system of claim 7, wherein the elastic storage pointer includes at least one of:

a DSN memory identifier associated with the elastic dispersed storage, a data object identifier, a namespace location associated with the elastic dispersed storage, a slice name associated with the plurality of elastic slices, a slice name associated with the encoded data slice, or a revision identifier associated with the encoded data slice.

10. The processing system of claim 7, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

dispersed storage encode a second data object to generate a second plurality of encoded data slices;

determine when a second encoded data slice of the second plurality of encoded data slices will not be stored in local dispersed storage;

when the second encoded data slice will not be stored in the local dispersed storage;

send the second encoded data slice to another DST processing unit associated with the elastic dispersed storage for dispersed storage encoding the second encoded data slice into a second plurality of elastic slices and storage of the second plurality of elastic slices in a plurality of storage units of the elastic dispersed storage;

generate a second elastic storage pointer indicating second retrieval information for retrieval of the second plurality of elastic slices of the encoded data slice from the plurality of storage units; and

store the second elastic storage pointer in the local dispersed storage.

11. The processing system of claim 7 wherein a write request response is received from the local dispersed storage in response to the storing of the elastic storage pointer.

12. The processing system of claim 7 wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

determine the elastic storage pointer for the encoded data slice is stored in the local dispersed storage in response to a read request for the encoded data slice;

extract the retrieval information from the elastic storage pointer;

retrieve the plurality of elastic slices from the plurality of storage units based on the retrieval information; and

restore the encoded data slice from the plurality of elastic slices.

13. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

dispersed storage encode a data object to generate a plurality of encoded data slices;

determine when an encoded data slice of the plurality of encoded data slices will not be stored in local dispersed storage;

when the encoded data slice will not be stored in the local dispersed storage;

dispersed storage encode the encoded data slice into a plurality of elastic slices;

store the plurality of elastic slices in a plurality of storage units of an elastic dispersed storage;

generate an elastic storage pointer indicating retrieval information for retrieval of the plurality of elastic slices of the encoded data slice from the plurality of storage units; and

store the elastic storage pointer in the local dispersed storage.

14. The non-transitory computer readable storage medium of claim 13 wherein determining when the encoded data slice will not be stored in local dispersed storage is based on at least one of:

an available storage capacity of the local dispersed storage, a comparison of elastic storage cost to local storage cost, a security requirement associated with the data object, an access parameter associated with the data object, or a status parameter of the local dispersed storage.

15. The non-transitory computer readable storage medium of claim 13 wherein the elastic storage pointer includes at least one of:

a DSN memory identifier associated with the elastic dispersed storage, a data object identifier, a namespace location associated with the elastic dispersed storage, a slice name associated with the plurality of elastic slices, a slice name associated with the encoded data slice, a revision identifier associated with the encoded data slice.

16. The non-transitory computer readable storage medium of claim 13, wherein the operational instructions, when executed by the processing system of the DSN, causes the processing system to:

dispersed storage encode a second data object to generate a second plurality of encoded data slices;

determine when a second encoded data slice of the second plurality of encoded data slices will not be stored in local dispersed storage;

when the second encoded data slice will not be stored in the local dispersed storage;

send the second encoded data slice to another DST processing unit associated with the elastic dispersed storage for dispersed storage encoding the second encoded data slice into a second plurality of elastic slices and storage of the second plurality of elastic slices in a plurality of storage units of the elastic dispersed storage;

generate a second elastic storage pointer indicating second retrieval information for retrieval of the second plurality of elastic slices of the encoded data slice from the plurality of storage units; and

store the second elastic storage pointer in the local dispersed storage.