Adjusting data ingest based on compaction rate in a dispersed storage network

• US 10,635,360 B1
• Filed: 10/29/2018
• Issued: 04/28/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Volvovski, Ilya, Baptist, Andrew D., Martin, Benjamin L., Viraraghavan, Praveen, Guo, Ying Z., Williams, Jordan H.
• Primary Examiner(s): Chaudry, Mujtaba M
• Application Number:
  US16/174,226
• Publication Number:
  US 20200133581A1
• Time in Patent Office: 547 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method for execution by a compaction management system includes determining observed compaction information based on compaction observed in at least one storage device during an observed timeframe. An estimated compaction rate is generated for a first future timeframe based on the observed compaction information. An updated ingest rate is generated for the first future timeframe based on a current ingest rate and the estimated compaction rate. A first proper subset of a set of data to be written to the at least one storage device is generated based on the updated ingest rate. Storage of the first proper subset in the at least one storage device is facilitated during the first future timeframe. Storage of a remaining proper subset of the set of data in an elastic buffer is facilitated during the first future timeframe, where the elastic buffer utilizes a memory of the compaction management system.

Claims

1. A method for execution by a compaction management system that includes a processor, the method comprises:

determining observed compaction information based on compaction observed in at least one storage device during an observed timeframe;

generating an estimated compaction rate for a first future timeframe based on the observed compaction information;

generating an updated ingest rate for the first future timeframe based on a current ingest rate and the estimated compaction rate;

selecting a first proper subset of a set of data to be written to the at least one storage device based on the updated ingest rate;

facilitating storage of the first proper subset in the at least one storage device during the first future timeframe; and

facilitating storage of a remaining proper subset of the set of data in an elastic buffer during the first future timeframe, wherein the elastic buffer utilizes a memory of the compaction management system that is different from the at least one storage device, and wherein the first proper subset and the remaining proper subset are mutually exclusive and collectively exhaustive with respect to the set of data to be written.

2. The method of claim 1, further comprising:

determining second observed compaction information based on compaction observed in the at least one storage device during the first future timeframe in response to the first future timeframe elapsing;

generating a second estimated compaction rate for a second future timeframe based on the observed compaction information, wherein the second future timeframe begins after the first future timeframe has elapsed;

generating a second updated ingest rate for the first future timeframe based on the updated ingest rate and the second estimated compaction rate;

selecting a second proper subset of a second set of data to be written to the at least one storage device based on the second updated ingest rate;

facilitating storage of the second proper subset in the at least one storage device during the second future timeframe; and

facilitating storage of a second remaining proper subset of the second set of data in the elastic buffer during the second future timeframe, wherein the second proper subset and the second remaining proper subset are mutually exclusive and collectively exhaustive with respect to the second set of data to be written.

3. The method of claim 2, wherein the second proper subset includes a third subset of the remaining proper subset, and wherein facilitating storage of the second proper subset includes retrieving the third subset of the remaining proper subset from the elastic buffer during the second future timeframe for storage in the at least one storage device.

4. The method of claim 1, wherein the at least one storage device includes at least one Append Only Memory Device (AOMD), and wherein determining the observed compaction information includes determining an amount of unreferenced space created as a result of overwrite and delete operations performed in the at least one AOMD during the observed timeframe.

5. The method of claim 4, wherein generating the estimated compaction rate includes estimating a number of overwrites and deletes that will be performed in the at least one AOMD based on age of data stored in the at least one AOMD, and further based on a number of overwrites and deletes that were performed in the at least one AOMD during the observed timeframe.

6. The method of claim 1, wherein generating the updated ingest rate includes:

determining whether utilizing the current ingest rate in the first future timeframe is expected to exceed a speed of space reclamation indicated by the estimated compaction rate; and

reducing the current ingest rate to generate the updated ingest rate in response to determining that the current ingest rate in the first future timeframe is expected to exceed the speed of space reclamation indicated by the estimated compaction rate.

7. The method of claim 6, wherein a magnitude of an amount that the updated ingest rate is reduced from the current ingest rate is determined as a monotonically increasing function of a magnitude of a difference between the current ingest rate and the estimated compaction rate.

8. The method of claim 6, wherein generating the updated ingest rate further includes:

determining whether utilizing the current ingest rate in the first future timeframe is expected to exceed a speed of space reclamation indicated by the estimated compaction rate; and

increasing the current ingest rate to generate the updated ingest rate in response to determining that the current ingest rate in the first future timeframe is not expected to exceed the speed of space reclamation indicated by the estimated compaction rate, wherein the updated ingest rate is not expected to exceed the speed of space reclamation.

9. The method of claim 1, wherein generating the updated ingest rate includes:

determining a maximum ingest rate that does not exceed a speed of space reclamation indicated by the estimated compaction rate; and

setting the updated ingest rate equal to the maximum ingest rate.

10. The method of claim 1, wherein generating the updated ingest rate includes:

determining a maximum ingest rate that does not exceed a speed of space reclamation indicated by the estimated compaction rate;

determining an amount of uncertainty associated with the estimated compaction rate;

determining a magnitude of a ingest rate buffer as a monotonically increasing function of the amount of uncertainty; and

reducing the maximum ingest rate by the magnitude of the ingest rate buffer to generate the updated ingest rate.

11. The method of claim 1, wherein determining the observed compaction information includes determining an observed compaction rate, and wherein generating the estimated compaction rate for the first future timeframe includes setting the estimated compaction rate equal to the observed compaction rate.

12. The method of claim 1, wherein the at least one storage device includes a set of storage units, wherein the set of data to be stored in the at least one storage device includes a set of encoded slices for storage in the set of storage units, wherein a data segment was dispersed storage error encoded to produce the set of encoded slices, wherein determining the observed compaction information includes receiving compaction information from the set of storage units via a network, and wherein facilitating storage of the first proper subset includes transmitting at least one of the set of encoded slices to at least one of the set of storage units via the network.

13. The method of claim 1, wherein the compaction management system and the at least one storage device is incorporated within one of a set of storage units, wherein the set of data to be stored includes one of a set of encoded data slices, wherein a data segment was dispersed storage error encoded to produce the set of encoded data slices for storage in the set of storage units, further comprising receiving the one of the set of encoded data slices via a network.

14. A processing system of a compaction management system comprises:

at least one processor;

a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to;

determine observed compaction information based on compaction observed in at least one storage device during an observed timeframe;

generate an estimated compaction rate for a first future timeframe based on the observed compaction information;

generate an updated ingest rate for the first future timeframe based on a current ingest rate and the estimated compaction rate;

select a first proper subset of a set of data to be written to the at least one storage device based on the updated ingest rate;

facilitate storage of the first proper subset in the at least one storage device during the first future timeframe; and

facilitate storage of a remaining proper subset of the set of data in an elastic buffer during the first future timeframe, wherein the elastic buffer utilizes a memory of the compaction management system that is different from the at least one storage device, and wherein the first proper subset and the remaining proper subset are mutually exclusive and collectively exhaustive with respect to the set of data to be written.

15. The processing system of claim 14, wherein the operational instructions, when executed by the at least one processor, further cause the processing system to:

determine second observed compaction information based on compaction observed in the at least one storage device during the first future timeframe in response to the first future timeframe elapsing;

generate a second estimated compaction rate for a second future timeframe based on the observed compaction information, wherein the second future timeframe begins after the first future timeframe has elapsed;

generate a second updated ingest rate for the first future timeframe based on the updated ingest rate and the second estimated compaction rate;

select a second proper subset of a second set of data to be written to the at least one storage device based on the second updated ingest rate;

facilitate storage of the second proper subset in the at least one storage device during the second future timeframe; and

facilitate storage of a second remaining proper subset of the second set of data in the elastic buffer during the second future timeframe, wherein the second proper subset and the second remaining proper subset are mutually exclusive and collectively exhaustive with respect to the second set of data to be written.

16. The processing system of claim 14, wherein the at least one storage device includes at least one Append Only Memory Device (AOMD), and wherein determining the observed compaction information includes determining an amount of unreferenced space created as a result of overwrite and delete operations performed in the at least one AOMD during the observed timeframe.

17. The processing system of claim 14, wherein generating the updated ingest rate includes:

determining whether utilizing the current ingest rate in the first future timeframe is expected to exceed a speed of space reclamation indicated by the estimated compaction rate; and

reducing the current ingest rate to generate the updated ingest rate in response to determining that the current ingest rate in the first future timeframe is expected to exceed the speed of space reclamation indicated by the estimated compaction rate.

18. The processing system of claim 14, wherein generating the updated ingest rate includes:

determining a maximum ingest rate that does not exceed a speed of space reclamation indicated by the estimated compaction rate; and

setting the updated ingest rate equal to the maximum ingest rate.

19. The processing system of claim 14, wherein the at least one storage device includes a set of storage units, wherein the set of data to be stored in the at least one storage device includes a set of encoded slices for storage in the set of storage units, wherein a data segment was dispersed storage error encoded to produce the set of encoded slices, wherein determining the observed compaction information includes receiving compaction information from the set of storage units via a network, and wherein facilitating storage of the first proper subset includes transmitting at least one of the set of encoded slices to at least one of the set of storage units via the network.

20. A computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to;

determine observed compaction information based on compaction observed in at least one storage device during an observed timeframe;

generate an estimated compaction rate for a first future timeframe based on the observed compaction information;

generate an updated ingest rate for the first future timeframe based on a current ingest rate and the estimated compaction rate;

select a first proper subset of a set of data to be written to the at least one storage device based on the updated ingest rate;

facilitate storage of the first proper subset in the at least one storage device during the first future timeframe; and

facilitate storage of a remaining proper subset of the set of data in an elastic buffer during the first future timeframe, wherein the elastic buffer utilizes a memory that is different from the at least one storage device, and wherein the first proper subset and the remaining proper subset are mutually exclusive and collectively exhaustive with respect to the set of data to be written.