Dynamic ingestion throttling of data log

• US 9,747,222 B1
• Filed: 03/31/2016
• Issued: 08/29/2017
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Armangau, Philippe, Caram, Bruce E., Seibel, Christopher A.
• Primary Examiner(s): Chappell, Daniel C
• Application Number:
  US15/086,644
• Time in Patent Office: 516 Days
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 12/0804 with main memory updating G06F12/0806 takes precedence
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 12/124 being minimized, e.g. non MRU
• CPC Class Codes: G06F 13/1642 with request queuing
• CPC Class Codes: G06F 13/4234 being a memory bus
• CPC Class Codes: G06F 2212/284 being distributed
• CPC Class Codes: G06F 2212/312 In storage controller
• CPC Class Codes: G06F 2212/314 In storage network, e.g. network attached cache
• CPC Class Codes: G06F 2212/502 using adaptive policy

Abstract

A technique for controlling acceptance of host application data into a data log in a data storage system includes selectively accepting or refusing newly arriving host data into the data log based on a comparison between an oldest entry in the data log and an age threshold. The age threshold is dynamically updated based on system heuristics. As long as the oldest log entry is younger than the age threshold, the data log continues to accept newly arriving host application data, acknowledging IO requests to host applications as the data specified in those requests is entered into the log. However, when the oldest log entry is older than the age threshold, new log entries are temporarily refused entry into the data log. Instead, newly arriving data are placed in a pending list, where they are kept until the data log is again accepting new log entries.

Claims

1. A method of dynamically controlling acceptance of host application data into a data log in a data storage system, the method comprising:

generating a throttle signal, the throttle signal having (i) a first value when an oldest log entry in the data log is younger than an age threshold and (ii) a second value when the oldest log entry in the data log is older than the age threshold, the data storage system dynamically updating the age threshold based on a set of heuristics pertaining to the data storage system;

in response to the throttle signal having the first value and the data storage system receiving a first IO request from a host application, (i) storing data specified in the first IO request in a first log entry of the data log and (ii) acknowledging completion of the first IO (Input/Output) request to the host application;

in response to the throttle signal having the second value and the data storage system receiving a second IO request from the host application, storing data specified in the second IO request in a pending list; and

after storing the data specified in the second IO request in the pending list and in response to the throttle signal then assuming the first value, (i) storing the data specified in the second IO request in a second log entry of the data log and (ii) acknowledging completion of the second IO request to the host application.

2. The method of claim 1, further comprising:

with the throttle signal having the second value, flushing multiple log entries from the data log, the log entries including the oldest log entry, such that a log entry younger than the oldest log entry becomes a new oldest log entry; and

changing the throttle signal from the second value to the first value in response to the new oldest log entry in the data log being older than the age threshold.

3. The method of claim 2, wherein the data storage system persistently stores the data log but non-persistently stores the pending list.

4. The method of claim 3, wherein the set of heuristics includes a measure of fullness of the data log, and wherein the method further comprises:

decreasing the age threshold in response to detecting that the measure of fullness of the data log has increased; and

increasing the age threshold in response to detecting that the measure of fullness of the data log has decreased.

5. The method of claim 3, wherein the set of heuristics includes an age of a current oldest entry in the data log, and wherein the method further comprises:

decreasing the age threshold in response to detecting that the age of the current oldest entry in the data log has increased; and

increasing the age threshold in response to detecting that the age of the current oldest entry in the data log has decreased.

6. The method of claim 3, further comprising, when storing data in log entries of the data log, also storing metadata describing those log entries in a volatile data cache, wherein the set of heuristics includes a measure of fullness of the volatile data cache, and wherein the method further comprises:

decreasing the age threshold in response to detecting that the measure of fullness of the volatile data cache has increased; and

increasing the age threshold in response to detecting that the measure of fullness of the volatile data cache has decreased.

7. The method of claim 3, further comprising:

when storing data in log entries of the data log, also storing metadata describing those log entries in a volatile data cache,

wherein the set of heuristics includes (i) a measure of fullness of the data log, (ii) an age of a current oldest entry in the data log, and (iii) a measure of fullness of the volatile data cache,

and wherein the method further comprises;

decreasing the age threshold in response to detecting an increase in any of (i) the measure of fullness of the data log, (ii) the age of the current oldest entry in the data log, and (iii) the measure of fullness of the volatile data cache; and

increasing the age threshold in response to detecting a decrease in any of (i) the measure of fullness of the data log, (ii) the age the current oldest entry in the data log, and (iii) the measure of fullness of the volatile data cache.

8. The method of claim 7, wherein dynamically updating the age threshold based on a set of heuristics includes regenerating the age threshold on a periodic basis.

9. The method of claim 8, wherein the method further comprises decreasing the age threshold by at least hundreds of milliseconds in response to the data log becoming full.

10. The method of claim 9, wherein the method further comprises decreasing the age threshold by at least hundreds of milliseconds in response to the data cache becoming full.

11. The method of claim 10, wherein the data storage system includes a data log manager that manages the data log, and wherein the method further comprises:

backing a set of log entries in the data log in a cache structure, the cache structure configured to flush each of the set of log entries to non-volatile storage either (i) to a first storage location or (ii) to a second storage location;

receiving, by the data log manager, an alert from the cache structure, the alert indicating that at least one of the set of log entries has been flushed to the first storage location; and

in response to receiving the alert, decreasing the age threshold by at least hundreds of milliseconds.

12. A data storage system, comprising control circuitry that includes a set of processing units coupled to memory, the control circuitry constructed and arranged to:

generate a throttle signal, the throttle signal having (i) a first value when an oldest log entry in the data log is younger than an age threshold and (ii) a second value when the oldest log entry in the data log is older than the age threshold, the data storage system configured to dynamically update the age threshold based on a set of heuristics pertaining to the data storage system;

in response to the throttle signal having the first value and the data storage system receiving a first IO request from a host application, (i) store data specified in the first IO (Input/Output) request in a first log entry of the data log and (ii) acknowledge completion of the first IO request to the host application;

in response to the throttle signal having the second value and the data storage system receiving a second IO request from the host application, store data specified in the second IO request in a pending list; and

after storing the data specified in the second IO request in the pending list and in response to the throttle signal then assuming the first value, (i) store the data specified in the second IO request in a second log entry of the data log and (ii) acknowledge completion of the second IO request to the host application.

13. A computer program product including a set of non-transitory, computer-readable media having instructions which, when executed by control circuitry of a data storage system, cause the control circuitry to perform a method for dynamically controlling acceptance of host application data into a data log in a data storage system, the method comprising:

generating a throttle signal, the throttle signal having (i) a first value when an oldest log entry in the data log is younger than an age threshold and (ii) a second value when the oldest log entry in the data log is older than the age threshold, the data storage system dynamically updating the age threshold based on a set of heuristics pertaining to the data storage system;

in response to the throttle signal having the first value and the data storage system receiving a first IO request from a host application, (i) storing data specified in the first IO request in a first log entry of the data log and (ii) acknowledging completion of the first IO (Input/Output) request to the host application;

in response to the throttle signal having the second value and the data storage system receiving a second IO request from the host application, storing data specified in the second IO request in a pending list; and

after storing the data specified in the second IO request in the pending list and in response to the throttle signal then assuming the first value, (i) storing the data specified in the second IO request in a second log entry of the data log and (ii) acknowledging completion of the second IO request to the host application.

14. The computer program product of claim 13, wherein the method further comprises:

with the throttle signal having the second value, flushing multiple log entries from the data log, the log entries including the oldest log entry, such that a log entry younger than the oldest log entry becomes a new oldest log entry; and

changing the throttle signal from the second value to the first value in response to the new oldest log entry in the data log being older than the age threshold.

15. The computer program product of claim 14, wherein the set of heuristics includes a measure of fullness of the data log, and wherein the method further comprises:

decreasing the age threshold in response to detecting that the measure of fullness of the data log has increased; and

increasing the age threshold in response to detecting that the measure of fullness of the data log has decreased.

16. The computer program product of claim 14, wherein the method further comprises:

when storing data in log entries of the data log, also storing metadata describing those log entries in a volatile data cache,

wherein the set of heuristics includes (i) a measure of fullness of the data log, (ii) an age of a current oldest entry in the data log, and (iii) a measure of fullness of the volatile data cache,

and wherein the method further comprises;

decreasing the age threshold in response to detecting an increase in any of (i) the measure of fullness of the data log, (ii) the age of the current oldest entry in the data log, and (iii) the measure of fullness of the volatile data cache; and

increasing the age threshold in response to detecting a decrease in any of (i) the measure of fullness of the data log, (ii) the age the current oldest entry in the data log, and (iii) the measure of fullness of the volatile data cache.

17. The computer program product of claim 16, wherein dynamically updating the age threshold based on a set of heuristics includes regenerating the age threshold on a periodic basis.

18. The computer program product of claim 17, wherein the method further comprises decreasing the age threshold by at least hundreds of milliseconds in response to the data log becoming full.

19. The computer program product of claim 18, wherein the method further comprises decreasing the age threshold by at least hundreds of milliseconds in response to the data cache becoming full.

20. The computer program product of claim 19, wherein the data storage system includes a data log manager that manages the data log, and wherein the method further comprises:

backing a set of log entries in the data log in a cache structure, the cache structure configured to flush each of the set of log entries to non-volatile storage either (i) to a first storage location or (ii) to a second storage location;

receiving, by the data log manager, an alert from the cache structure, the alert indicating that at least one of the set of log entries has been flushed to the first storage location; and

in response to receiving the alert, decreasing the age threshold by at least hundreds of milliseconds.