Quality of service management in a distributed storage system

• US 10,133,516 B2
• Filed: 05/19/2017
• Issued: 11/20/2018
• Est. Priority Date: 01/28/2016
• Status: Active Grant

• Current Assignee: Weka.IO Ltd.
• Original Assignee: Weka.IO Ltd.
• Inventors: Ben Dayan, Maor, Palmon, Omri, Zvibel, Liran, Arditti, Kanael, Filiba, Tomer
• Primary Examiner(s): Boccio, Vincent F
• Application Number:
  US15/599,737
• Publication Number:
  US 20170255642A1
• Time in Patent Office: 550 Days
• Field of Search: 707827, 709213, 709215
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 47/11 Identifying congestion
• CPC Class Codes: H04L 47/12 Avoiding congestion; Recovering from congestion
• CPC Class Codes: H04L 47/24 Traffic characterised by specific attributes, e.g. priority or QoS
• CPC Class Codes: H04L 47/30 in combination with information about buffer occupancy at either end or at transit nodes
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

One or more computing devices may comprise congestion management circuitry, one or more client file system request buffers, and DESS interface circuitry. The one or more client file system request buffers is/are operable to queue first client file system requests of a first priority level and second client file system requests of a second priority level, where the first priority level is higher priority than the second priority level. The DESS interface circuitry is operable to determine a choking level according to the load on a plurality of DESS resources. Individual load values of the DESS resources are mapped to a composite load value using a first function. The composite load value is mapped to a congestion contribution using a second function. And, the congestion contribution is mapped to a choking level using a third function.

Claims

1. A system comprising:

congestion management circuitry operable to determine an amount of congestion in a distributed electronic storage system (DESS);

one or more client file system request buffers operable to queue first client file system requests of a first priority level and second client file system requests of a second priority level, wherein the first priority level is higher priority than the second priority level; and

DESS interface circuitry operable to determine a choking level according to a load on a plurality of DESS resources, wherein individual load values of the plurality of DESS resources are mapped to a composite load value using a first function, the composite load value is mapped to a congestion contribution using a second function, and the congestion contribution is mapped to a choking level using a third function, wherein the DESS interface circuitry is operable to dynamically update the choking level according to a normalization of each file system request and a level of file system requests previously queued in the one or more client file system request buffers.

2. The system of claim 1, comprising queueing circuitry operable to:

receive a particular client file system request; and

determine whether the particular client file system request is one of the first client file system requests or one of the second client file system requests based on a mount point to which the particular client file system request is directed.

3. The system of claim 1, comprising queueing circuitry operable to:

receive a particular client file system request;

queue the particular client file system request in a first of the one or more buffers if the particular client file system request is directed at a first mount point; and

queue the particular client file system request in a second of the one or more buffers if the particular client file system request is directed at a second mount point.

4. The system of claim 1, wherein the first client file system requests are directed to a file system mounted at a first mount point and the second client file system requests are directed to a file system mounted at a second mount point.

5. The system of claim 4, wherein the file system mounted at the first mount point and the file system mounted at the second mount point are the same file system.

6. The system of claim 1, wherein the DESS interface circuitry is operable to fetch client file system requests from the one or more client file system request buffers in accordance with a round-robin schedule and one or more choking levels.

7. The system of claim 6, wherein the round-robin schedule is such that, in each round of the round-robin:

at least one client file system request directed to a file system mounted at a first mount point is fetched; and

at least one client file system request directed to a file system mounted at a second mount point is fetched.

8. The system of claim 1, comprising DESS priority management circuitry operable to determine, based on characteristics of a node of the DESS, where to mount a file system.

9. The system of claim 8, wherein the characteristics of the node comprise a client process that runs on the node.

10. The system of claim 1, wherein the DESS interface circuitry is operable to:

fetch both the first client file system requests and the second client file system requests when the amount of congestion in the DESS is below a threshold; and

fetch the first client file system requests but not the second client file system requests when the amount of congestion in the DESS is above the threshold.

11. One or more non-transitory machine-readable storage medium having code stored thereon, that, when executed by one or more of a plurality of computing devices of a distributed electronic storage system (DESS), configures the one or more computing devices to comprise:

congestion management circuitry operable to determine an amount of congestion in a distributed electronic storage system (DESS);

one or more client file system request buffers operable to queue first client file system requests of a first priority level and second client file system requests of a second priority level, wherein the first priority level is higher priority than the second priority level; and

DESS interface circuitry operable to determine a choking level according to a load on a plurality of DESS resources, wherein individual load values of the plurality of DESS resources are mapped to a composite load value using a first function, the composite load value is mapped to a congestion contribution using a second function, and the congestion contribution is mapped to a choking level using a third function, wherein the DESS interface circuitry is operable to dynamically update the choking level according to a normalization of each file system request and a level of file system requests previously queued in the one or more client file system request buffers.

12. The one or more non-transitory machine-readable storage medium of claim 11, wherein the code, when executed by the one or more computing devices, configures the one or more computing devices to comprise queueing circuitry operable to:

receive a particular client file system request; and

determine whether the particular client file system request is one of the first client file system requests or one of the second client file system requests based on a mount point to which the particular client file system request is directed.

13. The one or more non-transitory machine-readable storage medium of claim 11, wherein the code, when executed by the one or more computing devices, configures the one or more computing devices to comprise queueing circuitry operable to:

receive a particular client file system request;

queue the particular client file system request in a first of the one or more buffers if the particular client file system request is directed at a first mount point; and

queue the particular client file system request in a second of the one or more buffers if the particular client file system request is directed at a second mount point.

14. The one or more non-transitory machine-readable storage medium of claim 11, wherein the first client file system requests are directed to a file system mounted at a first mount point and the second client file system requests are directed to a file system mounted at a second mount point.

15. The one or more non-transitory machine-readable storage medium of claim 14, wherein the file system mounted at the first mount point and the file system mounted at the second mount point are the same file system.

16. The one or more non-transitory machine-readable storage medium of claim 11, wherein the DESS interface circuitry is operable to fetch client file system requests from the one or more client file system request buffers in accordance with a round-robin schedule and one or more choking levels.

17. The one or more non-transitory machine-readable storage medium of claim 16, wherein the round-robin schedule is such that, in each round of the round-robin:

at least one client file system request directed to a file system mounted at a first mount point is fetched; and

at least one client file system request directed to a file system mounted at a second mount point is fetched.

18. The one or more non-transitory machine-readable storage medium of claim 11, wherein the code, when executed by the one or more computing devices, configures the one or more computing devices to comprise DESS priority management circuitry operable to determine, based on characteristics of a node of the DESS, where to mount a file system.

19. The one or more non-transitory machine-readable storage medium of claim 18, wherein the characteristics of the node comprise a client process that runs on the node.

20. The one or more non-transitory machine-readable storage medium of claim 11, wherein the DESS interface circuitry is operable to:

fetch both the first client file system requests and the second client file system requests when the amount of congestion in the DESS is below a threshold; and

fetch the first client file system requests but not the second client file system requests when the amount of congestion in the DESS is above the threshold.