Congestion mitigation in a distributed storage system

• US 9,733,834 B1
• Filed: 02/11/2016
• Issued: 08/15/2017
• Est. Priority Date: 01/28/2016
• Status: Active Grant

• Current Assignee: Weka.IO Ltd.
• Original Assignee: Weka.IO Ltd.
• Inventors: Ben Dayan, Maor, Filiba, Tomer, Palmon, Omri, Zvibel, Liran, Arditti, Kanael
• Primary Examiner(s): Rojas, Midys
• Assistant Examiner(s): Doan, Khoa D
• Application Number:
  US15/041,123
• Publication Number:
  US 20170220260A1
• Time in Patent Office: 551 Days
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0673 Single storage device
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: H04L 43/0882 Utilisation of link capacity
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 47/12 Avoiding congestion; Recovering from congestion
• CPC Class Codes: H04L 47/6275 based on priority
• CPC Class Codes: H04L 47/70 Admission control; Resource allocation
• CPC Class Codes: H04L 47/83 : based on usage prediction
• CPC Class Codes: H04L 49/90 : Buffering arrangements
• CPC Class Codes: H04L 67/01 : Protocols
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A system comprises a plurality of computing devices that are communicatively coupled via a network and have a file system distributed among them, and comprises one or more file system request buffers residing on one or more of the plurality of computing devices. File system choking management circuitry that resides on one or more of the plurality of computing devices is operable to separately control: a first rate at which a first type of file system requests (e.g., one of data requests, data read requests, data write requests, metadata requests, metadata read requests, and metadata write requests) are fetched from the one or more buffers, and a second rate at which a second type of file system requests (e.g., another of data requests, data read requests, data write requests, metadata requests, metadata read requests, and metadata write requests) are fetched from the one or more buffers.

Claims

1. A system comprising:

a plurality of computing devices that are communicatively coupled via one or more network links and that have a file system distributed among them;

one or more file system request buffers that reside on one or more of said plurality of computing devices;

file system choking management circuitry that resides on one or more of said plurality of computing devices and is operable to separately control;

a first rate at which a first type of file system requests are fetched from said one or more buffers, said file system choking management circuitry being operable to control said first rate according to an adjustment of a first batch size setting; and

a second rate at which a second type of file system requests are fetched from said one or more buffers;

wherein said first type of file system requests comprise requests to read data from said file system and requests to write data to said file system; and

said second type of file system requests comprise requests to read metadata from said file system and requests to write metadata to said file system; and

wherein said file system choking management circuitry is operable to adjust;

a data batch timing setting;

a data batch size setting;

a metadata batch timing setting; and

a metadata batch size setting.

2. The system of claim 1, wherein said file system choking management circuitry is operable to control said first rate according to an adjustment of a first batch timing setting.

3. The system of claim 2, wherein said file system choking management circuitry is operable to adjust a second batch timing setting and a second batch size setting.

4. The system of claim 1, wherein:

said first type of file system requests comprise requests to read data from said file system and requests to read metadata from said file system;

and said second type of file system requests comprise requests to write data to said file system and requests to write metadata to said file system.

5. The system of claim 4, wherein said file system choking management circuitry is operable to adjust:

a read batch timing setting;

a read batch size setting;

a write batch timing setting; and

a write batch size setting.

6. The system of claim 1, wherein said file system choking management circuitry is operable to control, separately from said first rate and said second rate, a third rate at which a third type of file system requests are fetched from said one or more buffers.

7. The system of claim 6, wherein said file system choking management circuitry is operable to adjust:

a data read batch timing setting; and

a data read batch size setting;

a data write batch timing setting; and

a data write batch size setting;

a metadata batch timing setting; and

a metadata batch size setting.

8. The system of claim 1, wherein said file system choking management circuitry is operable to control according to current load on one or more resources of said plurality of computing devices and according to predicted load on one or more resources of said plurality of computing devices.

9. The system of claim 8, wherein:

said one or more resources comprise a storage device; and

a load on one or more resources of said plurality of computing devices is based on a depth of a buffer of said storage device.

10. The system of claim 1, wherein:

said file system choking management circuitry is operable to calculate a choking level based on current and/or predicted load on one or more resources of said plurality of computing devices; and

said control of said first rate and of said second rate is based on said choking level.

11. The system of claim 10, wherein said calculation of said choking level comprises a mapping, according to a first determined function, of a plurality of resource load values for resources of said plurality of computing devices to a plurality of congestion contribution values.

12. The system of claim 11, wherein said plurality of resource load values comprise two or more of:

a network load value;

a processing core load value;

a memory load value; and

a storage device load value.

13. The system of claim 12, wherein said storage device load value is determined based on a depth of a buffer of a storage device of said plurality of computing devices.

14. The system of claim 11, wherein said plurality of resource load values comprises a storage device read load value and a storage device write load value.

15. The system of claim 11 wherein:

said resources of said plurality of computing devices comprise a storage device which stores data and/or metadata of said file system; and

said first determined function is determined based on characteristics of said storage device.

16. The system of claim 15, wherein said characteristics of said storage device are determined from one or more of:

a manufacturer of said storage device;

a manufacturer of a component of said storage device;

a model of said storage device;

a model of a component of said storage device;

a serial number of said storage device; and

a serial number of a component of said storage device.

17. The system of claim 15, wherein said characteristics of said storage device comprise input/output operations per second (IOPS).

18. The system of claim 11, wherein said calculation of said choking level comprises a combining of said plurality of congestion contribution values according to a second determined function.

19. The system of claim 1, wherein said file system choking management circuitry is configured to provide an application programming interface via which one or more settings used by said file system choking management circuitry for said separate control said first rate and said second rate are configurable during runtime of said file system choking management circuitry.

20. The system of claim 1, wherein said file system choking management circuitry is operable to:

determine characteristics of hardware of said plurality of computing devices;

perform an initial configuration one or more settings used by said file system choking management circuitry for said separate control said first rate and said second rate based on said characteristics of said hardware; and

adapt said one or more settings as file system requests are serviced and/or queued for servicing by said file system.

21. One or more non-transitory machine-readable storage media having code stored thereon, wherein:

said code, when executed by one or more of a plurality of computing devices that are communicatively coupled via one or more network links and that have a file system distributed among them, configures said one or more of said plurality of computing devices to implement one or more file system request buffers;

said code, when executed by said one or more of said plurality of computing devices, configures said one or more of said plurality of computing devices to implement a file system choking management process; and

said file system choking management process is configured to separately control;

a first rate at which a first type of file system requests are fetched from said one or more buffers, said file system choking management circuitry being operable to control said first rate according to an adjustment of a first batch size setting; and

a second rate at which a second type of file system requests are fetched from said one or more buffers;

wherein said first type of file system requests comprise requests to read data from said file system and requests to write data to said file system; and

said second type of file system requests comprise requests to read metadata from said file system and requests to write metadata to said file system; and

wherein said file system choking management circuitry is operable to adjust;

a data batch timing setting;

a data batch size setting;

a metadata batch timing setting; and

a metadata batch size setting.