Management of latency and throughput in a cluster file system

• US 8,229,961 B2
• Filed: 05/05/2010
• Issued: 07/24/2012
• Est. Priority Date: 05/05/2010
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Whitehouse, Steven John
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US12/774,601
• Publication Number:
  US 20110276542A1
• Time in Patent Office: 811 Days
• Field of Search: 707600-831
• US Class Current: 707/791
• CPC Class Codes: G06F 16/1774 Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files

Abstract

Some embodiments of a system and a method to detect contention for resource in a cluster file system have been presented. For instance, a processing device may measure time spent performing actual operations by each of a set of nodes in a cluster file system when a respective node holds a lock on a resource and time spent performing overhead operations by the set of nodes without synchronization of clocks across the cluster file system. Then the processing device can determine latency and throughput of the cluster file system based on the time spent performing actual operations and the time spent performing overhead operations.

Claims

1. A method comprising:

measuring, by a computing system, time spent performing actual operations by a node in a cluster file system while the node holds a lock on a resource;

measuring, by the computing system, time spent performing overhead operations by the node without synchronization of clocks across the cluster file system; and

determining, by the computing system, latency and throughput of the cluster file system based on the time spent performing actual operations and the time spent performing overhead operations.

2. The method of claim 1, further comprising:

tracking the number of nodes in the cluster file system that are interested in the resource.

3. The method of claim 1, wherein measuring the time spent performing actual operations comprises:

starting a timer after a lock request from the node is granted and after initial cache filling has been completed; and

stopping the timer upon the node deciding to release the lock request, but before the lock request is actually released and before any overhead operations are executed.

4. The method of claim 1, further comprising:

increasing a time for which the node can keep a lock on the resource to adjust a ratio between the latency and throughput.

5. The method of claim 1, further comprising:

providing a first counter and a second counter, both the first and the second counters associated with the resource, wherein the first and the second counters can be updated regardless of a lock mode of the lock;

using the first counter to keep track of the number of nodes that are interested in the resource; and

using the second counter to measure the time spent performing actual operations and the time spent performing overhead operations.

6. The method of claim 1, further comprising:

converting a lock value block (LVB) of the cluster file system to a lock counter block (LCB) to count in both read lock mode and write lock mode.

7. The method of claim 1, further comprising:

determining the number of nodes that are interested in the resource and a total queue length of all lock request queues on all nodes of the plurality of nodes;

determining an average queue length based on the number of nodes interested in the resource and the total queue length; and

adjusting a time interval during which the node can hold the lock based on the average queue length.

8. An apparatus comprising:

a network interface to communicably couple to a plurality of nodes of a cluster file system; and

a processing device coupled to the network interface to;

measure time spent performing actual operations by a node in the cluster file system while the node holds a lock on a resource,

measure time spent performing overhead operations by the node without synchronization of clocks across the cluster file system, and

determine latency and throughput of the cluster file system based on the time spent performing actual operations and the time spent performing overhead operations.

9. The apparatus of claim 8, wherein the processing device tracks the number of nodes in the cluster file system that are interested in the resource.

10. The apparatus of claim 8, further comprising:

a timer, wherein the processing device starts the timer after a lock request from the node is granted and after initial cache filling has been completed, and stops the timer upon the node deciding to release the lock request, but before the lock request is actually released.

11. The apparatus of claim 8, wherein the processing device increases a time for which the node can keep a lock on the resource to adjust a ratio between the latency and throughput.

12. The apparatus of claim 8, further comprising:

a first counter; and

a second counter, wherein both the first and the second counters are associated with the resource and can be updated regardless of a lock mode of the lock, and wherein the processing device is also to;

use the first counter to keep track of the number of nodes that are interested in the resource, and

use the second counter to measure the time spent performing actual operations and the time spent performing overhead operations.

13. The apparatus of claim 8, wherein the processing device converts a lock value block (LVB) of the cluster file system to a lock counter block (LCB) to count in both read lock mode and write lock mode.

14. The apparatus of claim 8, wherein the processing device is also to:

determine the number of nodes that are interested in the resource and a total queue length of all lock request queues on all nodes of the plurality of nodes,

determine an average queue length based on the number of nodes interested in the resource and the total queue length, and

adjust a time interval during which the node can hold the lock based on the average queue length.

15. A non-transitory computer-readable storage medium embodying instructions that, when executed by a processing device, will cause the processing device to perform a method comprising:

measuring, by the processing device, time spent performing actual operations by a node in a cluster file system while the node holds a lock on a resource;

measuring, by the processing device, time spent performing overhead operations by the node without synchronization of clocks across the cluster file system; and

determining, by the processing device, latency and throughput of the cluster file system based on the time spent performing actual operations and the time spent performing overhead operations.

16. The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:

tracking the number of nodes in the cluster file system that are interested in the resource.

17. The non-transitory computer-readable storage medium of claim 15, wherein measuring the time spent performing actual operations comprises:

starting a timer after a lock request from the node is granted and after initial cache filling has been completed; and

stopping the timer upon the node deciding to release the lock request, but before the lock request is actually released and before any overhead operations are executed.

18. The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:

increasing a time for which the node can keep a lock on the resource to adjust a ratio between the latency and throughput.

19. The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:

providing a first counter and a second counter, both the first and the second counters associated with the resource, wherein the first and the second counters can be updated regardless of a lock mode of the lock;

using the first counter to keep track of the number of nodes that are interested in the resource; and

using the second counter to measure the time spent performing actual operations and the time spent performing overhead operations.

20. The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:

converting a lock value block (LVB) of the cluster file system to a lock counter block (LCB) to count in both read lock mode and write lock mode.

21. The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:

determining the number of nodes that are interested in the resource and a total queue length of all lock request queues on all nodes of the plurality of nodes;

determining an average queue length based on the number of nodes interested in the resource and the total queue length; and

adjusting a time interval during which the node can hold the lock based on the average queue length.