Distributed resource contention detection

• US 9,389,926 B2
• Filed: 05/05/2010
• Issued: 07/12/2016
• Est. Priority Date: 05/05/2010
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Whitehouse, Steven John
• Primary Examiner(s): VU, BAI DUC
• Application Number:
  US12/774,591
• Publication Number:
  US 20110276690A1
• Time in Patent Office: 2,260 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1774 Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 9/526 Mutual exclusion algorithms

Abstract

Some embodiments of a system and a method to detect contention for resource in a cluster file system have been presented. For instance, a processing device executable on a node in a cluster file system may put lock requests for a resource from the node into a local lock request queue, where multiple nodes in the cluster file system share the resource. Then the lock requests for the resource from this node and lock requests for the resource from other nodes may be tracked to measure congestion on the resource.

Claims

1. A method comprising:

tracking, by a processing device, lock requests for a single resource shared by a plurality of nodes of a cluster file system, wherein each node of the cluster file system has its own respective local queue to track its own lock requests for the single resource;

determining, by the processing device, a measure of congestion for the single resource, wherein the measure of congestion is in view of the respective local queues of nodes requesting a lock on the single resource; and

adjusting a relative lock hold time for a node of the cluster file system in view of the measure of congestion.

2. The method of claim 1, further comprising:

providing a first global counter and a second global counter, both the first and the second global counters associated with the single resource, wherein tracking the lock requests for the single resource comprises, each time a node is granted a lock on the single resource;

incrementing the first global counter before releasing a lock request from the node; and

incrementing the second global counter by a length of the local lock request queue when the lock is released.

3. The method of claim 1, wherein a task running on a node is allowed to read a counter associated with the single resource when the task receives a lock grant for the single resource.

4. The method of claim 1, further comprising:

adding to a lock value block (LVB) of the cluster file system a flag that indicates that the LVB is to be used as a lock counter block (LCB) comprising a plurality of counters.

5. The method of claim 1, further comprising:

determining, by the processing device, an average queue length of the respective local queues of nodes interested in the single resource; and

determining, by the processing device, a congestion factor for a node in view of the length of the respective local queue of the node and the average queue length.

6. The method of claim 1, further comprising:

compensating, by the processing device, for unequal workloads across the plurality of nodes of the cluster file system.

7. An apparatus comprising:

a network interface to communicably couple a plurality of nodes of a cluster file system, wherein each node of the cluster file system has its own respective local queue to track its own lock requests for a single resource shared by the plurality of nodes; and

a processing device operatively coupled to the network interface, to;

track lock requests for the single resource,

determine a measure of congestion for the single resource, wherein the measure of congestion is in view of the respective local queues of nodes requesting a lock on the single resource; and

adjust a relative lock hold time for a node of the cluster file system in view of the measure of congestion.

8. The apparatus of claim 7, further comprising:

a first global counter associated with the single resource; and

a second global counter associated with the single resource, wherein each time a node is granted a lock on the single resource, the first global counter is incremented before releasing a lock request from the node, and the second global counter is incremented by a length of the lock request queue.

9. The apparatus of claim 7, further comprising:

a plurality of counters to track the lock requests for the single resource, wherein a task running on a node is allowed to read a counter associated with the single resource when the task receives a lock grant for the single resource.

10. The apparatus of claim 7, further comprising:

a lock value block (LVB) comprising a plurality of counters; and

a flag to indicate that the LVB is to be used as a lock counter block (LCB);

wherein the plurality of counters of the LVB are updatable by tasks running on the plurality of nodes in both read lock mode and write lock mode.

11. The apparatus of claim 7, wherein the processing device is further to:

determine a number of nodes interested in the single resource and a total queue length of all respective local queues on all nodes of the plurality of nodes,

determine an average queue length of the respective local queues of nodes interested in the single resource, and

determine a congestion factor for a node in view of the length of the respective local queue of the node and the average queue length.

12. The apparatus of claim 7, wherein the processing device is further to compensate for unequal workloads across the plurality of nodes of the cluster file system.

13. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:

track, by the processing device, lock requests for a single resource shared by a plurality of nodes of a cluster file system, wherein each node of the cluster file system has its own respective local queue to track its own lock requests for the single resource;

determine, by the processing device, a measure of congestion for the single resource, wherein the measure of congestion is in view of the respective local queues of nodes requesting a lock on the single resource; and

adjust a relative lock hold time for a node of the cluster file system in view of the measure of congestion.

14. The non-transitory computer-readable storage medium of claim 13, the processing device further to provide a first global counter and a second global counter;

wherein both the first and the second global counters are associated with the single resource; and

wherein to track the lock requests for the single resource, the processing device is to, each time a node is granted a lock on the single resource;

increment the first global counter before releasing a lock request from the node, and

increment the second global counter by a length of the local lock request queue when the lock is released.

15. The non-transitory computer-readable storage medium of claim 13, wherein a task running on a node is allowed to read a counter associated with the single resource when the task receives a lock grant for the single resource.

16. The non-transitory computer-readable storage medium of claim 13, the processing device further to:

add to a lock value block (LVB) of the cluster file system a flag that indicates that the LVB is to be used as a lock counter block (LCB) comprising a plurality of counters.

17. The non-transitory computer-readable storage medium of claim 13, the processing device further to:

determine, by the processing device, an average queue length of the respective local queues of nodes interested in the single resource; and

determine, by the processing device, a congestion factor for a node in view of the length of the respective local queue of the node and the average queue length.

18. The non-transitory computer-readable storage medium of claim 13, the processing device further to:

compensate for unequal workloads across the plurality of nodes of the cluster file system.