Managing opportunistic locks in a distributed file system

• US 10,169,367 B2
• Filed: 06/06/2014
• Issued: 01/01/2019
• Est. Priority Date: 06/06/2014
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Lin, Yun, Taylor, John Richard
• Primary Examiner(s): Adams, Charles D
• Application Number:
  US14/298,496
• Publication Number:
  US 20150356110A1
• Time in Patent Office: 1,670 Days
• Field of Search: None
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1774 Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

The disclosed embodiments disclose techniques for managing opportunistic locks in a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in the cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a client sending a request to access a file from a cloud controller also requests an opportunistic lock for the file. If there are no other outstanding client accesses on the other cloud controllers that preclude the requested opportunistic lock, the cloud controller grants the client file access and the opportunistic lock.

Claims

1. A computer-implemented method for managing distributed opportunistic locks in a distributed filesystem, the method comprising:

collectively managing data in the distributed file system using 2 or more cloud controllers, wherein collectively managing the data comprises storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems;

receiving at a first cloud controller a request from a first client to access a file in the distributed filesystem, wherein the first client requests an opportunistic lock for the file from the first cloud controller, wherein the requested opportunistic lock requests shared read/write access permissions that are not exclusive to other client accesses, wherein the first cloud controller determines an owning cloud controller among the two or more cloud controllers that has been assigned to manage opportunistic locks for the file, wherein the owning cloud controller is distinct from the first cloud controller, wherein the owning cloud controller tracks multiple clients that have been granted opportunistic locks for the file via other cloud controllers that are distinct from the first cloud controller, wherein every cloud controller with clients presently accessing the file registers with the owning cloud controller to receive notifications of any updates to the file;

initiating via the owning cloud controller a sharing access check that determines that there are outstanding opportunistic locks that are presently being held by one or more other clients of the distributed filesystem that were granted via other distinct cloud controllers for the distributed filesystem and that at least one or more of the outstanding opportunistic locks are write locks, wherein the owning cloud controller sends a distributed oplock break request to the cloud controllers that manage granted outstanding opportunistic locks for the file, wherein cloud controllers receiving the distributed oplock break request send an opportunistic lock break notification to their respective clients that are caching the file to ensure that there are no outstanding unflushed modifications to the file, wherein upon receiving the opportunistic lock break notification said clients flush all buffered modifications for the file to their respective cloud controllers, wherein information describing the flushed modifications for the file is synchronized to the cloud storage systems and to all of the other cloud controllers that have registered to receive notifications of updates for the file; and

upon determining that none of the other outstanding client opportunistic locks for the file that remain granted via the other cloud controllers of the distributed filesystem are exclusive locks that conflict with the requested opportunistic lock and determining that all distributed modifications to the file have been synchronized via the distributed oplock break request to ensure that no client receives obsolete data for the file, granting the first client the requested opportunistic lock for the file, wherein multiple clients simultaneously hold opportunistic locks granting shared write access for the file via distinct cloud controllers.

2. The computer-implemented method of claim 1, wherein the first client uses the opportunistic lock to cache the file locally and perform buffered reads and/or writes for the file, thereby reducing network latencybetween the first client and the first cloud controller and improving the first client'"'"'s file access performance.

3. The computer-implemented method of claim 2,

wherein the request includes a requested file access type and a requested sharing mode;

wherein determining whether other outstanding client accesses preclude the requested opportunistic lock comprises contacting the owning cloud controller to determine whether the requested file access type and the requested sharing mode are compatible with any other current file handles that have been granted for the file.

4. The computer-implemented method of claim 3,

wherein upon determining that a second, remote client that is associated with a second cloud controller that is distinct from the first cloud controller currently holds an open file handle for the file, the owning cloud controller performs the sharing access check for the request;

wherein the sharing access check determines that the requested file access type and the requested sharing mode do not conflict with the open file handle and that the first client and the second client can simultaneously access the file;

wherein the owning cloud controller indicates to the first cloud controller that there are no sharing violations that preclude the requested opportunistic lock; and

wherein allowing multiple distributed clients to simultaneously perform buffered file operations enhances file access performance.

5. The computer-implemented method of claim 3,

wherein upon determining that a second, remote client that is associated with a second cloud controller that is distinct from the first cloud controller currently holds an open write file handle for the file, the owning cloud controller requests that the second cloud controller send an opportunistic lock break notification to the second client;

and

wherein upon receiving the opportunistic lock break notification, the second client is configured to flush all buffered modifications for the file to the second cloud controller.

6. The computer-implemented method of claim 5,

wherein an application executing on the second client that originally requested the open write file handle is still accessing the file;

wherein upon flushing all buffered modifications for the file in response to the opportunistic lock break notification the application continues using the open write file handle to write to the file;

wherein if, subsequent to initiating the sending of the opportunistic lock break notification to the second client, the owning cloud controller determined a sharing access check for the request that determines that the open write file handle conflicts with the request, the owning cloud controller would indicate to the first cloud controller that a sharing violation precludes the requested opportunistic lock.

7. The computer-implemented method of claim 6, wherein subsequent to initiating the sending of the opportunistic lock break notification the owning cloud controller at least one of disables or reducesthe opportunistic lock level for the file for the second client to ensure that subsequent writes to the file are not buffered on the second client.

8. The computer-implemented method of claim 5,

wherein an application executing on the second client that originally requested the open write file handle has completed accessing the file and closed the file, but the second client is still caching the open write file handle;

wherein upon flushing all buffered modifications for the file the second client closes the cached open write file handle and thus closes the open write file handle on the second cloud controller; and

wherein, subsequent to sending the opportunistic lock break notification to the second client via the second cloud controller, the owning cloud controller performs the sharing access check for the request that determines that there are no other client file handles for the file that conflict with the request and indicates to the first cloud controller that there are no sharing violations that preclude the requested opportunistic lock.

9. The computer-implemented method of claim 8, wherein if no sharing violations preclude granting the requested opportunistic lock, contacting the owning cloud controller further comprises transferring ownership of the file to the first cloud controller to enable the first cloud controller to manage opportunistic lock requests for the file that are initiated by other clients of the distributed filesystem via other cloud controllers.

10. The computer-implemented method of claim 8, wherein the method further comprises:

determining that the file needs strong read-after-write consistency;

sending a registration notification from the first cloud controller to the other cloud controllers of the distributed filesystem, wherein the registration notification requests that the first cloud controller receive immediate notification of any changes to the file;

wherein upon subsequently receiving a set of modifications to the file from the first client, sending a change notification to two or more cloud controllers that have registered to receive change notifications for the file; and

subsequently sending an incremental metadata snapshot that includes metadata updates associated with the set of modifications to the file to all of the other cloud controllers for the distributed filesystem.

11. The computer-implemented method of claim 10, wherein each of the two or more cloud controllers that have registered to receive change notifications for the file, upon receiving the change notification for the file from the first cloud controller:

sends an opportunistic lock break notification to local clients that are reading a cached version of the file to guarantee strong read-after-write consistency for the file;

upon receiving a subsequent read open request for the file from a local client, sends a synchronization update request to the first cloud controller to retrieve the flushed modifications that were written to the file by the first client; and

provides the modified file to the local client.

12. The computer-implemented method of claim 2,

wherein each specific cloud controller of the distributed filesystem includes a local fileserver component that tracks the set of opportunistic locks that have been granted to its local clients by the owning cloud controller andleveraging those granted looks when managing opportunistic lock requests and interactions between local clients that access the distributed filesystem via that respective specific cloud controller and escalating requests that cannot be handled locally by the respective cloud controller to the owning cloud controller;

and

wherein the cloud controllers of the distributed filesystem include a second fileserver component that hierarchically manages opportunistic lock requests and interactions between the owning cloud controller for the file and the other cloud controllers of the distributed filesystem to enable distributed oplock capabilities that enable file caching for the file for distributed clients that simultaneously access the file and the distributed filesystem via different cloud controllers.

13. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for managing distributed opportunistic locks in a distributed filesystem, the method comprising:

collectively managing data in the distributed file system using two or more cloud controllers, wherein collectively managing the data comprises storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems;

receiving at a first cloud controller a request from a first client to access a file in the distributed filesystem, wherein the first client requests an opportunistic lock for the file from the first cloud controller, wherein the requested opportunistic lock requests shared read/write access permissions that are not exclusive to other client accesses, wherein the first cloud controller determines an owning cloud controller among the two or more cloud controllers that has been assigned to manage opportunistic locks for the file, wherein the owning cloud controller is distinct from the first cloud controller, wherein the owning cloudcontroller tracks multiple clients that have been granted opportunistic locks for the file via other cloud controllers that are distinct from the first cloud controller, wherein every cloud controller with clients presently accessing for the file registers with the owning cloud controller to receive notifications of any updates to the file;

initiating via the owning cloud controller a sharing access check that determines that there are outstanding opportunistic locks that are presently being held by one or more other clients of the distributed filesystem that were granted via other distinct cloud controllers for the distributed filesystem and that at least one or more of the outstanding opportunistic locks are write locks, wherein the owning cloud controller sends a distributed oplock break request to the cloud controllers that manage granted outstanding opportunistic locks for the file, wherein cloud controllers receiving the distributed oplock break request send an opportunistic lock break notification to their respective clients that are caching the file to ensure that there are no outstanding unflushed modifications to the file, wherein upon receiving the opportunistic lock break notification said clients flush all buffered modifications for the file to their respective cloud controllers, wherein information describing the flushed modifications for the file is synchronized to the cloud storage systems and to all of the other cloud controllers that have registered to receive notifications of updates for the file; and

upon determining that none of the other outstanding client opportunistic locks for the file that remain granted via the other cloud controllers of the distributed filesystem are exclusive locks that conflict with the requested opportunistic lock and determining that all distributed modifications to the file have been synchronized via the distributed oplock break request to ensure that no client receives obsolete data for the file, granting the first client the requested opportunistic lock for the file, wherein multiple clients simultaneously holdopportunistic locks granting shared write access for the file via distinct cloud controllers.

14. The non-transitory computer-readable storage medium of claim 13, wherein the first client uses the opportunistic lock to cache the file locally and perform buffered reads and/or writes for the file, thereby reducing network latency between the first client and the first cloud controller and improving the first client'"'"'s file access performance.

15. The non-transitory computer-readable storage medium of claim 14,

wherein the request includes a requested file access type and a requested sharing mode;

wherein determining whether other outstanding client accesses preclude the requested opportunistic lock comprises contacting the owning cloud controller to determine whether the requested file access type and the requested sharing mode are compatible with any other current file handles that have been granted for the file.

16. The non-transitory computer-readable storage medium of claim 15,

wherein upon determining that a second, remote client that is associated with a second cloud controller that is distinct from the first cloud controller currently holds an open file handle for the file, the owning cloud controller performs the sharing access check for the request;

wherein the sharing access check determines that the requested file access type and the requested sharing mode do not conflict with the open file handle and that the first client and the second client can simultaneously access the file;

wherein the owning cloud controller indicates to the first cloud controller that there are no sharing violations that preclude the requested opportunistic lock; and

wherein allowing multiple distributed clients to simultaneously perform buffered file operations enhances file access performance.

17. The non-transitory computer-readable storage medium of claim 15,

wherein upon determining that a second, remote client that is associated with a second cloud controller that is distinct from the first cloud controller currently holds an open write file handle for the file, the owning cloud controller requests that the second cloud controller send an opportunistic lock break notification to the second client;

and

wherein upon receiving the opportunistic lock break notification, the second client is configured to flush all buffered modifications for the file to the second cloud controller.

18. The non-transitory computer-readable storage medium of claim 17,

wherein an application executing on the second client that originally requested the open write file handle is still accessing the file;

wherein upon flushing all buffered modifications for the file in response to the opportunistic lock break notification the application continues using the open write file handle to write to the file;

wherein if, subsequent to initiating the sending of the opportunistic lock break notification to the second client, the owning cloud controller determined a sharing access check for the request that determines that the open write file handle conflicts with the request, the owning cloud controller would indicate tothe first cloud controller that a sharing violation precludes the requested opportunistic lock.

19. The non-transitory computer-readable storage medium of claim 17,

wherein an application executing on the second client that originally requested the open write file handle has completed accessing the file and closed the file, but the second client is still caching the open write file handle;

wherein upon flushing all buffered modifications for the file the second client closes the cached open write file handle and thus closes the open write file handle on the second cloud controller; and

wherein, subsequent to sending the opportunistic lock break notification to the second client via the second cloud controller, the owning cloud controller performs the sharing access check for the request that determines that there are no other client file handles for the file that conflict with the request and indicates to the first cloud controller that there are no sharing violations that preclude the requested opportunistic lock.

20. A cloud controller that facilitates managing distributed opportunistic locks in a distributed file system, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem; and

a storage management mechanism;

wherein two or more cloud controllers collectively manage data in the distributed file system;

wherein the cloud controller is configured to receive a request from a client to access a file in the distributed filesystem, wherein the client requests an opportunistic lock for the file from the cloud controller, wherein the requestedopportunistic lock requests shared read/write access permissions that are not exclusive to other client accesses, wherein the cloud controller determines an owning cloud controller among the two or more cloud controllers that has been assigned to manage opportunistic locks for the file, wherein the owning cloud controller is distinct from the cloud controller, wherein the owning cloud controller tracks, wherein every cloud controller with clients presently accessing the file registers with the owning cloud controller to receive notifications of any updates to the file;

wherein the cloud controller is configured to initiate via the owning cloud controller a sharing access check that determines that there are outstanding opportunistic locks that are presently being held by one or more other clients of the distributed filesystem that were granted via other distinct cloud controllers for the distributed filesystem and that at least one or more of the outstanding opportunistic locks are write locks, wherein the owning cloud controller sends a distributed oplock break request to the cloud controllers that manage granted outstanding opportunistic locks for the file, wherein cloud controllers receiving the distributed oplock break request send an opportunistic lock break notification to their respective clients that are caching the file to ensure that there are no outstanding unflushed modifications to the file, wherein upon receiving the opportunistic lock break notification said clients flush all buffered modifications for the file to their respective cloud controllers, wherein information describing the flushed modifications for the file is synchronized to the cloud storage systems and to all of the other cloud controllers that have registered to receive notifications of updates for the file; and

wherein, upon determining that none of the other outstanding client opportunistic locks for the file that remain granted via the other cloud controllersof the distributed filesystem are exclusive locks that conflict with the requested opportunistic lock and determining that all distributed modifications to the file have been synchronized via the distributed oplock break request to ensure that no client receives obsolete data for the file, the storage management mechanism is configured to grant the client the requested opportunistic lock for the file, wherein multiple clients simultaneously hold opportunistic locks granting shared write access for the file via distinct cloud controllers.