EXECUTING A CLOUD COMMAND FOR A DISTRIBUTED FILESYSTEM

• US 20140006354A1
• Filed: 09/05/2013
• Published: 01/02/2014
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Davis, Andrew P., Taylor, John Richard, Chou, Randy Yen-pang, Parkison, Brian Christopher
• Granted Patent: US 9,811,532 B2
• Time in Patent Office: Days
• US Class Current: 707/649
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

The disclosed embodiments disclose techniques for executing a cloud command for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller presents a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem (e.g., using a file abstraction). Upon receiving a request from the client system to access and/or operate upon this file, the client controller executes an associated cloud command. More specifically, the cloud controller initiates a specially-defined operation that accesses additional functionality for the distributed filesystem that exceeds the scope of individual reads and writes to a typical data file.

Claims

1. A computer-implemented method for performing a distributed-filesystem-specific action, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems; and

caching data for the distributed filesystem in each cloud controller;

presenting a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem using a file abstraction;

receiving at a cloud controller a request from the client system to access the file; and

performing a distributed-filesystem-specific action in response to the file access.

2. The computer-implemented method of claim 1,

wherein the distributed management of the distributed filesystem is typically transparent to the client system;

and

wherein the distributed-filesystem-specific action initiates a specially-defined operation in a cloud controller that accesses additional functionality for the distributed filesystem that exceeds the scope of individual reads and writes to a file.

3. The computer-implemented method of claim 2, wherein the distributed-filesystem-specific action is a cloud-aware copy operation that comprises:

determining from the request a source file and a destination file for the cloud-aware copy operation;

using the metadata for the source file to generate metadata for the destination file that references the data blocks associated with the source file in a cloud storage system; and

distributing a metadata snapshot for the destination file to the other cloud controllers collectively managing the distributed filesystem;

wherein the data blocks for the source file are not transferred to the client system and the cloud controller during the cloud-aware copy operation, thereby substantially reducing the perceived command execution time for the client system.

4. The computer-implemented method of claim 3, wherein the method further comprises:

determining that one or more data blocks for the destination file are not currently being cached by the cloud controller; and

pre-fetching one or more uncached data blocks for the destination file from the cloud storage system.

5. The computer-implemented method of claim 2, wherein the distributed-filesystem-specific action is a user-initiated snapshot operation that comprises:

determining from the request a portion of the distributed filesystem to be snapshotted;

initiating a distributed snapshot operation for the portion of the distributed filesystem in every cloud controller associated with the distributed filesystem;

triggering an incremental metadata snapshot for each cloud controller associated with the distributed filesystem to share the resulting snapshot state between the cloud controllers, thereby snapshotting the entire state of the distributed filesystem at the time of the request.

6. The computer-implemented method of claim 5,

wherein the method further comprises exposing snapshot information to the client system using a snapshot directory;

wherein a user can access a snapshotted version of a file that was created during the user-initiated snapshot operation via the snapshot directory; and

wherein facilitating user access to snapshotted files facilitates reducing the administrative burden of system administrators for the distributed filesystem.

7. The computer-implemented method of claim 5,

wherein the user-initiated snapshot operation further comprises:

executing and initiating a virtual machine in the distributed filesystem environment; and

snapshotting the state and the data of the virtual machine;

wherein the distributed-filesystem-specific action further comprises cloning the snapshotted virtual machine; and

wherein cloning the snapshotted virtual machine facilitates reducing the overhead associated with instantiating a second virtual machine.

8. The computer-implemented method of claim 7, wherein cloning the snapshotted virtual machine further comprises performing a cloud-aware copy operation for the state and the data of the virtual machine.

9. The computer-implemented method of claim 5, wherein the distributed-filesystem-specific action is a database-backup operation that comprises:

synchronizing all of the in-memory data for a database to the distributed filesystem, wherein synchronizing the in-memory data to the distributed filesystem ensures that all of the data for the database is consistently stored in the distributed filesystem; and

performing the user-initiated snapshot operation for the database to ensure that all updated data blocks for the database are propagated to a cloud storage system;

wherein using the distributed filesystem to perform the database-backup operation facilitates backing up the data for the database while avoiding writing the database data to a separate database dump file.

10. The computer-implemented method of claim 2, wherein the distributed-filesystem-specific action is a cloud-aware archive operation that comprises:

determining from the request a file to be archived in an archival cloud storage system; and

transferring one or more cloud files containing data associated with the file from a first cloud storage system to the archival cloud storage system.

11. The computer-implemented method of claim 2, wherein the distributed-filesystem-specific action is a cloud-aware restore operation that comprises:

determining from the request a file that has been archived in an archival cloud storage system;

transferring one or more cloud files containing data associated with the file from an archival cloud storage system to a second cloud storage system; and

transferring the data blocks for the file to the cloud controller.

12. The computer-implemented method of claim 2, wherein the method further comprises determining a set of distributed-filesystem-specific capabilities that are presented to the client system based on a set of permissions associated with a user accessing the distributed filesystem via the client system.

13. The computer-implemented method of claim 12, wherein the method further comprises changing the set of cloud commands supported by the cloud controller without modifying the client system or the interface between the client system and the cloud controller.

14. The computer-implemented method of claim 12, wherein the method further comprises:

detecting that the file access is associated with a cloud command instead of a normal data file in the distributed filesystem; and

initiating an event handler and a set of program instructions that are associated distributed-filesystem-specific actions.

15. The computer-implemented method of claim 14,

wherein the distributed-filesystem-specific action provides status information for a data file in distributed filesystem;

wherein the status information for the data file is dynamically generated by the cloud controller; and

wherein the status information comprises;

a timestamp for the most recent snapshot that included the data file;

replication status for the data file;

the percentage of the data file'"'"'s data has been written to a cloud storage system;

the portions of the data file that are currently being cached in the cloud controller;

an estimated time interval needed to retrieve any uncached data blocks for the data file given the load of the cloud controller and an associated network; and

an indication of whether the data file has been archived and, if so, restore information for the data file.

16. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for performing a distributed-filesystem-specific action, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems; and

caching data for the distributed filesystem in each cloud controller;

presenting a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem using a file abstraction;

receiving at a cloud controller a request from the client system to access the file; and

performing a distributed-filesystem-specific action in response to the file access.

17. The non-transitory computer-readable storage medium of claim 16, wherein the distributed-filesystem-specific action is a cloud-aware copy operation that comprises:

determining from the request a source file and a destination file for the cloud-aware copy operation;

using the metadata for the source file to generate metadata for the destination file that references the data blocks associated with the source file in a cloud storage system; and

distributing a metadata snapshot for the destination file to the other cloud controllers collectively managing the distributed filesystem;

wherein the data blocks for the source file are not transferred to the client system and the cloud controller during the cloud-aware copy operation, thereby substantially reducing the perceived command execution time for the client system.

18. The non-transitory computer-readable storage medium of claim 16, wherein the distributed-filesystem-specific action is a user-initiated snapshot operation that comprises:

determining from the request a portion of the distributed filesystem to be snapshotted;

initiating a distributed snapshot operation for the portion of the distributed filesystem in every cloud controller associated with the distributed filesystem;

triggering an incremental metadata snapshot for each cloud controller associated with the distributed filesystem to share the resulting snapshot state between the cloud controllers, thereby snapshotting the entire state of the distributed filesystem at the time of the request.

19. The non-transitory computer-readable storage medium of claim 18,

wherein the user-initiated snapshot operation further comprises:

executing and initiating a virtual machine in the distributed filesystem environment; and

snapshotting the state and the data of the virtual machine;

wherein the distributed-filesystem-specific action further comprises cloning the snapshotted virtual machine; and

wherein cloning the snapshotted virtual machine facilitates reducing the overhead associated with instantiating a second virtual machine.

20. A cloud controller that performs a distributed-filesystem-specific action, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem; and

a storage management mechanism;

wherein two or more cloud controllers collectively manage the data of the distributed filesystem;

wherein the cloud controller is configured to present a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem using a file abstraction;

wherein the cloud controller is further configured to receive a request from the client system to access the file; and

wherein the cloud controller is further configured to perform a distributed-filesystem-specific action in response to the file access.