PROVIDING DISASTER RECOVERY FOR A DISTRIBUTED FILESYSTEM

• US 20130111262A1
• Filed: 12/21/2012
• Published: 05/02/2013
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Taylor, John Richard, Chou, Randy Yen-pang, Davis, Andrew P.
• Granted Patent: US 8,805,967 B2
• Time in Patent Office: Days
• US Class Current: 714/4.11
• CPC Class Codes: G06F 11/2005 using redundant communication controllers
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/1865 Transactional file systems

Abstract

The disclosed embodiments provide a system that distributes data for a distributed filesystem across multiple cloud storage systems. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. Whenever each cloud controller receives new data from a client, it outputs an incremental metadata snapshot for the new data that is propagated to the other cloud controllers and an incremental data snapshot containing the new data that is sent to a cloud storage system. During operation, a backup cloud controller associated with the distributed filesystem is also configured to receive each (incremental) metadata snapshot, such that, upon determining the failure of a cloud controller, the backup cloud controller can immediately begin receiving data requests from clients associated with the failed cloud controller.

Claims

1. A computer-implemented method for providing disaster recovery for a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

upon receiving in a cloud controller new data from a client, sending from the cloud controller an incremental metadata snapshot for the new data, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems; and

sending an incremental data snapshot containing the new data from the cloud controller to a cloud storage system;

receiving the incremental metadata snapshot at a backup cloud controller associated with the distributed filesystem;

upon determining a failure for the cloud controller, receiving data requests from clients associated with the cloud controller at the backup cloud controller.

2. The computer-implemented method of claim 1,

wherein the backup cloud controller and all of the two or more cloud controllers receive all of the incremental metadata snapshots sent by the two or more cloud controllers;

wherein the backup cloud controller is distinct from the two or more cloud controllers;

wherein storing the complete set of metadata for the distributed filesystem in each cloud controller facilitates accessing data from a cloud file in the cloud storage system; and

wherein storing the complete set of metadata for the distributed filesystem in the backup cloud controller facilitates responding to client requests received from clients previously associated with the cloud controller.

3. The computer-implemented method of claim 2, wherein receiving the incremental metadata snapshot at the backup cloud controller further comprises:

using the received incremental metadata snapshot to identify the new data that was uploaded to the cloud storage system by the cloud controller; and

downloading and caching the new data in the backup cloud controller.

4. The computer-implemented method of claim 3, wherein prior to the failure of the cloud controller, the method further comprises downloading and caching in the backup cloud controller the data most recently written to the cloud storage system by all of the two or more cloud controllers, thereby facilitating efficiently and seamlessly taking over the role of a failed cache controller.

5. The computer-implemented method of claim 3, wherein subsequent to the failure of the cloud controller the backup cloud controller is configured to manage its local cache to optimize the performance of the data requests received from its clients.

6. The computer-implemented method of claim 3,

wherein the two or more cloud controllers are configured to track client data usage to determine a set of data that is most commonly accessed by the clients of the distributed filesystem;

and

wherein prior to the failure of the cloud controller, the backup cloud controller is configured to download the determined set of data from the cloud storage provider and cache the determined set of data locally.

7. The computer-implemented method of claim 4, wherein the backup cloud controller only receives data requests from clients subsequent to the failure of the cloud controller.

8. The computer-implemented method of claim 7, wherein receiving data requests destined for the cloud controller at the backup cloud controller further comprises:

detecting the failure of the cloud controller; and

automatically rerouting client requests directed to the cloud controller to the backup cloud controller.

9. The computer-implemented method of claim 8,

wherein two or more backup cloud controllers are associated with the distributed filesystem;

and

wherein choosing one of the two or more backup cloud controllers to take over for the failed cloud controller involves considering at least one of network characteristics and network proximity to the failed cloud controller.

10. The computer-implemented method of claim 9,

wherein the cloud storage system is unaware of the organization and structure of the distributed filesystem;

wherein data stored in the distributed filesystem is indexed using a global address space;

wherein data is stored in the cloud storage system as cloud files, wherein each cloud file is uniquely indexed in the global address space; and

wherein using a metadata entry to download a cloud file that contains a desired data block from the cloud storage system comprises;

determining from the metadata entry that the desired data block is not presently stored in the requesting cloud controller;

using a global address stored in the metadata entry to identify the cloud file that includes the data block;

downloading the identified cloud file to the requesting cloud controller; and

using an offset stored in the metadata entry to determine the location of the data block in the cloud file.

11. The computer-implemented method of claim 10,

wherein a second cloud controller receives the incremental metadata snapshot;

and

wherein the second cloud controller uses the incremental metadata snapshot to retrieve data associated with the incremental data snapshot from the cloud storage system.

12. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for providing disaster recovery for a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

upon receiving in a cloud controller new data from a client, sending from the cloud controller an incremental metadata snapshot for the new data, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems; and

sending an incremental data snapshot containing the new data from the cloud controller to a cloud storage system;

receiving the incremental metadata snapshot at a backup cloud controller associated with the distributed filesystem;

upon determining a failure for the cloud controller, receiving data requests from clients associated with the cloud controller at the backup cloud controller.

13. The non-transitory computer-readable storage medium of claim 12,

wherein the backup cloud controller and all of the two or more cloud controllers receive all of the incremental metadata snapshots sent by the two or more cloud controllers;

wherein the backup cloud controller is distinct from the two or more cloud controllers;

wherein storing the complete set of metadata for the distributed filesystem in each cloud controller facilitates accessing data from a cloud file in the cloud storage system; and

wherein storing the complete set of metadata for the distributed filesystem in the backup cloud controller facilitates responding to client requests received from clients previously associated with the cloud controller.

14. The non-transitory computer-readable storage medium of claim 13, wherein receiving the incremental metadata snapshot at the backup cloud controller further comprises:

using the received incremental metadata snapshot to identify the new data that was uploaded to the cloud storage system by the cloud controller; and

downloading and caching the new data in the backup cloud controller.

15. The non-transitory computer-readable storage medium of claim 14, wherein prior to the failure of the cloud controller, the method further comprises downloading and caching in the backup cloud controller the data most recently written to the cloud storage system by all of the two or more cloud controllers, thereby facilitating efficiently and seamlessly taking over the role of a failed cache controller.

16. The non-transitory computer-readable storage medium of claim 14, wherein subsequent to the failure of the cloud controller the backup cloud controller is configured to manage its local cache to optimize the performance of the data requests received from its clients.

17. The non-transitory computer-readable storage medium of claim 14,

wherein the two or more cloud controllers are configured to track client data usage to determine a set of data that is most commonly accessed by the clients of the distributed filesystem;

and

wherein prior to the failure of the cloud controller, the backup cloud controller is configured to download the determined set of data from the cloud storage provider and cache the determined set of data locally.

18. The non-transitory computer-readable storage medium of claim 15, wherein the backup cloud controller only receives data requests from clients subsequent to the failure of the cloud controller, and wherein receiving data requests destined for the cloud controller at the backup cloud controller further comprises:

detecting the failure of the cloud controller; and

automatically rerouting client requests directed to the cloud controller to the backup cloud controller.

19. The non-transitory computer-readable storage medium of claim 18,

wherein two or more backup cloud controllers are associated with the distributed filesystem;

and

wherein choosing one of the two or more backup cloud controllers to take over for the failed cloud controller involves considering at least one of network characteristics and network proximity to the failed cloud controller.

20. A backup cloud controller that provides disaster recovery for a distributed filesystem, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem; and

a receiving mechanism;

wherein two or more cloud controllers collectively manage the data of the distributed filesystem, wherein collectively managing the data comprises;

upon receiving in a cloud controller new data from a client, sending from the cloud controller an incremental metadata snapshot for the new data, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems; and

sending an incremental data snapshot containing the new data from the cloud controller to a cloud storage system;

wherein the receiving mechanism is configured to receive the incremental metadata snapshot and received metadata is stored in the storage mechanism; and

wherein the receiving mechanism is further configured to, upon determining a failure for the cloud controller, receive data requests from clients associated with the cloud controller.