DISTRIBUTING DATA FOR A DISTRIBUTED FILESYSTEM ACROSS MULTIPLE CLOUD STORAGE SYSTEMS

• US 20130110778A1
• Filed: 12/21/2012
• Published: 05/02/2013
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Taylor, John Richard, Chou, Randy Yen-pang, Davis, Andrew P.
• Granted Patent: US 8,799,413 B2
• Time in Patent Office: Days
• US Class Current: 707/624
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035

Abstract

The disclosed embodiments provide a system that distributes data for a distributed filesystem across multiple cloud storage systems. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. Whenever each cloud controller receives new data from a client, it outputs an incremental metadata snapshot for the new data that is propagated to the other cloud controllers and an incremental data snapshot containing the new data that is sent to a cloud storage system. During operation, data stored in the distributed filesystem can be distributed across two or more cloud storage systems to optimize performance and/or cost for the distributed filesystem.

Claims

1. A computer-implemented method for distributing data for a distributed filesystem across multiple cloud storage systems, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

upon receiving in a cloud controller new data from a client, sending from the cloud controller an incremental metadata snapshot for the new data, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems; and

sending an incremental data snapshot containing the new data from the cloud controller to a first cloud storage system; and

distributing data stored in the distributed filesystem across the first cloud storage system and a second cloud storage system to optimize at least one of performance and cost of the distributed filesystem.

2. The computer-implemented method of claim 1, wherein distributing data across the first and second cloud storage system comprises:

initially storing the incremental data snapshot in the first cloud storage systems; and

at a subsequent time, migrating the incremental data snapshot from the first cloud storage system to the second cloud storage system.

3. The computer-implemented method of claim 2,

wherein the first cloud storage system provides a higher level of replication across multiple points of presence than the second cloud storage system;

wherein storing new and frequently used data on the first cloud storage system facilitates rapidly distributing the data to the other cloud controllers of the distributed filesystem;

wherein the storage cost for the first cloud system is higher than the storage cost for the second cloud system; and

wherein data that is no longer frequently accessed is migrated to the second cloud storage system and deleted from the first cloud storage system.

4. The computer-implemented method of claim 3,

wherein the cloud controller manages the migration of the incremental data snapshot from the first cloud storage system to the second cloud storage system;

and

wherein the cloud controller prioritizes the migration as a low-priority job to ensure that the file access performance of clients of the cloud controller is not negatively impacted.

5. The computer-implemented method of claim 4, wherein the migration of the incremental data snapshot is scheduled at a time when the load of the distributed filesystem is low and the cost of network bandwidth is reduced.

6. The computer-implemented method of claim 5, wherein migrating the incremental data snapshot further comprises determining whether to migrate the incremental data snapshot based on:

the cost of storage at the first cloud storage system;

the cost of storage at the second cloud storage system;

a network bandwidth cost for the migration;

the access frequency and access history for the incremental data snapshot;

the potential performance impact for clients accessing the incremental data snapshot from the second cloud storage system instead of the first cloud storage system;

the load of one or more cloud controllers; and

a locality policy for the first incremental data snapshot.

7. The computer-implemented method of claim 6,

wherein each cloud controller is configured to track access frequency and access patterns for cloud files that were created by that cloud controller;

and

wherein cloud controllers determine the set of cloud files to migrate to the second cloud storage system based on the track access frequencies and access patterns.

8. The computer-implemented method of claim 7, wherein migrating the incremental data snapshot further comprises:

determining that a cloud file in the incremental data snapshot should be migrated to the second cloud storage system;

calculating a new cloud virtual address (CVA) for the cloud file at the second cloud storage provider, wherein the new CVA reflects the location of the cloud file as being on the second cloud storage provider;

uploading the cloud file to the second cloud storage provider using an identifier based on the new CVA;

upon receiving confirmation of receipt for the cloud file from the second cloud storage provider, updating the metadata for all of the file blocks in the migrated cloud file to point to the new CVA;

distributing an updated incremental metadata snapshot to the other cloud controllers to notify them of the migration of the cloud file; and

after ensuring that the other cloud controllers have updated their local metadata to reflect the migration, deleting the cloud file from the first cloud storage provider.

9. The computer-implemented method of claim 1,

wherein distributing data across the first and second cloud storage systems comprises mirroring the incremental data snapshot to the first and second cloud storage systems;

and

wherein mirroring data across multiple cloud storage providers ensures that data remains available to clients across an outage in one of the first and second cloud storage systems.

10. The computer-implemented method of claim 9,

wherein the first cloud storage system has lower access costs than the second cloud storage system but is also less reliable than the second cloud storage system;

and

wherein the cloud controllers are configured to first attempt to read data from the first cloud storage system to minimize the access costs for the distributed filesystem.

11. The computer-implemented method of claim 10, wherein mirroring the incremental data snapshot to the first and second cloud storage systems comprises:

immediately writing the incremental data snapshot from the cloud controller to the first cloud storage system; and

caching the incremental data snapshot on the cloud controller and delaying transferring the incremental data snapshot from the cloud controller to the second cloud storage system until a timeframe in which the cost of network bandwidth between the cloud controller and the second cloud storage system is lower.

12. The computer-implemented method of claim 1,

wherein the first cloud storage system is associated with a first cloud storage provider;

wherein the second cloud storage system is associated with a second cloud storage provider; and

wherein the first cloud storage provider and the second cloud storage provider are distinct.

13. The computer-implemented method of claim 1,

wherein the first cloud storage system and the second cloud storage system are associated with a first cloud storage provider;

and

wherein the first cloud storage system and the second cloud storage system are associated with different levels of service.

14. The computer-implemented method of claim 1,

wherein a cloud storage system is unaware of the organization and structure of the distributed filesystem;

wherein data stored in the distributed filesystem is indexed using a global address space;

wherein data is stored in the cloud storage system as cloud files, wherein each cloud file is uniquely indexed in the global address space;

wherein a second cloud controller receiving the incremental metadata snapshot uses a metadata entry in the incremental metadata snapshot to retrieve data in the incremental data snapshot; and

wherein using the metadata entry to retrieve data in the incremental data snapshot comprises;

determining from the metadata entry that a desired data block in the incremental data snapshot is not presently stored in the second cloud controller;

using a global address stored in the metadata entry to identify a cloud file in the cloud storage system that includes the data block;

downloading the identified cloud file to the second cloud controller; and

using an offset stored in the metadata entry to determine the location of the data block in the cloud file.

15. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for distributing data for a distributed filesystem across multiple cloud storage systems, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

upon receiving in a cloud controller new data from a client, sending from the cloud controller an incremental metadata snapshot for the new data, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems; and

sending an incremental data snapshot containing the new data from the cloud controller to a first cloud storage system; and

distributing data stored in the distributed filesystem across the first cloud storage system and a second cloud storage system to optimize at least one of performance and cost of the distributed filesystem.

16. The non-transitory computer-readable storage medium of claim 15, wherein distributing data across the first and second cloud storage system comprises:

initially storing the incremental data snapshot in the first cloud storage systems; and

at a subsequent time, migrating the incremental data snapshot from the first cloud storage system to the second cloud storage system.

17. The non-transitory computer-readable storage medium of claim 16,

wherein the first cloud storage system provides a higher level of replication across multiple points of presence than the second cloud storage system;

wherein storing new and frequently used data on the first cloud storage system facilitates rapidly distributing the data to the other cloud controllers of the distributed filesystem;

wherein the storage cost for the first cloud system is higher than the storage cost for the second cloud system; and

wherein data that is no longer frequently accessed is migrated to the second cloud storage system and deleted from the first cloud storage system.

18. The non-transitory computer-readable storage medium of claim 17,

wherein the cloud controller manages the migration of the incremental data snapshot from the first cloud storage system to the second cloud storage system;

and

wherein the cloud controller prioritizes the migration as a low-priority job to ensure that the file access performance of clients of the cloud controller is not negatively impacted.

19. The non-transitory computer-readable storage medium of claim 18, wherein the migration of the incremental data snapshot is scheduled at a time when the load of the distributed filesystem is low and the cost of network bandwidth is reduced.

20. A cloud controller that distributes data for a distributed filesystem across multiple cloud storage systems, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem; and

a storage management mechanism;

wherein two or more cloud controllers collectively manage the data of the distributed filesystem, wherein collectively managing the data comprises;

upon receiving in a cloud controller new data from a client, sending from the cloud controller an incremental metadata snapshot for the new data, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the storage management mechanisms of the cloud controllers are configured to cache and ensure data consistency for data stored in the cloud storage systems; and

sending an incremental data snapshot containing the new data from the cloud controller to a first cloud storage system; and

wherein the storage management mechanism is further configured to distribute data stored in the distributed filesystem across the first cloud storage system and a second cloud storage system to optimize at least one of performance and cost of the distributed filesystem.