Distributing data for a distributed filesystem across multiple cloud storage systems

• US 8,799,413 B2
• Filed: 12/21/2012
• Issued: 08/05/2014
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Taylor, John Richard, Chou, Randy Yen-pang, Davis, Andrew P.
• Primary Examiner(s): Bates, Kevin
• Assistant Examiner(s): RAHMAN, SM AZIZUR
• Application Number:
  US13/725,738
• Publication Number:
  US 20130110778A1
• Time in Patent Office: 592 Days
• Field of Search: 709/219
• US Class Current: 709/219
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035

Abstract

The disclosed embodiments provide a system that distributes data for a distributed filesystem across multiple cloud storage systems. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. Whenever each cloud controller receives new data from a client, it outputs an incremental metadata snapshot for the new data that is propagated to the other cloud controllers and an incremental data snapshot containing the new data that is sent to a cloud storage system. During operation, data stored in the distributed filesystem can be distributed across two or more cloud storage systems to optimize performance and/or cost for the distributed filesystem.

Claims

1. A computer-implemented method for distributing data for a distributed filesystem across multiple cloud storage systems, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers by;

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients can only access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in a first remote cloud storage system using cloud files, wherein each cloud controller caches a subset of the file data from the first remote cloud storage system that is being actively accessed by that cloud controller'"'"'s respective clients, wherein all new file data received by each cloud controller from its clients is written to the first remote cloud storage system via the receiving cloud controller;

maintaining at each cloud controller a copy of the complete metadata for all of the files stored in the distributed filesystem, wherein each cloud controller communicates any changes to the metadata for the distributed filesystem to the full set of cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of each file in the distributed filesystem;

upon receiving in a cloud controller new file data from a client, storing the new file data for the distributed filesystem in the first remote cloud storage system, wherein the new file data includes two or more new files that are stored in a cloud file, wherein the cloud file is sent from the cloud controller to the first remote cloud storage system as part of an incremental data snapshot; and

upon receiving confirmation that the cloud file has been successfully stored in the first remote cloud storage system, sending from the cloud controller an incremental metadata snapshot that includes new metadata for the distributed filesystem that describes the new file data and links to the cloud file, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

and

distributing data stored in the distributed filesystem across the first remote cloud storage system and a second remote cloud storage system to optimize at least one of performance and cost of the distributed filesystem.

2. The computer-implemented method of claim 1, wherein distributing data across the first and second cloud storage system comprises:

initially storing the incremental data snapshot in the first remote cloud storage system; and

at a subsequent time, migrating the incremental data snapshot from the first remote cloud storage system to the second remote cloud storage system.

3. The computer-implemented method of claim 2,

wherein the first remote cloud storage system provides a higher level of replication across multiple points of presence than the second remote cloud storage system;

wherein storing new and frequently used data on the first remote cloud storage system facilitates rapidly distributing the data to the other cloud controllers of the distributed filesystem;

wherein the storage cost for the first remote cloud storage system is higher than the storage cost for the second remote cloud storage system; and

wherein data that is no longer frequently accessed is migrated to the second remote cloud storage system and deleted from the first remote cloud storage system.

4. The computer-implemented method of claim 3,

wherein the cloud controller manages the migration of the incremental data snapshot from the first remote cloud storage system to the second remote cloud storage system;

and

wherein the cloud controller prioritizes the migration as a low-priority job to ensure that the file access performance of clients of the cloud controller is not negatively impacted.

5. The computer-implemented method of claim 4, wherein the migration of the incremental data snapshot is scheduled at a time when the load of the distributed filesystem is low and the cost of network bandwidth is reduced.

6. The computer-implemented method of claim 5, wherein migrating the incremental data snapshot further comprises determining whether to migrate the incremental data snapshot based on:

the cost of storage at the first remote cloud storage system;

the cost of storage at the second remote cloud storage system;

a network bandwidth cost for the migration;

the access frequency and access history for the incremental data snapshot;

the potential performance impact for clients accessing the incremental data snapshot from the second remote cloud storage system instead of the first remote cloud storage system;

the load of one or more cloud controllers; and

a locality policy for the first incremental data snapshot.

7. The computer-implemented method of claim 6,

wherein each cloud controller is configured to track access frequency and access patterns for cloud files that were created by that cloud controller;

and

wherein cloud controllers determine the set of cloud files to migrate to the second remote cloud storage system based on the tracked access frequencies and access patterns.

8. The computer-implemented method of claim 7, wherein migrating the incremental data snapshot further comprises:

determining that a cloud file in the incremental data snapshot should be migrated to the second remote cloud storage system;

calculating a new cloud virtual address (CVA) for the cloud file at the second remote cloud storage system, wherein a CVA for the cloud file uniquely identifies the originating cloud controller that create the cloud file and the cloud storage system storing the cloud file, wherein the new CVA reflects the location of the cloud file as being on the second cloud storage system;

uploading the cloud file to the second remote cloud storage system using an identifier based on the new CVA;

upon receiving confirmation of receipt for the cloud file from the second remote cloud storage system, updating the metadata for all of the file blocks in the migrated cloud file to point to the new CVA;

distributing an updated incremental metadata snapshot to the other cloud controllers to notify them of the migration of the cloud file; and

after ensuring that the other cloud controllers have updated their local metadata to reflect the migration, deleting the cloud file from the first remote cloud storage system;

wherein allocating a new CVA and performing a multi-step migration to the second remote cloud storage system facilitates maintaining the data coherency for the distributed filesystem when migrating cloud files between remote cloud storage systems.

9. The computer-implemented method of claim 1,

wherein distributing data across the first and second remote cloud storage systems comprises mirroring the incremental data snapshot to the first and second remote cloud storage systems;

and

wherein mirroring data across multiple cloud storage providers ensures that data remains available to clients across an outage in one of the first and second cloud storage systems.

10. The computer-implemented method of claim 9,

wherein the first remote cloud storage system has lower access costs than the second remote cloud storage system but is also less reliable than the second remote cloud storage system;

and

wherein the cloud controllers are configured to first attempt to read data from the first remote cloud storage system to minimize the access costs for the distributed filesystem.

11. The computer-implemented method of claim 10, wherein mirroring the incremental data snapshot to the first and second remote cloud storage systems comprises:

immediately writing the incremental data snapshot from the cloud controller to the first remote cloud storage system; and

caching the incremental data snapshot on the cloud controller and delaying transferring the incremental data snapshot from the cloud controller to the second remote cloud storage system until a timeframe in which the cost of network bandwidth between the cloud controller and the second remote cloud storage system is lower.

12. The computer-implemented method of claim 1,

wherein the first remote cloud storage system is associated with a first cloud storage provider;

wherein the second remote cloud storage system is associated with a second cloud storage provider; and

wherein the first cloud storage provider and the second cloud storage provider are distinct.

13. The computer-implemented method of claim 1,

wherein the first remote cloud storage system and the second remote cloud storage system are associated with the same cloud storage provider;

and

wherein the first remote cloud storage system and the second remote cloud storage system are associated with different costs and levels of service that are provided by the same cloud storage provider.

14. The computer-implemented method of claim 1,

wherein a cloud storage system is unaware of the organization and structure of the distributed filesystem;

wherein data stored in the distributed filesystem is indexed using a global address space;

wherein data is stored in the cloud storage system as cloud files, wherein each cloud file is uniquely indexed in the global address space;

wherein a second cloud controller receiving the incremental metadata snapshot uses a metadata entry in the incremental metadata snapshot to retrieve data in the incremental data snapshot; and

wherein using the metadata entry to retrieve data in the incremental data snapshot comprises;

determining from the metadata entry that a desired data block in the incremental data snapshot is not presently stored in the second cloud controller;

using a global address stored in the metadata entry to identify a cloud file in the cloud storage system that includes the data block;

downloading the identified cloud file to the second cloud controller; and

using an offset stored in the metadata entry to determine the location of the data block in the cloud file.

15. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for distributing data for a distributed filesystem across multiple cloud storage systems, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers by;

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients can only access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in a first remote cloud storage system using cloud files, wherein each cloud controller caches a subset of the file data from the first remote cloud storage system that is being actively accessed by that cloud controller'"'"'s respective clients, wherein all new file data received by each cloud controller from its clients is written to the first remote cloud storage system via the receiving cloud controller;

maintaining at each cloud controller a copy of the complete metadata for all of the files stored in the distributed filesystem, wherein each cloud controller communicates any changes to the metadata for the distributed filesystem to the full set of cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of each file in the distributed filesystem;

upon receiving in a cloud controller new file data from a client, storing the new file data for the distributed filesystem in the first remote cloud storage system, wherein the new file data includes two or more new files that are stored in a cloud file, wherein the cloud file is sent from the cloud controller to the first remote cloud storage system as part of an incremental data snapshot; and

upon receiving confirmation that the cloud file has been successfully stored in the first remote cloud storage system, sending from the cloud controller an incremental metadata snapshot that includes new metadata for the distributed filesystem that describes the new file data and links to the cloud file, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

and

distributing data stored in the distributed filesystem across the first remote cloud storage system and a second remote cloud storage system to optimize at least one of performance and cost of the distributed filesystem.

16. The non-transitory computer-readable storage medium of claim 15, wherein distributing data across the first and second cloud storage system comprises:

initially storing the incremental data snapshot in the first remote cloud storage systems; and

at a subsequent time, migrating the incremental data snapshot from the first remote cloud storage system to the second remote cloud storage system.

17. The non-transitory computer-readable storage medium of claim 16,

wherein the first remote cloud storage system provides a higher level of replication across multiple points of presence than the second remote cloud storage system;

wherein storing new and frequently used data on the first remote cloud storage system facilitates rapidly distributing the data to the other cloud controllers of the distributed filesystem;

wherein the storage cost for the first remote cloud storage system is higher than the storage cost for the second remote cloud storage system; and

wherein data that is no longer frequently accessed is migrated to the second remote cloud storage system and deleted from the first remote cloud storage system.

18. The non-transitory computer-readable storage medium of claim 17,

wherein the cloud controller manages the migration of the incremental data snapshot from the first remote cloud storage system to the second remote cloud storage system;

and

wherein the cloud controller prioritizes the migration as a low-priority job to ensure that the file access performance of clients of the cloud controller is not negatively impacted.

19. The non-transitory computer-readable storage medium of claim 18, wherein the migration of the incremental data snapshot is scheduled at a time when the load of the distributed filesystem is low and the cost of network bandwidth is reduced.

20. A cloud controller that distributes data for a distributed filesystem across multiple cloud storage systems, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem; and

a storage management mechanism;

wherein two or more cloud controllers collectively manage the data of the distributed filesystem, wherein collectively managing the data comprises;

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients can only access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in a first remote cloud storage system using cloud files, wherein each cloud controller caches a subset of the file data from the first remote cloud storage system that is being actively accessed by that cloud controller'"'"'s respective clients, wherein all new file data received by each cloud controller from its clients is written to the first remote cloud storage system via the receiving cloud controller;

maintaining at each cloud controller a copy of the complete metadata for all of the files stored in the distributed filesystem, wherein each cloud controller communicates any changes to the metadata for the distributed filesystem to the full set of cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of each file in the distributed filesystem;

upon receiving in the cloud controller new file data from a client, storing the new file data for the distributed filesystem in the first remote cloud storage system, wherein the cloud file is sent from the cloud controller to the first remote cloud storage system as part of an incremental data snapshot; and

upon receiving confirmation that the cloud file has been successfully stored in the first remote cloud storage system, sending from the cloud controller an incremental metadata snapshot that includes new metadata for the distributed filesystem that describes the new file data and links to the cloud file, wherein the incremental metadata snapshot is received by the other cloud controllers of the distributed filesystem;

and

wherein the storage management mechanism is further configured to distribute data stored in the distributed filesystem across the first remote cloud storage system and a second remote cloud storage system to optimize at least one of performance and cost of the distributed filesystem.