Managing metadata and data storage for a cloud controller in a distributed filesystem

• US 9,792,298 B1
• Filed: 02/15/2013
• Issued: 10/17/2017
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Taylor, John Richard, Chou, Randy Yen-pang, Davis, Andrew P.
• Primary Examiner(s): Rahman, S M
• Application Number:
  US13/769,211
• Time in Patent Office: 1,705 Days
• Field of Search: 709219
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035

Abstract

The disclosed embodiments disclose techniques for managing metadata and data storage for a cloud controller in a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems. More specifically, the cloud controllers cache and ensure data consistency for the data stored in the cloud storage systems, with each cloud controller maintaining (e.g., storing) in a local storage device: (1) one or more metadata regions containing a metadata hierarchy that reflects the current state of the distributed filesystem; and (2) cached data for the distributed filesystem. During operation, the cloud controller receives an incremental metadata snapshot that references new data written to the distributed filesystem. The cloud controller stores updated metadata from this incremental metadata snapshot in one of the metadata regions on the local storage device.

Claims

1. A computer-implemented method for managing metadata and data storage for a cloud controller in a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in a remote cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the remote cloud storage system, wherein the cloud controller includes a local storage device, wherein the local storage device comprises a rotating disk drive that comprises one or more disk platters;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein the metadata hierarchy is stored in the local storage device, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem; and

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the remote cloud storage system, wherein cloud controllers cache in their local storage devices a subset of the file data from the remote cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients, wherein new file data received by each cloud controller from its clients is written to the remote cloud storage system, wherein the metadata hierarchy in the cloud controller tracks the location of distributed filesystem data blocks in the remote cloud storage system and cached distributed filesystem data blocks in the cloud controller, wherein the cloud controller uses the metadata hierarchy to locate and download requested, uncached data blocks in the distributed filesystem from the remote cloud storage system;

defining in a disk platter of the rotating disk drive two or more metadata regions in which the cloud controller stores metadata for the distributed filesystem, wherein the metadata regions are distinct from two or more allocated data regions that are defined in the disk platter that cache distributed filesystem data, wherein different regions of the disk platter in the local storage device have different levels of performance, wherein a metadata region is defined in an outer region of the disk platter that supports the highest access bandwidth and lower access latency;

receiving an incremental metadata snapshot that references new data written to the distributed filesystem;

storing a new metadata entry for the distributed filesystem from the incremental metadata snapshot in the metadata region on the disk platter; and

upon receiving a client request to access a new data block referenced in the incremental metadata snapshot, selecting a data region that is in near proximity to the metadata region and caching the new data block in that data region to ensure that the new metadata entry and the new data block are in relative proximity on the disk platter, thereby ensuring that associated metadata and data can be read without substantially degrading access performance, wherein the data region is distinct from the metadata region;

wherein the cloud controller predicts that the new metadata entry and the new data block are likely to be accessed frequently, wherein the cloud controller selects the metadata region and the data region for the new metadata entry and the new data block respectively because they are on an outer region of the disk platter and hence more favorable for frequent accesses, wherein outer regions of the disk platter have higher spatial density and hence higher effective data bandwidth that improves access rates for frequently accessed data stored in such regions.

2. The computer-implemented method of claim 1, wherein the method further comprises:

receiving a request from a client to access the new data block referenced by the updated metadata;

using the metadata hierarchy to download a cloud file containing the data block from the remote cloud storage system; and

caching the new data block in the local storage device of the cloud controller.

3. The computer-implemented method of claim 2,

wherein the metadata hierarchy grows as new data is added to the distributed filesystem, wherein the new data block written to the distributed filesystem is referenced by the new metadata entry in the metadata hierarchy;

and

wherein separating the metadata region from the data region facilitates preventing defragmentation of the data region by metadata over time.

4. The computer-implemented method of claim 3,

wherein the set of data cached in the local storage device changes frequently based on the data accesses of one or more clients associated with the cloud controller;

wherein a metadata entry in the metadata hierarchy is deleted from the local storage device of the cloud controller if the data block associated with the metadata entry is deleted from the distributed filesystem; and

wherein the deletion rate for files of the distributed filesystem is substantially lower than the cache replacement rate for the distributed filesystem data cached in the local storage device.

5. The computer-implemented method of claim 1, wherein the distribution of the metadata and data regions across the disk platter is prioritized based on an estimation of future metadata and data access trends.

6. The computer-implemented method of claim 5, wherein metadata is accessed more frequently than cached distributed filesystem data.

7. The computer-implemented method of claim 5,

wherein the local storage device comprises multiple storage devices;

and

wherein the method further comprises storing the new data block on a first storage device and the new metadata entry on a second storage device, thereby ensuring that the new data block and the new metadata entry can be simultaneously accessed without additional seek latency.

8. The computer-implemented method of claim 7, wherein the first storage device and the second storage device comprise separate RAIDs.

9. The computer-implemented method of claim 7,

wherein the amount of metadata for the distributed filesystem increases over time, and

wherein the method further comprises adjusting at least one of the size, number, or location of one or more metadata regions to accommodate the change in the set of metadata for the distributed filesystem.

10. The computer-implemented method of claim 9, wherein the method further involves defining the metadata region and the data region by dynamically weighting disk blocks in the local storage device towards metadata or data, thereby facilitating dynamically changing the size and borders of metadata and data regions.

11. The computer-implemented method of claim 10, wherein the location and organization for one or more metadata and data regions is determined using a locality policy.

12. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for managing metadata and data storage for a cloud controller in a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in a remote cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the remote cloud storage system, wherein the cloud controller includes a local storage device, wherein the local storage device comprises a rotating disk drive that comprises one or more disk platters;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein the metadata hierarchy is stored in the local storage device, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem; and

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the remote cloud storage system, wherein cloud controllers cache in their local storage devices a subset of the file data from the remote cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients, wherein new file data received by each cloud controller from its clients is written to the remote cloud storage system, wherein the metadata hierarchy in the cloud controller tracks the location of distributed filesystem data blocks in the remote cloud storage system and cached distributed filesystem data blocks in the cloud controller, wherein the cloud controller uses the metadata hierarchy to locate and download requested, uncached data blocks in the distributed filesystem from the remote cloud storage system;

defining in a disk platter of the rotating disk drive two or more metadata regions in which the cloud controller stores metadata for the distributed filesystem, wherein the metadata regions are distinct from two or more allocated data regions that are defined in the disk platter that cache distributed filesystem data, wherein different regions of the disk platter in the local storage device have different levels of performance, wherein a metadata region is defined in an outer region of the disk platter that supports the highest access bandwidth and lowest access latency;

receiving an incremental metadata snapshot that references new data written to the distributed filesystem;

storing a new metadata entry for the distributed filesystem from the incremental metadata snapshot in the metadata region on the disk platter; and

upon receiving a client request to access a new data block referenced in the incremental metadata snapshot, selecting a data region that is in near proximity to the metadata region and caching the new data block in that data region to ensure that the new metadata entry and the new data block are in relative proximity on the disk platter, thereby ensuring that associated metadata and data can be read without substantially degrading access performance, wherein the data region is distinct from the metadata region;

wherein the cloud controller predicts that the new metadata entry and the new data block are likely to be accessed frequently, wherein the cloud controller selects the metadata region and the data region for the new metadata entry and the new data block respectively because they are on an outer region of the disk platter and hence more favorable for frequent accesses, wherein outer regions of the disk platter have higher spatial density and hence higher effective data bandwidth that improves access rates for frequently accessed data stored in such regions.

13. The non-transitory computer-readable storage medium of claim 12, wherein the method further comprises:

receiving a request from a client to access the new data block referenced by the updated metadata;

using the metadata hierarchy to download a cloud file containing the data block from the remote cloud storage system; and

caching the new data block in the local storage device of the cloud controller.

14. The non-transitory computer-readable storage medium of claim 13,

wherein the metadata hierarchy grows as new data is added to the distributed filesystem, wherein the new data block written to the distributed filesystem is referenced by the new metadata entry in the metadata hierarchy;

and

wherein separating the metadata region from the data region facilitates preventing defragmentation of the data region by metadata over time.

15. The non-transitory computer-readable storage medium of claim 14,

wherein the set of data cached in the local storage device changes frequently based on the data accesses of one or more clients associated with the cloud controller;

wherein a metadata entry in the metadata hierarchy is deleted from the local storage device of the cloud controller if the data block associated with the metadata entry is deleted from the distributed filesystem; and

wherein the deletion rate for files of the distributed filesystem is substantially lower than the cache replacement rate for the distributed filesystem data cached in the local storage device.

16. The non-transitory computer-readable storage medium of claim 12, wherein the distribution of the metadata and data regions across the local storage device is prioritized based on an estimation of future metadata and data access trends.

17. The non-transitory computer-readable storage medium of claim 16,

wherein the local storage device comprises multiple storage devices;

and

wherein the method further comprises storing the new data block on a first storage device and the new metadata entry on a second storage device, thereby ensuring that the new data block and the new metadata entry can be simultaneously accessed without additional seek latency.

18. A cloud controller manages metadata and data storage for a distributed filesystem, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem, wherein the storage mechanism comprises a rotating disk drive that comprises one or more disk platters; and

a storage management mechanism;

wherein two or more cloud controllers collectively manage the data of the distributed filesystem, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in a remote cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the remote cloud storage system, wherein each cloud controller includes a local storage device;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein the metadata hierarchy is stored in the local storage device, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem; and

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the remote cloud storage system, wherein cloud controllers cache in their local storage devices a subset of the file data from the remote cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients, wherein new file data received by each cloud controller from its clients is written to the remote cloud storage system, wherein the metadata hierarchy in the cloud controller tracks the location of distributed filesystem data blocks in the remote cloud storage system and cached distributed filesystem data blocks in the cloud controller, wherein the cloud controller uses the metadata hierarchy to locate and download requested, uncached data blocks in the distributed filesystem from the remote cloud storage system; and

wherein the storage management mechanism is further configured to;

define in a disk platter of the rotating disk drive two or more metadata regions in which the cloud controller stores metadata for the distributed filesystem, wherein the metadata regions are distinct from two or more allocated data regions that are defined in the disk platter that cache distributed filesystem data, wherein different regions of the disk platter in the local storage device have different levels of performance, wherein a metadata region is defined in an outer region of the disk platter that supports the highest access bandwidth and lowest access latency;

receive an incremental metadata snapshot that references new data written to the distributed filesystem;

store a new metadata entry for the distributed filesystem from the incremental metadata snapshot in the metadata region on the disk platter; and

upon receiving a client request to access a new data block referenced in the incremental metadata snapshot, select a data region that is in near proximity to the metadata region and cache the new data block in that data region to ensure that the new metadata entry and the new data block are in relative proximity on the disk platter, thereby ensuring that associated metadata and data can be read without substantially degrading access performance, wherein the data region is distinct from the metadata region;

wherein the storage mechanism predicts that the new metadata entry and the new data block are likely to be accessed frequently, wherein the storage mechanism selects the metadata region and the data region for the new metadata entry and the new data block respectively because they are on an outer region of the disk platter and hence more favorable for frequent accesses, wherein outer regions of the disk platter have higher spatial density and hence higher effective data bandwidth that improves access rates for frequently accessed data stored in such regions.