Restoring an archived file in a distributed filesystem

• US 9,804,928 B2
• Filed: 09/05/2013
• Issued: 10/31/2017
• Est. Priority Date: 11/14/2011
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Davis, Andrew P., Parkison, Brian Christopher, Taylor, John Richard
• Primary Examiner(s): Rahman, S M
• Application Number:
  US14/019,270
• Publication Number:
  US 20140006357A1
• Time in Patent Office: 1,517 Days
• Field of Search: 709219
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

The disclosed embodiments disclose techniques for restoring an archived file in a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. Furthermore, cloud controllers may archive infrequently-accessed files in an archival cloud storage system. During operation, a cloud controller receives a request from a client system to access an archived file, and restores this archived file from the archival cloud storage system.

Claims

1. A computer-implemented method for restoring an archived file in a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem;

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the cloud storage systems, wherein cloud controllers cache in their local storage devices a subset of the file data from the remote cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients, wherein new file data received by each cloud controller from its clients is written to the cloud storage systems; and

archiving one or more infrequently accessed files for the distributed filesystem in an archival cloud storage system;

determining a billing model that discloses cost trade-offs for restore operations for the archival cloud storage system;

receiving at a cloud controller a request from a client system to access an archived file in the distributed filesystem, wherein the cloud controller tracks one or more distributed restore operations that are currently being performed by other cloud controllers for the distributed filesystem upon the archival cloud storage system; and

requesting to restore the archived file from the archival cloud storage system, wherein requesting to restore the archived file comprises;

determining a set of restore options for the archived file based on the billing model, wherein the set of restore options span a range of different restore time intervals and costs that are available for restoring the archived file; and

adjusting the restore behavior for the distributed filesystem by contacting one or more other cloud controllers for the distributed filesystem to collectively adjust one or more of the distributed restore operations for the distributed filesystem to ensure that the full set of restore operations for the distributed filesystem do not exceed a cost and bandwidth constraint for the billing model.

2. The computer-implemented method of claim 1, wherein receiving the request further involves:

receiving in the request a filename for a file in the distributed filesystem;

determining from metadata for the distributed filesystem that the file has been archived in the archival cloud storage system;

notifying the client system that the file is archived; and

receiving a restore request for the archived file from the client system.

3. The computer-implemented method of claim 1, wherein restoring the archived file comprises one or more of:

transferring one or more cloud files containing data associated with the archived file from the archival cloud storage system to a second cloud storage system; and

transferring the data blocks for the restored file to the cloud controller.

4. The computer-implemented method of claim 3, wherein restoring the archived file comprises:

tracking in the cloud controller status information for the cloud controller and the distributed filesystem; and

using the tracked status information to estimate a restore time interval for the archived file.

5. The computer-implemented method of claim 4, wherein restoring the archived file further comprises presenting the tracked status information and the estimated restore time interval to the client system via a status file that is associated with the archived file.

6. The computer-implemented method of claim 5,

wherein the status file represents a directory in the distributed filesystem and comprises status information and a restore time interval for multiple archived files in a sub-hierarchy of the distributed filesystem that can be accessed via the directory;

wherein the request from the client system initiates a restore operation for all of the multiple archived files in the sub-hierarchy; and

wherein the sub-hierarchy includes the archived file.

7. The computer-implemented method of claim 5, wherein restoring the archived file further comprises:

adjusting the restore behavior for the distributed filesystem to balance the performance and cost of restore operations across the cloud controllers for the distributed filesystem; and

ensuring that the cloud controllers collectively do not exceed a maximum restore bandwidth specified by the billing model for the archival cloud storage system.

8. The computer-implemented method of claim 7, wherein adjusting the restore behavior for the distributed filesystem comprises:

presenting the set of restore options to the client system using the status file;

and receiving a selected restore option from the client system.

9. The computer-implemented method of claim 7,

wherein the billing model comprises billing for restored data based on the maximum restore bandwidth that is used to access the archival cloud storage system in a given time interval;

and

wherein adjusting the restore behavior for the distributed filesystem comprises coordinating the restore bandwidth that is used by the set of cloud controllers that collectively manage the data of the distributed filesystem based on the billing model to ensure that restore operations for the distributed filesystem do not exceed a specified budget.

10. The computer-implemented method of claim 9, wherein tracking status information for the distributed filesystem further comprises tracking outstanding restore requests and in-progress restore operations for multiple restore operations that are distributed across multiple cloud controllers for the distributed filesystem in the context of the billing model to provide an estimated restore time interval that meets the budget constraints of the distributed filesystem.

11. The computer-implemented method of claim 10,

wherein a restore request for the archived file indicates a high priority for the archived file and an increased restore budget for the distributed filesystem;

and

wherein restoring the archived file further comprises increasing the maximum restore bandwidth that is used to access the archival cloud storage system to reduce the restore time interval for the archived file.

12. The computer-implemented method of claim 10,

wherein a restore request for the archived file indicates a high priority restore operation for the archived file;

and

wherein restoring the archived file further comprises adjusting the scheduling of pending and in-progress restore operations across other distributed cloud controllers for the distributed filesystem to reduce the estimated restore time interval for the archived file.

13. The computer-implemented method of claim 7, wherein the method further comprises:

checking for an update to the billing model for the archival cloud storage system;

retrieving an updated billing model for the archival cloud storage system; and

adjusting the schedule of pending and in-progress restore operations for the distributed filesystem to optimize restore performance for the distributed filesystem while ensuring that the restore operations for the distributed filesystem do not exceed the specified budget in the context of the updated billing model.

14. The computer-implemented method of claim 7,

wherein the billing model imposes a minimum storage term for a set of stored data;

wherein the billing model determines the restore bandwidth available for the distributed filesystem based on the amount of distributed filesystem data stored in the archival cloud storage system; and

wherein, upon receiving a request to delete the set of stored data from the archival cloud storage system before the end of the minimum storage term, the method further comprises delaying the deletion of data in the archival cloud storage system for the extent of the minimum storage term to maximize the restore bandwidth available for the distributed filesystem.

15. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for restoring an archived file in a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem;

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the cloud storage systems, wherein cloud controllers cache in their local storage devices a subset of the file data from the remote cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients, wherein new file data received by each cloud controller from its clients is written to the cloud storage systems; and

archiving one or more infrequently accessed files for the distributed filesystem in an archival cloud storage system;

determining a billing model that discloses cost trade-offs for restore operations for the archival cloud storage system;

receiving at a cloud controller a request from a client system to access an archived file in the distributed filesystem, wherein the cloud controller tracks one or more distributed restore operations that are currently being performed by other cloud controllers for the distributed filesystem upon the archival cloud storage system; and

requesting to restore the archived file from the archival cloud storage system, wherein requesting to restore the archived file comprises;

determining a set of restore options for the archived file based on the billing model, wherein the set of restore options span a range of different restore time intervals and costs that are available for restoring the archived file; and

adjusting the restore behavior for the distributed filesystem by contacting one or more other cloud controllers for the distributed filesystem to collectively adjust one or more of the distributed restore operations for the distributed filesystem to ensure that the full set of restore operations for the distributed filesystem do not exceed a cost and bandwidth constraint for the billing model.

16. The non-transitory computer-readable storage medium of claim 15, wherein receiving the request further involves:

receiving in the request a filename for a file in the distributed filesystem;

determining from metadata for the distributed filesystem that the file has been archived in the archival cloud storage system;

notifying the client system that the file is archived; and

receiving a restore request for the archived file from the client system.

17. The non-transitory computer-readable storage medium of claim 15, wherein restoring the archived file comprises one or more of:

transferring one or more cloud files containing data associated with the archived file from the archival cloud storage system to a second cloud storage system; and

transferring the data blocks for the restored file to the cloud controller.

18. The non-transitory computer-readable storage medium of claim 17, wherein restoring the archived file comprises:

tracking in the cloud controller status information for the cloud controller and the distributed filesystem; and

using the tracked status information to estimate a restore time interval for the archived file.

19. The non-transitory computer-readable storage medium of claim 18, wherein restoring the archived file further comprises presenting the tracked status information and the estimated restore time interval to the client system via a status file that is associated with the archived file.

20. A cloud controller that restores an archived file in a distributed filesystem, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem; and

a storage management mechanism;

wherein two or more cloud controllers collectively manage the data of the distributed filesystem by;

storing the data for the distributed filesystem in one or more cloud storage systems, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage systems;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem; and

collectively presenting a unified namespace for the distributed filesystem to the clients of the distributed filesystem via the two or more cloud controllers, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the cloud storage systems, wherein cloud controllers cache in their local storage devices a subset of the file data from the remote cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients, wherein new file data received by each cloud controller from its clients is written to the cloud storage systems;

wherein the cloud controller is configured to archive one or more infrequently accessed files for the distributed filesystem in an archival cloud storage system;

wherein the cloud controller is configured to determine a billing model that discloses cost trade-offs for restore operations for the archival cloud storage system;

wherein the cloud controller is further configured to receive a request from a client system to access an archived file in the distributed filesystem, wherein the cloud controller tracks one or more distributed restore operations that are currently being performed by other cloud controllers for the distributed filesystem upon the archival cloud storage system; and

wherein the cloud controller is further configured to request to restore the archived file from the archival cloud storage system, wherein requesting to restore the archived file comprises the cloud controller;

determining a set of restore options for the archived file based on the billing model, wherein the set of restore options span a range of different restore time intervals and costs that are available for restoring the archived file; and

adjusting the restore behavior for the distributed filesystem by contacting one or more other cloud controllers for the distributed filesystem to collectively adjust one or more of the distributed restore operations for the distributed filesystem to ensure that the full set of restore operations for the distributed filesystem do not exceed a cost and bandwidth constraint for the billing model.