Avoiding client timeouts in a distributed filesystem

• US 9,852,150 B2
• Filed: 08/20/2013
• Issued: 12/26/2017
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Sharpe, Richard, Taylor, John Richard, Chou, Randy Yen-pang
• Primary Examiner(s): Rahman, Sm
• Application Number:
  US13/971,621
• Publication Number:
  US 20130339407A1
• Time in Patent Office: 1,589 Days
• Field of Search: 709234
• CPC Class Codes: G06F 12/0868 Data transfer between cache memory and other subsystems, e.g. storage devices or host systems
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 2212/314 In storage network, e.g. network attached cache
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

The disclosed embodiments disclose techniques that facilitate of avoiding client timeouts in a distributed filesystem. Multiple cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem in a local storage pool. During operation, a cloud controller receives from a client system a request for a data block in a target file that is stored in the distributed filesystem. Although the cloud controller is already caching the requested data block, the cloud controller delays transmission of the cached data block; this additional delay gives the cloud controller more time to access uncached data blocks for the target file from a cloud storage system, thereby ensuring that subsequent requests of such data blocks do not exceed a timeout interval on the client system.

Claims

1. A computer-implemented method for reducing client timeouts for a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in a cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage system, wherein each cloud controller includes a local storage space; and

caching data for the distributed filesystem in the local storage pool of each cloud controller, wherein due to space constraints each cloud controller cannot cache the entirety of the distributed filesystem in its local storage space, wherein the cloud controllers are configured to mask the latency of accessing the cloud storage system to provide clients with an abstraction that all of the data in the distributed filesystem is available locally;

receiving at a cloud controller a request from a client system for a target file in a set of less-frequently accessed files in the distributed filesystem, wherein the cloud controller caches one or more initial blocks for each file in the set to ensure that at least one block of data can be returned to a client that requests to access a file in the set without incurring the latency of contacting the cloud storage system;

determining that a first set of data blocks for the target file are currently cached in the local storage space of the cloud controller and that subsequent data blocks for the target file are not yet being cached by the cloud controller and are only available via the cloud storage system;

requesting the subsequent data blocks from the cloud storage system, wherein a local network connection between the client system and the cloud controller is less resource-constrained than a wide-area network connection between the cloud controller and the cloud storage system, wherein requesting a subsequent data block comprises estimating the time needed for the cloud storage system to receive the request and return the subsequent data block to the cloud controller, wherein the client is configured with a timeout interval and will abort the file request if any requested data block is not received within the timeout interval;

determining that the cached first block can be sent to the client within the timeout interval but that requesting a subsequent requested data block in the target file from the cloud storage system will exceed that subsequent request'"'"'s associated client timeout interval; and

delaying transmitting one or more cached data blocks for the target file that are already available at the cloud controller to the client system to ensure that the subsequent requested data blocks requested from the cloud storage system are received by the cloud controller from the cloud storage system and can be sent to the client system before their respective timeout intervals on the client system are exceeded, wherein the spacing of the transmission of the cached data blocks is determined based on network characteristics for the wide-area network connection, wherein the cloud controller delays the transmission of the cached data block but ensures that this delay does not result in exceeding the client timeout interval associated with the request for the cached data block.

2. The computer-implemented method of claim 1,

wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers for the distributed filesystem to ensure that the clients of the distributed filesystem share a consistent view of the files in the distributed filesystem;

wherein the two or more cloud controllers collectively present a unified namespace for the distributed filesystem to the clients of the distributed filesystem, wherein the clients access the distributed filesystem via the cloud controllers, wherein the file data for the distributed filesystem is stored in the cloud storage system, wherein cloud controllers cache in their local storage devices a subset of the file data from the cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients, wherein new file data received by each cloud controller from its clients is written to the cloud storage system; and

wherein delaying the transmission of a cached data block for the target file from the cloud controller to the client system facilitates providing more time for the cloud controller to access one or more uncached data blocks for the target file from a cloud storage system, thereby ensuring that subsequent requests of uncached data by the client system do not exceed a timeout interval on the client system.

3. The computer-implemented method of claim 2, wherein delaying the transmission of the cached data block further comprises:

calculating a delay interval for sending the requested data block from the cloud controller to the client system; and

sending the requested data block from the cloud controller to the client system after the delay interval to ensure that the uncached data blocks can be retrieved by the cloud controller from the cloud storage system before the timeout interval on the client system is exceeded.

4. The computer-implemented method of claim 3, wherein calculating the delay interval further comprises:

determining the number of data blocks for the target file that are not cached by the cloud controller;

determining an expected arrival rate for an uncached data block; and

setting the delay interval to ensure that, upon receiving a subsequent request for the uncached data block from the client system, the cloud controller has sufficient time to receive the uncached data block from the cloud storage system and forward the uncached data block to the client system before the timeout interval is exceeded on the client system.

5. The computer-implemented method of claim 4, wherein calculating the delay interval further comprises:

tracking in the cloud controller a round-trip delay that measures the amount of time needed to receive previously requested data blocks from the cloud storage system; and

dynamically adjusting the delay interval for one or more subsequent requests of data blocks based on the round-trip delay for the previously requested data blocks.

6. The computer-implemented method of claim 5, wherein calculating the delay interval further comprises:

using metadata for the distributed filesystem to;

determine a cloud file that includes the uncached data block; and

determine the location of the uncached data block in the cloud file to determine the portions of the cloud file that need to be transferred to the cloud controller; and

determining the delay interval based on the amount of data from the cloud file that needs to be transferred to the cloud controller to access the uncached data block, wherein the amount of cloud file data to be transferred substantially exceeds the size of the uncached data block.

7. The computer-implemented method of claim 6, wherein calculating the delay interval further comprises using conservative estimates for the round-trip delay for an initial set of uncached data blocks to ensure that changes in network characteristics and cloud controller load that delay the transmission of the uncached data block do not result in a timeout on the client system.

8. The computer-implemented method of claim 6, wherein using a conservative estimate comprises adding an additional safety margin to the delay interval.

9. The computer-implemented method of claim 6, wherein dynamically adjusting the delay interval comprises considering:

an estimated network bandwidth available to the cloud controller;

the number of uncached data blocks for the target file that the cloud controller has received from the cloud storage system;

the number of uncached data blocks for the target file that remain to be downloaded to the cloud controller; and

the rate at which the cloud controller is receiving requests from the client system for data blocks of the target file.

10. The computer-implemented method of claim 6, wherein the method further involves:

tracking file and data block access patterns on the cloud controller; and

using the tracked access patterns to determine the set of data blocks to be cached on the cloud controller.

11. The computer-implemented method of claim 10,

wherein a first set of cached data blocks for the target file are transferred from the cloud controller to the client system without delay to reduce client-perceived file access latency;

and

wherein a second set of cached data blocks for the target file are delayed for the delay interval to ensure that additional uncached data blocks for the target file can be accessed by the client system without exceeding the timeout interval.

12. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for reducing client timeouts for a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in a cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage system, wherein each cloud controller includes a local storage space; and

caching data for the distributed filesystem in the local storage space of each cloud controller, wherein due to space constraints each cloud controller cannot cache the entirety of the distributed filesystem in its local storage space, wherein the cloud controllers are configured to mask the latency of accessing the cloud storage system to provide clients with an abstraction that all of the data in the distributed filesystem is available locally;

receiving at a cloud controller a request from a client system for a target file in a set of less-frequently accessed files in the distributed filesystem, wherein the cloud controller caches one or more initial blocks for each file in the set to ensure that at least one block of data can be returned to a client that requests to access a file in the set without incurring the latency of contacting the cloud storage system;

determining that a first set of data blocks for the target file are currently cached in the local storage space of the cloud controller and that subsequent data blocks for the target file are not yet being cached by the cloud controller and are only available via the cloud storage system;

requesting the subsequent data blocks from the cloud storage system, wherein a local network connection between the client system and the cloud controller is less resource-constrained than a wide-area network connection between the cloud controller and the cloud storage system, wherein requesting a subsequent data block comprises estimating the time needed for the cloud storage system to receive the request and return the subsequent data block to the cloud controller, wherein the client is configured with a timeout interval and will abort the file request if any requested data block is not received within the timeout interval;

determining that the cached first block can be sent to the client within the timeout interval but that requesting a subsequent requested data block in the target file from the cloud storage system will exceed that subsequent request'"'"'s associated client timeout interval; and

delaying transmitting one or more cached data blocks for the target file that are already available at the cloud controller to the client system to ensure that the subsequent requested data blocks requested from the cloud storage system are received by the cloud controller from the cloud storage system and can be sent to the client system before their respective timeout intervals on the client system are exceeded, wherein the spacing of the transmission of the cached data blocks is determined based on network characteristics for the wide-area network connection, wherein the cloud controller delays the transmission of the cached data block but ensures that this delay does not result in exceeding the client timeout interval associated with the request for the cached data block.

13. The non-transitory computer-readable storage medium of claim 12, wherein delaying the transmission of a cached data block for the target file from the cloud controller to the client system facilitates providing more time for the cloud controller to access one or more uncached data blocks for the target file from a cloud storage system, thereby ensuring that subsequent requests of uncached data by the client system do not exceed a timeout interval on the client system.

14. The non-transitory computer-readable storage medium of claim 13, wherein delaying the transmission of the cached data block further comprises:

determining that a requested data block is currently cached by the cloud controller;

calculating a delay interval for sending the requested data block from the cloud controller to the client system; and

sending the requested data block from the cloud controller to the client system after the delay interval to ensure that the uncached data blocks can be retrieved by the cloud controller from the cloud storage system before the timeout interval on the client system is exceeded.

15. The non-transitory computer-readable storage medium of claim 14, wherein calculating the delay interval further comprises:

determining the number of data blocks for the target file that are not cached by the cloud controller,

determining an expected arrival rate for an uncached data block; and

setting the delay interval to ensure that, upon receiving a subsequent request for the uncached data block from the client system, the cloud controller has sufficient time to receive the uncached data block from the cloud storage system and forward the uncached data block to the client system before the timeout interval is exceeded on the client system.

16. The non-transitory computer-readable storage medium of claim 15, wherein calculating the delay interval further comprises:

tracking in the cloud controller a round-trip delay that measures the amount of time needed to receive previously requested data blocks from the cloud storage system; and

dynamically adjusting the delay interval for one or more subsequent requests of data blocks based on the round-trip delay for the previously requested data blocks.

17. The non-transitory computer-readable storage medium of claim 16, wherein calculating the delay interval further comprises:

using metadata for the distributed filesystem to;

determine a cloud file that includes the uncached data block; and

determine the location of the uncached data block in the cloud file to determine the portions of the cloud file that need to be transferred to the cloud controller; and

determining the delay interval based on the amount of data from the cloud file that needs to be transferred to the cloud controller to access the uncached data block, wherein the amount of cloud file data to be transferred substantially exceeds the size of the uncached data block.

18. The non-transitory computer-readable storage medium of claim 17, wherein calculating the delay interval further comprises using conservative estimates for the round-trip delay for an initial set of uncached data blocks to ensure that changes in network characteristics and cloud controller load that delay the transmission of the uncached data block do not result in a timeout on the client system.

19. The non-transitory computer-readable storage medium of claim 17, wherein using a conservative estimate comprises adding an additional safety margin to the delay interval.

20. A cloud controller that reduces client timeouts for a distributed filesystem, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem; and

a storage management mechanism that includes a local storage space that caches distributed filesystem data, wherein due to space constraints each cloud controller cannot cache the entirety of the distributed filesystem in its local storage space, wherein the cloud controllers are configured to mask the latency of accessing the cloud storage system to provide clients with an abstraction that all of the data in the distributed filesystem is available locally;

wherein two or more cloud controllers collectively manage the data of the distributed filesystem, wherein the cloud controllers cache and ensure data consistency for data stored in a cloud storage system;

wherein the cloud controller is configured to;

receive a request from a client system for a target file in a set of less-frequently accessed files in the distributed filesystem, wherein the cloud controller caches one or more initial blocks for each file in the set to ensure that at least one block of data can be returned to a client that requests to access a file in the set without incurring the latency of contacting the cloud storage system;

determine that a first set of data blocks for the target file are currently cached in the local storage space of the cloud controller and that subsequent data blocks for the target file are not yet being cached by a cloud controller and are only available via the cloud storage system;

request the subsequent data blocks from the cloud storage system, wherein a local network connection between the client system and the cloud controller is less resource-constrained than a wide-area network connection between the cloud controller and the cloud storage system, wherein requesting a subsequent data block comprises estimating the time needed for the cloud storage system to receive the request and return the subsequent data block to the cloud controller, wherein the client is configured with a timeout interval and will abort the file request if any requested data block is not received within the timeout interval;

determine that the cached first block can be sent to the client within the timeout interval but that requesting a subsequent requested data block in the target file from the cloud storage system will exceed that subsequent request'"'"'s associated client timeout interval; and

delay transmitting one or more cached blocks for the target file that are already available at the cloud controller to the client system to ensure that the subsequent requested data blocks are received by the cloud controller from the cloud storage system and can be sent to the client system before their respective timeout intervals on the client system are exceeded, wherein the spacing of the transmission of the cached data blocks is determined based on network characteristics for the wide-area network connection, wherein the cloud controller delays the transmission of the cached data block but ensures that this delay does not result in exceeding the client timeout interval associated with the request for the cached data block.