Throughput throttling for distributed file storage services with varying connection characteristics

• US 9,860,317 B1
• Filed: 04/30/2015
• Issued: 01/02/2018
• Est. Priority Date: 04/30/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Gupta, Tarang
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): Krishnan, Raji
• Application Number:
  US14/701,464
• Time in Patent Office: 978 Days
• CPC Class Codes: H04L 45/125 based on throughput or bandwidth
• CPC Class Codes: H04L 47/25 with rate being modified by the source upon detecting a change of network conditions
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

During a throttling parameter determination, a throttling manager of a file storage service transmits a maximum throughput allotment to an access node of the service. The allotment is based at least in part on the estimated size of a file system instance. The access node assigns respective portions of the allotment to individual ones of a set of persistent client connections classified as sustainable-demand connections, such that the connections are able to sustain their throughputs for a time period. The remainder of the allotment is distributed among one or more high-demand persistent client connections. The access node enforces the throughput limits assigned to the connections.

Claims

1. A system, comprising:

a plurality of access nodes (ANs) of a distributed file storage service implemented at a provider network, including a first AN designated as a member of a first AN group collectively responsible for processing received client requests directed to at least a first file system instance (FSI); and

one or more throttling managers implemented at one or more computing devices, including a first throttling manager associated with at least the first FSI;

wherein the first throttling manager is configured to perform a plurality of throttling parameter determination iterations (TPDIs), wherein a particular TPDI of the plurality of TPDIs comprises transmitting, to the first AN, a throttling parameter set including a maximum throughput allotment (MTA) for the first FSI, wherein the MTA is determined based at least in part on one or more of (a) an estimated size of the first FSI or (b) a population of a token collection associated with the first FSI, wherein the tokens of the token collection correspond to respective amounts of throughput; and

wherein the first AN is configured to;

classify, based at least in part on respective sets of throughput metrics, individual ones of a plurality of persistent connections established to enable clients of the file storage service to access the first FSI into (a) a sustainable-demand (SD) category or (b) a high-demand (HD) category;

assign, to one or more SD persistent connections, a portion of the MTA enabling the respective SD persistent connections to sustain respective throughput rates for at least a particular time corresponding to the particular TPDI;

assign, to one or more HD persistent connections for at least the particular time, a portion of a remainder throughput allotment, wherein the remainder throughput allotment is obtained by subtracting, from the MTA, at least a sum of throughputs assigned to the one or more SD persistent connections; and

enforce throughput assignments of the plurality of persistent connections.

2. The system as recited in claim 1, wherein the first throttling manager is further configured to:

receive, from the first AN, an indication of a measured throughput associated with a particular connection of the plurality of persistent connections;

estimate, based on inputs received from one or more ANs including the first AN, an aggregate throughput with respect to the first FSI;

remove one or more tokens from the token collection based at least in part on the aggregate throughput; and

add one or more tokens to the token collection based at least in part on a token refill rate associated with the first FSI.

3. The system as recited in claim 1, wherein the first AN group comprises a second AN, wherein the first throttling manager is configured to:

transmit, to the second AN during the first TPDI, a second throttling parameter set; and

receive, from the first AN and the second AN, respective indications of measured throughputs of one or more persistent connections associated with the first FSI.

4. The system as recited in claim 1, wherein the distributed file storage service comprises a second AN associated with a second FSI, wherein the first throttling manager is configured to:

transmit, to the second AN during a different TPDI associated with the second FSI, a different throttling parameter set corresponding to the second FSI; and

receive, from the second AN, respective indications of measured throughputs of one or more persistent connections established at the second AN and associated with the second FSI.

5. The system as recited in claim 1, wherein the distributed file storage service comprises a second throttling manager associated with the first FSI, wherein the first throttling manager is configured to:

transmit, to the second throttling manager, a query for cached throughput metrics pertaining to at least the first FSI; and

receive, from the second throttling manager, a response comprising one or more throughput metrics obtained at the second throttling manager from a different AN.

6. A method, comprising:

performing, by a first throttling manager associated with a first file system instance of a distributed file storage service, throttling parameter determinations, wherein a particular throttling parameter determination comprises transmitting, to a first access node of a plurality of access nodes of the service associated with the first file system instance, a throttling parameter set including a maximum throughput allotment for the first file system instance, wherein the maximum throughput allotment is determined based at least in part on an updated estimate of a size of the first file system instance, wherein the updated estimate is obtained for the particular throttling parameter determination;

assigning, by the first access node, to one or more sustainable-demand connections of a plurality of persistent connections established for client access to the first file system instance, a respective portion of the maximum throughput allotment, enabling the respective sustainable-demand connections to sustain respective throughput rates for at least a particular time;

assigning, by the first access node, to one or more high-demand connections of the plurality of persistent connections, for at least the particular time, a portion of a remainder throughput allotment, wherein the reminder throughput allotment is obtained by subtracting, from the maximum throughput allotment, at least a sum of throughputs associated with the one or more sustainable-demand connections; and

enforcing throughput assignments of the plurality of persistent connections.

7. The method as recited in claim 6, further comprising:

classifying, by the first access node based at least in part on one or more of (a) a throughput metric of a particular connection of the plurality of persistent connections or (b) a parameter indicated in the throttling parameter set, the particular connection into a sustainable-demand category or a high-demand category.

8. The method as recited in claim 6, wherein the maximum throughput allotment is further based at least in part on a population of a token collection associated with the first file system instance.

9. The method as recited in claim 8, further comprising:

receiving, by the first throttling manager from the first access node, an indication of a measured throughput associated with a particular connection of the plurality of persistent connections;

estimating, by the first throttling manager based on inputs received from one or more access nodes including the first access node, an aggregate throughput with respect to the first file system instance;

removing, by the first throttling manager, one or more tokens from the token collection based at least in part on the aggregate throughput; and

adding, by the first throttling manager, one or more tokens to the token collection based at least in part on a token refill rate associated with the first file system instance.

10. The method as recited in claim 9, further comprising:

modifying, by the first throttling manager, the token refill rate based at least in part on another updated estimate of the size of the first file system instance.

11. The method as recited in claim 6, wherein the distributed file storage service comprises a plurality of throttling managers including a second throttling manager associated with the first file system instance, further comprising:

storing, by the first throttling manager, at least a portion of the throttling parameter set in a persistent repository accessible by the second throttling manager.

12. The method as recited in claim 6, wherein the distributed file storage service comprises a second access node associated with the first file system instance, further comprising:

transmitting, by the first throttling manager to the second access node, a second throttling parameter set; and

transmitting, to the first throttling manager by the first access node and the second AN, respective indications of measured throughputs of one or more persistent connections associated with the first file system instance.

13. The method as recited in claim 6, wherein the distributed file storage service comprises a second access node associated with a second file system instance, further comprising:

transmitting, by the first throttling manager to the second access node, a different throttling parameter set corresponding to the second file system instance; and

transmitting, to the first throttling manager by the second access node, respective indications of measured throughputs of one or more persistent connections established at the second access node and associated with the second file system instance.

14. The method as recited in claim 6, wherein the distributed file storage service comprises a second throttling manager associated with the first file system instance, further comprising:

transmitting, to the second throttling manager by the first throttling manager a query for throughput metrics pertaining to at least the first file system instance; and

receiving, from the second throttling manager at the first throttling manager, a response comprising one or more throughput metrics obtained at the second throttling manager from a different AN.

15. The method as recited in claim 14, wherein the first throttling manager is implemented at a host associated with a first availability container of a provider network, and wherein the second throttling manager is implemented at a different host associated with a second availability container of the provider network.

16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implement a first access node of a file storage service, wherein the first access node is a member of a first access node group comprising more than one access nodes collectively responsible for processing received client requests directed to at least a first file system instance, wherein the first access node is configured to perform a plurality of throughput throttling determinations, wherein a particular throughput throttling determination of the plurality of throughput throttling determinations comprises:

classifying, based at least in part on respective sets of throughput metrics, individual ones of a plurality of persistent connections associated with the first file system instance into (a) a sustainable-demand category, or (b) a high-demand category;

receiving an indication of a maximum throughput allocation associated with the first file system instance, wherein the maximum throughput allotment is based at least in part on an estimated size of the first file system instance;

assigning, to one or more sustainable-demand persistent connections, a portion of the maximum throughput allotment, to enable the respective sustainable-demand persistent connections to sustain respective throughput rates for at least a particular time;

assigning, to one or more high-demand persistent connections, respective portions of a remainder throughput allotment, wherein the remainder throughput allotment is obtained by subtracting, from the maximum throughput allotment, at least a sum of throughputs assigned to the one or more sustainable-demand persistent connections; and

enforcing throughput assignments of the plurality of persistent connections.

17. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the client requests directed to at least the first file system instance are formatted in accordance with one of:

(a) a version of an NFS (Network File System) protocol or (b) a version of an SMB (Server Message Block) protocol.

18. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the indication of the maximum throughput allotment is received at the first access node from a first throttling manager of the file system service, and wherein the first access node is configured to transmit, to the first throttling manager, respective indications of throughput metrics of one or more persistent connections associated with the first file system instance.

19. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implement a first throttling manager of a file storage service, wherein the first throttling manager is configured to perform a plurality of throttling parameter determinations, wherein a particular throttling parameter determination of the plurality of throttling parameter determinations comprises:

receiving respective throughput updates from one or more access nodes of the file storage service, wherein a particular throughput update comprises an indication of a measured throughput of at least one persistent connection established to access a first file system instance at a first access node of the one or more access nodes;

computing a maximum throughput allotment for the first file system instance based at least in part on one or more of (a) an estimate of a size of the first file system instance or (b) a population of a token collection associated with the first file system instance; and

transmitting a first throttling parameter set to at least one access node of the one or more access nodes.

20. The non-transitory computer-accessible storage medium as recited in claim 19, wherein the particular throttling parameter determination comprises:

estimating an aggregate measured throughput with respect to the first file system instance;

removing one or more tokens from the token collection based at least in part on the aggregate throughput; and

adding one or more tokens to the token collection based at least in part on a token refill rate associated with the first file system instance.

21. The non-transitory computer-accessible storage medium as recited in claim 20, wherein the first throttling manager is further configured to:

modify the token refill rate based at least in part on an updated estimate of the size of the first file system instance.

22. The non-transitory computer-accessible storage medium as recited in claim 19, wherein the one or more access nodes include a second access node associated with the first file system instance and a third access node associated with a second file system instance, wherein the first throttling manager is further configured to:

transmit, to the second access node, a second throttling parameter set corresponding to the first file system instance; and

transmit, to the third access node, a third throttling parameter set corresponding to the second file system instance.