Proactive optimizations at multi-tier file systems

• US 10,127,234 B1
• Filed: 03/27/2015
• Issued: 11/13/2018
• Est. Priority Date: 03/27/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Krishnan, Karthikeyan, Sait, Abdul Sathar
• Primary Examiner(s): Aspinwall, Evan
• Application Number:
  US14/672,086
• Time in Patent Office: 1,327 Days
• Field of Search: 706 12, 711170, 709213, 709224, 707602, 707741
• CPC Class Codes: G06F 16/119 Details of migration of file systems migration mechanisms in storage systems G06F3/0647
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1847 specifically adapted to static storage, e.g. adapted to flash memory or SSD
• CPC Class Codes: G06F 16/90324 using system suggestions
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A recommendations manager (RM) of a file system service identifies a file system object group (FSOG) for which proactive placement recommendations are to be generated using statistical analyzes. Usage metrics collected from the FSOG are used to train a model. Predictions obtained from the model are used to generate a recommendation to transfer a file system object proactively from one storage device group of the service to another. The recommendations are provided to an object migrator of the service to initiate the transfer.

Claims

1. A system, comprising:

a recommendations manager (RM) of a file system service, implemented at one or more computing devices of a provider network;

one or more object migrators of the file system service; and

a plurality of storage device groups (SDG) of the provider network, including a first SDG with a first storage performance capability level and a second SDG with a different storage performance capability level;

wherein the RM is configured to;

identify one or more file system object groups (FSOGs) for which respective sets of proactive placement recommendations (PPRs) are to be generated, including a first FSOG associated with a first client, and a second FSOG associated with a second client;

determine a first constraint on resources of the provider network to be used to generate a first set of PPRs for the first FSOG, and a second constraint on resources to be used to generate a second set of PPRs for the second FSOG;

obtain a first collection of usage metrics of the first FSOG, and a second collection of usage metrics of the second FSOG;

perform, (a) a first statistical analysis using the first collection of usage metrics and resources selected in accordance with the first constraint, and (b) a second statistical analysis using the second collection of usage metrics and resources selected in accordance with the first constraint;

generate, based at least in part on one or more predictions obtained from the first statistical analysis, the first set of PPRs, including at least a first recommendation to transfer a first file system object of the first FSOG from a particular storage device of the first SDG to a second storage device of the second SDG;

generate, based at least in part on one or more predictions obtained from the second statistical analysis, the second set of PPRs, including at least a second recommendation to transfer a second file system object of the second FSOG from a particular storage device of the second SDG to a different storage device of the first SDG; and

wherein the one or more object migrators are configured to;

initiate respective transfers of the first and second file system objects in accordance with the first and second recommendations.

2. The system as recited in claim 1, wherein the first constraint is based at least in part on one or more of:

(a) an identity of a client on whose behalf the first FSOG is created, (b) a size of the first FSOG, (c) an analysis of billing records associated with the first FSOG, (d) an available computing capacity at a machine learning service implemented at a provider network.

3. The system as recited in claim 1, wherein the first statistical analysis comprises an implementation of a first modeling methodology selected from a methodology set comprising:

(a) regression, (b) classification or (c) time series modeling, and wherein the second statistical analysis comprises an implementation of a different modeling methodology of the methodology set.

4. The system as recited in claim 1, wherein the RM is further configured to:

classify, using a clustering model, users of the file system service into a plurality of categories based at least in part on respective file access patterns of the users, wherein the first set of PRRs is based at least in part on a result of the clustering model.

5. The system as recited in claim 1, wherein to identify the first FSOG, the RM is configured to:

receive, via a programmatic interface of the file system service, an indication from a client of one or more file system objects for which the client has opted in for proactive placement recommendations.

6. A method, comprising:

performing, by a recommendations manager (RM) of a file system service whose data is distributed among a plurality of storage device groups (SDGs);

identifying one or more file system object groups (FSOGs) for which respective sets of proactive placement recommendations (PPRs) are to be generated using one or more machine learning models, including a first FSOG and a second FSOG;

training, (a) a first machine learning model using a first collection of usage metrics of the first FSOG, and (b) a second machine learning model using a second collection of usage metrics of the second FSOG; and

generating, based at least in part on one or more predictions obtained from the first machine learning model, a first set of PPRs for the first FSOG, including at least one recommendation to transfer a particular file system object of the first FSOG from its current SDG to a different SDG; and

generating, based at least in part on one or more predictions obtained from the second machine learning model, a second set of PPRs for the second FSOG, including at least one recommendation to transfer a particular file system object of the second FSOG from its current SDG to a different SDG.

7. The method as recited in claim 6, further comprising:

determining, by the RM, a first constraint on machine learning resources of a provider network to be used to generate the first set of PPRs, wherein the training of the first machine learning model is performed using resources selected in accordance with the first constraint.

8. The method as recited in claim 7, wherein the first constraint is based at least in part on one or more of:

(a) an identity of a client on whose behalf the first FSOG is created, (b) a size of the first FSOG, (c) an analysis of billing records associated with the first FSOG, (d) an available computing capacity at a machine learning service implemented at a provider network.

9. The method as recited in claim 7, further comprising:

generating, by the RM, an addition set of PRRs for the first FSOG using a set of resources selected in accordance with a different constraint, wherein the different constraint is determined by the RM subsequent to determining the first constraint.

10. The method as recited in claim 6, wherein the first machine learning model implements a first modeling methodology selected from a methodology set comprising:

(a) regression, (b) classification or (c) time series modeling, and wherein the second machine learning model implements a different modeling methodology of the methodology set.

11. The method as recited in claim 6, further comprising performing, by the RM:

classifying, using a clustering model, users of the first FSOG into a plurality of categories based at least in part on respective file access patterns of the users, wherein the first set of PRRs is based at least in part on a result of the clustering model.

12. The method as recited in claim 6, further comprising performing, by the RM:

determining, based at least in part on the first collection of usage metrics, that an application associated with the first FSOG is (a) I/O (input/output)-bound and (b) running at an execution platform with a first computing capacity; and

generating a recommendation to transfer the application to a different execution platform with a different computing capacity.

13. The method as recited in claim 6, wherein the plurality of SDGs comprises:

(a) locally-attached solid state drives (SSD) of instance hosts of the provider network, (b) locally-attached rotating magnetic disk drives of instance hosts of the provider network, (c) network-accessible solid state drives of a service implementing a block-level programmatic interface, (d) network-accessible magnetic disk drives of a service implementing a block-level programmatic interface, (e) devices of an object storage service implementing a web-services interface, (f) devices of a third-party storage service implemented at least in part outside the provider network, or (g) storage devices attached to un-virtualized hosts of the provider network.

14. The method as recited in claim 6, wherein the identifying of the first FSOG comprises:

receiving, via a programmatic interface, an indication from a client of one or more file system objects for which the client has approved generation of PPRs.

15. The method as recited in claim 6, further comprising performing, by the RM:

obtaining at least some usage metrics of the first collection of usage metrics from an operating-system-agnostic file access tracing module installed at one or more virtualization hosts of the provider network.

16. The method as recited in claim 6, further comprising performing, by the RM:

selecting a particular storage device of the different SDG to which the particular file system object of the first FSOG is to be transferred based at least in part on one of;

(a) a durability requirement associated with the first FSOG or (b) an availability requirement associated with the first FSOG.

17. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors implements a recommendations manager (RM) of a file system service, wherein the recommendations manager is configured to:

identify a file system object group (FSOG) for which one or more proactive placement recommendations (PPRs) are to be generated using one or more machine learning models;

train one or more machine learning models, including a first machine learning model, using a first collection of usage metrics of the first FSOG;

generate, based at least in part on one or more predictions obtained from the first machine learning model, a particular PPR to transfer a particular file system object of the first FSOG from a first storage device group of the file system service to a different storage device group; and

transmit the particular PPR to an object migrator component of the file system service configured to initiate the transfer of the particular file system object.

18. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the RM is further configured to:

determine a first constraint on machine learning resources of a provider network to be used to generate one or more PRRs for the first FSOG, wherein the training of the first machine learning model is performed using resources selected in accordance with the first constraint.

19. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the first constraint is based at least in part on one or more of:

(a) an identity of a client on whose behalf the first FSOG is created, (b) a size of the first FSOG, (c) an analysis of billing records associated with the first FSOG, (d) an available computing capacity at a machine learning service implemented at a provider network.

20. The non-transitory computer-accessible storage medium as recited in claim 17, wherein the machine learning model implements a first modeling methodology selected from a methodology set comprising:

(a) regression, (b) classification or (c) time series modeling.