Dynamic evaluation and selection of file system pre-fetch policy

• US 10,474,635 B1
• Filed: 12/21/2018
• Issued: 11/12/2019
• Est. Priority Date: 12/21/2018
• Status: Active Grant

• Current Assignee: Qumulo, Incorporated
• Original Assignee: Qumulo, Incorporated
• Inventors: Unger, Thomas R., Youngblut, Eric E., Rothschilds, Thomas Gregory, Drouillard, Jeremy
• Primary Examiner(s): Hasan, Syed H
• Application Number:
  US16/231,354
• Time in Patent Office: 326 Days
• CPC Class Codes: G06F 11/3457 Performance evaluation by simulation
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning

Abstract

Embodiments are directed to managing data in a file system. An engine may receive read requests associated with a file system that includes pre-fetch storage and file storage. The engine may employ a pre-fetch policy selected from a plurality of pre-fetch policies to copy blocks to pre-fetch storage. The engine may evaluate the selected pre-fetch policy and also evaluate simulated performances for each unselected pre-fetch policy based on subsequent read requests. The engine may provide a performance score based on the evaluation of the selected pre-fetch policy and it may provide other performance scores based on the simulated evaluation of each unselected pre-fetch policy If one of the other performance scores for one of the unselected pre-fetch policies exceeds the performance score of the selected pre-fetch policy, the engine may select that unselected pre-fetch policies as a new selected pre-fetch policy.

Claims

1. A method for managing data in a file system over a network using one or more processors that execute instructions to perform actions, comprising:

instantiating a pre-fetch engine to perform actions including;

receiving one or more read requests associated with the file system that includes a pre-fetch storage tier and a file storage tier of one or more storage devices;

employing a pre-fetch policy selected from a plurality of pre-fetch policies to copy one or more blocks to the pre-fetch storage tier, wherein the selection of the pre-fetch policy is based on one or more characteristics of the one or more read requests, and wherein each pre-fetch policy includes instructions for copying one or more blocks on the file storage tier to the pre-fetch storage tier;

evaluating a performance of the selected pre-fetch policy to fulfill one or more subsequent read requests and also evaluating a simulated performance of each unselected pre-fetch policy based on the one or more subsequent read requests;

providing a performance score based on the evaluation of the selected pre-fetch policy and providing one or more other performance scores based on the simulated evaluation of each unselected pre-fetch policy; and

when one of the other performance scores for one of the unselected pre-fetch policies exceeds the performance score of the selected pre-fetch policy, selecting the one of the unselected pre-fetch policies as a new selected pre-fetch policy for further one or more subsequent read requests, wherein the new selected pre-fetch policy improves efficiency and performance of the one or more processors by increasing an amount of the further one or more subsequent read requests being fulfilled by the one or more blocks copied to the pre-fetch storage tier.

2. The method of claim 1, wherein providing the performance score, further comprises:

increasing the performance score based on a number of the one or more blocks that are read by the one or more subsequent read requests;

decreasing the performance score based on a number of the one or more blocks that remain unread by the one or more subsequent read requests; and

decreasing the performance score based on a number of one or more other blocks located on the file storage tier that are read by the one or more subsequent read requests, wherein the one or more other blocks are unselected by the selected pre-fetch policy.

3. The method of claim 1, wherein determining the selected pre-fetch policy, further comprises:

instantiating a machine learning engine to perform actions including;

generating one or more input parameters based on the one or more characteristics of the one or more read requests;

providing the one or more input parameters to one or more machine learning models; and

determining the selected pre-fetch policy based on one or more results from the one or more machine learning models.

4. The method of claim 1, wherein the selection of the pre-fetch policy is based on its association with a highest performance score that is associated with one or more previous read requests that are similar to the one or more read requests.

5. The method of claim 1, wherein the pre-fetch engine performs further actions including:

storing one or more associations of performance scores and the one or more characteristics of the one or more read requests; and

further determining the selected pre-fetch policy based on the one or more associations.

6. The method of claim 1, wherein determining the selected pre-fetch policy, further comprises, selecting a no-copy pre-fetch policy that refrains from copying any blocks to the pre-fetch storage tier to minimize waste based on a prediction that pre-fetching the one or more blocks is unnecessary to improve performance.

7. The method of claim 1, wherein the pre-fetch engine performs further actions, including:

instantiating a machine learning engine to perform actions including;

training one or more machine learning models based on one or more previous read requests and one or more previous performance scores, wherein the one or more machine learning models are trained to select the pre-fetch policy based on a first read request; and

deploying the one or more trained machine learning models to select the pre-fetch policy based on the first read request.

8. The method of claim 1, wherein the one or more characteristics of the one or more read requests, includes, information that includes one or more of block identifiers, user information, directory or folder information, volume identifier, storage device type identifiers, application hints, filenames, file extensions, file size, time of day, file access properties, network addresses, or network ports.

9. A processor readable non-transitory storage media that includes instructions for managing data in a file system over a network, wherein execution of the instructions by one or more processors on one or more network computers performs actions, comprising:

instantiating a pre-fetch engine to perform actions including;

receiving one or more read requests associated with the file system that includes a pre-fetch storage tier and a file storage tier of one or more storage devices;

employing a pre-fetch policy selected from a plurality of pre-fetch policies to copy one or more blocks to the pre-fetch storage tier, wherein the selection of the pre-fetch policy is based on one or more characteristics of the one or more read requests, and wherein each pre-fetch policy includes instructions for copying one or more blocks on the file storage tier to the pre-fetch storage tier;

evaluating a performance of the selected pre-fetch policy to fulfill one or more subsequent read requests and also evaluating a simulated performance of each unselected pre-fetch policy based on the one or more subsequent read requests;

providing a performance score based on the evaluation of the selected pre-fetch policy and providing one or more other performance scores based on the simulated evaluation of each unselected pre-fetch policy; and

when one of the other performance scores for one of the unselected pre-fetch policies exceeds the performance score of the selected pre-fetch policy, selecting the one of the unselected pre-fetch policies as a new selected pre-fetch policy for further one or more subsequent read requests, wherein the new selected pre-fetch policy improves efficiency and performance of the one or more processors by increasing an amount of the further one or more subsequent read requests being fulfilled by the one or more blocks copied to the pre-fetch storage tier.

10. The media of claim 9, wherein providing the performance score, further comprises:

increasing the performance score based on a number of the one or more blocks that are read by the one or more subsequent read requests;

decreasing the performance score based on a number of the one or more blocks that remain unread by the one or more subsequent read requests; and

decreasing the performance score based on a number of one or more other blocks located on the file storage tier that are read by the one or more subsequent read requests, wherein the one or more other blocks are unselected by the selected pre-fetch policy.

11. The media of claim 9, wherein determining the selected pre-fetch policy, further comprises:

instantiating a machine learning engine to perform actions including;

generating one or more input parameters based on the one or more characteristics of the one or more read requests;

providing the one or more input parameters to one or more machine learning models; and

determining the selected pre-fetch policy based on one or more results from the one or more machine learning models.

12. The media of claim 9, wherein the selection of the pre-fetch policy is based on its association with a highest performance score that is associated with one or more previous read requests that are similar to the one or more read requests.

13. The media of claim 9, wherein the pre-fetch engine performs further actions including:

storing one or more associations of performance scores and the one or more characteristics of the one or more read requests; and

further determining the selected pre-fetch policy based on the one or more associations.

14. The media of claim 9, wherein determining the selected pre-fetch policy, further comprises, selecting a no-copy pre-fetch policy that refrains from copying any blocks to the pre-fetch storage tier to minimize waste based on a prediction that pre-fetching the one or more blocks is unnecessary to improve performance.

15. The media of claim 9, wherein the pre-fetch engine performs further actions, including:

instantiating a machine learning engine to perform actions including;

training one or more machine learning models based on one or more previous read requests and one or more previous performance scores, wherein the one or more machine learning models are trained to select the pre-fetch policy based on a first read request; and

deploying the one or more trained machine learning models to select the pre-fetch policy based on the first read request.

16. A system for managing data in a file system comprising:

a network computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

instantiating a pre-fetch engine to perform actions including;

receiving one or more read requests associated with the file system that includes a pre-fetch storage tier and a file storage tier of one or more storage devices;

employing a pre-fetch policy selected from a plurality of pre-fetch policies to copy one or more blocks to the pre-fetch storage tier, wherein the selection of the pre-fetch policy is based on one or more characteristics of the one or more read requests, and wherein each pre-fetch policy includes instructions for copying one or more blocks on the file storage tier to the pre-fetch storage tier;

evaluating a performance of the selected pre-fetch policy to fulfill one or more subsequent read requests and also evaluating a simulated performance of each unselected pre-fetch policy based on the one or more subsequent read requests;

providing a performance score based on the evaluation of the selected pre-fetch policy and providing one or more other performance scores based on the simulated evaluation of each unselected pre-fetch policy; and

when one of the other performance scores for one of the unselected pre-fetch policies exceeds the performance score of the selected pre-fetch policy, selecting the one of the unselected pre-fetch policies as a new selected pre-fetch policy for further one or more subsequent read requests, wherein the new selected pre-fetch policy improves efficiency and performance of the one or more processors by increasing an amount of the further one or more subsequent read requests being fulfilled by the one or more blocks copied to the pre-fetch storage tier; and

a client computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

providing one or more of the one or more read requests.

17. The system of claim 16, wherein providing the performance score, further comprises:

increasing the performance score based on a number of the one or more blocks that are read by the one or more subsequent read requests;

decreasing the performance score based on a number of the one or more blocks that remain unread by the one or more subsequent read requests; and

decreasing the performance score based on a number of one or more other blocks located on the file storage tier that are read by the one or more subsequent read requests, wherein the one or more other blocks are unselected by the selected pre-fetch policy.

18. The system of claim 16, wherein determining the selected pre-fetch policy, further comprises:

instantiating a machine learning engine to perform actions including;

generating one or more input parameters based on the one or more characteristics of the one or more read requests;

providing the one or more input parameters to one or more machine learning models; and

determining the selected pre-fetch policy based on one or more results from the one or more machine learning models.

19. The system of claim 16, wherein the selection of the pre-fetch policy is based on its association with a highest performance score that is associated with one or more previous read requests that are similar to the one or more read requests.

20. The system of claim 16, wherein the pre-fetch engine performs further actions including:

storing one or more associations of performance scores and the one or more characteristics of the one or more read requests; and

further determining the selected pre-fetch policy based on the one or more associations.

21. The system of claim 16, wherein determining the selected pre-fetch policy, further comprises, selecting a no-copy pre-fetch policy that refrains from copying any blocks to the pre-fetch storage tier to minimize waste based on a prediction that pre-fetching the one or more blocks is unnecessary to improve performance.

22. The system of claim 16, wherein the pre-fetch engine performs further actions, including:

instantiating a machine learning engine to perform actions including;

training one or more machine learning models based on one or more previous read requests and one or more previous performance scores, wherein the one or more machine learning models are trained to select the pre-fetch policy based on a first read request; and

deploying the one or more trained machine learning models to select the pre-fetch policy based on the first read request.

23. The system of claim 16, wherein the one or more characteristics of the one or more read requests, includes, information that includes one or more of block identifiers, user information, directory or folder information, volume identifier, storage device type identifiers, application hints, filenames, file extensions, file size, time of day, file access properties, network addresses, or network ports.

24. A network computer for managing data in a file system, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

instantiating a pre-fetch engine to perform actions including;

receiving one or more read requests associated with the file system that includes a pre-fetch storage tier and a file storage tier of one or more storage devices;

employing a pre-fetch policy selected from a plurality of pre-fetch policies to copy one or more blocks to the pre-fetch storage tier, wherein the selection of the pre-fetch policy is based on one or more characteristics of the one or more read requests, and wherein each pre-fetch policy includes instructions for copying one or more blocks on the file storage tier to the pre-fetch storage tier;

evaluating a performance of the selected pre-fetch policy to fulfill one or more subsequent read requests and also evaluating a simulated performance of each unselected pre-fetch policy based on the one or more subsequent read requests;

providing a performance score based on the evaluation of the selected pre-fetch policy and providing one or more other performance scores based on the simulated evaluation of each unselected pre-fetch policy; and

when one of the other performance scores for one of the unselected pre-fetch policies exceeds the performance score of the selected pre-fetch policy, selecting the one of the unselected pre-fetch policies as a new selected pre-fetch policy for further one or more subsequent read requests, wherein the new selected pre-fetch policy improves efficiency and performance of the one or more processors by increasing an amount of the further one or more subsequent read requests being fulfilled by the one or more blocks copied to the pre-fetch storage tier.

25. The network computer of claim 24, wherein providing the performance score, further comprises:

increasing the performance score based on a number of the one or more blocks that are read by the one or more subsequent read requests;

decreasing the performance score based on a number of the one or more blocks that remain unread by the one or more subsequent read requests; and

decreasing the performance score based on a number of one or more other blocks located on the file storage tier that are read by the one or more subsequent read requests, wherein the one or more other blocks are unselected by the selected pre-fetch policy.

26. The network computer of claim 24, wherein determining the selected pre-fetch policy, further comprises:

instantiating a machine learning engine to perform actions including;

generating one or more input parameters based on the one or more characteristics of the one or more read requests;

providing the one or more input parameters to one or more machine learning models; and

determining the selected pre-fetch policy based on one or more results from the one or more machine learning models.

27. The network computer of claim 24, wherein the selection of the pre-fetch policy is based on its association with a highest performance score that is associated with one or more previous read requests that are similar to the one or more read requests.

28. The network computer of claim 24, wherein the pre-fetch engine performs further actions including:

storing one or more associations of performance scores and the one or more characteristics of the one or more read requests; and

further determining the selected pre-fetch policy based on the one or more associations.

29. The method of claim 24, wherein determining the selected pre-fetch policy, further comprises, selecting a no-copy pre-fetch policy that refrains from copying any blocks to the pre-fetch storage tier to minimize waste based on a prediction that pre-fetching the one or more blocks is unnecessary to improve performance.

30. The network computer of claim 24, wherein the pre-fetch engine performs further actions, including:

instantiating a machine learning engine to perform actions including;

training one or more machine learning models based on one or more previous read requests and one or more previous performance scores, wherein the one or more machine learning models are trained to select the pre-fetch policy based on a first read request; and

deploying the one or more trained machine learning models to select the pre-fetch policy based on the first read request.