Dynamic scheduling of distributed storage management tasks using predicted system characteristics

• US 10,168,953 B1
• Filed: 05/20/2016
• Issued: 01/01/2019
• Est. Priority Date: 05/20/2016
• Status: Active Grant

• Current Assignee: Nutanix Incorporated
• Original Assignee: Nutanix Incorporated
• Inventors: Gupta, Karan, Arora, Varun Kumar, Shukla, Himanshu, Beedu, Bharath Kumar, Nagpal, Abhinay
• Primary Examiner(s): Schnee, Hal
• Application Number:
  US15/160,246
• Time in Patent Office: 956 Days
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0629 Configuration or reconfiguration of storage systems
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06N 20/20 Ensemble learning

Abstract

Systems and methods for scheduling storage management tasks over predicted user tasks in a distributed storage system. A method commences upon receiving a set of historical stimulus records that characterize management tasks that are run in the storage system. A corresponding set of historical response records comprising system metrics associated with execution of the system tasks is also received. A learning model is formed from the stimulus records and the response records and formatted to be used as a predictor. A set of forecasted user tasks is input as new stimulus records to the predictor to determine a set of forecasted system metrics that would result from running the forecasted user tasks. Management tasks are selected so as not to impact the forecasted user tasks. Management tasks can be selected based on non-contentions resource usage between historical management task resource usage and predictions of resource usage by the user tasks.

Claims

1. A method for scheduling storage management tasks over predicted user tasks in a distributed storage system, the method comprising:

receiving a set of historical stimulus records, comprising system task data records that characterize one or more system tasks that have been executed on the distributed storage system that comprises at least a first node and a second node, wherein the distributed storage system comprising a plurality of storage devices of a cluster, wherein any node distributed across the cluster of nodes that has a controller virtual machine utilizes its respective controller virtual machine to read and write to content on the plurality of storage devices in a storage pool;

receiving, a set of historical response records comprising one or more system metrics associated with execution of the system tasks on the first node of the distributed storage system of the cluster, wherein a user task executed at the first node is observable at different nodes within the cluster, the one or more system metrics comprising a first portion that corresponds to measured metrics at the first node and a second portion that corresponds to results measured at a second node that are produced by executing the user task on the first node;

generating a prediction model for the distributed storage system of the cluster based on a learning model formed from at least two stimulus records of the set of historical stimulus records and at least two response records of the set of historical response records;

generating a set of forecasted user tasks predicted to be executed on the distributed storage system of cluster;

applying the set of forecasted user tasks as new stimulus records to the prediction model to determine a set of forecasted system metrics for the distributed storage system of cluster, the set of forecasted system metrics being predicted to result from running the set of forecasted user tasks on the distributed storage system of cluster the distributed storage system of cluster; and

selecting one or more distributed storage management tasks to be scheduled for execution on certain nodes of the distributed storage system of cluster based at least in part on a comparison between management task parameters and the set of forecasted system metrics, wherein the certain nodes of the distributed storage system are identified as being relevant to the one or more distributed storage management tasks.

2. The method of claim 1, wherein selecting the one or more management tasks is based at least in part on an inter-relationship between a first one of the one or more management tasks and a second one of the one or more management tasks.

3. The method of claim 2, wherein selecting the one or more management tasks is based at least in part on a time of release of resources.

4. The method of claim 1, wherein the learning model characterizes one or more correlations between at least two user tasks, management tasks, or system metrics.

5. The method of claim 1, wherein the management task parameters are based at least in part on historical data from running management tasks.

6. The method of claim 1, further comprising generating a task list based at least in part on the system metrics.

7. The method of claim 6, wherein the one or more management tasks are selected from the task list.

8. The method of claim 6, wherein a first set of the one or more management tasks are prioritized based at least in part on a set of scoring rules.

9. The method of claim 8, wherein the first set of the one or more management tasks do not exceed a set of resource constraints, the set of historical stimulus records comprise time-sequenced stimulus records, and the set of historical response records comprise time-sequenced response records.

10. The method of claim 6, further comprising determining an execution schedule for at least two of the one or more management tasks.

11. A computer readable medium, embodied in a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by a processor causes the processor to perform a set of acts for scheduling storage management tasks over predicted user tasks in a distributed storage system, the set of acts comprising:

receiving a set of historical stimulus records, comprising system task data records that characterize one or more system tasks that have been executed on the distributed storage system that comprises at least a first node and a second node, wherein the distributed storage system comprising a plurality of storage devices of a duster, wherein any node distributed across the cluster of nodes that has a controller virtual machine utilizes its respective controller virtual machine to read and write to content on the plurality of storage devices in a storage pool;

receiving, a set of historical response records comprising one or more system metrics associated with execution of the system tasks on the first node of the distributed storage system of the cluster, wherein a user task executed at the first node is observable at different nodes within the cluster, the one or more system metrics comprising a first portion that corresponds to measured metrics at the first node and a second portion that corresponds to results measured at a second node that are produced by executing the user task on the first node;

generating a prediction model for the distributed storage system of the cluster based on a learning model formed from at least two stimulus records of the set of historical stimulus records and at least two response records of the set of historical response records;

generating a set of forecasted user tasks predicted to be executed on the distributed storage system of cluster;

applying the set of forecasted user tasks as new stimulus records to the prediction model to determine a set of forecasted system metrics for the distributed storage system of cluster, the set of forecasted system metrics being predicted to result from running the set of forecasted user tasks on the distributed storage system of cluster the distributed storage system of cluster; and

selecting one or more distributed storage management tasks to be scheduled for execution on certain nodes of the distributed storage system of cluster based at least in part on a comparison between management task parameters and the set of forecasted system metrics, wherein the certain nodes of the distributed storage system are identified as being relevant to the one or more distributed storage management tasks.

12. The computer readable medium of claim 11, wherein selecting the one or more management tasks is based at least in part on an inter-relationship between a first one of the one or more management tasks and a second one of the one or more management tasks.

13. The computer readable medium of claim 12, wherein selecting the one or more management tasks is based at least in part on a time of release of resources.

14. The computer readable medium of claim 11, wherein the learning model characterizes one or more correlations between at least two user tasks, management tasks, or system metrics.

15. The computer readable medium of claim 11, wherein the management task parameters are based at least in part on historical data from running management tasks.

16. The computer readable medium of claim 11, wherein the set of acts further comprise generating a task list based at least in part on the system metrics.

17. The computer readable medium of claim 16, wherein the one or more management tasks are selected from the task list.

18. A system for scheduling storage management tasks over predicted user tasks in a distributed storage system comprising:

a storage medium having stored thereon a sequence of instructions; and

a processor or processors that execute the sequence of instructions to cause the processor or processors to perform a set of acts, the set of acts comprising;

receiving a set of historical stimulus records, comprising system task data records that characterize one or more system tasks that have been executed on the distributed storage system that comprises at least a first node and a second node, wherein the distributed storage system comprising a plurality of storage devices of a cluster, wherein any node distributed across the cluster of nodes that has a controller virtual machine utilizes its respective controller virtual machine to read and write to content on the plurality of storage devices in a storage pool;

receiving, a set of historical response records comprising one or more system metrics associated with execution of the system tasks on the first node of the distributed storage system of the cluster, wherein a user task executed at the first node is observable at different nodes within the cluster, the one or more system metrics comprising a first portion that corresponds to measured metrics at the first node and a second portion that corresponds to results measured at a second node that are produced by executing the user task on the first node;

generating a prediction model for the distributed storage system of the cluster based on a learning model formed from at least two stimulus records of the set of historical stimulus records and at least two response records of the set of historical response records;

generating a set of forecasted user tasks predicted to be executed on the distributed storage system of cluster;

applying the set of forecasted user tasks as new stimulus records to the prediction model to determine a set of forecasted system metrics for the distributed storage system of cluster, the set of forecasted system metrics being predicted to result from running the set of forecasted user tasks on the distributed storage system of cluster the distributed storage system of cluster; and

selecting one or more distributed storage management tasks to be scheduled for execution on certain nodes of the distributed storage system of duster based at least in part on a comparison between management task parameters and the set of forecasted system metrics, wherein the certain nodes of the distributed storage system are identified as being relevant to the one or more distributed storage management tasks.

19. The system of claim 18, wherein selecting the one or more management tasks is based at least in part on an inter-relationship between a first one of the one or more management tasks and a second one of the one or more management tasks.

20. The system of claim 19, wherein selecting the one or more management tasks is based at least in part on a time of release of resources.