Distributed storage data repair air via partial data rebuild within an execution path

• US 10,203,986 B2
• Filed: 11/30/2015
• Issued: 02/12/2019
• Est. Priority Date: 06/25/2014
• Status: Active Grant

• Current Assignee: Rubrik, Inc.
• Original Assignee: Igneous Systems, Inc.
• Inventors: Rakitzis, Triantaphyllos Byron, Lemar, Eric Michael, Hughes, Jeffrey Douglas, Bhageshpur, Kiran V., Engineer, Anu
• Primary Examiner(s): Wu, Benjamin C
• Application Number:
  US14/954,616
• Publication Number:
  US 20160085588A1
• Time in Patent Office: 1,170 Days
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 9/48 Program initiating; Program switching, e.g. by interrupt
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Claims

1. A method for managing tasks for a storage system over a network using a network computer that executes instructions that perform actions, comprising:

employing two or more of a plurality of storage computers in a single physical chassis of two or more physical chassis of the storage system to provide a distributed ordering of an execution path for one or more tasks based on a type of a plurality of operations that are used to perform the one or more tasks when each storage computer in the execution path is also in the single physical chassis, wherein the distributed ordering of the execution path is based on one or more factors that optimize performance of the type of operations used for the one or more tasks including limiting a number of times that traversal of the execution path requires accessing an external network to the single physical chassis that is slower than an internal network to the single physical chassis, and wherein one or more processors of the two or more of the plurality of storage computers execute instructions to perform actions, including;

communicating current status information from each of the plurality of storage computers in the single physical chassis to the network computer, wherein the current status information is based on monitoring performance of the one or more tasks; and

employing one or more conditions for execution of the one or more tasks that are met for execution by each storage computer in the distributed ordering of the execution path to perform further actions, including;

employing a first storage computer in the distributed ordering of the execution path to provide a working set of intermediate results for the one or more tasks executing on data from the first storage computer;

iteratively communicating the provided working set of intermediate results to a next storage computer in the distributed ordering of the execution path, wherein the next storage computer employs the provided working set and data from the next storage computer to generate a next working set of intermediate results for the one or more tasks executing on the next storage computer'"'"'s data until each storage computer'"'"'s data in the distributed ordering of the execution path is employed to provide a corresponding working set of intermediate results based on a previously provided working set of intermediate results; and

providing one or more results for the one or more tasks based on the corresponding working sets of intermediate results provided by each storage computer in the distributed ordering of the execution path.

2. The method of claim 1, wherein providing the working set of intermediate results further comprises, providing erasure coding information as part of a storage repair task.

3. The method of claim 1, further comprising storing the one or more results includes on one or more other storage computers that are separate from the two or more storage computers in the distributed ordering of the execution path.

4. The method of claim 1, wherein the optimization of the performance of the execution path is further based on one or more of:

a type of the one or more tasks;

a characteristic of the storage system;

or

a characteristic of the network; and

wherein the type of operations include one or more of associative or direct sequence.

5. A storage system for managing tasks over a network, comprising:

a network computer comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store logic; and

a plurality of processors that are operative to execute the logic to perform actions, including;

enabling two or more of a plurality of storage computers in a single physical chassis of two or more physical chassis of the storage system to provide a distributed ordering of an execution path for one or more tasks based on a type of a plurality of operations that are used to perform the one or more tasks when each storage computer in the execution path is also in the single physical chassis, wherein the distributed ordering of the execution path is based on one or more factors that optimize performance of the type of operations used for the one or more tasks including limiting a number of times that traversal of the execution path requires accessing an external network to the single physical chassis that is slower than an internal network to the single physical chassis;

enabling one or more of the plurality of storage computers to communicate current status information from each of the plurality of storage computers in the single physical chassis to the network computer, wherein the current status information is based on monitoring performance of the one or more tasks; and

enabling one or more processors of the two or more of the plurality of storage computers, to perform actions, including;

employing one or more conditions for execution of the one or more tasks that are met for execution by each storage computer in the distributed ordering of the execution path, performing further actions, including;

employing a first storage computer in the distributed ordering of the execution path to provide a working set of intermediate results for the one or more tasks executing on data from the first storage computer;

iteratively communicating the provided working set of intermediate results to a next storage computer in the distributed ordering of the execution path, wherein the next storage computer employs the provided working set and data from the next storage computer to generate a next working set of intermediate results for the one or more tasks executing on the next storage computer'"'"'s data until each storage computer'"'"'s data in the distributed ordering of the execution path is employed to provide a corresponding working set of intermediate results based on a previously provided working set of intermediate results; and

providing one or more results for the one or more tasks to the network computer, wherein the one or more results are based on the corresponding working sets of intermediate results provided by each storage computer in the distributed ordering of the execution path.

6. The system of claim 5, wherein providing the working set of intermediate results further comprises, providing erasure coding information as part of a storage repair task.

7. The system of claim 5, further comprising storing the one or more results includes on one or more other storage computers that are separate from the two or more storage computers in the distributed ordering of the execution path.

8. The system of claim 5, wherein the optimization of the performance of the execution path is further based on one or more of:

a type of the one or more tasks;

a characteristic of the storage system;

or

a characteristic of the network; and

wherein the type of operations include one or more of associative or direct sequence.

9. A processor readable non-transitory storage media that includes logic for managing tasks for a storage system over a network, wherein execution of the logic by one or more processors performs actions, comprising:

employing two or more of a plurality of storage computers in a single physical chassis of two or more physical chassis of the storage system to provide a distributed ordering of an execution path for one or more tasks based on a type of a plurality of operations that are used to perform the one or more tasks when each storage computer in the execution path is also in the single physical chassis, wherein the distributed ordering of the execution path is based on one or more factors that optimize performance of the type of operations used for the one or more tasks including limiting a number of times that traversal of the execution path requires accessing an external network to the single physical chassis that is slower than an internal network to the single physical chassis;

employing one or more of the plurality of storage computers to communicate current status information from each of the plurality of storage computers in the single physical chassis to the network computer, wherein the current status information is based on monitoring performance of the one or more tasks; and

employing one or more processors of the two or more of the plurality of storage computers, to perform actions, including;

employing one or more conditions for execution of the one or more tasks that are met for execution by each storage computer in the distributed ordering of the execution path, performing further actions, including;

employing a first storage computer in the execution path to provide a working set of intermediate results for the one or more tasks executing on data from the first storage computer;

iteratively communicating the provided working set of intermediate results to a next storage computer in the distributed ordering of the execution path, wherein the next storage computer employs the provided working set and data from the next storage computer to generate a next working set of intermediate results for the one or more tasks executing on the next storage computer'"'"'s data until each storage computer'"'"'s data in the distributed ordering of the execution path is employed to provide a corresponding working set of intermediate results based on a previously provided working set of intermediate results; and

providing one or more results for the one or more tasks based on the corresponding working sets of intermediate results provided by each storage computer in the distributed ordering of the execution path.

10. The processor readable non-transitory storage media of claim 9, wherein providing the working set of intermediate results further comprises, providing erasure coding information as part of a storage repair task.

11. The processor readable non-transitory storage media of claim 9, further comprising storing the one or more results includes on one or more other storage computers that are separate from the two or more storage computers in the distributed ordering of the execution path.

12. The processor readable non-transitory storage media of claim 9, wherein the optimization of the performance of the execution path is further based on one or more of:

a type of the one or more tasks;

a characteristic of the storage system;

or

a characteristic of the network; and

wherein the type of operations include one or more of associative or direct sequence.

13. A storage computer that is operative for managing tasks in a storage system over a network, wherein the storage system includes a plurality of storage computers, and wherein each storage computer comprises:

a transceiver that is operative to communicate over a network;

a memory that is operative to store logic; and

one or more processors that are operative to execute instructions that enable actions, including;

enabling two or more of a plurality of storage computers in a single physical chassis of two or more physical chassis of the storage system to provide a distributed ordering of an execution path for one or more tasks based on a type of a plurality of operations that are used to perform the one or more tasks when each storage computer in the execution path is also in the single physical chassis, wherein the distributed ordering of the execution path is based on one or more factors that optimize performance of the type of operations used for the one or more tasks including limiting a number of times that traversal of the execution path requires accessing an external network to the single physical chassis that is slower than an internal network to the single physical chassis;

communicating current status information from each of the plurality of storage computers in the single physical chassis to the network computer, wherein the current status information is based on monitoring performance of the one or more tasks; and

enabling one or more processors of the two or more of the plurality of storage computers, to enable actions, including;

employing one or more conditions for execution of the one or more tasks that are met for execution by each storage computer in the distributed ordering of the execution path to perform further actions, including;

employing a first storage computer in the execution path to provide a working set of intermediate results for the one or more tasks executing on data from the first storage computer;

iteratively communicating of the provided working set of intermediate results to a next storage computer in the execution path, wherein the next storage computer employs the provided working set and data from the next storage computer to generate a next working set of intermediate results for the one or more tasks executing on the next storage computer'"'"'s data until each storage computer'"'"'s data in the distributed ordering of the execution path is employed to provide a corresponding working set of intermediate results based on a previously provided working set of intermediate results; and

providing of one or more results for the one or more tasks based on the corresponding working sets of intermediate results provided by each storage computer in the distributed ordering of the execution path.

14. The storage computer of claim 13, wherein providing the working set of intermediate results further comprises, providing erasure coding information as part of a storage repair task.

15. The storage computer of claim 13, further comprising storing the one or more results includes on one or more other storage computers that are separate from the two or more storage computers in the distributed ordering of the execution path.

16. The storage computer of claim 13, wherein the optimization of the performance of the execution path is further based on one or more of:

a type of the one or more tasks;

a characteristic of the storage system;

or

a characteristic of the network; and

wherein the type of operations include one or more of associative or direct sequence.