Distributed storage data repair air via partial data rebuild within an execution path

• US 9,201,735 B1
• Filed: 06/25/2014
• Issued: 12/01/2015
• Est. Priority Date: 06/25/2014
• Status: Active Grant

• Current Assignee: Rubrik, Inc.
• Original Assignee: Igneous Systems, Inc.
• Inventors: Rakitzis, Triantaphyllos Byron, Lemar, Eric Michael, Hughes, Jeffrey Douglas, Bhageshpur, Kiran V., Engineer, Anu
• Primary Examiner(s): Puente, Emerson
• Assistant Examiner(s): WU, BENJAMIN C
• Application Number:
  US14/315,139
• Time in Patent Office: 524 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 9/48 Program initiating; Program switching, e.g. by interrupt
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Embodiments are directed towards managing the distribution of tasks in a storage system. An execution path for tasks may be generated based on the type of the task and characteristics of the storage system such that the execution path includes storage computers in a storage system. The tasks may be provided to each storage computer in the execution path. A working set of intermediate results may be generated on the storage computer in the execution path. If there is more than one storage computer in the execution path, working sets may be iteratively communicated to a next storage computer in the execution path such that the next storage computer employs a previously generated working set to generate a next working set until each storage computer in the execution path has been employed to generate a working set. The results may be stored on the storage computers.

Claims

1. A method for managing tasks for a storage system over a network using a network computer that executes instructions that perform actions, comprising:

generating an execution path for at least one task based on a type of the at least one task and at least one characteristic of the storage system, wherein the execution path includes a plurality of storage computers of the storage system;

providing the at least one task to each storage computer in the execution path, wherein each storage computer comprises a CPU complex and a storage device; and

employing the CPU complexes of the plurality of storage computers, to perform actions, including;

communicating current status information from the plurality of storage computers to the network computer, wherein the status information is based on monitoring performance of the at least one task; and

when one or more conditions for execution of the at least one task are met for execution by each storage computer in the execution path, performing further actions, including;

generating a working set of intermediate results from the at least one task executing on data from a storage computer initially in the execution path;

iteratively communicating a previously generated working set to a next storage computer in the execution path, wherein the next storage computer employs the previously generated working set and data from the next storage computer to generate a next working set of intermediate results for the at least one task executing on the next storage computer'"'"'s data until each storage computer'"'"'s data in the execution path has been employed to generate each corresponding working set;

when each storage computer in the execution path is in a single physical chassis, determining a distributed execution path within the single physical chassis to provide working set information to each of the storage computers in the single physical chassis;

generating at least one result for the at least one task based on each working set from each storage computer in the execution path; and

storing the at least one result in at least one of the storage computers in the storage system.

2. The method of claim 1, wherein generating each working set further comprises, generating erasure coding information as part of a storage repair task.

3. The method of claim 1, wherein storing the at least one result in the at least one storage computer in the storage system further comprises, employing the at least one task to determine storing of the at least one result on a storage computer other than one of each storage computer in the execution path.

4. The method of claim 1, wherein generating the execution path, further comprises, determining the execution path based on at least one network characteristic, including at least one of network congestion, network bandwidth, network latency, error rate, jitter, quality-of-service, or throughput.

5. The method of claim 1, further comprising:

storing at least one data chunk to at least one destination storage computer; and

employing the at least one data chunk stored on each destination storage computer to generate each working set.

6. The method of claim 1, wherein providing the at least one task to each storage computer in the execution path further comprises, concurrently providing two or more separate tasks to at least one storage computer.

7. The method of claim 1, wherein generating the execution path, further comprises, determining each storage device in the same storage computer to include in the execution path.

8. The method of claim 1, further comprising, if the at least one task includes at least one task portion that is independent of the execution of other task portions, those task portions are executed as soon as the task is obtained by each storage computer.

9. A system for managing tasks for a storage system over a network, comprising:

a network computer comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

generating an execution path for at least one task based on a type of the at least one task and at least one characteristic of the storage system, wherein the execution path includes a plurality of storage computers of the storage system;

providing the at least one task to each storage computer in the execution path, wherein each storage computer comprises a CPU complex and a storage device; and

employing the CPU complexes of the plurality of storage computers, to perform actions, including;

communicating current status information from the plurality of storage computers to the network computer, wherein the status information is based on monitoring performance of the at least one task; and

when one or more conditions for execution of the at least one task are met for execution by each storage computer in the execution path, performing further actions, including;

generating a working set of intermediate results from the at least one task executing on data from a storage computer initially in the execution path;

iteratively communicating a previously generated working set to a next storage computer in the execution path, wherein the next storage computer employs the previously generated working set and data from the next storage computer to generate a next working set of intermediate results for the at least one task executing on the next storage computer'"'"'s data until each storage computer'"'"'s data in the execution path has been employed to generate each corresponding working set;

when each storage computer in the execution path is in a single physical chassis, determining a distributed execution path within the single physical chassis to provide working set information to each of the storage computers in the single physical chassis;

generating at least one result for the at least one task based on each working set from each storage computer in the execution path; and

storing the at least one result in at least one of the storage computers in the storage system; and

a client computer, comprising;

a transceiver that is operative to communicate over the network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

providing the at least one task to the network computer.

10. The system of claim 9, wherein generating each working set further comprises, generating erasure coding information as part of a storage repair task.

11. The system of claim 9, wherein storing the at least one result in the at least one storage computer in the storage system further comprises, employing the at least one task to determine storing of the at least one result on a storage computer other than one of each storage computer in the execution path.

12. The system of claim 9, wherein generating the execution path, further comprises, determining the execution path based on at least one network characteristic, including at least one of network congestion, network bandwidth, network latency, error rate, jitter, quality-of-service, or throughput.

13. The system of claim 9, wherein the network computer processor device is operative to enable actions, further comprising:

storing at least one data chunk to at least one destination storage computer; and

employing the at least one data chunk stored on each destination storage computer to generate each working set.

14. The system of claim 9, wherein providing the at least one task to each storage computer in the execution path further comprises, concurrently providing two or more separate tasks to at least one storage computer.

15. The system of claim 9, wherein generating the execution path, further comprises, determining each storage device in the same storage computer to include in the execution path.

16. The system of claim 9, further comprising, if the at least one task includes at least one task portion that is independent of the execution of other task portions, those task portions are executed as soon as the task is obtained by each storage computer.

17. A processor readable non-transitory storage media that includes instructions for managing tasks for a storage system over a network, wherein execution of the instructions by a processor device enables actions, comprising:

generating an execution path for at least one task based on a type of the at least one task and at least one characteristic of the storage system, wherein the execution path includes a plurality of storage computers of the storage system;

providing the at least one task to each storage computer in the execution path, wherein each storage computer comprises a CPU complex and a storage device; and

employing the CPU complexes of the plurality of storage computers, to perform actions, including;

communicating current status information from the plurality of storage computers to the network computer, wherein the status information is based on monitoring performance of the at least one task; and

when one or more conditions for execution of the at least one task are met for execution by each storage computer in the execution path, performing further actions, including;

generating a working set of intermediate results from the at least one task executing on data from a storage computer initially in the execution path;

iteratively communicating a previously generated working set to a next storage computer in the execution path, wherein the next storage computer employs the previously generated working set and data from the next storage computer to generate a next working set of intermediate results for the at least one task executing on the next storage computer'"'"'s data until each storage computer'"'"'s data in the execution path has been employed to generate each corresponding working set;

when each storage computer in the execution is in a single physical chassis, determining a distributed execution path within the single physical chassis to provide working set information to each of the storage computers in the single physical chassis;

generating at least one result for the at least one task based on each working set from each storage computer in the execution path; and

storing the at least one result in at least one of the storage computers in the storage system.

18. The media of claim 17, wherein generating each working set further comprises, generating erasure coding information as part of a storage repair task.

19. The media of claim 17, wherein storing the at least one result in the at least one storage computer in the storage system further comprises, employing the at least one task to determine storing of the at least one result on a storage computer other than one of each storage computer in the execution path.

20. The media of claim 17, wherein generating the execution path, further comprises, determining the execution path based on at least one network characteristic, including at least one of network congestion, network bandwidth, network latency, error rate, jitter, quality-of-service, or throughput.

21. The media of claim 17, further comprising:

storing at least one data chunk to at least one destination storage computer; and

employing the at least one data chunk stored on each destination storage computer to generate each working set.

22. The media of claim 17, wherein providing the at least one task to each storage computer in the execution path further comprises, concurrently providing two or more separate tasks to at least one storage computer.

23. The media of claim 17, wherein generating the execution path, further comprises, determining each storage device in the same storage computer to include in the execution path.

24. A network computer that is operative for managing tasks for a storage system over a network using a network computer, comprising:

a transceiver that is operative to communicate over a network;

a memory that is operative to store at least instructions; and

a processor device that is operative to execute instructions that enable actions, including;

generating an execution path for at least one task based on a type of the at least one task and at least one characteristic of the storage system, wherein the execution path includes a plurality of storage computers of the storage system;

providing the at least one task to each storage computer in the execution path, wherein each storage computer comprises a CPU complex and a storage device; and

employing the CPU complexes of the plurality of storage computers, to perform actions, including;

communicating current status information from the plurality of storage computers to the network computer, wherein the status information is based on monitoring performance of the at least one task; and

when one or more conditions for execution of the at least one task are met for execution by each storage computer in the execution path, performing further actions, including;

 

generating a working set of intermediate results from the at least one task executing on data from a storage computer initially in the execution path;

 

iteratively communicating a previously generated working set to a next storage computer in the execution path, wherein the next storage computer employs the previously generated working set and data from the next storage computer to generate a next working set of intermediate results for the at least one task executing on the next storage computer'"'"'s data until each storage computer'"'"'s data in the execution path has been employed to generate each corresponding working set;

 

when each storage computer in the execution path is in a single physical chassis, determining a distributed execution path within the single physical chassis to provide working set information to each of the storage computers in the single physical chassis;

 

generating at least one result for the at least one task based on each working set from each storage computer in the execution path; and

 

storing the at least one result in at least one of the storage computers in the storage system.

25. The network computer of claim 24, wherein generating each working set further comprises, generating erasure coding information as part of a storage repair task.

26. The network computer of claim 24, wherein storing the at least one result in the at least one storage computer in the storage system further comprises, employing the at least one task to determine storing of the at least one result on a storage computer other than one of each storage computer in the execution path.

27. The network computer of claim 24, wherein generating the execution path, further comprises, determining the execution path based on at least one network characteristic, including at least one of network congestion, network bandwidth, network latency, error rate, jitter, quality-of-service, or throughput.

28. The network computer of claim 24, wherein the network computer processor device is operative to enable actions, further comprising:

storing at least one data chunk to at least one destination storage computer; and

employing the at least one data chunk stored on each destination storage computer to generate each working set.

29. The network computer of claim 24, wherein providing the at least one task to each storage computer in the execution path further comprises, concurrently providing two or more separate tasks to the at least one storage computer.

30. The network computer of claim 24, further comprising, if the at least one task includes at least one task portion that is independent of the execution of other task portions, those task portions are executed as soon as the task is obtained by each storage computer.