Functional fail-over apparatus and method of operation thereof

• US 6,934,880 B2
• Filed: 11/21/2001
• Issued: 08/23/2005
• Est. Priority Date: 11/21/2001
• Status: Active Grant

• Current Assignee: Dell Global B.V. - Singapore Branch (Dell Technologies Inc.)
• Original Assignee: Exanet, Inc. (Dell Technologies Inc.)
• Inventors: Hofner, Robert
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): PUENTE, EMERSON C
• Application Number:
  US09/989,231
• Publication Number:
  US 20030097610A1
• Time in Patent Office: 1,371 Days
• Field of Search: 714/4, 714/6, 714/25, 714/10, 714/11, 714/12, 714/13
• US Class Current: 714/10
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A system and method for a failover system where in case of a failure of an element within the system, only that element is shutdown, rather then shutting down an entire node. The solution is of particular use in network systems, networked storage systems as well as location independent file systems.

Claims

1. A system comprising:

a first network node and a second network node connected via a communication link;

a plurality of processes resident on said first network node and presently executing;

a plurality of monitors for said plurality of processes, each of said plurality of processes having one corresponding monitor from said plurality of monitors, said corresponding monitors resident on said second network node and presently executing,

each monitor being configured for detecting failure of its corresponding process on said first network node and, if its corresponding process has failed, causing said failed process to execute on said second network node as a new process having a corresponding monitor executing on said first network node, while the remaining processes continue to execute on said first network node.

2. The system of claim 1, wherein said first and second network nodes are central processing units.

3. The system of claim 1, wherein said first and second network nodes are computer hosts.

4. The system of claim 1, wherein said first and second network nodes are computer servers.

5. The system of claim 1, wherein said first and second network nodes are storage nodes.

6. The system of claim 1, wherein said first and second network nodes are printer nodes.

7. The system of claim 1, wherein said first and second network nodes are file systems.

8. The system of claim 1, wherein said first and second network nodes are location independent file systems.

9. The system of claim 1, wherein said communication link is a local area network.

10. The system of claim 1, wherein said communication link is a wide area network.

11. The system of claim 1, wherein said monitors periodically check their corresponding processes executing on said first network node in order to detect a failure.

12. The system of claim 11, wherein said periodic checking comprises sending a key to said corresponding process and receiving a predefined response.

13. The system of claim 11, wherein said periodic checking comprises monitoring heartbeat signals sent at a periodic rate.

14. The system of claim 11, wherein, when one of said corresponding monitors detects a failure, said one corresponding monitor initiates a process swap, said process swap comprising:

terminating said failed process from executing on said first network node;

initiating said failed process as a new process executing on said second network node;

initiating a new corresponding monitor on said first network node that monitors said new process executing on said second network node; and

terminating said one corresponding monitor executing on said second network node.

15. The system of claim 1, wherein said process is selected from the group consisting of a service, a task and a thread.

16. A system comprising:

a first plurality of network nodes connected via a first communication link;

a second plurality of network nodes connected via a second communication link;

said first communication link and said second communication link connected through a third communication link,

a plurality of processes resident on one of the network nodes and presently executing;

a plurality of monitors for said plurality of processes, each of said plurality of processes having one corresponding monitor from said plurality of monitors, said corresponding monitors resident on another one of the network nodes and presently executing,

each corresponding monitor being configured for detecting failure of its corresponding process and, if its corresponding process has failed, causing said failed process to execute on another of the network nodes as a new process having a corresponding monitor executing on said one network node while the remaining processes continue to execute on said one network node.

17. The system of claim 16, wherein said network nodes are central processing units.

18. The system of claim 16, wherein said network nodes are computer hosts.

19. The system of claim 16, wherein said network nodes are computer servers.

20. The system of claim 16, wherein said network nodes are storage nodes.

21. The system of claim 16, wherein said network nodes are printer nodes.

22. The system of claim 16, wherein said network nodes are file systems.

23. The system of claim 16, wherein said network nodes are location independent file systems.

24. The system of claim 16, wherein said first communication link and said second communication link are local area networks.

25. The system of claim 16, wherein said third communication link is a wide area network.

26. The system of claim 16, wherein said corresponding monitors periodically check their corresponding processes executing on said one node of said first plurality of network nodes in order to detect a failure.

27. The system of claim 26, wherein said periodic checking comprises sending a key to said corresponding process and receiving a predefined response.

28. The system of claim 26, wherein said periodic checking comprises monitoring heartbeat signals sent at a periodic rate.

29. The system of claim 26, wherein, when one of said corresponding monitors detects a failure, said one corresponding monitor initiates a process swap, said process swap comprising:

terminating said failed process from executing on said one network node;

transferring and initiating said failed process as a new process executing on another network node;

initiating a new corresponding monitor on another network node that is not the same node as the node to which the failed process was transferred, wherein said new corresponding monitor monitors said new process; and

terminating said corresponding monitor executing on said one network node.

30. The system of claim 29, wherein, if said failed process initially executed on a network node connected to said first communication link, then said new process execution is initiated on a network node connected to said second communication link.

31. The system of claim 29, wherein, if said failed process initially executed on a network node connected to said second communication link, then said new process execution is initiated on a network node connected to said first communication link.

32. The system of claim 29, wherein if said one corresponding monitor initially executed on a network node connected to said first communication link, then execution of said new corresponding monitor is initiated on a node connected to said second communication link.

33. The system of claim 29, wherein if said one corresponding monitor initially executed on a network node connected to said second communication link, then execution of said new corresponding monitor is initiated on a network node connected to said first communication link.

34. The system of claim 16, wherein said process is selected from the group consists of a service, a task and a thread.

35. A method for operating a failover system, wherein failover does not require the termination of all the processes executing on a first network node, the method comprising:

executing a plurality of processes resident on the first network node;

executing a plurality of monitors for said plurality of processes, each of said plurality of processes having one corresponding monitor from said plurality of monitors resident on a second network node, said second network node connected to said first network node via a communications link;

periodically checking the operation of said plurality of processes, said periodic checking performed by said plurality of corresponding monitors;

if an execution failure of a process of said plurality of processes is detected by its corresponding monitor, then

terminating execution of said failed process from executing on said first network node while the remaining processes continue to execute on said first network node;

transferring and initiating execution of said failed process as a new process on said second network node;

initiating execution of a new corresponding monitor for said new process on said first network node; and

terminating said corresponding monitor of said failed monitor from executing on said first network node.

36. The method of claim 35, wherein said first and second network nodes are a central processing units.

37. The method of claim 35, wherein said first and second network nodes are computer hosts.

38. The method of claim 35, wherein said first and second network nodes are computer servers.

39. The method of claim 35, wherein said first and second network nodes are storage nodes.

40. The method of claim 35, wherein said first and second network nodes are printer nodes.

41. The method of claim 35, wherein said first and second network nodes are file systems.

42. The method of claim 35, wherein said first and second network nodes are location independent file systems.

43. The method of claim 35, wherein said communication link is a local area network.

44. The method of claim 35, wherein said communication link is a wide area network.

45. The method of claim 35, wherein said process is selected from the group consisting of a service, a task and a thread.

46. A computer system for controlling failover so that the termination of all the executing processes is not required, the computer system comprising:

a first network node and a second network node;

a memory comprising software instructions that enable the computer system to perform;

executing a plurality of processes resident on the first network node;

executing a plurality of monitors for said plurality of processes, each of said plurality of processes having one corresponding monitor from said plurality of monitors resident on a second network node, said second network node connected to said first network node via a communications link;

periodically checking the operation of said plurality of processes, said periodic checking performed by said plurality of corresponding monitors;

if an execution failure of one process of said plurality of processes is detected by its corresponding monitor, then

terminating execution of said failed process from executing on said first network node while the remaining processes continue to execute on said first network node;

transferring and initiating execution of said failed process as a new process on said second network node;

initiating execution of a new corresponding monitor for said new process on said first network node; and

terminating said corresponding monitor of said failed process from executing on said first network node.

47. A computer software product for a computer system comprising a first network node and a second network node to control failover so that the termination of all the processes executing on said first network node is not required, the computer program product comprising:

software instructions for enabling the computer system to perform predetermined operations, and a computer readable medium bearing the software instructions, said predetermined operations comprising;

executing a plurality of processes resident on the first network node;

executing a plurality of monitors for said plurality of processes, each of said plurality of processes having one corresponding monitor from said plurality of monitors resident on a second network node, said second network node connected to said first network node via a communications link;

periodically checking the operation of said plurality of processes, said periodic checking performed by said plurality of corresponding monitors;

if an execution failure of a process of said plurality of processes is detected its corresponding monitor, then

terminating execution of said failed process from executing on said first network node while the remaining processes continue to execute on said first network node;

transferring and initiating execution of said failed process as a new process on said second network node;

initiating execution of a new corresponding monitor for said new process on said first network node; and

terminating said corresponding monitor of said failed process from executing on said first network node.

48. A system comprising:

a first network node and a second network node connected via a communication link;

at least one application comprising a plurality of sub-processes resident on said first network node and presently executing;

a plurality of monitors for said plurality of sub-processes, each of said plurality of sub-processes having one corresponding monitor from said plurality of monitors, said plurality of monitors resident on said second network node and presently executing,

each corresponding monitor being configured for detecting failure of its corresponding sub-process of said application on said first network node and, if a corresponding sub-process has failed, causing said failed sub-process to execute on said second network node as a new sub-process having a corresponding monitor executing on said first network node while the remaining sub-processes continue to execute on said first network node.

49. The system of claim 48, wherein said first and second network nodes are from the group comprising central processing units, computer hosts, computer servers, storage nodes, printer nodes, file systems and location independent file systems.

50. The system of claim 48, wherein said communication link is a local area network or a wide area network.

51. The system of claim 48, wherein said corresponding monitors periodically check their corresponding sub-processes executing on said first network node in order to detect a failure.

52. The system of claim 51, wherein said periodic checking comprises sending a key to said corresponding sub-process and receiving a predefined response.

53. The system of claim 51, wherein said periodic checking comprises monitoring heartbeat signals sent at a periodic rate.

54. The system of claim 51, wherein, when one of said corresponding monitors detects a failure, said one corresponding monitor initiates a sub-process swap, said sub-process swap comprising:

terminating said failed sub-process from executing on said first network node;

initiating said failed sub-process as a new sub-process executing on said second network node;

initiating a new corresponding monitor on said first network node that monitors said new sub-process executing on said second network node; and

terminating said one corresponding monitor executing on said second network node.

55. A system comprising:

a first plurality of network nodes connected via a first communication link;

a second plurality of network nodes connected via a second communication link;

said first communication link and said second communication link connected through a third communication link,

at least one application comprising a plurality of sub-processes resident on one of the network nodes and presently executing;

a plurality of monitors for said plurality of sub-processes, each of said plurality of sub-processes having one corresponding monitor from said plurality of monitors, said plurality of monitors resident on one of the network nodes and presently executing,

each corresponding monitor being configured for detecting failure of its corresponding sub-process and, if a corresponding sub-process has failed, causing said failed sub-process to execute on another of the network nodes as a new sub-process having a corresponding monitor while the remaining sub-processes continue to execute on said one network node where resident.

56. The system of claim 55, wherein said network nodes are from the group comprising central processing units, computer hosts, computer servers, storage nodes, printer nodes, file systems and location independent file systems.

57. The system of claim 55, wherein said first communication link, said second communications link and said third communication link are local area networks, wide area networks or a combination of both a local area network and a wide are network.

58. The system of claim 55, wherein said corresponding monitors periodically check said sub-process executing on said one node of said first plurality of network nodes in order to detect a failure.

59. The system of claim 58, wherein said periodic checking comprises sending a key to said sub-process and receiving a predefined response.

60. The system of claim 58, wherein said periodic checking comprises monitoring heartbeat signals sent at a periodic rate.

61. The system of claim 58, wherein, when one of said corresponding monitors detects a failure, said one corresponding monitor initiates a sub-process swap, said sub-process swap comprising:

terminating said failed sub-process from executing on said one network node;

transferring and initiating said failed sub-process as a new sub-process executing on another network node;

initiating a new corresponding monitor on another network node that is not the same node as the node to which the failed sub-process was transferred, wherein said new corresponding monitors said new sub-process; and

terminating said one corresponding monitor executing on said one network node.

62. The system of claim 61, wherein, if said failed sub-process initially executed on a network node connected to said first communication link, then said new sub-process execution is initiated on a network node connected to said second communication link.

63. The system of claim 61, wherein, if said failed sub-process initially executed on a network node connected to said second communication link, then said new sub-process execution is initiated on a network node connected to said first communication link.

64. The system of claim 61, wherein, if said one corresponding monitor initially executed on a network node connected to said first communication link, then execution of said new corresponding monitor is initiated on a node connected to said second communication link.

65. The system of claim 61, wherein, if said one corresponding monitor initially executed on a network node connected to said second communication link, then execution of said new corresponding monitor is initiated on a network node connected to said first communication link.

66. A method for operating a failover system, wherein failover does not require the termination of all the sub-processes executing on a first network node, the method comprising:

executing at least one application comprising a plurality of sub-processes, said at least one application resident on the first network node;

executing a plurality of monitors for said plurality of sub-processes, each of said plurality of sub-processes having one corresponding monitor from said plurality of monitors, said plurality of monitors resident on a second network node, said second network node connected to said first network node via a communications link;

periodically checking the operation of said application, said periodic checking performed by said plurality of corresponding monitors;

if an execution failure of a sub-process of said application is detected by its corresponding monitor, then

terminating execution of said failed sub-process from executing on said first network node while the remaining sub-processes continue to execute on said first network node;

transferring and initiating execution of said failed sub-process as a new sub-process on said second network node;

initiating execution of a new corresponding monitor for said new sub-process on said first network node; and

terminating said corresponding monitor of said failed sub-process from executing on said first network node.

67. The method of claim 66, wherein said first and second network nodes are from the group comprising central processing units, computer hosts, computer servers, storage nodes, printer nodes, file systems and location independent file systems.

68. The method of claim 66, wherein said communication link is a local area network or a wide area network.

69. A computer system for controlling failover so that the termination of all the executing processes is not required, the computer system comprising:

a first network node and a second network node;

a memory comprising software instructions that enable the computer system to perform;

executing at least one application comprising a plurality of sub-processes, said at least one application resident on the first network node;

executing a plurality of monitors for said plurality of sub-processes, each of said plurality of sub-processes having one corresponding monitor from said plurality of monitors, said plurality of monitors resident on a second network node, said second network node connected to said first network node via a communications link;

periodically checking the operation of said application, said periodic checking performed by said plurality of corresponding monitors;

if an execution failure of a sub-process of said application is detected by its corresponding monitor, then

terminating execution of said failed sub-process from executing on said first network node while the remaining sub-processes continue to execute on said first network node;

transferring and initiating execution of said failed sub-process as a new sub-process on said second network node;

initiating execution of a new corresponding monitor for said new sub-process on said first network node; and

terminating said corresponding monitor of said failed sub-process from executing on said first network node.

70. A computer software product for a computer system comprising a first network node and a second network node to control failover so that the termination of all the sub-processes executing on said first network node is not required, the computer program product comprising:

software instructions for enabling the computer system to perform predetermined operations, and a computer readable medium bearing the software instructions, said predetermined operations comprising;

executing at least one application comprising a plurality of sub-processes, said at least one application resident on the first network node;

executing a plurality of monitors for said plurality of sub-processes, each of said plurality of sub-processes having one corresponding monitor from said plurality of monitors, said plurality of monitors resident on a second network node, said second network node connected to said first network node via a communications link;

periodically checking the operation of said application, said periodic checking performed by said plurality of corresponding monitors;

if an execution failure of a sub-process of said application is detected by its corresponding monitor, then

terminating execution of said failed sub-process from executing on said first network node while the remaining sub-processes continue to execute on said first network node;

transferring and initiating execution of said failed sub-process as a new sub-process on said second network node;

initiating execution of a new corresponding monitor for said new sub-process on said first network node; and

terminating said corresponding monitor of said failed sub-process from executing on said first network node.