Negotiating takeover in high availability cluster

• US 6,728,897 B1
• Filed: 07/25/2000
• Issued: 04/27/2004
• Est. Priority Date: 07/25/2000
• Status: Active Grant

• Current Assignee: Impax Laboratories Incorporated (Amneal Pharmaceuticals, LLC), NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Cramer, Samuel M., Schoenthal, Scott
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): MASKULINSKI, MICHAEL C
• Application Number:
  US09/625,234
• Time in Patent Office: 1,372 Days
• Field of Search: 714/4-6, 714/10
• US Class Current: 714/6.1
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence

Abstract

The invention provides a method and system in which a node in a 2-node high availability cluster can declare itself impaired. In doing so, an impaired node can self diagnose or request another node in the cluster assist in diagnosing the problem. An impaired node can then request another node takeover the impaired node'"'"'s tasks whereby the impaired node is able to shut down gracefully.

Claims

1. An apparatus comprising:

a first filer;

a second filer;

an interconnect between said first filer and said second filer;

a means of detecting a soft failure state at said first filer;

a means of communicating said soft failure state to said second filer;

a means of responding to said soft failure state when and if said second filer can do so usefully; and

a means of taking over after said responding to said soft failure state;

wherein a means of taking over after said responding to said soft failure state further comprises a means of shutting down said first filer;

wherein a countdown timer counts for a predetermined duration sufficient to allow said first filer to shut down gracefully; and

wherein notification of shutdown is transmitted from said first filer to said second filer prior to expiration of said countdown timer.

2. The apparatus of claim 1, wherein said first filer comprises a file server capable of responding to requests for data transfer.

3. The apparatus of claim 1, wherein said second filer comprises a file server capable of responding to requests for data transfer.

4. The apparatus of claim 1, wherein said interconnect facilitates communication between said first filer and said second filer.

5. The apparatus of claim 4, wherein said communication is bi-directional.

6. The apparatus of claim 1, wherein said means of detecting a soft failure state at said first filer is accomplished by said first filer.

7. The apparatus of claim 1, wherein said means of communicating said soft failure state further comprises;

a means of diagnosing said first filer;

a means of requesting help from said second filer.

8. The apparatus of claim 7, wherein said diagnosis of said first filer involves self diagnosis by said first filer.

9. The apparatus of claim 7, wherein said diagnosis of said first filer involves comparison of said second filer'"'"'s operational characteristics with those of said first filer.

10. The apparatus of claim 7, wherein said diagnosis of said first filer involves testing said second filer'"'"'s ability to perform tasks said first filer has identified in said information relating to a failure of said first filer it cannot perform.

11. The apparatus of claim 7, wherein said means of requesting help comprises a message containing information relating to said soft failure state of said first filer as identified by said first filer.

12. The apparatus of claim 11, wherein said message is transmitted from said first filer to said second filer.

13. The apparatus of claim 12, wherein said message is transmitted via said interconnect.

14. The apparatus of claim 1, wherein said means of responding to said soft failure state when and if said second filer can do so usefully further comprises a means of determining whether said second filer is impaired.

15. The apparatus of claim 1, wherein said means of responding to said soft failure state when and if said second filer can do so usefully further comprises a means of determining whether said second filer is recovering from recent impairment.

16. The apparatus of claim 1, wherein a request to shut down is sent from said second filer to said first filer.

17. The apparatus of claim 1, wherein said first filer fails to shut down prior to expiration of said countdown timer causing said second filer to send messages determined to force said first filer to shut down.

18. A method of operating a filer cluster comprising the steps of:

operating a first filer;

operating a second filer;

creating an interconnect between said first filer and said second filer;

detecting a soft failure state at said first filer

communication said soft failure state to said second filer

responding to said soft failure state when and if said second filer can do so usefully; and

taking over after said responding to said soft failure state;

wherein said step of taking over after said responding to said soft failure state further comprises the steps of shutting down said first filer and assuming the tasking of said first filer by said second filer;

wherein said step of shutting down said first filer is accomplished by sending a shutdown request from said second filer to said first filer;

wherein a countdown timer counts for a predetermined duration sufficient to allow said first filer to shut down gracefully; and

wherein notification of shutdown is transmitted from said first filer to said second filer.

19. The method of claim 18, wherein said step of operating a first filer is responsive to requests for data transfer.

20. The method of claim 18, wherein said step of operating a second filer is responsive to requests for data transfer.

21. The method of claim 18, wherein said step of creating an interconnect between said first filer and said second filer facilitates communication between said first filer and said second filer.

22. The method of claim 21, wherein said communication is bidirectional.

23. The method of claim 18, wherein step of detecting a soft failure state at said first filer is accomplished by said first filer.

24. The method of claim 18, wherein said step of communicating said software failure state further comprises the steps of;

diagnosing said first filer;

requesting help from said second filer.

25. The method of claim 24, wherein said step of diagnosing said first filer is accomplished by said first filer.

26. The method of claim 24, wherein said step of diagnosing said first filer is accomplished by comparison of said second filer'"'"'s operational characteristics with those of said first filer.

27. The method of claim 24, wherein said step of diagnosing said first filer is accomplished by testing said second filer'"'"'s ability to perform tasks said first filer has identified is said information relating to a failure of said first filer it cannot perform.

28. The method of claim 24, wherein said step of requesting help creates a message containing information relating to said soft failure state of said first filer as identified by said first filer.

29. The method of claim 28, wherein said message is transmitted from said first filer to said second filer.

30. The method of claim 29, wherein said message is transmitted via said interconnect.

31. The method of claim 18, wherein said step of responding to said soft failure state when and if said second filer can do so usefully further comprises the step of determining whether said second filer is impaired.

32. The method of claim 18, wherein said step of responding to said soft failure state when and if said second filer can do so usefully further comprises the step of determining whether said second filer is recovering from recent impairment.

33. The method of claim 18, wherein said countdown timer expires and said second filer sends messages determined to force said first filer to shut down.