Method and system for asymmetrically maintaining system operability

• US 6,658,595 B1
• Filed: 10/19/1999
• Issued: 12/02/2003
• Est. Priority Date: 10/19/1999
• Status: Expired due to Fees

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Thamattoor, Ajoy K.
• Primary Examiner(s): Beausoliel, Robert
• Assistant Examiner(s): Bonzo, Bryce P.
• Application Number:
  US09/422,213
• Time in Patent Office: 1,505 Days
• Field of Search: 714/4, 714/10, 714/11, 714/23, 714/51, 714/55, 370/219
• US Class Current: 714/11
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component

Abstract

A system is provided for asymmetrically maintaining system operability that includes a first processing element and a second processing element coupled to the first processing element by a communication link. The first processing element is operable to perform at least one function. The second processing element is operable to perform at least one function of the first processing element in the event the first processing element fails, and further operable to expect and receive keepalive inquiries at an expected rate from the first processing element and to send responses in response to the inquiries to the first processing element. The second processing element is further operable to take remedial action after not receiving any inquiries within a first predetermined time period. In another embodiment of the present invention, the first processing element is operable to take remedial action after not receiving any response to any inquiries sent within a second predetermined time period, wherein the first predetermined time period is larger than the second predetermined time period. In other embodiments of the present invention, the first and second processing elements are routers.

Claims

1. An asymmetric failure detection system, comprising:

a first processing element operable to perform at least one function; and

a second processing element coupled to the first processing element by a communication link, the second processing element operable to perform the at least one function of the first processing element in the event the first processing element fails, the second processing element further operable to expect and receive keepalive inquiries at an expected rate from the first processing element and to send responses in response to the inquiries to the first processing element, the second processing element further operable to take remedial action after not receiving any inquiries within a first predetermined time period; and

wherein the first processing element is operable to take remedial action after not receiving any response to any inquiries sent within a second predetermined time period, the first predetermined time period being larger than the second predetermined time period, and wherein the first predetermined time period is determined based on an amount of time needed by the first processing element to take remedial action after not receiving any response to inquiries, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

2. The system of claim 1, wherein the first processing element and the second processing element each comprise a portion of a processing unit operable to perform at least one function.

3. The system of claim 1, wherein the remedial action includes generating a signal operable to disrupt the operation of the other processing element on a dedicated communication path to the other processing element.

4. The system of claim 1, wherein the remedial action includes transmitting a message operable to disrupt the operation of the other processing element.

5. The system of claim 1, wherein the first processing element detects a failure of one of the second processing element or the communication link after not receiving any response to any inquiries sent within the second predetermined time period.

6. The system of claim 1, wherein the first processing element and second processing element are each routers.

7. The system of claim 1, wherein the communication link comprises an EHSA bus.

8. An asymmetric failure detection system, comprising:

a first processing element, operable to transmit keepalive inquiries at an expected rate and to receive responses;

a second processing element coupled to the first processing element by a communication link, the second processing element operable to expect and receive the inquiries at the expected rate and to send the responses, the second processing element further operable to take remedial action after not receiving any inquiries within a first predetermined time period; and

wherein the remedial action includes transmitting a message operable to disrupt the operation of the other processing element and wherein the first processing element is operable to take remedial action after not receiving any response to any inquiries sent within a second predetermined time period, the first predetermined time being period larger than the second predetermined time period, and wherein the first predetermined time period is determined based on an amount of time needed by the first processing element to take remedial action after not receiving any response to inquiries, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

9. The system of claim 8, wherein the first processing element and the second processing element each comprise a portion of a processing unit operable to perform at least one function.

10. The system of claim 8, wherein the first and second processing elements are routers.

11. The system of claim 8, wherein the first predetermined time period is approximately at least twice the second predetermined time period.

12. The system of claim 8, wherein the first processing element detects a failure of one of the second processing element or the communication link after not receiving any response to any inquiries sent within the second predetermined time period.

13. The system of claim 8, wherein the message resets the other processing element.

14. A method for asymmetric failure detection, comprising:

providing a first processing element operable to transmit keepalive inquiries at an expected rate and receive responses in response to the transmitted inquiries;

receiving the inquiries at a second processing element coupled to the first processing element at the expected rate;

sending the responses in response to the transmitted inquiries;

taking remedial action by the second processing element after failing to receive any inquiries within a first predetermined time period; and

wherein taking remedial action includes transmitting a message operable to disrupt the operation of the other processing element; and

wherein the first processing element is operable to take remedial action after not receiving any response to any inquiries sent within a second predetermined time period, the first predetermined time period being larger than the second predetermined time period, and wherein the first predetermined time period is determined based on an amount of time needed for the first processing element to take remedial action, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

15. The method of claim 16, further comprising detecting the failure of one of the second processing element or a communication link coupling the first processing element to the second processing element after not receiving any response to any inquiries sent within the second predetermined time period.

16. The method of claim 14, wherein the message resets the other processing element.

17. The method of claim 14, wherein the first predetermined time period is approximately twice the second predetermined time period.

18. The method of claim 14, wherein the first processing element and the second processing element comprise a portion of a telecommunications network.

19. The method of claim 14, wherein the first processing element and the second processing element each comprise a portion of a processing unit operable to perform at least one function.

22. The asymmetric failure detector of claim 17, wherein the message resets the other processing element.

20. An asymmetric failure detector, comprising:

a computer-readable storage medium; and

an asymmetric failure detector resident on the computer-readable storage medium and operable to;

provide a first processing element operable to transmit keepalive inquiries at an expected rate and receive responses in response to the transmitted inquiries;

receive the inquiries at a second processing element coupled to the first processing element at the expected rate;

send the responses in response to the transmitted inquiries;

take remedial action by the second processing element after not receiving any inquiries within a first predetermined time period; and

wherein taking remedial action includes transmitting a message operable to disrupt the operation of the other processing element; and

wherein the first processing element is operable to take remedial action after not receiving any response to any inquiries sent within a second predetermined time period, the first predetermined time period being larger than the second predetermined time period, and wherein the first predetermined time period is determined based on an amount of time needed for the first processing element to take remedial action, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

21. The asymmetric failure detector of claim 20, further comprising detecting the failure of one of the second processing element or a communication link coupling the first processing element to the second processing element after not receiving any response to any inquiries sent within the second predetermined time period.

23. The asymmetric failure detector of claim 20, wherein the first predetermined time period is approximately twice the second predetermined time period.

24. The asymmetric failure detector of claim 20, wherein the first processing element and the second processing element comprise a portion of a telecommunications network.

25. The asymmetric failure detector of claim 20, wherein the first processing element and the second processing element each comprise a portion of a processing unit operable to perform at least one function.

26. An asymmetric failure detection system, comprising:

a first processing element operable to perform at least one function of a second processing element in the event the second processing element fails; and

logic residing within the first processing element operable to expect and to receive keepalive inquiries at an expected rate from the second processing element and to send responses in response to the inquiries to the second processing element, the logic further operable to cause remedial action to be taken after not receiving any inquiries within a first predetermined time period; and

wherein the first predetermined time period is larger than a second predetermined time period after which operation of the first processing element is disrupted if no responses to any inquiries are sent within the second predetermined time period; and

wherein the first predetermined time period is determined based on an amount of time needed for the first processing element to cause remedial action to be taken, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

27. The system of claim 26, wherein the first processing element comprises a portion of a processing unit operable to perform at least one function.

28. The system of claim 26, wherein the remedial action includes generating a signal operable to disrupt the operation of the other processing element on a dedicated communication path to the other processing element.

29. The system of claim 26, wherein the remedial action includes transmitting a message operable to disrupt the operation of the other processing element.

30. The system of claim 26, wherein the first processing element is a router.

31. The system of claim 26, wherein the inquiries are received via an EHSA bus.

32. An asymmetric failure detection system, comprising:

a first processing element;

logic within the first processing element operable to expect and to receive keepalive inquiries at an expected rate from a second processing element and to send responses to the inquiries, the first processing element further operable to cause remedial action to be taken after not receiving any inquiries within a first predetermined time period; and

wherein the remedial action includes transmitting a message operable to disrupt the operation of the other processing element; and

wherein the first predetermined time period is larger than a second predetermined time period after which operation of the first processing element is disrupted if no responses to any inquiries are sent within the second predetermined time period, and wherein the first predetermined time period is based on an amount of time needed for the first processing element to cause remedial action to be taken, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

33. The system of claim 32, wherein the first processing element comprises a portion of a processing unit operable to perform at least one function.

34. The system of claim 32, wherein the first processing element is a router.

35. The system of claim 32, wherein the message resets the other processing element.

36. The system of claim 32, wherein the first predetermined time period is approximately at least twice the second predetermined time period.

37. A method for asymmetric failure detection, comprising:

receiving keepalive inquiries at an expected rate at a first processing element from a second processing element;

sending responses in response to the transmitted inquiries;

taking remedial action by the first processing element after failing to receive any inquiries within a first predetermined time period, wherein the first predetermined time period is based on an amount of time needed for the second processing element to take remedial action; and

wherein taking remedial action includes transmitting a message operable to disrupt the operation of the other processing element; and

disrupting the operation of the first processing element after a second predetermined time period that is smaller than the first predetermined time period if no responses to any inquiries are sent within the second predetermined time period, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

38. The method of claim 37, wherein the first predetermined time period is approximately twice the second predetermined time period.

39. The method of claim 37, wherein the message resets the other processing element.

40. The method of claim 37, wherein the first processing element comprises a portion of a telecommunications network.

41. The method of claim 37, wherein the first processing element comprises a portion of a processing unit operable to perform at least one function.

42. An asymmetric failure detector, comprising:

a computer-readable storage medium; and

an asymmetric failure detector resident on the computer-readable storage medium and operable to;

receive keepalive inquiries at an expected rate at first processing element from a second processing element;

send responses in response to the transmitted inquiries;

take remedial action by the first processing element after not receiving any inquiries within a first predetermined time period; and

wherein taking remedial action includes causing a message operable to disrupt the operation of the other processing element to be transmitted; and

wherein the first predetermined time period is larger than a second predetermined time period after which operation of the first processing element is disrupted if no responses to any inquiries are sent within the second predetermined time period; and

wherein the first predetermined time period is determined based on an amount of time needed to disrupt the operation of the first processing element, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

43. The asymmetric failure detector of claim 42, wherein the first predetermined time period is approximately twice the second predetermined time period.

44. The asymmetric failure detector of claim 42, wherein the message resets the other processing element.

45. The asymmetric failure detector of claim 42, wherein the first processing element comprises a portion of a telecommunications network.

46. The asymmetric failure detector of claim 42, wherein the first processing element comprises a portion of a processing unit operable to perform at least one function.

47. An asymmetric failure detection system, comprising:

a first processing element operable to perform at least one function of a second processing element in the event the second processing element fails; and

logic residing within the first processing element operable to transmit keepalive inquiries to and to receive responses to the inquiries from the second processing element, the logic further operable to cause remedial action to be taken after not receiving any responses within a first predetermined time period; and

wherein the first predetermined time period is smaller than a second predetermined time period after which operation of the first processing element is disrupted if no inquiries are sent within the second predetermined time period and wherein the first predetermined time period is smaller than a second predetermined time period, and wherein the second predetermined time period is based on an amount of time needed for the first processing element to cause remedial action to be taken, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link.

48. The system of claim 47, wherein the first processing element comprises a portion of a processing unit operable to perform at least one function.

49. The system of claim 47, wherein the remedial action includes generating a signal operable to disrupt the operation of the other processing element on a dedicated communication path to the other processing element.

50. The system of claim 47, wherein the remedial action includes transmitting a message operable to disrupt the operation of the other processing element.

51. The system of claim 47, wherein the first processing element is a router.

52. The system of claim 47, wherein the logic detects a failure of one of the second processing element or a communication link coupling the first processing element to the second processing element if no responses to any inquiries are sent within the first predetermined time period.

53. The system of claim 52, wherein the communication link comprises an EHSA bus.

54. An asymmetric failure detection system, comprising:

a first processing element;

logic residing within the first processing element operable to transmit keepalive inquiries to and to receive responses to the inquiries from a second processing element, the logic further operable to cause remedial action to be taken after not receiving any responses within a first predetermined time period;

wherein the first predetermined time period is smaller than a second predetermined time period after which operation of the first processing element is disrupted if no inquiries are sent within the second predetermined time period and wherein the second predetermined time period is based on an amount of time needed for the first processing element to cause remedial action to be taken, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link; and

wherein the remedial action includes transmitting a message operable to disrupt the operation of the other processing element.

55. The system of claim 54, wherein the first processing element comprises a portion of a processing unit operable to perform at least one function.

56. The system of claim 54, wherein the first processing element is a router.

57. The system of claim 54, wherein the message resets the other processing element.

58. The system of claim 54, wherein the second predetermined time period is approximately at least twice the first predetermined time period.

59. The system of claim 54, wherein the logic detects a failure of one of the second processing element or a communication link coupling the first processing element to the second processing element if no responses to any inquiries are sent within the first predetermined time period.

60. A method for asymmetric failure detection, comprising:

transmitting keepalive inquiries from a first processing element to a second processing element;

receiving responses in response to the transmitted inquiries;

taking remedial action by the first processing element after failing to receive any responses within a first predetermined time period;

wherein the first predetermined time period is smaller than a second predetermined time period after which operation of the first processing element is disrupted if no inquiries are sent within the second predetermined time period and wherein the second predetermined time period is based on an amount of time needed for the first processing element to take remedial action, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link; and

wherein taking remedial action includes transmitting a message operable to disrupt the operation of the other processing element.

61. The method of claim 60, further comprising detecting a failure of one of the second processing element or a communication link coupling the first processing element to the second processing element if no responses to any inquiries are sent within the second predetermined time period.

62. The method of claim 60, wherein the second predetermined time period is approximately twice the first predetermined time period.

63. The method of claim 60, wherein the message resets the other processing element.

64. The method of claim 60, wherein the first processing element comprises a portion of a telecommunications network.

65. The method of claim 60, wherein the first processing element comprises a portion of a processing unit operable to perform at least one function.

66. An asymmetric failure detector, comprising:

a computer-readable storage medium; and

an asymmetric failure detector resident on the computer-readable storage medium and operable to;

transmit keepalive inquiries from a first processing element to a second processing element;

receive responses in response to the transmitted inquiries;

take remedial action by the first processing element after not receiving any responses within a first predetermined time period;

wherein the first predetermined time period is smaller than a second predetermined time period after which operation of the first processing element is disrupted if no inquiries are sent within the second predetermined time period; and

wherein the second predetermined time period is based on an amount of time needed for the first processing element to take remedial action, the first predetermined time period and the second predetermined time period selected to prevent simultaneous shutdown of both the first processing element and the second processing element in the event of a failure of the communication link; and

wherein taking remedial action includes causing a message operable to disrupt the operation of the other processing element to be transmitted.

67. The asymmetric failure detector of claim 66, wherein the first processing element detects a failure of one of the second processing element or a communication link coupling the first processing element to the second processing element if no responses to any inquiries are sent within the first predetermined time period.

68. The asymmetric failure detector of claim 66, wherein the first predetermined time period is approximately twice the second predetermined time period.

69. The asymmetric failure detector of claim 66, wherein the message resets the other processing element.

70. The asymmetric failure detector of claim 66, wherein the first processing element comprises a portion of a telecommunications network.

71. The asymmetric failure detector of claim 66, wherein the first processing element comprises a portion of a processing unit operable to perform at least one function.