FAILURE RECOVERY SYSTEM AND SERVER

• US 20070220303A1
• Filed: 03/01/2007
• Published: 09/20/2007
• Est. Priority Date: 03/02/2006
• Status: Active Grant

• Current Assignee: Alaxala Networks Corporation (Fortinet Inc.)
• Original Assignee: Alaxala Networks Corporation (Fortinet Inc.)
• Inventors: Miyamoto, Takahisa, KIMURA, Hiroyasu
• Granted Patent: US 7,827,446 B2
• Time in Patent Office: Days
• US Class Current: 714/4
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A server 200 includes a scenario table in which object to-be-monitored information that indicates one or more network devices A, B and C being objects for failure recovery, failure information for identifying contents of failures, countermeasure information against failures, and frequence information that indicates the number of times of the recoveries from the failures based on the countermeasure information are correspondingly stored. The network device A 300 detects the failure of the network device itself and transmits a failure event to the server 200. The server 200 selects the countermeasure information item in descending order of the frequence information items, by reference to the scenario table, and transmits the selected countermeasure information item to the network device A 300. The server 200 repeats the selections and transmissions of the pertinent information item until the reception of the failure event from the network device A 300 stops.

Claims

1. A failure recovery system comprising:

one or more network devices constituting a network; and

a server connected to said network devices, and including a scenario table in which object to-be-monitored information that indicates said network device or devices being objects for failure recovery, failure information for identifying contents of failures, and countermeasure information against the failures, and frequence information that indicates the number of times of recoveries from the failures based on the countermeasure information are correspondingly stored;

wherein;

said network device detects the failure of said network device itself, and transmits to said server, a failure event which contains the object to-be-monitored information indicating said device itself, and the failure information for identifying content of the failure;

said server receives the failure event, searches for one or more countermeasure information items corresponding to the object to-be-monitored information and the failure information which are contained in the failure event, by reference to the scenario table, and selects from among the pertinent countermeasure information items, one countermeasure information item as to which the corresponding frequence information is the largest, or, is equal to or larger than a predetermined value;

said server transmits the selected countermeasure information item to said network device;

said network device receives the countermeasure information item, and reflects the countermeasure information item or alters its setting on the basis of the countermeasure information item; and

when said server judges that the failure event is not received again within a predetermined time period since the transmission of the selected countermeasure information item, said server increases the frequence information item corresponding to the selected countermeasure information item, by reference to the scenario table.

3. A failure recovery system as defined in claim 1, wherein:

said server receives the failure event again, the failure event being transmitted when said network device detects the failure again after said network device has reflected the countermeasure information item or has altered its setting on the basis of the countermeasure information item; and

when said server receives the failure event again, said server selects from among one or more countermeasure information items corresponding to the object to-be-monitored information and the failure information which are contained in the failure event, the countermeasure information item in descending order of the corresponding frequence information items, by reference to the scenario table, and transmits the selected countermeasure information item to said network device, the selections and the transmissions being repeated until the re-reception of the failure event stops.

4. A failure recovery system as defined in claim 1, wherein the transmission to said server includes that:

said network device monitors logs outputted beforehand and containing logs of route computations, so as to evaluate the number of times of the route computations having occurred within a fixed time period;

said network device detects the failure of frequent occurrences of route learning, because the evaluated number of times of the route computations is larger than a preset threshold value; and

said network device transmits to said server, the failure event which contains the object to-be-monitored information indicating the device itself, and the failure information for identifying the frequent occurrences of the route learning.

13. A failure recovery system as defined in claim 1, further comprising

a client device which is connected to said server, wherein:

said client device inputs object to-be-monitored information, failure information and countermeasure information from an input unit, and transmits the inputted information to said server;

said server stores in the scenario table, the object to-be-monitored information, the failure information and the countermeasure information which have been received from said client device; and

said server initializes the frequence information which corresponds to the respective stored information items.

16. A failure recovery system as defined in claim 1, further comprising

a client device which is connected to said server, wherein:

said server notifies to said client device in a case where the unselected countermeasure information item has become nonexistent by repeatedly selecting pertinent countermeasure information items in the scenario table; and

said client device displays on a display unit or outputs to an output unit, nonexistence of the countermeasure information.

19. A failure recovery system as defined in claim 1, wherein the scenario table is further stored therein manual information relevant to the failure information.

2. A server comprising:

an interface for communicating with one or more network devices which constitute a network;

a scenario table in which object to-be-monitored information that indicates the network device or devices being objects for failure recoveries, failure information for identifying contents of failures, countermeasure information against the failures, and frequence information that indicates the number of times of recoveries from the failures based on the countermeasure information are correspondingly stored; and

a processing unit;

wherein;

said processing unit receives through said interface, a failure event which is transmitted when the network device detects the failure, and which contains the object to-be-monitored information indicating the network device, and the failure information for identifying the content of the failure;

said processing unit searches for one or more countermeasure information items corresponding to the object to-be-monitored information and the failure information that are contained in the failure event, by reference to said scenario table, and selects from among the pertinent countermeasure information items, one countermeasure information item as to which the corresponding frequence information is the largest, or, is equal to or larger than a predetermined value;

said processing unit transmits the selected countermeasure information item to the network device through said interface; and

in a case where the failure event is not received again within a predetermined time period since the transmission of the selected countermeasure information item, said processing unit increases the frequence information item corresponding to the selected countermeasure information item, by reference to said scenario table.

5. A failure recovery system comprising:

a first network device constituting a network;

a second network device connected to said first network device, and constituting the network; and

a server connected to said first and second network devices, and including a scenario table in which object to-be-monitored information that indicates said first and second network devices being objects for recovery from a failure, failure information that is determined by a combination of a status of said first network device and a status of said second network device, countermeasure information against the failure, and frequence information that indicates the number of times of the recoveries from the failure based on the countermeasure information are correspondingly stored;

wherein;

said first network device transmits to said server, a first event which contains first object to-be-monitored information indicating said device itself, and first status information indicating the status of said device itself;

said second network device transmits to said server, a second event which contains second object to-be-monitored information indicating said device itself, and second status information indicating the status of said device itself;

said server receives the first and second events, judges existence or nonexistence of the failure on the basis of the first status information and the second status information, and finds failure information;

said server searches for one or more countermeasure information items which correspond to the first and second object to-be-monitored information items respectively contained in the first and second events, and the found failure information, by reference to the scenario table, and selects from among the pertinent countermeasure information items, one countermeasure information item as to which the corresponding frequence information is the largest, or, is equal to or larger than a predetermined value;

said server transmits the selected countermeasure information item to said first and second network devices, respectively;

said first and second network devices receive the countermeasure information item, and reflect said countermeasure information item or alter their settings on the basis of the countermeasure information item, respectively; and

when the failure is avoided, said server increases the frequence information item corresponding to the selected countermeasure information item, by reference to the scenario table.

7. A failure recovery system as defined in claim 5, wherein:

said server receives from said first network device, a third event which contains the first object to-be-monitored information, and third status information indicating the status of said first network device after the reflection of the countermeasure information item or the alteration of the setting;

said server receives from said second network device, a fourth event which contains the second object to-be-monitored information, and fourth status information indicating the status of said second network device after the reflection of the countermeasure information item or the alteration of the setting;

said server judges if the failure has been avoided, on the basis of the third status information and the fourth status information; and

when it is judged that the failure has not been avoided, said server selects from among one or more countermeasure information items corresponding to the first and second object to-be-monitored information items and the failure information, the countermeasure information item in descending order of the corresponding frequence information items, by reference to the scenario table, and transmits the selected countermeasure information item to said first and second network devices, the selections and the transmissions being repeated until the avoidance of the failure is judged.

8. A failure recovery system as defined in claim 5, wherein:

said first and second network devices operate in a redundant configuration;

the finding of the failure information contains that, in a case where both the first status information of said first network device and the second status information of said second network device indicate masters, or where both of them indicate backups, said server judges the existence of the failure, and that said server finds the failure information indicating “

double master”

or “

double backup”

; and

the countermeasure information contains information for bringing one of said first network device and said second network device into the master, and the other into the backup.

14. A failure recovery system as defined in claim 5, further comprising

a client device which is connected to said server, wherein:

said client device inputs object to-be-monitored information, failure information and countermeasure information from an input unit, and transmits the inputted information to said server;

said server stores in the scenario table, the object to-be-monitored information, the failure information and the countermeasure information which have been received from said client device; and

said server initializes the frequence information which corresponds to the respective stored information items.

17. A failure recovery system as defined in claim 5, further comprising

a client device which is connected to said server, wherein:

said server notifies to said client device in a case where the unselected countermeasure information item has become nonexistent by repeatedly selecting pertinent countermeasure information items in the scenario table; and

said client device displays on a display unit or outputs to an output unit, nonexistence of the countermeasure information.

6. A server comprising:

an interface for communicating with first and second network devices which constitute a network;

a scenario table in which object to-be-monitored information that indicate the first and second network devices being objects for recovery from a failure, failure information that is determined by a combination of a status of the first network device and a status of the second network device, countermeasure information against the failure, and frequence information that indicates the number of times of the recoveries from the failure based on the countermeasure information are correspondingly stored; and

a processing unit;

wherein;

said processing unit receives a first event which contains first object to-be-monitored information indicating the first network device, and first status information indicating the status of the first network device, from the first network device and through said interface;

said processing unit receives a second event which contains second object to-be-monitored information indicating the second network device, and second status information indicating the status of the second network device, from the second network device and through said interface;

said processing unit judges existence or nonexistence of the failure on the basis of the first status information and the second status information, and finds failure information;

said processing unit searches for one or more countermeasure information items which correspond to the first and second object to-be-monitored information items respectively contained in the first and second events, and the found failure information, by reference to said scenario table, and selects from among the pertinent countermeasure information items, one countermeasure information item as to which the corresponding frequence information is the largest or, is equal to or larger than a predetermined value;

said processing unit transmits the selected countermeasure information item to the first and second network devices through said interface, respectively; and

when the failure is avoided, said processing unit increases the frequence information corresponding to the selected countermeasure information item, by reference to said scenario table.

9. A failure recovery system comprising:

a first network device constituting a network;

a second network device constituting the network;

a third network device connected to the network through said first network device, and connected to the network through said second network device; and

a server connected to said first and second network devices, and including a scenario table in which object to-be-monitored information that indicates the network device or devices being objects for failure recovery, failure information for identifying contents of failures, countermeasure information against the failures, and frequence information that indicates the number of times of recoveries from the failures based on the countermeasure information are correspondingly stored;

wherein;

when said third network device detects that a failure has occurred in a transfer to the network, due to the failure of said first or second network device, said third network device transmits to said server, a failure event which contains the object to-be-monitored information indicating said device itself, and failure information for identifying the failure of a transfer function;

said server receives the failure event, searches for one or more countermeasure information items corresponding to the object to-be-monitored information and the failure information which are contained in the failure event, by reference to the scenario table, and selects from among the pertinent countermeasure information items, one countermeasure information item as to which the corresponding frequence information is the largest, or, is equal to or larger than a predetermined value;

said server transmits the countermeasure information item to said first and second network devices in conformity with the selected countermeasure information item;

said first and second network devices receive the countermeasure information item, and reflect the countermeasure information item or alter their settings on the basis of the countermeasure information item; and

when said server does not receive the failure event again within a predetermined time period since the transmission of the selected countermeasure information item, said server increases the frequence information corresponding to the selected countermeasure information item, by reference to the scenario table.

11. A failure recovery system as defined in claim 9,wherein:

said server receives the failure event again, the failure event being transmitted when the failure is detected again by the third network device after the first and second network devices have reflected the countermeasure information item or have altered their settings on the basis of the countermeasure information item;

when the failure event is received again, said server selects from among one or more countermeasure information items corresponding to the object to-be-monitored information and the failure information that are contained in the failure event, the countermeasure information item in descending order of the corresponding frequence information items, by reference to said scenario table, and transmits the selected countermeasure information item to said first and second network devices, the selections and the transmissions being repeated until the re-reception of the failure event from the third network device stops.

12. A failure recovery system as defined in claim 9, wherein:

the first network device is set as a rendezvous point of multicast routing;

the transmission to said server includes that said third network device detects occurrence of a failure in a multicast routing function of said device itself, due to a failure of a rendezvous point function of said first network device, and transmits the failure event to said server; and

the countermeasure information item contains information for setting the second network device as a rendezvous point of the multicast routing.

15. A failure recovery system as defined in claim 9, further comprising

a client device which is connected to said server, wherein:

said client device inputs object to-be-monitored information, failure information and countermeasure information from an input unit, and transmits the inputted information to said server;

said server stores in the scenario table, the object to-be-monitored information, the failure information and the countermeasure information which have been received from said client device; and

said server initializes the frequence information which corresponds to the respective stored information items.

18. A failure recovery system as defined in claim 9, further comprising

a client device which is connected to said server, wherein:

said server notifies to said client device in a case where the unselected countermeasure information item has become nonexistent by repeatedly selecting pertinent countermeasure information items in the scenario table; and

said client device displays on a display unit or outputs to an output unit, nonexistence of the countermeasure information.

10. A server comprising:

an interface for communicating with first, second and third network devices which constitute a network;

a scenario table in which object to-be-monitored information that indicates the network device or devices being objects for failure recovery, failure information for identifying contents of failures, countermeasure information against the failures, and frequence information that indicates the numbers of times of recoveries from the failures based on the countermeasure information are correspondingly stored; and

a processing unit;

wherein;

said processing unit receives through said interface, a failure event which is transmitted when the third network device detects that a failure has occurred in a transfer to the network, due to the failure of the first or second network device, and which contains the object to-be-monitored information indicating the third network device, and the failure information for identifying the failure of a transfer function;

said processing unit searches for one or more countermeasure information items which correspond to the object to-be-monitored information and the failure information that are contained in the failure event, by reference to said scenario table, and selects from among the pertinent countermeasure information items, one countermeasure information item as to which the corresponding frequence information is the largest, or, is equal to or larger than a predetermined value;

said processing unit transmits the countermeasure information item to the first and second network devices through said interface, in conformity with the selected countermeasure information item; and

when the failure event is not received again within a predetermined time period since the transmission of the selected countermeasure information item, said processing unit increases the frequence information corresponding to the selected countermeasure information item, by reference to said scenario table.