SYNCHRONIZING DEVICE ERROR INFORMATION AMONG NODES

• US 20090300436A1
• Filed: 06/03/2008
• Published: 12/03/2009
• Est. Priority Date: 06/03/2008
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hood, James Lamar, Sorenson, Todd Charles, Rinaldi, Brian Anthony, Robison, Micah
• Granted Patent: US 7,904,752 B2
• Time in Patent Office: Days
• US Class Current: 714/57
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/0677 Localisation of faults

Abstract

Provided are a method, system, and article of manufacture for synchronizing device error information among nodes. A first node performs an action with respect to a first node error counter for a device in communication with the first node and a second node. The first node transmits a message to the second node indicating the device and the action performed with respect to the first node error counter for the device. The second node performs the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device.

Claims

1. A method, comprising:

performing, by a first node, an action with respect to a first node error counter for a device in communication with the first node and a second node;

transmitting, by the first node, a message to the second node indicating the device and the action performed with respect to the first node error counter for the device;

performing, by the second node, the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device.

2. The method of claim 1, wherein the action indicated in the message comprises an open action, wherein performing, by the second node, the open action comprises creating the second node error counter for the device in response to the message.

3. The method of claim 1, wherein the action indicated in the message comprises an expire action, wherein performing, by the second node, the expire action comprises expiring the second error counter for the device in response to the message.

4. The method of claim 1, further comprising:

detecting, by the first node, an error at the device, wherein the action performed by the first node comprises incrementing the first node error counter for the device in response to detecting the error, wherein the action indicated in the message comprises an increment action, wherein performing, by the second node, the increment action comprises incrementing the second node error counter for the device at the second node.

5. The method of claim 1, further comprising:

initiating, by the first node, an error recovery operation in response to the error counter value reaching an error threshold.

6. The method of claim 1, wherein the first node operates as an owner of the device performing error handling for the device, further comprising:

taking over, by the second node, ownership of the device; and

incrementing, by the second node, the second node error counter, indicating errors at the device detected by the first node, in response to the second node detecting an error at the device.

7. The method of claim 6, wherein the second node takes over the ownership of the device in response to a failure of the first node.

8. The method of claim 1, wherein the device comprises a first device, the first node error counter comprises a first node first device error counter, and the second node error counter comprises a second error first device counter copy, further comprising:

maintaining, by the second node, a second node second device error counter indicating a number of errors at a second device in communication with the first node and the second node;

maintaining, by the first node, a first node second device error counter indicating a number of errors at the second device including errors detected by the second node.

9. The method of claim 8, wherein the first and second nodes comprise first and second processing clusters in a server that communicate over a first interface in the server, wherein the first and second devices comprise first and second network adaptors, and wherein the first and second processing clusters communicate with the first and second adaptors over a second interface.

10. The method of claim 8, further comprising:

receiving, by the second node, an error message from the second device;

incrementing, by the second node, the second node second device error counter;

transmitting, by the second node, an increment message to the first node for the second device;

incrementing, by the first node, the first node second device error counter to the second error counter value in response to the increment message.

11. The method of claim 8, wherein the first node operates as an owner of the first device performing error handling for the first device and wherein the second node operates as an owner of the second device performing error handling for the second device, further comprising:

taking over, by the second node, ownership of the first device, wherein the second node uses the second node first device error counter to perform error management for the first device; and

taking over, by the first node, ownership of the second device, wherein the first node uses the first node second device error counter to perform error management for the second device.

12. The method of claim 8, further comprising:

initiating a synchronization operation at the first and second nodes;

sending, by the first node a synchronization message to the second node indicating a value of the first node first device error counter to the second node in response to the synchronization operation; and

sending, by the second node a synchronization message to the first node indicating a value of the second node second device error counter to the first node in response to the synchronization operation.

13. The method of claim 12, further comprising:

updating, by the first node, the first node second device error counter with the value of the second node second device error counter indicated in the synchronization message sent to the first node in response to the first node determining that the value indicated in the synchronization message is greater than the first node second device error counter; and

updating, by the second node, the second node first device error counter with the value of the first node first device error counter indicated in the synchronization message sent to the second node in response to the second node determining that the value indicated in the synchronization message sent to the second node is greater than the second node first device error counter.

14. The method of claim 12, further comprising:

generating, by the first node, the first node second device error counter in response to determining that first node does not include the first node second device error counter for the second device;

setting, by the first node, the first node second device error counter to the value indicated in the synchronization message sent to the first node;

generating, by the second node, the second node first device error counter in response to determining that the second node does not include the second node first device error counter for the first device;

setting, by the second node, the second node first device error counter to the value indicated in the synchronization message sent to the second node.

15. The method of claim 1, further comprising:

initiating, by the second node, a reinitialization operation;

sending, by the second node, a join message to the first node as part of the reinitialization operation;

sending, by the first node, a message to the second node indicating a value of the first node error counter for the device in response to the join message; and

updating, by the second node, the second node error counter to the value indicated in the message sent in response to the join message.

16. A system in communication with a device, comprising:

a first node executing code to perform operations, the operations comprising;

performing an action with respect to a first node error counter for the device in communication with the first node and a second node;

transmitting a message to the second node indicating the device and the action performed with respect to the first node error counter for the device; and

a second node executing code to perform operations, the operations comprising;

performing the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device.

17. The system of claim 16, wherein the first node further detects an error at the device, wherein the action performed by the first node comprises incrementing the first node error counter for the device in response to detecting the error, wherein the action indicated in the message comprises an increment action, wherein performing, by the second node, the increment action comprises incrementing the second node error counter for the device at the second node.

18. The system of claim 16, wherein the first node operates as an owner of the device performing error handling for the device, wherein the second node further performs:

taking over ownership of the device; and

incrementing the second node error counter, indicating errors at the device detected by the first node, in response to the second node detecting an error at the device.

19. The system of claim 16, wherein the device comprises a first device, the first node error counter comprises a first node first device error counter, and the second node error counter comprises a second error first device counter copy,

wherein the second node further maintains a second node second device error counter indicating a number of errors at a second device in communication with the first node and the second node;

and

wherein the first node further maintains a first node second device error counter indicating a number of errors at the second device including errors detected by the second node.

20. The system of claim 19, wherein the first node operates as an owner of the first device performing error handling for the first device and wherein the second node operates as an owner of the second device performing error handling for the second device,

wherein the second node further takes over ownership of the first device, wherein the second node uses the second node first device error counter to perform error management for the first device;

and

wherein the first node further takes over ownership of the second device, wherein the first node uses the first node second device error counter to perform error management for the second device.

21. An article of manufacture including code executed by a first node and second node, wherein a device is in communication with the first node and the second node, wherein the code is executed to cause operations, the operations comprising:

performing, by the first node, an action with respect to a first node error counter for the device;

transmitting, by the first node, a message to the second node indicating the device and the action performed with respect to the first node error counter for the device;

performing, by the second node, the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device.

22. The article of manufacture of claim 21, wherein the operations further comprise:

detecting, by the first node, an error at the device, wherein the action performed by the first node comprises incrementing the first node error counter for the device in response to detecting the error, wherein the action indicated in the message comprises an increment action, wherein performing, by the second node, the increment action comprises incrementing the second node error counter for the device at the second node.

23. The article of manufacture of claim 21, wherein the first node operates as an owner of the device performing error handling for the device, wherein the operations further comprise:

taking over, by the second node, ownership of the device; and

incrementing, by the second node, the second node error counter, indicating errors at the device detected by the first node, in response to the second node detecting an error at the device.

24. The article of manufacture of claim 21, wherein the device comprises a first device, the first node error counter comprises a first node first device error counter, and the second node error counter comprises a second error first device counter copy, wherein the operations further comprise:

maintaining, by the second node, a second node second device error counter indicating a number of errors at a second device in communication with the first node and the second node;

maintaining, by the first node, a first node second device error counter indicating a number of errors at the second device including errors detected by the second node.

25. The article of manufacture of claim 24, wherein the first node operates as an owner of the first device performing error handling for the first device and wherein the second node operates as an owner of the second device performing error handling for the second device, wherein the operations further comprise:

taking over, by the second node, ownership of the first device, wherein the second node uses the second node first device error counter to perform error management for the first device; and

taking over, by the first node, ownership of the second device, wherein the first node uses the first node second device error counter to perform error management for the second device.