Synchronizing device error information among nodes

• US 7,904,752 B2
• Filed: 06/03/2008
• Issued: 03/08/2011
• Est. Priority Date: 06/03/2008
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hood, James Lamar, Sorenson, Todd Charles, Rinaldi, Brian Anthony, Robison, Micah
• Primary Examiner(s): McCarthy; Christopher S
• Application Number:
  US12/132,550
• Publication Number:
  US 20090300436A1
• Time in Patent Office: 1,008 Days
• Field of Search: None
• US Class Current: 714/16
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/0677 Localisation of faults

Abstract

Provided are a method, system, and article of manufacture for synchronizing device error information among nodes. A first node performs an action with respect to a first node error counter for a device in communication with the first node and a second node. The first node transmits a message to the second node indicating the device and the action performed with respect to the first node error counter for the device. The second node performs the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device.

Claims

1. A method, comprising:

performing, by a first node, an action with respect to a first node error counter for a device in communication with the first node and a second node, wherein the first node operates as an owner of the device performing error handling for the device;

transmitting, by the first node, a message to the second node indicating the device and the action performed with respect to the first node error counter for the device;

performing, by the second node, the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device;

taking over, by the second node, ownership of the device; and

incrementing, by the second node, the second node error counter, indicating errors at the device detected by the first node, in response to the second node detecting an error at the device.

2. The method of claim 1, further comprising:

detecting, by the first node, an error at the device, wherein the action performed by the first node comprises incrementing the first node error counter for the device in response to detecting the error, wherein the action indicated in the message comprises an increment action, wherein performing, by the second node, the increment action comprises incrementing the second node error counter for the device at the second node.

3. The method of claim 1, further comprising:

initiating, by the first node, an error recovery operation in response to the error counter value reaching an error threshold.

4. The method of claim 1, wherein the second node takes over the ownership of the device in response to a failure of the first node.

5. A method, comprising:

performing, by a first node, an open action with respect to a first node error counter for a device in communication with the first node and a second node;

transmitting, by the first node, a message to the second node indicating the device and the open action the first node performed with respect to the first node error counter for the device;

performing, by the second node, the open action indicated in the message by creating a second node error counter for the device in response to the message, wherein the second node error counter corresponds to the first node error counter for the device.

6. A method, comprising:

performing, by a first node, an expire action with respect to a first node error counter for a device in communication with the first node and a second node;

transmitting, by the first node, a message to the second node indicating the device and the expire action performed with respect to the first node error counter for the device; and

performing, by the second node, the expire action by expiring a second node error counter for the device in response to the message, wherein the second node error counter corresponds to the first node error counter for the device.

7. A method, comprising:

performing, by a first node, an action with respect to a first node first device error counter for a first device in communication with the first node and a second node;

transmitting, by the first node, a message to the second node indicating the first device and the action performed with respect to the first node first device error counter for the first device;

performing, by the second node, the action indicated in the message with respect to a second node first device error counter for the first device indicated in the message, wherein the second node first device error counter corresponds to the first node first device error counter for the first device;

maintaining, by the second node, a second node second device error counter indicating a number of errors at a second device in communication with the first node and the second node; and

maintaining, by the first node, a first node second device error counter indicating a number of errors at the second device including errors detected by the second node.

8. The method of claim 7, wherein the first and second nodes comprise first and second processing clusters in a server that communicate over a first interface in the server, wherein the first and second devices comprise first and second network adaptors, and wherein the first and second processing clusters communicate with the first and second adaptors over a second interface.

9. The method of claim 7, further comprising:

receiving, by the second node, an error message from the second device;

incrementing, by the second node, the second node second device error counter;

transmitting, by the second node, an increment message to the first node for the second device;

incrementing, by the first node, the first node second device error counter to the second error counter value in response to the increment message.

10. The method of claim 7, wherein the first node operates as an owner of the first device performing error handling for the first device and wherein the second node operates as an owner of the second device performing error handling for the second device, further comprising:

taking over, by the second node, ownership of the first device, wherein the second node uses the second node first device error counter to perform error management for the first device; and

taking over, by the first node, ownership of the second device, wherein the first node uses the first node second device error counter to perform error management for the second device.

11. The method of claim 7, further comprising:

initiating a synchronization operation at the first and second nodes;

sending, by the first node, a synchronization message to the second node indicating a value of the first node first device error counter to the second node in response to the synchronization operation; and

sending, by the second node, a synchronization message to the first node indicating a value of the second node second device error counter to the first node in response to the synchronization operation.

12. The method of claim 11, further comprising:

updating, by the first node, the first node second device error counter with the value of the second node second device error counter indicated in the synchronization message sent to the first node in response to the first node determining that the value indicated in the synchronization message is greater than the first node second device error counter; and

updating, by the second node, the second node first device error counter with the value of the first node first device error counter indicated in the synchronization message sent to the second node in response to the second node determining that the value indicated in the synchronization message sent to the second node is greater than the second node first device error counter.

13. The method of claim 11, further comprising:

generating, by the first node, the first node second device error counter in response to determining that first node does not include the first node second device error counter for the second device;

setting, by the first node, the first node second device error counter to the value indicated in the synchronization message sent to the first node;

generating, by the second node, the second node first device error counter in response to determining that the second node does not include the second node first device error counter for the first device;

setting, by the second node, the second node first device error counter to the value indicated in the synchronization message sent to the second node.

14. A method, comprising:

performing, by a first node, an action with respect to a first node error counter for a device in communication with the first node and a second node;

transmitting, by the first node, a message to the second node indicating the device and the action performed with respect to the first node error counter for the device;

performing, by the second node, the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device;

initiating, by the second node, a reinitialization operation;

sending, by the second node, a join message to the first node as part of the reinitialization operation;

sending, by the first node, a message to the second node indicating a value of the first node error counter for the device in response to the join message; and

updating, by the second node, the second node error counter to the value indicated in the message sent in response to the join message.

15. A system in communication with a device, comprising:

a first node; and

a second node,

wherein the first node executes code to perform operations, the operations comprising;

performing an action with respect to a first node error counter for the device in communication with the first node and the second node, wherein the first node operates as an owner of the device performing error handling for the device;

transmitting a message to the second node indicating the device and the action performed with respect to the first node error counter for the device; and

wherein the second node executes code to perform operations, the operations comprising;

performing the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device;

taking over ownership of the device; and

incrementing the second node error counter, indicating errors at the device detected by the first node, in response to the second node detecting an error at the device.

16. The system of claim 15, wherein the first node further detects an error at the device, wherein the action performed by the first node comprises incrementing the first node error counter for the device in response to detecting the error, wherein the action indicated in the message comprises an increment action, wherein performing, by the second node, the increment action comprises incrementing the second node error counter for the device at the second node.

17. A system in communication with a first device and a second device, comprising:

a first node; and

a second node,

wherein the first node executes code to perform operations, the operations comprising;

performing an action with respect to a first node first device error counter for the first device in communication with the first node and the second node;

transmitting a message to the second node indicating the first device and the action performed with respect to the first node first device error counter for the first device; and

maintains maintaining a first node second device error counter indicating a number of errors at the second device including errors detected by the second node; and

wherein the second node executes code to perform operations, the operations comprising;

performing the action indicated in the message with respect to a second node first device error counter for the first device indicated in the message, wherein the second node first device error counter corresponds to the first node first device error counter for the first device; and

maintaining a second node second device error counter indicating a number of errors at the second device in communication with the first node and the second node.

18. The system of claim 17, wherein the first node operates as an owner of the first device performing error handling for the first device and wherein the second node operates as an owner of the second device performing error handling for the second device,

wherein the second node further takes over ownership of the first device, wherein the second node uses the second node first device error counter to perform error management for the first device;

and

wherein the first node further takes over ownership of the second device, wherein the first node uses the first node second device error counter to perform error management for the second device.

19. An article of manufacture comprising a computer readable storage medium including code executed by a first node and a second node, wherein a device is in communication with the first node and the second node, wherein the code is executed to cause operations, the operations comprising:

performing, by the first node, an action with respect to a first node error counter for the device, wherein the first node operates as an owner of the device performing error handling for the device;

transmitting, by the first node, a message to the second node indicating the device and the action performed with respect to the first node error counter for the device;

performing, by the second node, the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device;

taking over, by the second node, ownership of the device; and

incrementing, by the second node, the second node error counter, indicating errors at the device detected by the first node, in response to the second node detecting an error at the device.

20. The article of manufacture of claim 19, wherein the operations further comprise:

detecting, by the first node, an error at the device, wherein the action performed by the first node comprises incrementing the first node error counter for the device in response to detecting the error, wherein the action indicated in the message comprises an increment action, wherein performing, by the second node, the increment action comprises incrementing the second node error counter for the device at the second node.

21. An article of manufacture comprising a computer readable storage medium including code executed by a first node and a second node, wherein a first device and a second device are in communication with the first node and the second node, wherein the code is executed to cause operations, the operations comprising:

performing, by the first node, an action with respect to a first node first device error counter for the first device;

transmitting, by the first node, a message to the second node indicating the first device and the action performed with respect to the first node first device error counter for the first device;

performing, by the second node, the action indicated in the message with respect to a second node first error counter for the first device indicated in the message, wherein the second node first device error counter corresponds to the first node first device error counter for the first device;

maintaining, by the second node, a second node second device error counter indicating a number of errors at the second device in communication with the first node and the second node; and

maintaining, by the first node, a first node second device error counter indicating a number of errors at the second device including errors detected by the second node.

22. The article of manufacture of claim 21, wherein the first node operates as an owner of the first device performing error handling for the first device and wherein the second node operates as an owner of the second device performing error handling for the second device, wherein the operations further comprise:

taking over, by the second node, ownership of the first device, wherein the second node uses the second node first device error counter to perform error management for the first device; and

taking over, by the first node, ownership of the second device, wherein the first node uses the first node second device error counter to perform error management for the second device.

23. A system in communication with a device, comprising:

a first node; and

a second node,

wherein the first node executes code to perform operations, the operations comprising;

performing an open action with respect to a first node error counter for the device in communication with the first node and the second node;

transmitting a message to the second node indicating the device and the open action performed with respect to the first node error counter for the device; and

wherein the second node executes code to perform the open action indicated in the message by creating a second node error counter for the device in response to the message, wherein the second node error counter corresponds to the first node error counter for the device.

24. A system in communication with a device, comprising:

a first node; and

a second node,

wherein the first node executes code to perform operations, the operations comprising;

performing an expire action with respect to a first node error counter for the device in communication with the first node and the second node;

transmitting a message to the second node indicating the device and the expire action performed with respect to the first node error counter for the device; and

wherein the second node executes code to perform the expire action by expiring a second node error counter for the device in response to the message, wherein the second node error counter corresponds to the first node error counter for the device.

25. A system in communication with a device, comprising:

a first node; and

a second node,

wherein the first node executes code and the second node executes code to perform operations, the operations comprising;

performing, by the first node, an action with respect to a first node error counter for the device in communication with the first node and the second node;

transmitting, by the first node, a message to the second node indicating the device and the action performed with respect to the first node error counter for the device;

performing, by the second node, the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device;

initiating, by the second node, a reinitialization operation;

sending, by the second node, a join message to the first node as part of the reinitialization operation;

sending, by the first node, a message to the second node indicating a value of the first node error counter for the device in response to the join message; and

updating, by the second node, the second node error counter to the value indicated in the message sent in response to the join message.

26. An article of manufacture comprising a computer readable storage medium including code executed by a first node and a second node, wherein a device is in communication with the first node and the second node, wherein the code is executed to cause operations, the operations comprising:

performing, by the first node, an open action with respect to a first node error counter for the device;

transmitting, by the first node, a message to the second node indicating the device and the open action performed with respect to the first node error counter for the device;

performing, by the second node, the open action indicated in the message by creating a second node error counter for the device in response to the message, wherein the second node error counter corresponds to the first node error counter for the device.

27. An article of manufacture comprising a computer readable storage medium including code executed by a first node and a second node, wherein a device is in communication with the first node and the second node, wherein the code is executed to cause operations, the operations comprising:

performing an expire action with respect to a first node error counter for the device in communication with the first node and the second node;

transmitting a message to the second node indicating the device and the expire action performed with respect to the first node error counter for the device; and

performing, by the second node, the expire action by expiring a second node error counter for the device in response to the message, wherein the second node error counter corresponds to the first node error counter for the device.

28. An article of manufacture comprising a computer readable storage medium including code executed by a first node and a second node, wherein a device is in communication with the first node and the second node, wherein the code is executed to cause operations, the operations comprising:

performing, by the first node, an action with respect to a first node error counter for the device in communication with the first node and the second node;

transmitting, by the first node, a message to the second node indicating the device and the action performed with respect to the first node error counter for the device;

performing, by the second node, the action indicated in the message with respect to a second node error counter for the device indicated in the message, wherein the second node error counter corresponds to the first node error counter for the device;

initiating, by the second node, a reinitialization operation;

sending, by the second node, a join message to the first node as part of the reinitialization operation;

sending, by the first node, a message to the second node indicating a value of the first node error counter for the device in response to the join message; and

updating, by the second node, the second node error counter to the value indicated in the message sent in response to the join message.