Failover mechanisms in RDMA operations

• US 20060045005A1
• Filed: 12/20/2004
• Published: 03/02/2006
• Est. Priority Date: 08/30/2004
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Blackmore, Robert S., Govindaraju, Rama K., Hochschild, Peter H., Chaudhary, Piyush, Chang, Fu Chung, Houston, John S., Martin, Steven J., Goscinski, Jason E., Helmer, Leonard W. Jr., Grice, Donald G.
• Granted Patent: US 8,023,417 B2
• Time in Patent Office: Days
• US Class Current: 370/216
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: H04L 1/0079 Formats for control data H04L1/16 takes precedence; training sequences H04L25/00 and H04L27/00
• CPC Class Codes: H04L 1/1809 Selective-repeat protocols
• CPC Class Codes: H04L 2001/0097 Relays

Abstract

In remote direct memory access transfers in a multinode data processing system in which the nodes communicate with one another through communication adapters coupled to a switch or network, failures in the nodes or in the communication adapters can produce the phenomenon known as trickle traffic, which is data that has been received from the switch or from the network that is stale but which may have all the signatures of a valid packet data. The present invention addresses the trickle traffic problem in two situations: node failure and adapter failure. In the node failure situation randomly generated keys are used to reestablish connections to the adapter while providing a mechanism for the recognition of stale packets. In the adapter failure situation, a round robin context allocation approach is used with adapter state contexts being provided with state information which helps to identify stale packets. In another approach to handling the adapter failure situation counts are assigned which provide an adapter failure number to the node which will not match a corresponding number in a context field in the adapter, thus enabling the identification of stale packets.

Claims

1. A method for recovery from failure in a node in a data processing system which includes said node and a communications adapter coupled to the node, said method comprising the steps of:

restarting said failed node and generating keys, for data packets to be transferred to said adapter, based on a random number seeded key generator;

checking, by at least one of said adapters, that a key provided within data packet headers received by said at least one adapter matches a key loaded by a device driver into a table resident in the respective one of said nodes;

dropping said packet if said checking indicates a failed match.

2. The method of claim 1 further including the step of signaling to said restarted node that a fatal error has occurred with respect to said packet.

3. The method of claim 2 in which said node reprocesses said packet.

4. The method of claim 3 in which said reprocessing comprises retransmission of said packet.

5. The method of claim 3 in which said reprocessing comprises a request for retransmission of said packet.

6. The method of claim 1 in which said data packets are transferred in accordance via remote direct memory access protocol.

7. The method of claim 1 in which said headers include a data structure used to define portable address mappings between adapters for remote direct memory access.

8. The method of claim 1 in which said node resident table is a translation control entry table.

9. A method for recovery from failure in a communications adapter coupled to a node in a data processing system which includes said node and said communications adapter, said method comprising the steps of:

initiating recovery operations within said failed adapter;

acquiring, by said adapter, from said node, adapter state information from a set of available adapter state contexts used for remote direct memory access, said state contexts being supplied in round robin fashion from said set of available state contexts;

firstly checking, by said failed adapter, to insure that said acquired state context is owned by a current window by checking a window identifier field in said state context in memory in said adapter; and

secondly, checking, by said failed adapter, to verify that a state context identifier key for said remote direct memory access matches a state context identifier key field in adapter memory.

10. The method of claim 9 further including the step of discarding stale remote direct memory access packets which fail said second checking.

11. The method of claim 9 in which each state context stores sate information for a single active remote direct memory access operation.

12. The method of claim 9 in which each state context includes data selected from at least one of the group consisting of a linked list pointer, a local adapter window identifier, two remote direct memory access enabling addresses, payload length, and an identifier for said adapter and an adapter window identifier.

13. The method of claim 9 in which said window identifier field and said state context identifier key field in adapter memory are initialized by a device driver running on said node when said state contexts are assigned to specific windows.

14. A method for determining failure in a communications adapter coupled to a node in a data processing system which includes said node and said communications adapter, said method comprising the steps of:

maintaining within said node a count of adapter recovery events for said adapter;

establishing an adapter state information context residing within said node and in said adapter, said state information context including said count;

transferring data packets with packet header information including said count; and

comparing said count in said data packet header with said count established in said adapter to determine that an adapter failure has occurred.

15. The method of claim 14 in which said maintaining is carried out by a device driver residing within said node.

16. The method of claim 14 in which there is a plurality of adapters coupled to said node and a count of adapter recovery events is maintained for each adapter.

17. The method of claim 14 in which said packet header further includes a designated name for said state information context.

18. A multinode data processing system comprising:

a plurality of data processing nodes together with a respective plurality of communications adapters coupled thereto which enable said nodes to communicate through a switch or network to which said adapters are coupled; and

programming means within said nodes and said adapters for restarting a failed node and for generating keys for data packets to be transferred to said adapters, based on a random number seeded key generator; and

for checking, by at least one of said adapters, that a key provided within data packet headers received by said at least one adapter matches a key loaded by a device driver into a table resident in the respective one of said nodes; and

for dropping said packet if said checking indicates a failed match.

19. A multinode data processing system comprising:

a plurality of data processing nodes together with a respective plurality of communications adapters coupled thereto which enable said nodes to communicate through a switch or network to which said adapters are coupled; and

programming means within said nodes and said adapters for initiating recovery operations within a failed adapter;

for acquiring, by said failed adapter, from the node coupled thereto, adapter state information from a set of available adapter state contexts used for remote direct memory access, said state contexts being supplied in round robin fashion from said set of available state contexts; and

for firstly checking, by said failed adapter, to insure that said acquired state context is owned by a current window by checking a window identifier field in said state context in memory in said adapter; and

secondly, for checking, by said failed adapter, to verify that a state context identifier key for said remote direct memory access matches a state context identifier key field in adapter memory.

20. A multinode data processing system comprising:

a plurality of data processing nodes together with a respective plurality of communications adapters coupled thereto which enable said nodes to communicate through a switch or network to which said adapters are coupled; and

programming means within said nodes and said adapters for maintaining within said node a count of adapter recovery events for said adapter; and

for establishing adapter state information contexts residing within said nodes and in said adapters, said state information context including said count; and

for transferring data packets with packet header information including said count; and

for comparing said count in said data packet header with said count established in said adapter to determine that an adapter failure has occurred.