Failover mechanisms in RDMA operations

• US 8,023,417 B2
• Filed: 12/20/2004
• Issued: 09/20/2011
• Est. Priority Date: 08/30/2004
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Grice, Donald G., Blackmore, Robert S., Govindaraju, Rama K., Hochschild, Peter H., Chaudhary, Piyush, Chang, Fu Chung, Houston, John S., Martin, Steven J., Goscinski, Jason E., Helmer, Leonard W. Jr.
• Primary Examiner(s): Ngo; Ricky
• Assistant Examiner(s): HAN, CLEMENCE S
• Application Number:
  US11/017,574
• Publication Number:
  US 20060045005A1
• Time in Patent Office: 2,465 Days
• Field of Search: None
• US Class Current: 370/242
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: H04L 1/0079 Formats for control data H04L1/16 takes precedence; training sequences H04L25/00 and H04L27/00
• CPC Class Codes: H04L 1/1809 Selective-repeat protocols
• CPC Class Codes: H04L 2001/0097 Relays

Abstract

In remote direct memory access transfers in a multinode data processing system in which the nodes communicate with one another through communication adapters coupled to a switch or network, failures in the nodes or in the communication adapters can produce the phenomenon known as trickle traffic, which is data that has been received from the switch or from the network that is stale but which may have all the signatures of a valid packet data. The present invention addresses the trickle traffic problem in two situations: node failure and adapter failure. In the node failure situation randomly generated keys are used to reestablish connections to the adapter while providing a mechanism for the recognition of stale packets. In the adapter failure situation, a round robin context allocation approach is used with adapter state contexts being provided with state information which helps to identify stale packets.

Claims

1. A method for recovery from failure in a sending node in a data processing system which includes said sending node and at least one communications adapter coupled to the sending node, said method comprising:

receiving at a receiving adapter of the data processing system a data packet transferred via remote direct memory access protocol from the sending node after restarting of the sending node, the data packet comprising a data packet header with a new key generated by the sending node, after restarting thereof, based on a random number seeded key generator, the new key being provided in a data structure of the data packet header of the data packet received by said receiving adapter;

checking, by said receiving adapter, that the new key provided within the data structure of the data packet header of the data packet received by said receiving adapter matches a key loaded by a device driver into a table of a plurality of tables resident in a node in said data processing system coupled to said receiving adapter, the plurality of tables shared by the receiving adapter and one or more other adapters coupled to the node and mapping to memory buffers for remote direct memory access via the receiving adapter and the one or more other adapters, the data structure referencing the table of the plurality of tables, and the checking comprising using, by the receiving adapter, the data structure to reference the table of the plurality of tables to obtain the key loaded therein by the device driver; and

responsive to the checking indicating a failed match, dropping the received data packet at the receiving adapter and signaling to the sending node that a fatal error has occurred with respect to the data packet transferred via remote direct memory access protocol.

2. The method of claim 1 in which the sending node reprocesses the data packet responsive to the receiving node signaling the sending node.

3. The method of claim 2 in which said reprocessing comprises retransmission, via remote direct memory access protocol, of said data packet.

4. The method of claim 2 in which said reprocessing is responsive to a request for retransmission of said data packet made by said node coupled to said receiving adapter.

5. The method of claim 1 in which the data structure in the data packet header is further used to define portable address mappings between the receiving adapter and the one or more other adapters for remote direct memory access.

6. The method of claim 1 in which said node resident table is a translation control entry table.

7. A method for recovery from failure in a communications adapter coupled to a node in a data processing system which includes said node and said communications adapter, said method comprising the steps of:

initiating recovery operations within said failed adapter;

acquiring, by said adapter, from said node, adapter state information from a set of available adapter state contexts used for remote direct memory access, said state contexts being supplied in round robin fashion from said set of available state contexts;

firstly checking, by said failed adapter, to insure that said acquired adapter state information is owned by a current window by checking a window identifier field in memory in said adapter; and

secondly, checking, by said failed adapter, to verify that a state context identifier key for a received remote direct memory access data packet matches a state context identifier key field in adapter memory.

8. The method of claim 7 further including the step of discarding stale remote direct memory access packets which fail said second checking.

9. The method of claim 7 in which each state context stores state information for a single active remote direct memory access operation.

10. The method of claim 7 in which each state context includes data selected from at least one of the group consisting of a linked list pointer, a local adapter window identifier, two remote direct memory access enabling addresses, payload length, an identifier for said adapter and an adapter window identifier.

11. The method of claim 7 in which said window identifier field and said state context identifier key field in adapter memory are initialized by a device driver running on said node when said state contexts are assigned to specific windows.

12. A method for determining failure in a communications adapter coupled to a node in a data processing system which includes said node and said communications adapter, said method comprising the steps of:

maintaining within said node a count of adapter recovery events for said adapter;

establishing an adapter state information context residing within said node and in said adapter, said state information context including an initial indication of said count;

transferring data packets with packet header information including count information present in said node; and

comparing said count information of said packet header information with said count included in said established adapter state information context residing in said adapter to determine that an adapter failure has occurred.

13. The method of claim 12 in which said maintaining is carried out by a device driver residing within said node.

14. The method of claim 12 in which there is a plurality of adapters coupled to said node and a count of adapter recovery events is maintained for each adapter.

15. The method of claim 12 in which said packet header further includes a designated name for said state information context.

16. A multinode data processing system comprising:

a plurality of data processing nodes together with a respective plurality of communications adapters coupled respectively thereto which enable said nodes to communicate through a switch or network to which said adapters are coupled; and

programming means within said nodes and said adapters for receiving at a receiving adapter of the data processing system a data packet transferred via remote direct memory access protocol from a sending node after restarting of the sending node, the data packet comprising a data packet header with a new key generated by the sending node, after restarting thereof, based on a random number seeded key generator, the new key being provided in a data structure of the data packet header of the data packet received by said receiving adapter; and

for checking, by the receiving adapter, that the new key provided within the data structure of the data packet header of the data packet received by said receiving adapter matches a key loaded by a device driver into a table of a plurality of tables resident in the node coupled to said receiving adapter, the plurality of tables shared by the receiving adapter and one or more other adapters coupled to the node and mapping to memory buffers for remote direct memory access via the receiving adapter and the one or more other adapters, the data structure referencing the table of the plurality of tables, and the checking comprising using, by the receiving adapter, the data structure to reference the table of the plurality of tables to obtain the key loaded therein by the device driver; and

responsive to said checking indicating a failed match, dropping the received data packet at the receiving adapter and signaling to the sending node that a fatal error has occurred with respect to the data packet transferred via remote direct memory access protocol.

17. A multinode data processing system comprising:

a plurality of data processing nodes together with a respective plurality of communications adapters coupled respectively thereto which enable said nodes to communicate through a switch or network to which said adapters are coupled; and

programming means within said nodes and said adapters for;

initiating recovery operations within a failed adapter;

for acquiring, by said failed adapter, from its respectively coupled node, adapter state information from a set of available adapter state contexts used for remote direct memory access, said state contexts being supplied in round robin fashion from said set of available state contexts; and

for firstly checking, by said failed adapter, to insure that said acquired adapter state information is owned by a current window by checking a window identifier field in said adapter state information in memory in said adapter; and

secondly, for checking, by said failed adapter, to verify that a state context identifier key for a received remote direct memory access data packet matches a state context identifier key field in adapter memory.

18. A multinode data processing system comprising:

a plurality of data processing nodes together with a respective plurality of communications adapters coupled respectively thereto which enable said nodes to communicate through a switch or network to which said adapters are coupled; and

programming means within said nodes and said adapters for;

maintaining, within said nodes, counts of adapter recovery events for respective ones of said adapters; and

for establishing adapter state information contexts residing within said nodes and in said adapters, said adapter state information contexts including respective ones of said counts of adapter recovery events; and

for transferring data packets with packet header information including count information present in the nodes; and

for comparing said count information of said packet header information with said counts of adapter recovery events included in said established adapter state information contexts residing in said adapters to determine that an adapter failure has occurred.