Inifiniband channel adapter for performing direct DMA between PCI bus and inifiniband link

• US 6,594,712 B1
• Filed: 10/20/2000
• Issued: 07/15/2003
• Est. Priority Date: 10/20/2000
• Status: Expired due to Term

• Current Assignee: Intel Corporation
• Original Assignee: Banderacom Inc (Intel Corporation)
• Inventors: Pettey, Christopher, Rubin, Lawrence H.
• Primary Examiner(s): HUYNH, KIM NGOC
• Application Number:
  US09/693,405
• Time in Patent Office: 998 Days
• Field of Search: 710/5-7, 710/22-28, 710/36-51, 709/208-216
• US Class Current: 710/22
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 13/385 for adaptation of a particular data processing system to different peripheral devices
• CPC Class Codes: G06F 13/404 with address mapping

Abstract

An Infiniband channel adapter for performing direct data transfers between a PCI bus and an Infiniband link without double-buffering the data in system memory. A local processor programs the channel adapter to decode addresses in a range of the PCI bus address space dedicated to direct transfers. When an I/O controller attached to the PCI bus transfers data from an I/O device to an address in the dedicated range, the channel adapter receives the data into an internal buffer and creates an Infiniband RDMA Write packet for transmission to virtual address within a remote Infiniband node. When the channel adapter receives an Infiniband RDMA Read Response packet, the channel adapter provides the packet payload data to the I/O controller at a PCI address in the dedicated range. A plurality of programmable address range registers facilitates multiple of the direct transfers concurrently by dividing the dedicated address range into multiple sub-ranges. The address range registers enable random mapping between the address sub-ranges and multiple internal buffers for receiving and transmitting Infiniband RDMA packets.

Claims

1. An Infiniband channel adapter, comprising:

a local bus interface, for coupling the channel adapter to an I/O controller by a local bus, configured to receive data from the I/O controller if a local bus address of said data is within a predetermined address range of the local bus address space; and

a bus router, in communication with said local bus interface, configured to create an Infiniband remote direct memory access (RDMA). Write packet including said data in response to said local bus interface receiving said data from the I/O controller for transmission to a remote Infiniband node previously requesting said data.

2. The Infiniband channel adapter of claim 1, further comprising:

a plurality of programmable registers, corresponding to a plurality of address sub-ranges within said predetermined address range of the local bus address space.

3. The Infiniband channel adapter of claim 2, wherein said bus router is further configured to associate said local bus address with a work queue element based on which one of said plurality of address sub-ranges said local bus address is within.

4. The Infiniband channel adapter of claim 3, wherein said work queue element includes a virtual address of a location in a memory in the remote Infiniband node for storing said data received from the I/O controller.

5. The Infiniband channel adapter of claim 4, wherein said bus router is further configured to populate a virtual address field in said Infiniband RDMA Write packet with said virtual address in said work queue element.

6. The Infiniband channel adapter of claim 2, wherein each of said plurality of programmable registers includes a transfer length portion, for indicating a quantity of said data to be provided by the I/O controller for inclusion in said Infiniband RDMA Write packet.

7. The Infiniband channel adapter of claim 6, wherein each of said plurality of programmable registers further includes a count portion for indicating a quantity of said data currently transferred from the I/O controller to the channel adapter by said local bus interface.

8. The Infiniband channel adapter of claim 7, wherein said bus router is further configured to transmit said Infiniband RDMA Write packet to the remote Infiniband node in response to said count portion being equal to said transfer length portion.

9. The Infiniband channel adapter of claim 2, wherein each of said plurality of programmable registers includes a valid indicator, wherein said local bus interface is configured to receive said data from the I/O controller only if said valid indicator is programmed to a predetermined value.

10. The Infiniband channel adapter of claim 2, further comprising:

a memory, coupled to said local bus interface, for receiving said data from said local bus interface.

11. The Infiniband channel adapter of claim 10, wherein each of said plurality of programmable registers includes a block number portion, wherein said block number portion specifies one of a plurality of blocks within said memory for storing Infiniband RDMA Write packets.

12. The Infiniband channel adapter of claim 11, wherein said local bus interface is configured to translate said local bus address into an address of a location within one of said plurality of blocks within said memory specified by said block number portion.

13. The Infiniband channel adapter of claim 12, wherein said local bus interface is configured to translate said local bus address into said memory block address based on a size of a header of said Infiniband RDMA Write packet.

14. The Infiniband channel adapter of claim 11, wherein said local bus interface is configured to translate said local bus address into said memory block address based on a size of said plurality of address sub-ranges.

15. The Infiniband channel adapter of claim 1, wherein said local bus interface is a PCI bus interface.

16. The Infiniband channel adapter of claim 1, further comprising:

a programmable configuration register, for specifying a base address and length of said predetermined address range of the local bus address space.

17. The Infiniband channel adapter of claim 1, wherein said local bus interface is further configured to receive second data from a second I/O controller if a second local bus address of said second data is within said predetermined address range of the local bus address space, wherein said bus router is further configured to include said second data in said Infiniband RDMA Write packet.

18. An Infiniband channel adapter, comprising:

a bus router, configured to receive an Infiniband RDMA Read Response packet, having a payload of data, transmitted by a remote Infiniband node; and

a local bus interface, in communication with said bus router, configured to provide said payload of data to an I/O controller coupled to said local bus interface by a local bus if a local bus address specified by the I/O controller is within a predetermined address range of the local bus address space.

19. The Infiniband channel adapter of claim 18, further comprising:

a plurality of programmable registers, corresponding to a plurality of address sub-ranges within said predetermined address range of the local bus address space.

20. The Infiniband channel adapter of claim 19, wherein each of said plurality of programmable registers includes a valid indicator, wherein said bus router is configured to set said valid indicator upon reception of said Infiniband RDMA Read Response packet.

21. The Infiniband channel adapter of claim 20, wherein said local bus interface is configured to provide said data to the I/O controller in response to said valid indicator being set by said bus router.

22. The Infiniband channel adapter of claim 19, wherein each of said plurality of programmable registers includes a payload length portion, wherein said bus router is configured to store a length of said payload of data of said Infiniband RDMA Read Response packet in said payload length portion of said programmable register in response to reception of said packet.

23. The Infiniband channel adapter of claim 19, wherein said bus router is configured to locate a work queue element associated with said Infiniband RDMA Read Response packet in response to reception of said packet, wherein said work queue element includes an indication of which of said plurality of address sub-ranges is allocated to said Infiniband RDMA Read Response packet.

24. The Infiniband channel adapter of claim 19, further comprising:

a memory, coupled to said local bus interface, for receiving said Infiniband RDMA Read Response packet.

25. The Infiniband channel adapter of claim 24, wherein each of said plurality of programmable registers includes a block number portion, wherein said block number portion specifies one of a plurality of blocks within said memory for storing Infiniband RDMA Read Response packets.

26. The Infiniband channel adapter of claim 25, wherein said bus router is configured to store in said block number portion an indication of which of said plurality of blocks said Infiniband RDMA Read Response packet is stored in.

27. The Infiniband channel adapter of claim 18, wherein said local bus interface is configured to command the I/O controller to retry a request for said payload of data if said Infiniband RDMA Read Response packet has not been received by said bus router.

28. The Infiniband channel adapter of claim 18, wherein said local bus interface is configured to provide a first portion of said payload of data to the I/O controller and to provide a second portion of said payload of data to a second I/O controller coupled to said local bus interface if a second local bus address specified by the second I/O controller is within said predetermined address range of the local bus address space.

29. An Infiniband I/O unit, comprising:

an Infiniband channel adapter;

an I/O controller, coupled to said channel adapter by a local bus; and

a processor, for programming said I/O controller to transfer data to said channel adapter on the local bus at an address within a predetermined address range of the local bus address space dedicated for direct data transfers from said I/O controller to said channel adapter;

wherein said channel adapter is configured to receive said data from said I/O controller and to create an Infiniband RDMA Write packet including said data for transmission to a remote Infiniband node only if said address is within said predetermined address range.

30. The Infiniband I/O unit of claim 29, wherein said processor is further configured to provide to said channel adapter a virtual address of a memory location in the remote Infiniband node for inclusion by said channel adapter in said Infiniband RDMA Write packet.

31. The Infiniband I/O unit of claim 29, wherein said processor is further configured to allocate one of a plurality of address sub-ranges of said predetermined address range to a work queue element containing information used to create said Infiniband RDMA Write packet.

32. The Infiniband I/O unit of claim 29, wherein said channel adapter comprises a memory for receiving said data from said I/O controller.

33. The Infiniband I/O unit of claim 32, wherein said channel adapter is further configured to create said Infiniband RDMA Write packet including said data within said memory.

34. The Infiniband I/O unit of claim 29, wherein said processor is configured to program said I/O controller to transfer said data to said channel adapter at said address and to program said channel adapter to receive said data from said I/O controller and to create said Infiniband RDMA Write packet in response to receiving an out-of-band I/O read request Infiniband packet from the remote Infiniband node.

35. An Infiniband I/O unit, comprising:

an Infiniband channel adapter, for receiving an Infiniband RDMA Read Response packet including a payload of data transmitted from a remote Infiniband node;

an I/O controller, coupled to said channel adapter by a local bus; and

a processor, for programming said I/O controller to transfer said data in said payload from said channel adapter on the local bus at an address within a predetermined address range of the local bus address space dedicated for direct data transfers from said channel adapter to said I/O controller;

wherein said channel adapter is configured to provide said data to said I/O controller only if said address is within said predetermined address range.

36. The Infiniband I/O unit of claim 35, wherein said processor is further configured to program said channel adapter to transmit an Infiniband RDMA Read Request packet to the remote Infiniband node in response to reception of an out-of-band I/O write request from the remote Infiniband node prior to receiving said Infiniband RDMA Read Response packet.

37. The Infiniband I/O unit of claim 35, wherein said channel adapter is further configured to determine from said Infiniband RDMA Read Response packet which of a plurality of address sub-ranges of said predetermined address range was previously allocated for use by said I/O controller for transferring said data.

38. A method for translating Infiniband remote virtual addresses to local addresses, comprising:

receiving in a first Infiniband packet a first virtual address of a first memory location in a remote Infiniband node, by a local Infiniband node;

allocating a local address within a local address space of a local bus on said local node for transferring first data directly between an I/O controller of said local node and an Infiniband channel adapter of said local node in response to said receiving said first virtual address;

receiving in a second Infiniband packet a second virtual address of a second memory location in said remote Infiniband node, by said local Infiniband node, wherein said first and second virtual addresses are spatially disparate; and

allocating said local address for transferring second data directly between said I/O controller and said channel adapter in response to said receiving said second virtual address.

39. The method of claim 38, further comprising:

programming a configuration register for specifying an address range within said local address space dedicated to direct transfers of data between said I/O controller and said channel adapter, wherein said local address is within said address range, prior to said receiving said first virtual address.

40. A method for translating Infiniband remote virtual addresses to local addresses, comprising:

receiving in a first Infiniband packet a first virtual address of a first memory location in a first remote Infiniband node, by a local Infiniband node;

allocating a local address within a local address space of a local bus on said local node for transferring first data directly between an I/O controller of said local node and an Infiniband channel adapter of said local node in response to said receiving said first virtual address;

receiving in a second Infiniband packet a second virtual address of a second memory location in a second remote Infiniband node, by said local Infiniband node; and

allocating said local address for transferring second data directly between said I/O controller and said channel adapter in response to said receiving said second virtual address.

41. A method for translating Infiniband remote virtual addresses to local addresses, comprising:

receiving in a first Infiniband packet a virtual address of a memory location in a remote Infiniband node, by a local Infiniband node;

allocating a first local address within a local address space of a local bus on said local node for transferring first data directly between an I/O controller of said local node and an Infiniband channel adapter of said local node in response to said receiving said virtual address in said first packet;

receiving in a second Infiniband packet said virtual address of said memory location in said remote Infiniband node, by said local Infiniband node; and

allocating a second local address for transferring second data directly between said I/O controller and said channel adapter in response to said receiving said virtual address in said second packet.