System and method for supporting access to multiple I/O hub nodes in a host bridge

• US 6,920,519 B1
• Filed: 05/10/2000
• Issued: 07/19/2005
• Est. Priority Date: 05/10/2000
• Status: Expired due to Term

• Current Assignee: Intellectual Discovery Co., Ltd.
• Original Assignee: International Business Machines Corporation
• Inventors: Jackowski, Stefan Peter, Verdoorn, Jr., William Garrett, Williams, Phillip G, Beukema, Bruce Leroy, Bronson, Timothy Carl, Fuhs, Ronald Edward, Gilda, Glenn David, Bybell, Anthony J
• Primary Examiner(s): Rinehart, Mark H.
• Assistant Examiner(s): Lee, Christopher E.
• Application Number:
  US09/569,059
• Time in Patent Office: 1,896 Days
• Field of Search: 710/48, 710/260, 710/307, 710/308, 710/311, 710/22, 711/206, 711/144, 711/170, 711/154, 711/202, 709/1, 713/201, 712/228, 370/402
• US Class Current: 710/306
• CPC Class Codes: G06F 12/0284 Multiple user address space allocation, e.g. using different base addresses interprocessor communication G06F15/163
• CPC Class Codes: G06F 12/1027 using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB]
• CPC Class Codes: G06F 13/404 with address mapping
• CPC Class Codes: G06F 2212/682 Multiprocessor TLB consistency

Abstract

Dynamic routing of data to multiple processor complexes. PCI address space is subdivided among a plurality of processor complexes. Translation table entries at each processor complex determine which processor complex is to receive a DMA transfer, thereby enabling routing of DMA data to one I/O hub node while accessing translation table entries at another I/O hub node. Further, interrupt requests may be dynamically routed to multiple processor complexes.

Claims

1. An input/output system comprising:

a host bridge connected by a bus to I/O adapters;

a plurality of input/output hub nodes;

means for subdividing address space for said bus among said plurality of input/output hub nodes;

a translation table containing a plurality of translation table entries at each of a plurality of said hub nodes;

a translation table entry in said translation table containing a hub ID field for determining which of said hub nodes is to receive a data transfer from said host bridge;

means for routing of data from an I/O device to one hub node using said hub ID field from said translation table entry while accessing said translation table entry at another hub node using said means for subdividing address space;

a plurality of registers containing a hub ID field and a system memory address field;

an address associated with said bus originating from said I/O adapter;

means responsive to said means for subdividing address space for selecting one of said plurality of registers;

means responsive to said hub ID field from said selected register for determining which hub node to use for accessing said translation table entry; and

means for combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said hub node.

2. The input/output system of claim 1, a plurality of said hub nodes comprising a processor complex.

3. The input/output system of claim 1, further comprising a cache of said translation table entries in said host bridge for dynamically routing data to a plurality of said hub nodes.

4. A method for operating an input/output system including a host bridge and a bus connecting said host bridge to a plurality of input/output hub nodes, comprising the steps of:

operating a translation table containing a plurality of translation table entries at each of a plurality of said hub nodes;

subdividing address space for said bus among said plurality of input/output hub nodes for determining which said hub node and corresponding translation table to use to translate an address;

providing a translation table entry in said translation table for determining which of said hub nodes is to receive a data transfer;

routing data from said host bridge to one hub node responsive to said translation table entry while accessing translation table entries at another hub node responsive to said subdivided address space;

providing a plurality of resisters containing a hub ID field and a system memory address field;

receiving an address associated with said bus originating from said I/O adapter;

responsive to said subdividing address space, selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which hub node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said hub node.

5. An I/O method, comprising:

providing at each of a plurality of processor complexes a translation table containing a plurality of translation table entries;

operating a host bridge to fetch and cache said translation table entries containing hub identifiers and receive input/output data transfer requests;

connecting said host bridge over a bus to said plurality of processor complexes;

subdividing address space for said bus among said plurality of processor complexes for determining which hub node and corresponding translation table will be used to translate a requested address;

said host bridge fetching translation table entries from said complex selected responsive to said subdivided address space;

providing a plurality of registers containing a hub ID field and a system memory address field;

receiving an address associated with said bus originating from an I/O adapter;

responsive to said subdividing address space, selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which hub node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said hub node.

6. A coherency system, comprising:

a plurality of processor complexes, each complex having a translation table containing a plurality of translation table entries;

a plurality of host bridges for fetching and caching said translation table entries and receiving input/output data transfer requests;

a plurality of translation table caches associated with said host bridges;

at least one DMA read module associated with said host bridge;

a plurality of DMA read data sources, at least one said source associated with each said complex;

a coherency protocol for assuring coherency for cached translation table entries and DMA read data originating from at least two of said complexes, with translation data fetched from one said complex used to transfer data with another said complex;

a plurality of resisters containing a hub ID field and a system memory address field;

an address originating from an I/O adapter;

means for subdividing address space for said bus;

means responsive to said means for subdividing address space for selecting one of said plurality of registers;

means responsive to said hub ID field from said selected register for determining which complex to use for accessing said translation table entry; and

means for combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry within said address space.

7. An I/O system, comprising:

a PCI memory address space;

a system memory address space;

a host bridge node for translating addresses from said PCI memory address space to said system memory address space including

a plurality of registers containing a hub ID field and a system memory address field;

an address originating from an I/O adapter;

means for subdividing address space for said bus;

means responsive to said means for subdividing address space for selecting one of said plurality of registers;

means responsive to said hub ID field from said selected register for determining which complex to use for accessing said translation table entry; and

means for combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry within said address space

said host bridge node further for fetching translation table entries from a first processor complex node to determine a location in said system memory address space to which a direct memory address operation will be directed to transfer data to a second processor complex node;

said host bridge node, said first processor complex node and said second processor complex node interconnected by a packet-based connection protocol; and

means for responding to a failure of said second processor complex node by dynamically reconfiguring access to said first processor complex node due to a failure.

8. The system of claim 7, further comprising:

said host bridge node being further operable for simultaneously flowing data packets to at least two of said host bridge node, first processor complex node and second processor complex node;

said translation table entry containing node identifier fetched by said host bridge node; and

means for accessing translation table entries from a first I/O hub on said first processor complex node while routing DMA data responsive to said node identifier from said translation table entry to a second I/O hub on said second processor complex node.

9. An I/O method for transferring packets with respect to PCI memory address space and system memory address space including a plurality of nodes interconnected by a packet-based connection protocol, comprising the steps of:

operating a host bridge node to translate addresses from said PCI memory address space to said system memory address space, including;

providing a translation table containing a plurality of translation table entries at each of a plurality of said nodes;

providing a plurality of registers containing a hub ID field and an associated system memory address field;

receiving an address associated with said PCI memory address space originating from an I/O adapter;

subdividing said PCI memory address space, and responsive to said subdividing, selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which hub node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said hub node;

fetching translation table entries from a first processor complex node for determining a location in said system memory space to which a direct memory address operation will be directed to transfer data to a second processor complex node; and

responsive to a failure of said second processor complex node, dynamically reconfiguring access to said first processor complex node.

10. A high availability I/O system, comprising:

a plurality of input/output nodes;

a host bridge including an input/output adapter with connections to at least one PCI bus;

a communication link connecting said host bridge to said plurality of input/output nodes;

said host bridge including a plurality of TCE address registers, each said register associated with a PCI address range on each of said PCI busses and including node ID information; and

means for responding to said node ID information for fetching TCE table entries for associating PCI address range with a system memory address from system memory at any of a plurality of said input/output nodes, including

an address associated with said PCI bus originating from said input/output adapter;

a plurality of registers containing a hub ID field and a system memory address field;

means for subdividing address space for said PCI bus among said plurality of input/output nodes;

means responsive to said means for subdividing address space for selecting one of said plurality of registers;

a translation table containing a plurality of translation table entries at each said input/output node;

means responsive to said hub ID field from said selected register for determining which input/output node to use for accessing said translation table entry; and

means for combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said input/output node.

11. The system of claim 10, a plurality of said input/output nodes including a processor complex.

12. The system of claim 11, each said processor complex including an input/output hub as a connection point for connecting a multiprocessor and a system memory to said communication link.

13. The system of claim 12, said input/output hub being operable for tracking the state of TCE entries cached with a host bridge and issuing a block dkill packet when a TCE entry has been updated in said system memory.

14. The system of claim 13, said host bridge being operable in response to said block dkill packet for invaliding matching TCE entries at said host bridge.

15. The system of claim 12, said input/output hub being operable responsive to modification of data in said system memory for invalidating stale copies of that data in said host bridge.

16. The system of claim 10, said host bridge further comprising means for forking write operations to at least two of said input/output nodes selectively including processor complexes, system memories, and input/output hubs.

17. The system of claim 10, further comprising a plurality of host adapters, each host adapter for connecting a host to a host bridge connected to said communication link.

18. The system of claim 10, further comprising:

means for responding to one or more bits from a PCI address by routing a TCE fetch request packet to one of said plurality of input/output nodes.

19. The system of claim 18, further comprising:

means for responding to one or more bits selected from a TCE entry by identifying an input/output node for routing DMA data.

20. The system of claim 19, further comprising said input/output node for routing DMA data being different from the input/output node to which said TCE fetch request packet is directed.

21. The system of claim 10, said host bridge including an interrupt level control register for routing interrupts to different input/output nodes.

22. The system of claim 10, further comprising:

means for selectively providing in a destination node identifier field in a packet a node identifier for routing a DMA operation or a node identifier for a TCE fetch operation.

23. The system of claim 22, further comprising means for obtaining said node identifier for said TCE fetch operation from one of a plurality of TCE address registers at said host bridge selected responsive to said PCI address range.

24. The system of claim 23, further comprising means for obtaining said node identifier for said DMA operation from a hub ID identifier returned responsive to said TCE fetch operation.

25. The system of claim 23, further comprising means for obtaining said node identifier for an interrupt from an interrupt register at said host bridge, one such interrupt register maintained for each of a plurality of host bridge domains.

26. A method for operating an I/O system including a plurality of I/O nodes, a host, a host bridge connected to said host, and a communication link connecting said host bridge to said plurality of I/O nodes, comprising the steps of:

associating each of a plurality of address registers containing node identifier indicia at said host bridge with an address range; and

responsive to said node ID information, fetching a TCE table entry, said TCE table entry associating a PCI address with a system memory address at any of a plurality of said I/O nodes;

said associating and fetching steps including

subdividing said address range among said plurality of I/O nodes;

providing a translation table containing a plurality of translation table entries at each of a plurality of said I/O nodes;

providing a plurality of registers containing a hub ID field and an associated system memory address field;

receiving an address associated with said address range originating from an I/O adapter;

responsive to said subdividing, selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which I/O node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said I/O node.

27. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for operating an input/output system including a plurality of input/output hubs connected to a bus, said method steps comprising:

subdividing address space for said bus among said plurality of input/output hub nodes for determining which hub node and corresponding translation table will be used to translate a requested address;

operating a translation table containing a plurality of translation table entries at each of a plurality of said hub nodes for determining which of said hub nodes is to receive a data transfer;

routing data to one hub node using said translation table entries from said translation table while accessing said translation table entries at another hub node using said subdivided address space for determining location of said translation table entries;

providing a plurality of registers containing a hub ID field and an associated system memory address field;

receiving an address associated with said address range originating from an input/output adapter;

responsive to said subdividing selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which input/output hub node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said input/output hub node.

28. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for operating an I/O system, said method steps comprising:

providing at each of a plurality of processor complex nodes a translation table containing a plurality of translation table entries;

providing a host bridge node connected to said plurality of processor complex nodes over a bus for fetching and caching said translation table entries containing hub identifiers and receiving input/output data transfer requests;

subdividing address space for said bus for determining which hub node and corresponding translation table to use to translate a requested address;

said host bridge fetching said translation table entries from said complex node selected responsive to said subdivided address space by

providing a plurality of registers containing a hub ID field and a system memory address field;

receiving an address associated with said bus originating from an I/O adapter;

responsive to said subdividing address space, selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which hub node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said hub node.

29. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for operating an interrupt system, said method steps comprising;

interconnecting a plurality of processor complex nodes and host bridge nodes;

interconnecting an I/O device through an I/O adapter to a host bridge nodes;

setting hub node identifier fields identifying said hub node to send interrupt packets;

setting pointer fields, said pointer fields pointing to said hub node identifier fields and being identifiable with different interrupt levels; and

said setting steps including

providing a plurality of registers containing a hub ID field and a system memory address field;

receiving an address associated with said bus originating from said I/O adapter;

subdividing address space, and responsive to said subdividing address space selecting one of said plurality of registers;

providing at each of said plurality of processor complex nodes a translation table containing a plurality of translation table entries;

determining responsive to said hub ID field from a selected register which hub node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said hub node;

routing interrupts from said host bridge node responsive to said pointer fields selectable by interrupt level and said hub node identifier fields selectable by said pointer fields to dynamically determine routing of interrupts from said I/O device to a dynamically selected one of said plurality of processor complex nodes.

30. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for transferring packets with respect to PCI memory address space and system memory address space including a plurality of nodes, said method steps comprising:

operating a host bridge node to translate addresses from said PCI memory address space to said system memory address space, including;

providing a translation table containing a plurality of translation table entries at each of a plurality of said nodes;

providing a plurality of registers containing a hub ID field and an associated system memory address field;

receiving an address associated with said PCI memory address space originating from an I/O adapter;

subdividing said PCI memory address space, and responsive to said subdividing, selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which hub node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said hub node;

fetching translation table entries from a first processor complex node for determining a location in said system memory space to which a direct memory address operation will be directed to transfer data to a second processor complex node;

interconnecting said host bridge node, said first processor complex node and said second processor complex node by a packet-based connection protocol; and

responsive to a failure of said second processor complex node dynamically reconfiguring access to said first processor complex node.

31. A program storage device readable by a machine, tangibly embodying a program of instructions executable by a machine to perform method steps for operating an I/O system including a plurality of I/O nodes, a host, a host bridge connected to said host, and a communication link connecting said host bridge to said plurality of I/O nodes, said method steps comprising:

associating each of a plurality of address registers at said host bridge with an address range; and

responsive to node ID information, fetching a TCE table entry, said TCE table entry associating a PCI address with a system memory address at any of a plurality of said I/O nodes;

said associating and fetching steps including

subdividing said address range among said plurality of I/O nodes;

providing a translation table containing a plurality of translation table entries at each of a plurality of said I/O nodes;

providing a plurality of registers containing a hub ID field and an associated system memory address field;

receiving an address associated with said address range originating from an I/O adapter;

responsive to said subdividing, selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which I/O node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said I/O node.

32. A computer program product or computer program element for operating a computer according to the steps of:

providing at each of a plurality of processor complexes a translation table containing a plurality of translation table entries;

providing a host bridge connected to said plurality of processor complexes by a bus for fetching and caching said translation table entries containing hub identifiers and receiving input/output data transfer requests;

subdividing address space for said bus among said plurality of processor complexes for determining which hub node and corresponding translation table to use to translate a requested address;

said host bridge fetching said translation table entries from said complex selected responsive to said subdivided address space by

providing a plurality of registers containing a hub ID field and a system memory address field;

receiving an address associated with said bus originating from an I/O adapter;

responsive to said subdividing address space, selecting one of said plurality of registers;

determining responsive to said hub ID field from said selected register which hub node to use for accessing said translation table entry; and

combining said system memory address field from said selected register with a portion of said address for determining a system memory address of said translation table entry in said hub node.