Interrupt architecture for a non-uniform memory access (NUMA) data processing system

• US 6,148,361 A
• Filed: 12/17/1998
• Issued: 11/14/2000
• Est. Priority Date: 12/17/1998
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: deBacker, Philippe Louis, Dean, Mark Edward, Carpenter, Gary Dale, Glasco, David Brian, Rockhold, Ronald Lynn
• Primary Examiner(s): Follansbee, John A.
• Assistant Examiner(s): EL HADY, NABIL M
• Application Number:
  US09/213,998
• Time in Patent Office: 698 Days
• Field of Search: 710/260-269, 712/1
• US Class Current: 710/260
• CPC Class Codes: G06F 9/4812 by interrupt, e.g. masked

Abstract

A non-uniform memory access (NUMA) computer system includes at least two nodes coupled by a node interconnect, where at least one of the nodes includes a processor for servicing interrupts. The nodes are partitioned into external interrupt domains so that an external interrupt is always presented to a processor within the external interrupt domain in which the interrupt occurs. Although each external interrupt domain typically includes only a single node, interrupt channeling or interrupt funneling may be implemented to route external interrupts across node boundaries for presentation to a processor. Once presented to a processor, interrupt handling software may then execute on any processor to service the external interrupt. Servicing external interrupts is expedited by reducing the size of the interrupt handler polling chain as compared to prior art methods. In addition to external interrupts, the interrupt architecture of the present invention supports inter-processor interrupts (IPIs) by which any processor may interrupt itself or one or more other processors in the NUMA computer system. IPIs are triggered by writing to memory mapped registers in global system memory, which facilitates the transmission of IPIs across node boundaries and permits multicast IPIs to be triggered simply by transmitting one write transaction to each node containing a processor to be interrupted. The interrupt hardware within each node is also distributed for scalability, with the hardware components communicating via interrupt transactions conveyed across shared communication paths.

Claims

1. A data processing system, comprising:

a plurality of interrupt domains that each include at least one of a plurality of interconnected nodes, wherein each interrupt domain includes at least one processor capable of receiving external interrupts and at least one interrupt source capable of generating external interrupts, each of said plurality of interrupt domains having respective interrupt hardware that receives an external interrupt generated within that interrupt domain and first presents said external interrupt to a processor within that interrupt domain, wherein said processor to which said external interrupt is first presented invokes servicing of said external interrupt and a processor within a different interrupt domain services said external interrupt;

wherein said interrupt hardware within each interrupt domain includes a globally-accessible memory mapped register utilized to communicate interrupts between interrupt domains, and wherein said interrupts include inter-processor interrupts.

2. The data processing system of claim 1, said interrupt hardware within each of said plurality of interrupt domains including an interrupt destination unit that presents interrupts to processors only within its interrupt domain and at least one interrupt source unit that receives interrupts from interrupt sources.

3. The data processing system of claim 2, wherein said interrupt destination unit and said interrupt source unit communicate interrupt information via a shared interconnect.

4. The data processing system of claim 2, wherein for at least one interrupt domain among said plurality of interrupt domains, at least one interrupt source unit and said interrupt destination unit are located in different ones of said plurality of interconnected nodes.

5. The data processing system of claim 4, wherein the node containing said at least one interrupt source unit contains no processors to which external interrupts can be first presented.

6. The data processing system of claim 2, wherein at least one of said plurality of interrupt domains includes a plurality of interrupt source units.

7. The data processing system of claim 1, wherein said globally-accessible memory mapped register of each of said interrupt domains is assigned a respective physical address, and wherein the physical address of the globally-accessible memory mapped register of each interrupt domain has a uniform offset from a memory area allocated to a node containing said globally-accessible memory mapped register.

8. The data processing system of claim 1, and further comprising:

a plurality of local interconnects that are each located within a respective one of said plurality of interconnected nodes;

a system interconnect; and

a plurality of node controllers that each couple one of said plurality of local interconnects to said system interconnect for communication between nodes.

9. The data processing system of claim 1, wherein:

said data processing system further comprises memory within one or more of said processing nodes that stores an operating system and a first level interrupt handler and a second level interrupt handler; and

said first level interrupt handler is executed by said processor to which said external interrupt is first presented to invoke servicing of said external interrupt and said second level interrupt handler can be scheduled by said operating system to execute on any processor in said data processing system.

10. A method for handling an external interrupt in a data processing system, said method comprising:

establishing a plurality of interrupt domains that each include at least one of a plurality of interconnected nodes, wherein each interrupt domain includes at least one processor capable of receiving an external interrupt and at least one interrupt source capable of generating an external interrupt, each of said plurality of interrupt domains having respective interrupt hardware;

within a particular interrupt domain among said plurality of interrupt domains, receiving, at said interrupt hardware, an external interrupt generated by an interrupt source within said particular interrupt domain and first presenting said external interrupt to a processor within said particular interrupt domain by said interrupt hardware;

invoking interrupt servicing utilizing said processor of said particular interrupt domain to which said external interrupt is first presented;

communicating said external interrupt and inter-processor interrupts between interrupt domains utilizing a globally-accessible memory mapped register within said interrupt hardware; and

servicing said external interrupt utilizing a processor within a different one of said plurality of interrupt domains than said particular interrupt domain.

11. The method of claim 10, said interrupt hardware within each of said plurality of interrupt domains including an interrupt destination unit and at least one interrupt source unit, wherein receiving an external interrupt comprises receiving said external interrupt at said at least one interrupt source unit, and wherein presenting said external interrupt comprises presenting said external interrupt to said at least one processor utilizing said interrupt destination unit.

12. The method of claim 11, and further comprising communicating interrupt information between said interrupt destination unit and said interrupt source unit via a shared interconnect.

13. The method of claim 12, wherein for at least one interrupt domain among said plurality of interrupt domains, communicating interrupt information via a shared interconnect comprises communicating interrupt information via a shared interconnect interconnecting at least two of said plurality of nodes.

14. The method of claim 13, wherein establishing a plurality of interrupt domains includes establishing at least one interrupt domain in which one of said plurality of interconnected nodes contains at least one interrupt source unit and no processors to which external interrupts may be first presented.

15. The method of claim 11, establishing a plurality of interrupt domains comprises establishing at least one of said plurality of interrupt domains including a plurality of interrupt source units.

16. The data processing system of claim 10, and further comprising:

assigning said globally-accessible memory mapped register of each of said interrupt domains a respective physical address, wherein the physical address of the globally-accessible memory mapped register of each interrupt domain has a uniform offset from a memory area allocated to a node containing said globally-accessible memory mapped register.

17. The method of claim 10, wherein invoking servicing of said external interrupt comprises executing a first level interrupt handler and servicing said external interrupt comprises executing a second level interrupt handler.

18. A method of processing an interrupt within a data processing system including a plurality of interconnected nodes, said method comprising:

establishing a plurality of interrupt domains that each include at least one of the plurality of interconnected nodes, wherein each interrupt domain includes at least one processor capable of receiving an interrupt and at least one device capable of generating an interrupt, and wherein devices in multiple of said plurality of interconnected nodes can generate interrupts of the same level;

in response to presentation of an interrupt to a processor for servicing, said interrupt having a level, obtaining a list of devices capable of generating an interrupt of said level;

polling only devices within said list located within a same interrupt domain as said processor in order to identify which device within said list generated said interrupt; and

prior to presentation of said interrupt, creating and storing said list in a global memory space accessible to all of said plurality of interconnected nodes.

19. The method of claim 18, and further comprising thereafter executing an interrupt handler associated with said identified device.

20. The method of claim 18, wherein said list contains only devices within a single interrupt domain.

21. A data processing system, comprising:

a plurality of interrupt domains that each include at least one of a plurality of interconnected nodes, wherein each of said plurality of interrupt domains includes at least one processor capable of receiving an interrupt and at least one device that generates interrupts, wherein devices in multiple of said plurality of interconnected nodes can generate interrupts of the same level;

interrupt handler software stored within said data processing system and executable by said processor, wherein said interrupt handler software, in response to presentation of an interrupt having a level to said processor, obtains a list of devices capable of generating an interrupt of said level and polls only devices within said list located within a same interrupt domain as said processor in order to identify which device within said list generated said interrupt; and

a global memory space accessible to all of said plurality of interconnected nodes, wherein said list is stored in said global memory space prior to presentation of said interrupt.

22. The data processing system of claim 21, wherein said interrupt handler software is a first level interrupt handler, said data processing system further comprising a second level interrupt handler stored within said data processing system and executable by said processor, wherein said second level interrupt handler is associated with said device, and wherein said first level interrupt handler calls said second level interrupt handler to service said identified device.

23. The data processing system of claim 21, wherein said list contains only devices within a single interrupt domain.

24. A program product for use by a data processing system including a plurality of interrupt domains that each include at least one of a plurality of interconnected nodes, wherein each of said plurality of interrupt domains includes at least one processor capable of receiving an interrupt and at least one device that generates interrupts, and wherein devices in multiple of said plurality of interconnected nodes can generate interrupts of the same level, said program product comprising:

a computer usable medium;

interrupt handler software encoded within said computer usable, medium and executable by the data processing system, wherein said interrupt handler software, in response to presentation of an interrupt having a level to said processor, obtains a list of devices capable of generating an interrupt of said level and polls only devices within said list located within a same interrupt domain as said processor in order to identify which device within said list generated said interrupt; and

a configuration routine, encoded with said computer usable medium, that creates said list in global memory space accessible to all of said plurality of nodes prior to presentation of said interrupt.

25. The program product of claim 24, wherein said interrupt handler software is a first level interrupt handler, said program product further comprising a second level interrupt handler encoded within said computer usable medium, wherein said second level interrupt handler is associated with said device, and wherein said first level interrupt handler calls said second level interrupt handler to service said identified device.

26. The program product of claim 24, wherein said configuration routine includes within said list only devices within a single interrupt domain.