Ultrascalable petaflop parallel supercomputer

• US 7,761,687 B2
• Filed: 06/26/2007
• Issued: 07/20/2010
• Est. Priority Date: 06/26/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chiu, George, Hall, Shawn, Chen, Dong, Gara, Alan G., Ohmacht, Martin, Heidelberger, Philip, Takken, Todd, Giampapa, Mark E., Blumrich, Matthias A., Salapura, Valentina, Coteus, Paul W., Haring, Rudolf A., Kopcsay, Gerard V., Sugavanam, Krishnan, Cipolla, Thomas M.
• Primary Examiner(s): Coleman; Eric
• Application Number:
  US11/768,905
• Publication Number:
  US 20090006808A1
• Time in Patent Office: 1,120 Days
• Field of Search: None
• US Class Current: 712/11
• CPC Class Codes: G06F 15/17337 Direct connection machines, e.g. completely connected computers, point to point communication networks coupling between buses G06F13/4004

Abstract

A massively parallel supercomputer of petaOPS-scale includes node architectures based upon System-On-a-Chip technology, where each processing node comprises a single Application Specific Integrated Circuit (ASIC) having up to four processing elements. The ASIC nodes are interconnected by multiple independent networks that optimally maximize the throughput of packet communications between nodes with minimal latency. The multiple networks may include three high-speed networks for parallel algorithm message passing including a Torus, collective network, and a Global Asynchronous network that provides global barrier and notification functions. These multiple independent networks may be collaboratively or independently utilized according to the needs or phases of an algorithm for optimizing algorithm processing performance. The use of a DMA engine is provided to facilitate message passing among the nodes without the expenditure of processing resources at the node.

Claims

1. A scalable, massively parallel computer system comprising:

a plurality of processing nodes interconnected by independent networks, each node including one or more processing elements for performing computation or communication activity as required when performing parallel algorithm operations, each of said processing nodes including a direct memory access (DMA) element operable for providing a plurality of functions for said processing node; and

,

a first of said multiple independent networks comprising an n-dimensional torus network including communication links interconnecting said nodes in a manner optimized for providing high-speed, low latency point-to-point and multicast packet communications among said nodes or sub-sets of nodes, said processing node DMA element providing a communications message passing interface enabling communication of messages among said nodes;

a second of said multiple independent networks including a scalable collective network comprising nodal interconnections that facilitate simultaneous global operations among nodes or sub-sets of nodes of said network; and

,

partitioning means for dynamically configuring one or more combinations of independent processing networks according to needs of one or more algorithms, each independent network including a configurable sub-set of processing nodes interconnected by divisible portions of said first and second networks,

each of said configured independent processing networks is utilized to enable simultaneous collaborative processing for optimizing algorithm processing performance, and,

wherein each said DMA element at said nodes comprises;

a processor interface for interfacing with the at least one processor, a DMA controller logic device, a memory interface for interfacing with a memory structure for storing information, a DMA network interface for interfacing with the network, one or more injection and reception byte counters, and injection and reception FIFO metadata associated with a injection FIFO and reception FIFO, respectively,

wherein said DMA element supports message-passing operation as controlled from an application via an Injection FIFO Metadata describing multiple Injection FIFOs, where each Injection FIFO may contain an arbitrary number of message descriptors to process messages with a fixed processing overhead irrespective of the number of message descriptors comprising the Injection FIFO,

said (DMA) element is operable for Direct Memory Access functions for point-to-point, multicast, and all-to-all communications amongst said nodes.

2. The massively parallel computer system as claimed in claim 1, wherein

an injection FIFO at each DMA element maintains memory locations of the injection FIFO metadata memory locations, including its current head and tail, and,

a reception FIFO at each DMA element maintains the reception FIFO metadata memory locations, including its current head and tail, wherein the injection byte counters and reception byte counters may be shared between messages.

3. The massively parallel computer system as claimed in claim 2, wherein each of the injection and reception byte counters maintain a base memory address and a byte counter value, respectively, for a message being passed through the compute node associated with same.

4. The massively parallel computer system as claimed in claim 3, herein each of the injection and reception byte counters is shared by multiple messages by first setting a base address in the shared byte counter to a lower bound and setting its associated counter value to zero, and wherein an initial offset in a message descriptor comprising the message is equivalent to difference between the messages start address the base address offset.

5. The massively parallel computer system as claimed in claim 3, wherein a processor at a processing node increments the counter value maintained in the shared byte counter by the current message length, and the DMA engine automatically carries out the increment operation such that immediately after, the shared byte counter includes a sum of the number of bytes to be injected or received, minus the number of bytes already injected or received.

6. The massively parallel computer system as claimed in claim 3, wherein said DMA element is configured for passing at least one remote message triggered by a first single source compute node (A) sending a single message wherein each said DMA element provides for putting remote descriptors from the first single source compute node (A) into an injection FIFO at remote second compute node (B).

7. The massively parallel computer system as claimed in claim 6, wherein the first single source compute node (A) implements a 3^rd party send by sending a single remote first message including at least one remote message descriptor to a remote second compute node (B), to control the remote second compute node (B) to send a second data message, and wherein the remote second compute node (B) sends said second data message to third compute node (C).

8. The massively parallel computer system as claimed in claim 7, wherein the first compute node (A) implements at least one 3^rd party send by sending a single remote get message including at least one remote message descriptor to a remote second compute node (B), wherein upon receipt of said at least one remote message descriptor, said DMA element passing at least one data message corresponding to the at least one remote message descriptor.

9. The massively parallel computer system as claimed in claim 8, wherein remote second compute node (B) sends said multiple data messages to a third remote compute node (C).

10. The massively parallel computer system as claimed in claim 1, wherein said direct memory access element in the node includes a memory having at least an injection fifo and a receive buffer,

said DMA element utilized for optimizing collective operations in said system by:

said DMA element providing at least a byte counter for each submessage of a message, the byte counter including at least a base address in memory for storing associated submessage and a counter value, the direct memory access controller operable to update the counter value as a result of receiving one or more bytes of the associated submessage into the node; and

,

said one or more processing elements operable to monitor the counter value and, when a predetermined number of bytes of the submessage is received, the one or more processing elements further operable to process a block of data comprising the received predetermined number of bytes of the submessage.

11. The massively parallel computer system as claimed in claim 10, wherein said one or more processors are operable to resend the block of data by updating a message descriptor in the injection fifo with information about the block of data, said information about the block of data including at least a counter identifier associated with the submessage, the counter identifier including at least a base address in memory, and an offset from the base address in memory where the block of data is stored.

12. The massively parallel computer system as claimed in claim 10, wherein the message is a collective operation message, said one or more processors are operable to perform a reduction operation on the block of data and, operable to send a result of the reduction operation on the block of data.

13. The massively parallel computer system as claimed in claim 1, wherein said plurality of interconnected processing nodes operates a global message-passing application for performing communications across the network, each of the processing nodes comprising one or more individual processors with memories, wherein local instances of the global message-passing application operate at each compute node to carry out local processing operations independent of processing operations carried out at other processing nodes.

14. The massively parallel computer system as claimed in claim 13, wherein each said DMA element at an interconnected processing node interacts with the application via Injection FIFO Metadata describing an Injection FIFO containing an arbitrary number of message descriptors in order to process messages with a fixed processing overhead irrespective of the number of message descriptors comprising the Injection FIFO.

15. The massively parallel computer system as claimed in claim 14, wherein the DMA element accepts Injection FIFO Metadata for multiple Injection FIFOs.

16. The massively parallel computer system as claimed in claim 15, wherein the DMA element comprises:

a producer pointer and a consumer pointer, said producer and consumer pointers pointing to all message descriptors to be provided to the circular buffers beginning at a first circular buffer position.

17. The massively parallel computer system as claimed in claim 1, wherein a third of said multiple independent networks includes a scalable global signal network comprising nodal interconnections for enabling asynchronous global operations among nodes or sub-sets of nodes of said network.

18. The massively parallel computer system as claimed in claim 17, wherein asynchronous global operations include barrier and notification operations.

19. The massively parallel computer system as claimed in claim 1, wherein each node comprises a system-on-chip Application Specific Integrated Circuit (ASIC) comprising four processing elements each capable of individually or simultaneously working on any combination of computation or communication activity, or both, as required when performing particular classes of algorithms.

20. The massively parallel computer system as claimed in claim 19, further including means for enabling overlapping of computation and communication among one or more configured independent processing networks when performing particular classes of algorithms.

21. The massively parallel computer system as claimed in claim 1, wherein each processing element includes one or more cache memories associated therewith, a processing node further comprising:

a node coherence architecture comprising;

a snoop filter sub-system associated with each of said processing elements, each snoop filter sub-system having a plurality of dedicated inputs for receiving coherence requests from dedicated memory writing sources at said processing node and,

a point-to-point interconnect means comprising communication links for directly connecting a memory writing source to a corresponding dedicated input of each snoop filter sub-system associated with each processing element,

wherein each said snoop filter sub-system further comprises;

a snoop filter means each coupled to a corresponding one of said plurality of dedicated inputs for filtering respective received coherence requests from respective dedicated memory writing sources, each snoop filter means implementing one or more parallel operating sub-filter elements operative for determining those requests likely residing in a cache memory at said associated processing element and filtering out those coherence requests for data that are determined not cached locally at said associated processing element, whereby a number of coherence requests forwarded to a processing element is significantly reduced thereby increasing performance of said computing environment.

22. The massively parallel computing system as claimed in claim 21, wherein said node coherence architecture implements a write-invalidate cache coherence protocol, said memory writing source generating a write-invalidation coherence request signal.

23. The massively parallel computing system as claimed in claim 21, further comprising a mechanism for single stepping snoop filter operations relating to processing of respective received coherence requests from respective dedicated memory writing sources.

24. A massively parallel computing system comprising:

a plurality of processing nodes interconnected by independent networks, each processing node comprising a system-on-chip Application Specific Integrated Circuit (ASIC) comprising four or more processing elements each capable of performing computation or message passing operations, each of said processing nodes including a direct memory access (DMA) element operable for providing a plurality of functions for said processing node, and,

each said processing element of a processing node including a shared memory storage structure, said shared memory storage structure is programmed into a plurality of memory bank structures a processing node including a programmable means for sacrificing an amount memory to enable processing parallelism;

one or more first logic devices associated with a respective said processor element, each one or more first logic devices for receiving physical memory address signals and programmable for generating a respective memory storage structure select signal upon receipt of pre-determined address bit values at selected physical memory address bit locations; and

,

a second logic device responsive to each said respective select signal for generating an address signal used for selecting a memory storage structure for processor access, wherein each processor device of said computing system is enabled memory storage access distributed across said one or more memory storage structures; and

,

means receiving unselected bit values of said received physical memory address signal for generating an offset bit vector signal used to enable processor element access to memory locations within a selected memory storage structure; and

,

a first independent network comprising an n-dimensional torus network including communication links interconnecting said nodes in a manner optimized for providing high-speed, low latency point-to-point and multicast packet communications among said nodes or sub-sets of nodes of said network;

a second of said multiple independent networks includes a scalable global collective network comprising nodal interconnections that facilitate simultaneous global operations among nodes or sub-sets of nodes of said network; and

,

partitioning means for dynamically configuring one or more combinations of independent processing networks according to needs of one or more algorithms, each independent network including a configured sub-set of processing nodes interconnected by divisible portions of said first and second networks, and,

means enabling rapid coordination of processing and message passing activity at each said processing element in each independent processing network, wherein one, or more, of the processing elements performs calculations needed by the algorithm, while the other, or more, of the processing elements performs message passing activities for communicating with other nodes of said network, as required when performing particular classes of algorithms, a processing node DMA element providing a communications message passing interface enabling communication of messages among said nodes, wherein each of said configured independent processing networks and node processing elements thereof are dynamically utilized to enable collaborative processing for optimizing algorithm processing performance.

25. The massively parallel computing system as claimed in claim 24, further implementing a message passing interface (MPI) protocol wherein a processing element performs its own MPI task independently, wherein said programmable means enables a first mode usage wherein, each processing node performs up to four MPI tasks, one task per processing element, with each processing element utilizing up to ¼

of the node'"'"'s memory storage structure.

26. The massively parallel computing system as claimed in claim 24, wherein said programmable means enables a second mode usage wherein a single processing element performs a single MPI task and utilizes some or all of the node'"'"'s memory storage structure.

27. The massively parallel computing system as claimed in claim 24, wherein said programmable means enables a third mode usage wherein each processing element implements an MPI task in a respective associated processing thread, with each of said four processing elements sharing all the node'"'"'s memory storage structure.

28. The massively parallel computing system as claimed in claim 24, wherein said programmable means enables a fourth mode usage wherein two processing elements each implements a first MPI task in a first associated processing thread and, another two processing elements each implements a second MPI task in a second associated processing thread.

29. The massively parallel computing structure as claimed in claim 24, wherein each processing node further comprises a shared resource in a memory accessible by said processing units configured for lock exchanges to prevent bottlenecks in said processing elements.

30. The massively parallel computing structure as claimed in claim 29, wherein each processing node further comprises:

means for receiving a physical memory address from an issuing processing element, said memory address including instruction bits specifying an atomic counter operation to be performed at a memory location in said memory storage structure; and

,

means for removing said instruction bits from said received memory address to form a further memory address used in selecting said memory address location in said memory storage structure, wherein an atomic counter operation is to be performed at said further memory address according to said instruction bits.

31. The massively parallel computing structure as claimed in claim 30, wherein said memory storage structure comprises an SRAM provided at said node.

32. The massively parallel computing structure as claimed in claim 24, further comprising:

interface means for accessing high speed data from an internal or external data source, each said node implementing functionality for providing a persistent data eye for monitoring capturing of said high speed data signals and adjusting a clock strobe associated with data signal capture at said node.

33. The massively parallel computing structure as claimed in claim 24, further comprising:

a performance monitoring unit (PMU) for monitoring performance of events occurring at a processing node, wherein each processing element generates signals representing occurrences of events at said processor device, said PMU comprising a plurality of performance counters each for counting said signals representing occurrences of events from one or more said plurality of processor units in said multiprocessor system.

34. The massively parallel computing structure as claimed in claim 33, further wherein said PMU further comprises a plurality of input devices for receiving said event signals from one or more processor devices of said processing elements, said plurality of input devices programmable to select event signals for receipt by one or more of said plurality of performance counters for counting, wherein said PMU is shared between multiple processing units, or within a group of processors in said multiprocessing system.

35. In a massively parallel computing structure comprising a plurality of processing nodes interconnected by multiple independent networks, each processing node comprising:

a system-on-chip Application Specific Integrated Circuit (ASIC) comprising four or more processing elements each capable of performing computation or message passing operations;

means enabling rapid coordination of processing and message passing activity at each said processing element, wherein one or both of the processing elements performs calculations needed by the algorithm, while the other or both processing element performs message passing activities for communicating with other interconnected nodes of a network, as required when performing particular classes of algorithms, and,

means for supporting overlap of communication and computation with non-blocking communication primitives

wherein each of said plurality of processing nodes includes a direct memory access (DMA) element operable for providing a communications message passing interface enabling communication of messages among said nodes, each said DMA element at said nodes comprising;

a processor interface for interfacing with the at least one processor, a DMA controller logic device, a memory interface for interfacing with a memory structure for storing information, a DMA network interface for interfacing with the network, one or more injection and reception byte counters, and injection and reception FIFO metadata associated with a injection FIFO and reception FIFO, respectively,

wherein said DMA element supports message-passing operation as controlled from an application via an Injection FIFO Metadata describing multiple Injection FIFOs, where each Injection FIFO may contain an arbitrary number of message descriptors to process messages with a fixed processing overhead irrespective of the number of message descriptors comprising the Injection FIFO,

said (DMA) element is operable for Direct Memory Access functions for point-to-point, multicast, and all-to-all communications amongst said nodes.

36. The massively parallel computing structure as claimed in claim 35, wherein said supporting means supports point-to-point messaging (application programming interface) API with active messages.

37. The massively parallel computing structure as claimed in claim 35, wherein said supporting means enables non-blocking collective operations, where a collective call is initiated by a node and then progresses on the network while a processing element performs a computing operation.

38. The massively parallel computing structure as claimed in claim 35, wherein said supporting means supports synchronous and asynchronous collective operations.

39. The massively parallel computing structure as claimed in claim 38, wherein, in support of said asynchronous collective operations, said structure enabling many simultaneous collective operations messaging in flight on the network, said structure further comprising:

means at each node for distinguishing among packets associated with different collective operations messages.

40. A system for providing an operating environment in a parallel computer system, comprising:

a system administration and management subsystem including at least core monitoring and control database operable to store information associated with a plurality of nodes in a parallel computer system;

a partition and job management subsystem operable to allocate one or more nodes in the parallel computer system to one or more job partitions to provide job management in the parallel computer system;

an application development and debug tools subsystem including at least one ore more debugging environments, one or more application performance monitoring and tuning tools, one or more compilers and a user interface, the application development and debug tools subsystem operable to provide one or more application development tools in the parallel computer system;

a compute node kernel and services subsystem operable to provide an environment for execution of one or more user processes; and

input/output kernel and services subsystem operable to communicate with one or more file servers and other computer systems, the input/output kernel and services subsystem further operable to provide input and output functionality between said one or more user processes and said one or more file servers and other computer systems,

wherein the core monitoring and control database includes;

a configuration database operable to provide a representation of a plurality of hardware on the parallel computer system;

an operational database including information and status associated with one or more interactions in the plurality of hardware on the parallel computer system;

an environmental database including current values for the plurality of hardware on the parallel computer system;

or

a reliability, availability, and serviceability database operable to collect information associated with reliability, availability, and serviceability of the parallel computer system;

or combinations thereof,

wherein the system administration and management subsystem uses the configuration database, the operational database, the environmental database, or the reliability, availability, and serviceability database, or combinations thereof, in managing the parallel computer system.

41. The system of claim 40, wherein each of said job partitions is dedicated to a single user job.

42. The system of claim 40, wherein the environment for execution of one or more user processes includes process and thread creation and management, memory management, dynamically linked libraries, process debugging, or reliability, availability and serviceability management, or combinations thereof.

43. The system of claim 40, wherein the core monitoring and control database resides with a service node of the parallel computer system.

44. The system of claim 40, wherein the compute node kernel and services subsystem resides in a plurality of compute nodes in the parallel computer system, the plurality of compute nodes operable to process one or more user application jobs.