DISTRIBUTED SYMMETRIC MULTIPROCESSING COMPUTING ARCHITECTURE

• US 20110125974A1
• Filed: 11/15/2010
• Published: 05/26/2011
• Est. Priority Date: 11/13/2009
• Status: Active Grant

• Current Assignee: Richard S. Anderson
• Original Assignee: Richard S. Anderson
• Inventors: Anderson, Richard S.
• Granted Patent: US 8,607,004 B2
• Time in Patent Office: Days
• US Class Current: 711/153
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 15/17331 Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]

Abstract

Example embodiments of the present invention includes systems and methods for implementing a scalable symmetric multiprocessing (shared memory) computer architecture using a network of homogeneous multi-core servers. The level of processor and memory performance achieved is suitable for running applications that currently require cache coherent shared memory mainframes and supercomputers. The architecture combines new operating system extensions with a high-speed network that supports remote direct memory access to achieve an effective global distributed shared memory. A distributed thread model allows a process running in a head node to fork threads in other (worker) nodes that run in the same global address space. Thread synchronization is supported by a distributed mutex implementation. A transactional memory model allows a multi-threaded program to maintain global memory page consistency across the distributed architecture. A distributed file access implementation supports non-contentious file I/O for threads. These and other functions provide a symmetric multiprocessing programming model consistent with standards such as Portable Operating System Interface for Unix (POSIX).

Claims

1. A symmetric multiprocessing (SMP) computing system comprising:

a plurality of SMP nodes tightly coupled together to form a cluster, each SMP node including;

one or more multi-core processors;

memory, partitioned into a local and global partition, with the global partitions together forming a single global memory accessible by all processors in the cluster;

a communication adapter supporting data rates of at least 40 Gb/s and remote direct memory access (RDMA).

2. The system of claim 1 wherein the cluster forms a single SMP computer that includes a number of processors equal to the sum of the processors in the SMP nodes in the cluster;

and a size of memory equal to at least two-thirds of the sum of the total memory in the SMP nodes in the cluster.

3. The system of claim 1 wherein the cluster forms a single SMP computer that has a single user interface provided by a SMP node that is designated as the head node of the cluster.

4. The system of claim 1 wherein the cluster supports memory accesses with latencies of less than 4 microseconds.

5. The system of claim 1 wherein the cluster includes N SMP nodes, each SMP node includes N−

1 communication ports, and the nodes are coupled together directly via the communication ports, without the use of a communication switch.

6. The system of claim 1 wherein the cluster includes N SMP nodes that are coupled together via at least one communication switch supporting at least N ports.

7. The system of claim 6 wherein the switch or switches support a communication speed of at least 40 Gb/s and a latency of 150 nanoseconds or less.

8. The system of claim 1 wherein the local and global memories form a transactional distributed shared memory (T-DSM) that supports memory page coherency.

9. The system of claim 8 wherein the T-DSM maintains fixed logical addressing in the global memory for a given memory page, wherein the fixed logical address:

(a) translates to a physical page address in the global memory partition of a particular SMP node in the cluster;

(b) eliminates any need to search for memory pages in the global memory; and

(c) enables use of one-sided RDMA operations in moving the given memory page between the local memory and the global memory.

10. The system of claim 8 wherein the T-DSM is configured to maintain a consistent global representation of a logical address space of a running program in global memory by copying a memory page modified in the local memory partition of any node to global memory.

11. The system of claim 8 wherein the system is configured to enable a running program or one of its threads to lock for exclusive access a given memory page through a set of operating system primitives.

12. The system of claim 11 wherein a given SMP node is configured to lock any global memory page stored in the global memory partition of the given SMP node upon receipt of a lock request from a requester.

13. The system of claim 11 wherein the given SMP node is further configured to:

(a) broadcast a message to all other SMP nodes, invalidating all local copies of the given memory page, to lock the given memory page in global memory, and to forward an acceptance of the lock request to the requester if the lock request is granted; and

(b) forward a rejection of the lock to the requester if the lock request cannot be granted.

14. The system of claim 11 wherein the given SMP node is further configured to broadcast a message to all SMP nodes in the cluster unlocking the given memory page when the lock is released by the requester.

15. The system of claim 8 wherein the system is configured to force immediate availability of a modified memory page to global memory by issuing a sync request using an operating system primitive.

16. The system of claim 1 wherein the global memory includes a reference copy of every memory page in the logical address space used by a running application program.

17. A method of performing symmetric multiprocessing (SMP) computing, the method comprising:

forming a cluster of SMP compute nodes (servers), with the memory in each node similarly divided into a local and global partition, where the global partitions of each node combine together to form a single global memory;

starting and executing an application program in the local memory partition of the head SMP node in the cluster;

allowing additional threads of the program to be launched and executed by processor cores in the head node or in worker nodes;

identify and copy a given memory page from the global memory to the local memory partition of a given SMP node when needed by the executing program thread; and

enabling a thread to lock for exclusive access a given memory page in the global memory.

18. The method of claim 17 wherein identifying and copying the given memory page occurs with a latency of less than 4 microseconds.

19. The method of claim 17 wherein identifying and copying the given memory page includes transmitting data between a given pair of SMP nodes in the cluster via a directly coupled communications link between the given pair of SMP nodes.

20. The method of claim 17 wherein identifying and copying the given memory page includes transmitting data between a given pair of SMP nodes in the cluster via a switch.

21. The method of claim 17 further including:

maintaining a fixed logical address in the global memory for a given memory page, wherein the fixed logical address;

(a) translates to a physical page address in the global memory partition of a particular SMP node in the cluster;

(b) eliminates any need to search for memory pages in the global memory; and

(c) enables use of one-sided remote direct memory access operations in moving the given memory page between the local memory and the global memory.

22. The method of claim 17 further including:

copying a modified memory page from the local memory partition to global memory to maintain a consistent representation of an application program logical address space.

23. The method of claim 17 wherein enabling a program thread to lock a given global memory page for exclusive access by the thread, by issuing a lock request via an operating system primitive, and further including:

handling the lock request with the SMP node whose global memory partition stores a reference copy of the given memory page.

24. The method of claim 23 further including:

broadcasting a message invalidating and locking the targeted memory page from the SMP node whose global memory partition hosts the reference copy of the targeted memory page, to other SMP nodes in the cluster if the lock request is granted.

25. The method of claim 23 further including:

broadcasting a message to SMP nodes in the cluster unlocking the given memory page when the lock is released.

26. The method of claim 17, further including:

allowing a thread to force immediate availability of a modified memory page to global memory by using a memory page synchronization operating system primitive.

27. The method of claim 17, wherein enabling the programmer to avoid false sharing of independent variables by placing these variables on different memory pages using an aligned allocation operating system primitive that forces a new memory block to begin on a new memory page.

28. The method of claim 17, wherein distributed thread management is implemented on a network of symmetric multiprocessing (SMP) nodes that support distributed shared memory, the method comprising:

reproducing an SMP program execution model across multiple nodes;

starting all application programs on a head node;

launching additional execution threads that run in the head node or in a worker node; and

maintaining a global thread table of threads executing on worker nodes in the head node.

29. A method as in claim 28 further including:

duplicating the process environment of an application program running on the head, including page tables, on to a worker node, to allow the launching and execution of threads on the work node.

30. A method as in claim 28 further including:

launching a thread on the worker node from the head node; and

receiving a notification when execution of the thread is complete.

31. A method of claim 17, wherein distributed mutual exclusion lock (mutex) management is implemented on an underlying network of symmetric multiprocessing servers, the method comprising:

maintaining a global mutex table configured to track remote mutex activity at each node; and

filtering all locks or waits on a mutex through the global mutex table to handle those cases where the mutex is remotely addressed.

32. A method for the implementation of distributed disk input/output on an underlying network of symmetric multiprocessing (SMP) nodes, comprising:

performing all direct disk access from a head node of the network of SMP nodes;

extending disk access to threads running in the head node or non-head nodes of the network of SMP nodes through the use of message queues; and

providing non-contentious access to disk writes that do not require waiting on a mutual exclusion lock by use of a disk output queue; and

allowing multiple simultaneous blocking disk reads by providing a read request message queue.

33. A method of claim 32, further including a distributed message queue hosted on the head node that can receive messages from any node in the network and support a maximum message size that equals the size of a memory page.

34. A method of performing symmetric multiprocessing (SMP) computing, the method comprising:

executing an application thread in a local memory of a given SMP server node in a cluster of SMP server nodes using a multi-core processor associated with the given SMP node;

maintaining fixed logical addressing in the global memory for a given memory page;

upon request, identifying a given memory page in global memory during execution of the application thread using the fixed logical addressing, the global memory being formed of respective global memory partitions in the SMP nodes in the cluster, the fixed logical addressing translating to a physical page address in the global memory portion of a particular SMP node in the cluster;

copying the given memory page from the global memory to the local memory of the given SMP node; and

enabling a programmer to lock the given memory page in the global memory via a set of operating system primitives.