High speed remote storage controller

• US 20020083299A1
• Filed: 12/22/2000
• Published: 06/27/2002
• Est. Priority Date: 12/22/2000
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Mak, Pak-Kin, Blake, Michael A., Van Huben, Gary A.
• Granted Patent: US 6,738,870 B2
• Time in Patent Office: Days
• US Class Current: 712/29
• CPC Class Codes: G06F 15/17381 Two dimensional, e.g. mesh, torus

Abstract

A high speed remote storage controller system for a computer system has cluster nodes of symmetric multiprocessors. A plurality of clusters of symmetric multiprocessors each of has a plurality of processors, a shared cache memory, a plurality of I/O adapters and a main memory accessible from the cluster. Each cluster has an interface for passing data between cluster nodes of the symmetric multiprocessor system. Each cluster has a local interface and interface controller. The system provides one or more remote storage controllers each having a local interface controller and a local-to-remote data bus. A remote resource manager manages the interface between clusters of symmetric multiprocessors. The remote store controller is responsible for processing data accesses across a plurality of clusters and processes data storage operations involving shared memory. A macro is provided for processing a plurality of simultaneous data storage operations either synchronously through interaction with a sequential multistage centralized pipeline to serialize requests and provide address interlocking services or asynchronously whereby main memory accesses bypass a centralized system pipeline. These accesses can occur in parallel with other remote storage operations.

Claims

1. A high speed remote storage controller system for a computer system having cluster nodes of symmetric multiprocessors, comprising

one or more remote storage controllers each having a local interface controller, and a local-to-remote data bus, and responsive to a remote resource manager managing the local interface controller for data accesses between clusters of symmetric multiprocessors which are controlled for inter nodal storage operations by one of said remote storage controller, each of which clusters has a plurality of processors, a shared cache memory, a plurality of I/O adapters and a main memory accessible from the cluster,

said remote store controller while processing data accesses across a plurality of clusters for data storage operations involving shared memory and inter nodal storage operations enlisting a single one of said local interface controllers to perform remote cast outs, store requests from an I/O adapter, main storage padding operations, and main memory move page operations and while performing remote data storage operations to main memory, said enlisted single local interface controller also handles cross cluster invalidations associated with maintaining inter-nodal cache coherency for said computer system having cluster nodes of symmetric multiprocessors.

2. The remote storage controller system according to claim 1 in which a single macro is capable of processing a plurality of simultaneous data storage operations either synchronously through interaction with a sequential multistage centralized pipeline to serialize requests and provide address interlocking services or asynchronously whereby main memory accesses bypass the centralized pipeline and can occur in parallel with other remote storage operations.

3. The remote storage controller system according to claim 2 in which said single macro contains a plurality of remote storage resources, each of which has an affinity to a local storage resource on said local cluster and wherein each storage resource is general purpose and capable of processing all types of storage accesses including synchronous and asynchronous operations.

4. The remote storage controller system according to claim 2 whereby each storage resource contained therein can process asynchronous memory storage operations thus improving system performance by eliminating the need to perform address interlock detection with other storage controller operations and avoiding wasteful retry attempts resulting from pipeline reject situations.

5. The remote storage controller system according to claim 1 capable of supporting sectored changed cache lines thus permitting data storage accesses to require half the amount of regular data byte transfers in situations where only half of the cache line has been updated and requires casting out to main memory.

6. The remote storage controller system according to claim 1 comprising an advanced memory arbitration unit responsible for generating data storage requests to main memory in an efficient manner wherein said arbitration unit employs an LRU scheme based on memory bank availability to arbitrate between a plurality of remote storage resources contending for a shared memory bus.

7. The remote storage controller system according to claim 1 utilizing an improved mechanism for handling remote I/O storage requests which reduces resource utilization by ensuring only one pair of local and remote resources are required for any type of I/O Store operation including those which target shared main memory and/or the shared L2 cache on the remote cluster.

8. The remote storage controller system according to claim 2 which supports concurrent main memory page move operations among a plurality of remote storage resources contained within said remote storage controller macro.

9. The remote storage controller system according to claim 1 which supports retry capability for any remote storage operation which fails to progress due to some type of hardware error, and in cases where said hardware failure is intermittent, the retry capability permits the operation to eventually succeed without negatively impacting customer availability.

10. The remote storage controller system according to claim 1 which contains a deadlock avoidance mechanism designed to detect inter-nodal deadlocks which normally result from one resource on the local cluster waiting for a second resource on a remote cluster which is deadlocked against a third resource on said remote cluster waiting for a fourth resource on the local cluster which is deadlocked against said first resource on the local cluster.

11. The remote storage controller system according to claim 10 which upon detecting said inter-nodal deadlocks employs a systemic method to alleviate the deadlock and avoid a hang condition which would otherwise ensue if said deadlock was not successfully broken.

12. The remote storage controller system according to claim 10 which employs a fast hang quiesce mechanism embedded in each remote storage resource which works in conjunction with similar fast hang quiesce mechanisms throughout the System Controller to prevent system-wide hangs either caused by the failure of the current remote storage resource to make forward progress or by the failure of another operation somewhere in the Storage Controller to make forward progress.

13. The remote storage controller system according to claim 12 which enables communication with the other elements of the Storage Controller to signal the need for other concurrent operations in the Storage Controller to quiesce in order to permit the current remote storage resource to make forward progress in situations where lack of forward progress is detected.

14. The remote storage controller system according to claim 12 which enables the remote storage resource to react to the request from other controllers within the System Controller and quiesce the current remote storage operation in order that other operations in the System Controller may make forward progress in situations where lack of forward progress is detected.