Low occupancy protocol for managing concurrent transactions with dependencies

• US 6,154,816 A
• Filed: 10/24/1997
• Issued: 11/28/2000
• Est. Priority Date: 10/24/1997
• Status: Expired due to Term

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Compaq Computer Corporation (HP Inc.)
• Inventors: Steely, Simon C., Sharma, Madhumitra, VanDoren, Stephen R.
• Primary Examiner(s): Nguyen, Hiep T.
• Application Number:
  US08/957,565
• Time in Patent Office: 1,131 Days
• Field of Search: 711/121, 711/117, 711/118, 711/141, 711/142, 711/143, 711/144, 711/145, 711/146, 711/149, 711/150, 711/151, 711/168, 711/169, 710/131
• US Class Current: 711/150
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/0828 with concurrent directory accessing, i.e. handling multiple concurrent coherency transactions
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores

Abstract

An architecture and coherency protocol for use in a large SMP computer system includes a hierarchical switch structure which allows for a number of multi-processor nodes to be coupled to the switch to operate at an optimum performance. Within each multi-processor node, a simultaneous buffering system is provided that allows all of the processors of the multi-processor node to operate at peak performance. A memory is shared among the nodes, with a portion of the memory resident at each of the multi-processor nodes. Each of the multi-processor nodes includes a number of elements for maintaining memory coherency, including a victim cache, a directory and a transaction tracking table. The victim cache allows for selective updates of victim data destined for memory stored at a remote multi-processing node, thereby improving the overall performance of memory. Memory performance is additionally improved by including, at each memory, a delayed write buffer which is used in conjunction with the directory to identify victims that are to be written to memory. An arb bus coupled to the output of the directory of each node provides a central ordering point for all messages that are transferred through the SMP. The messages comprise a number of transactions, and each transaction is assigned to a number of different virtual channels, depending upon the processing stage of the message. The use of virtual channels thus helps to maintain data coherency by providing a straightforward method for maintaining system order. Using the virtual channels and the directory structure, cache coherency problems that would previously result in deadlock may be avoided.

Claims

1. A multi-processing system comprising a shared memory and a plurality of multi-processor nodes coupled via a switch, each of plurality of the multi-processor nodes further comprising at least one processor, the multi-processing system comprising:

a portion of the shared memory located in each multi-processor node and apportioned into a plurality of blocks;

a directory in each node having a plurality of entries corresponding in number to the plurality of blocks of the shared memory, each entry in the directory for identifying which of the plurality of multi-processor nodes stores copies of the data block; and

a serialization point coupled to the directory for ordering accesses to the plurality of blocks thereby allowing the multi-processing system to concurrently execute multiple references to each of the plurality of blocks.

2. The multi-processing system according to claim 1, wherein the multiple references to each of the plurality of blocks may be operating on a different one of at least one version of the block substantially simultaneously.

3. The multi-processing system according to claim 2, wherein each one of the at least one version of the block represents the results of a write operation to the block.

4. The multi-processing system according to claim 1, wherein the directory is accessed only once for each reference to one of the plurality of blocks.

5. The multi-processing system according to claim 4, wherein the directory is accessed atomically before the completion of the reference such that the directory reflects the status of the associated block of memory upon completion of the reference.

6. The multi-processing system according to claim 5, wherein one of the references is a probe type reference for reading one of the blocks of memory, and wherein one of the references is an update type operation for writing data to a block of memory, and wherein the each of the nodes of the multi-processing system includes means for determining a serialization order of a probe type operation and an update type operation.

7. The multi-processing system according to claim 6, wherein a probe type reference to a given block that is serialized behind an update type reference to the given block is received at the at least one processor of the multi-processing node before data associated with the update type reference is returned to the at least one processor, and wherein the multi-processor node includes means for stalling the probe type reference until the previously serialized update type reference is completed.

8. The multi-processing node according to claim 6, wherein the probe type reference to the given block is serialized before the update type reference to the given block and wherein data associated with the update type reference is received before the execution of the probe type reference, and wherein the multi-processor node includes means for selectively executing the probe type reference using the data associated with the update type reference.

9. The multi-processing system according to claim 5, further comprising means for guaranteeing completion of a reference that has atomically accessed the directory.

10. The multi-processing system according to claim 9, wherein the means for guaranteeing completion further comprises:

means, at each of the at least one processors of each of the plurality of multi-processor nodes, for temporary storage of a subset of modified blocks of shared memory until each modified block is written to the shared memory and until all probe type references to the modified block that are serialized before the update reference associated with the modified block have been completed.

11. The multi-processing system according to claim 9 wherein the means for guaranteeing completion further comprises:

means, at each of the plurality of multi-processor nodes, for temporary storage of a subset of modified blocks of shared memory until each modified block is written to the shared memory and until all probe type references to the modified block that are serialized before the update reference associated with the modified block have been completed.

12. The multi-processing system according to claim 6, wherein each of the references comprises a plurality of transactions and where each of the transactions of each of the references are forwarded on a corresponding one of a plurality of channels in the multi-processing system, and wherein the means for determining the serialization order comprises means for maintaining an order of transactions of references on at least one of the plurality of channels.

13. The multi-processing system according to claim 12, wherein the ordered one of the plurality of channels carries the information indicating a relative operating status of the update type references and probe type references.

14. The multi-processing system according to claim 9, wherein the means for guaranteeing further comprises:

means, at each of the plurality of multi-processing nodes, for delaying one of the multiple references until a desired version of the block of shared memory is returned to the multi-processing node.

15. The multi-processing system according to claim 9, wherein the means for guaranteeing further comprises:

means, at each of the plurality of multi-processing nodes, for delaying execution of one of the multiple references until a desired version of the block of shared memory is returned to the multi-processing node.

16. The multi-processing system according to claim 9, wherein each of the multiple references comprise a plurality of stages of transactions and wherein the means for guaranteeing further comprises means for indicating the stage of transaction of each of the references.

17. The multi-processing system according to claim 16, wherein each of the stages of transactions are forwarded on different channels, and wherein the means for indicating the stage of transaction of the references includes means for forwarding a packet on a channel associated with the stage of transaction to multi-processing nodes awaiting completion of the reference.

18. The multi-processing system according to claim 17, wherein the reference is a read reference forwarded from a first one of the multi-processing nodes to the directory of a second one of the multi-processing nodes, and the packet is a marker packet indicating that the read reference accessed the directory in the second one of the multi-processing nodes.

19. The multi-processing system according to claim 1, further comprising a victim cache for temporary storage of a subset of the plurality of blocks of shared memory until the subset of the plurality of blocks of shared memory are written to the shared memory, wherein each of the plurality of blocks of the subset has been updated by one of the at least one processor of the multi-processing node.

20. A method for allowing multiple references to a common block in a shared memory to be executing simultaneously in a multi-processing system, the multi-processing system comprising a plurality of multi-processor nodes coupled via a switch, each of plurality of the multi-processor nodes further comprising at least one processor, a portion of the shared memory apportioned into a plurality of blocks and a serialization unit, the serialization unit comprising a plurality of entries corresponding in number to the plurality of blocks of the portion of shared memory, the method comprising the steps of:

ordering all references to the common block as they are received at the serialization unit of multi-processor node associated with the common block, where each reference visits the serialization unit only once during execution; and

delaying completion of references to the common block, the common block stored at a destination, until a desired version of the block of shared memory is returned to the destination.

21. The method according to claim 20, wherein the multiple references to each of the plurality of blocks may be operating on a different one of at least one version of the block substantially simultaneously.

22. The method according to claim 21, wherein each one of the at least one version of the block represents the results of a write operation to the block.

23. The method according to claim 20, wherein the directory is accessed only once for each reference to one of the plurality of blocks.

24. The method according to claim 23, wherein the directory is accessed atomically before the completion of the reference such that the directory reflects the status of the associated block of memory upon completion of the reference.

25. The method according to claim 20, wherein one of the references is a probe type reference for reading one of the blocks of memory, and wherein one of the references is an update type operation for writing data to a block of memory, and wherein the method includes the steps of each of the nodes of the multi-processing determining a serialization order of a probe type operations and an update type operations at the respective node.

26. The method to claim 25, wherein a probe type reference to a given block that is serialized behind an update type reference to the given block is received at the at least one processor of the multi-processing node before data associated with the update type reference is returned to the at least one processor, and wherein the method includes the step of stalling the probe type reference until the previously serialized update type reference is completed.

27. The method according to claim 25, wherein the probe type reference to the given block is serialized before the update type reference to the given block and wherein data associated with the update type reference is received before the execution of the probe type reference, and wherein the method includes the step of selectively executing the probe type reference using the data associated with the update type reference.

28. The method according to claim 24, further comprising the step of guaranteeing completion of a reference that has atomically accessed the directory.

29. The method according to claim 28, wherein the step of guaranteeing completion further comprises the step of:

temporarily storing, at each of the at least one processors of each of the plurality of multi-processor nodes, a subset of modified blocks of shared memory until each modified block is written to the shared memory and until all probe type references to the modified block that are serialized before the update reference associated with the modified block have been completed.

30. The method according to claim 28 wherein the step of guaranteeing completion further comprises:

temporarily storing, at each of the plurality of multi-processor nodes, a subset of modified blocks of shared memory until each modified block is written to the shared memory and until all probe type references to the modified block that are serialized before the update reference associated with the modified block have been completed.

31. The method according to claim 28, further comprising the step of temporarily storing, at each of the plurality of multi-processor nodes, a subset of the plurality of blocks of shared memory modified by the corresponding at least one processor of the multi-processor node, until the subset of the plurality of blocks of shared memory are written to the shared memory.

32. The method according to claim 20, wherein each of the multiple references comprise a plurality of stages of transactions and wherein the method further comprises the step of indicating the stage of transaction of each of the references.

33. The method according to claim 32, wherein each of the stages of transactions are forwarded on different channels, and wherein the step of indicating the stage of transaction of the references includes the step of forwarding a packet on a channel associated with the stage of transaction to multi-processing nodes awaiting completion of the reference.

34. The method according to claim 33, wherein the reference is a read reference forwarded from a first one of the multi-processing nodes to the serialization unit of a second one of the multi-processing nodes, and the packet is a marker packet indicating that the read reference accessed the serialization unit in the second one of the multi-processing nodes.