Multi-node computer system in which networks in different nodes implement different conveyance modes

• US 7,225,298 B2
• Filed: 03/31/2004
• Issued: 05/29/2007
• Est. Priority Date: 04/11/2003
• Status: Active Grant

• Current Assignee: Oracle America, Inc. (Oracle Corporation)
• Original Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Inventors: Cypher, Robert E.
• Primary Examiner(s): Sough; Hyung
• Assistant Examiner(s): PATEL, KAUSHIKKUMAR M
• Application Number:
  US10/813,891
• Publication Number:
  US 20040215895A1
• Time in Patent Office: 1,154 Days
• Field of Search: None
• US Class Current: 711/141
• CPC Class Codes: G06F 12/0817 using directory methods
• CPC Class Codes: G06F 12/0833 in combination with broadcast means (e.g. for invalidation or updating)

Abstract

A system may include several nodes coupled by in inter-node network. Each node includes several active devices coupled by an address network. The address network included in one of the nodes may be configured to convey address packets specifying a particular coherency unit in broadcast mode. The address network included in a different one of the nodes may be configured to convey address packets specifying that coherency unit in point-to-point mode.

Claims

1. A system, comprising:

a plurality of nodes, wherein each node of the plurality of nodes includes a plurality of active devices, one or more memory subsystems, and an address network that is configured to couple the plurality of active devices and the one or more memory subsystems together; and

an inter-node network coupled to the address network and configured to convey communications between the plurality of nodes;

wherein each address network is independently configured to convey address packets specifying a coherency unit using a broadcast mode and a point-to-point mode; and

wherein each address network is further configured to independently select one of the broadcast mode and the point-to-point mode dependent upon whether the coherency unit maps to any of the one or more memory subsystems included in the node.

2. The system of claim 1, wherein in response to determining the coherency unit does not map to any of the one or more memory subsystems included in the node, the address network in the node is configured to convey all the address packets specifying the coherency unit in the broadcast mode.

3. The system of claim 2, wherein each node of the plurality of nodes includes an interface coupled to the address network, wherein the interface is configured to send and receive coherency messages on the inter-node network, wherein the interface is configured to store a record for each of the address packets specifying the coherency unit in an outstanding transaction queue, and wherein the interface is configured to send a coherency message specifying the coherency unit via the inter-node network to a home node for the coherency unit in response to the record being placed in the outstanding transaction queue.

4. The system of claim 2, wherein in response to determining the coherency unit does map to any of the one or more memory subsystems included in the node, the address network is configured to convey all address packets specifying the coherency unit in the point-to-point mode.

5. The system of claim 1, wherein each node includes a transmission mode unit configured to store an indication that controls whether a given address packet is transmitted through the address network in the broadcast mode or the point-to-point mode.

6. The system of claim 5, wherein the transmission mode unit included in each node is configured to store a plurality of additional indications to control whether address packets other than the given address packet are transmitted through the address network in point-to-point mode or in broadcast mode.

7. The system of claim 5, wherein the transmission mode unit is configured to dynamically update the indication to switch a conveyance mode of address packets specifying a given coherency unit from the point-to-point mode to the broadcast mode, and vice versa.

8. The system of claim 1, wherein each node further includes a respective directory, wherein the directory includes a plurality of entries, each entry corresponding to a different coherency unit that is mapped to a memory subsystem included in the node, wherein each entry contains an indication of whether a cached copy of a corresponding coherency unit has been created in one or more of the plurality of active devices included in the node.

9. The system of claim 8, wherein address packets conveyed by the address network in broadcast mode are broadcast to each of the plurality of active devices included in the node regardless of information contained within the directory.

10. The system of claim 8, wherein address packets conveyed by the address network in point-to-point mode are conveyed by the address network to a memory subsystem included in the node;

and

wherein the memory subsystem is configured to access the directory and responsively send address packets to one or more of the plurality of active devices dependent upon information contained within the directory.

11. The system of claim 1, wherein address packets specifying some coherency units are transmitted through the address network in the point-to-point mode while address packets specifying other coherency units are transmitted through the address network in the broadcast mode.

12. The system of claim 8, wherein the address network is further configured to select whether to transmit address packets in the broadcast mode or in the point-to-point mode dependent on an address specified in each of the address packets.

13. The system of claim 1, wherein one of the plurality of active devices included in the node is configured to transition an ownership responsibility for the coherency unit at a different time than an access right associated with the coherency unit transitions.

14. A method of operating a multi-node computer system, wherein the multi-node computer system includes a plurality of nodes coupled together by an inter-node network, wherein each node includes a plurality of active devices, one or more memory subsystems, and an address network coupling together the plurality of active devices and the one or more memory subsystems, the method comprising:

each address network included in a node of the plurality of nodes independently conveying address packets specifying a coherency unit using a broadcast mode and a point-to-point mode; and

each address network independently selecting one of the broadcast mode and the point-to-point mode dependent upon whether the coherency unit maps to any of the one or more memory subsystems included in the node.

15. The method of claim 14, further comprising:

the address network in the node conveying all the address packets specifying the coherency unit in the broadcast mode, in response to determining the coherency unit does not map to any of the one or more memory subsystem included in the node.

16. The method of claim 15, further comprising:

an interface included in each node and coupled to the address network storing a record for each of the address packets specifying the coherency unit in an outstanding transaction queue; and

the interface sending a coherency message specifying the coherency unit via the inter-node network to a home node for the coherency unit in response to the record being placed in the outstanding transaction queue.

17. The method of claim 15, further comprising in response to determining the coherency unit does map to any of the one or more memory subsystems included in the node, each address network conveying all address packets specifying the coherency unit in the point-to-point mode.

18. The method of claim 14, further comprising a transmission mode unit included in the node storing an indication that controls whether a given address packet is transmitted through the address network in the broadcast mode or the point-to-point mode.

19. The method of claim 18, further comprising the transmission mode unit included in each node storing a plurality of additional indications to control whether address packets other than the given address packet are transmitted through the address network in point-to-point mode or in broadcast mode.

20. The method of claim 18, further comprising the transmission mode unit dynamically updating the indication to switch a conveyance mode of address packets specifying a given coherency unit from the point-to-point mode to the broadcast mode, and vice versa.

21. The method of claim 15, wherein each node further includes a directory, wherein the directory includes a plurality of entries, each entry corresponding to a different coherency unit that is mapped to one or more of the memory subsystems included in the node, wherein each entry contains an indication of whether a cached copy of a corresponding coherency unit has been created in one or more of the plurality of active devices included in the node.

22. The method of claim 21, wherein conveying address packets in the broadcast mode includes conveying the address packets to each of the plurality of active devices included in the node regardless of information contained within the directory.

23. The method of claim 21, further comprising the address network conveying an address packet to at least one of the memory subsystems included in the node when the address packet is conveyed by the address network in the point-to-point mode;

and

in response to receiving the address packet, the at least one of the memory subsystems accessing the directory and responsively sending corresponding address packets to one or more of the plurality of active devices dependent upon information contained within the directory.

24. The method of claim 14, further comprising the address network conveying address packets specifying some coherency units in the point-to-point mode and conveying address packets specifying other coherency units in the broadcast mode.

25. The method of claim 24, further comprising the address network selecting whether to transmit an address packet in the broadcast mode or in the point-to-point mode dependefit on an address specified in the address packet.

26. The method of claim 14, further comprising at least one of the plurality of active devices included in the node transitioning an ownership responsibility for the coherency unit at a different time than an access right associated with the coherency unit transitions.