FAULT-TOLERANT CACHE COHERENCE OVER A LOSSY NETWORK

• US 20190205252A1
• Filed: 12/29/2017
• Published: 07/04/2019
• Est. Priority Date: 12/29/2017
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: LOEWENSTEIN, PAUL N., WALKER, DAMIEN, MITRA, PRIYAMBADA, VAHIDSAFA, ALI, COHEN, MATTHEW, EBERGEN, JOSEPHUS, BROCK, ANDREW
• Granted Patent: US 10,452,547 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/0824 Distributed directories, e.g. linked lists of caches
• CPC Class Codes: G06F 12/0828 with concurrent directory accessing, i.e. handling multiple concurrent coherency transactions
• CPC Class Codes: G06F 12/128 adapted to multidimensional cache systems, e.g. set-associative, multicache, multiset or multilevel
• CPC Class Codes: G06F 2212/1032 Reliability improvement, data loss prevention, degraded operation etc
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/621 Coherency control relating to peripheral accessing, e.g. from DMA or I/O device

Abstract

A cache coherence system manages both internode and intranode cache coherence in a cluster of nodes. Each node in the cluster of nodes is either a collection of processors running an intranode coherence protocol between themselves, or a single processor. A node comprises a plurality of coherence ordering units (COUs) that are hardware circuits configured to manage intranode coherence of caches within the node and/or internode coherence with caches on other nodes in the cluster. Each node contains one or more directories which tracks the state of cache line entries managed by the particular node. Each node may also contain one or more scoreboards for managing the status of ongoing transactions. The internode cache coherence protocol implemented in the COUs may be used to detect and resolve communications errors, such as dropped message packets between nodes, late message delivery at a node, or node failure. Additionally, a transport layer manages communication between the nodes in the cluster, and can additionally be used to detect and resolve communications errors.

Claims

1. A method, comprising:

sending, by a first hardware unit of a first plurality of hardware units on a requester node of a cluster of nodes, a first request for data to a home node of the cluster of nodes, wherein the first request comprises a first sequence number;

wherein each node of the cluster of nodes comprises a plurality of hardware units wherein each hardware unit of the plurality of hardware units is coupled to a particular memory and a particular cache and each particular hardware unit of the plurality of hardware units is configured as a cache controller of the particular memory and the particular cache;

determining, by the first hardware unit, that no data message has been received by the first hardware unit from the home node in a specified time out period for the first request for data;

based on the determining that no data message has been received, sending, by the first hardware unit on the requester node, a second request for data to the home node, wherein the second request comprises a second sequence number;

receiving, by the first hardware unit on the requester node, a first data message containing the requested data from the home node; and

sending, by the first hardware unit on the requester node, an acknowledgement message to the home node.

2. The method of claim 1, further comprising:

storing a first cache line state of the requested data in a first directory on the requester node; and

storing a second cache line state of the requested data in a second directory on the home node.

3. The method of claim 2, further comprising:

in response to receiving the data message at the requester node, the first hardware unit updating the first cache line state of the requested data on the requester node to a shared clean state; and

in response to receiving the acknowledgement message at a second hardware unit on the home node, the second hardware unit updating the second cache line state of the requested data on the home node to a shared clean state.

4. The method of claim 1, further comprising:

storing, in a requester scoreboard of the requester node, a scoreboard comprising a plurality of scoreboard entries, wherein each scoreboard entry of the plurality of scoreboard entries comprises a transaction identifier, sequence number, source node identifier, destination node identifier, and cache line address; and

storing, based on the sending of the first request for data, by the first hardware unit, a first scoreboard entry in the requester scoreboard for the first request for data, wherein the first scoreboard entry comprises a first transaction identifier, the first sequence number, a first source node identifier that identifies the requester node, a first destination node identifier that identifies the home node, and a first cache line address that identifies the data in the first request.

5. The method of claim 4, further comprising:

storing, based on the sending of the second request for data, by the first hardware unit, a second scoreboard entry in the requester scoreboard for the first request for data, wherein the second scoreboard entry comprises the first transaction identifier, a second sequence number, the first source node identifier that identifies the requester node, the first destination node identifier that identifies the home node, and the first cache line address.

6. The method of claim 5, further comprising:

in response to receiving the first data message containing the requested data from the home node, deleting the first scoreboard entry and the second scoreboard entry from the requester scoreboard.

7. A first node for use in a distributed computing system, the first node comprising:

one or more hardware units, wherein each hardware unit of the one or more hardware units comprises one or more processors, registers, content-addressable memories, and/or other computer-implemented hardware circuity;

wherein each hardware unit of the one or more hardware units is coupled to a particular memory and a particular cache and each particular hardware unit of the one or more hardware units is configured as a cache controller of the particular memory and the particular cache;

wherein a first hardware unit, of the one or more hardware units, is configured to;

send a first request for data to a second node, wherein the first request comprises a first sequence number;

determine that no data message has been received by the first hardware unit from the second node in a specified time out period for the first request for data;

based on the determination that no data message has been received, send, by the first hardware unit, send a second request for data to the second node, wherein the second request comprises a second sequence number;

receive a first data message containing the requested data from the second node; and

send an acknowledgement message to the second node.

8. The first node of claim 7, wherein the first hardware unit is further configured to:

store a first cache line state of the requested data in a first directory on the first node.

9. The first node of claim 8, wherein the first hardware unit is further configured to:

in response to receiving the data message at the requester node, the first hardware unit updating the first cache line state of the requested data on the first node to a shared clean state.

10. The first node of claim 7, wherein the first hardware unit is further configured to:

store, in a requester scoreboard of the first node, a scoreboard comprising a plurality of scoreboard entries, wherein each scoreboard entry of the plurality of scoreboard entries comprises a transaction identifier, sequence number, source node identifier, destination node identifier, and cache line address; and

store, based on the sending of the first request for data, a first scoreboard entry in the requester scoreboard for the first request for data, wherein the first scoreboard entry comprises a first transaction identifier, the first sequence number, a first source node identifier that identifies the requester node, a first destination node identifier that identifies the first node, and a first cache line address that identifies the data in the first request.

11. The first node of claim 10, wherein the first hardware unit is further configured to:

store, based on the sending of the second request for data a second scoreboard entry in the requester scoreboard for the first request for data, wherein the second scoreboard entry comprises the first transaction identifier, a second sequence number, the first source node identifier that identifies the requester node, the first destination node identifier that identifies the home node, and the first cache line address.

12. The first node of claim 11, wherein the first hardware unit is further configured to:

in response to receiving the first data message containing the requested data from the home node, delete the first scoreboard entry and the second scoreboard entry from the requester scoreboard.

13. One or more non-transitory computer-readable storage media storing instructions, which when executed by one or more processors, cause:

sending, by a first hardware unit of a first plurality of hardware units on a requester node of a cluster of nodes, a first request for data to a home node of the cluster of nodes, wherein the first request comprises a first sequence number;

wherein each node of the cluster of nodes comprises a plurality of hardware units wherein each hardware unit of the plurality of hardware units is coupled to a particular memory and a particular cache and each particular hardware unit of the plurality of hardware units is configured as a cache controller of the particular memory and the particular cache;

determining, by the first hardware unit, that no data message has been received by the first hardware unit from the home node in a specified time out period for the first request for data;

based on the determining that no data message has been received, sending, by the first hardware unit on the requester node, a second request for data to the home node, wherein the second request comprises a second sequence number;

receiving, by the first hardware unit on the requester node, a first data message containing the requested data from the home node; and

sending, by the first hardware unit on the requester node, an acknowledgement message to the home node.

14. The one or more non-transitory computer-readable storage media of claim 13, further comprising instructions, which when executed by the one or more processors, cause:

storing a first cache line state of the requested data in a first directory on the requester node; and

storing a second cache line state of the requested data in a second directory on the home node.

15. The one or more non-transitory computer-readable storage media of claim 14, further comprising instructions, which when executed by the one or more processors, cause:

in response to receiving the data message at the requester node, the first hardware unit updating the first cache line state of the requested data on the requester node to a shared clean state; and

in response to receiving the acknowledgement message at a second hardware unit on the home node, the second hardware unit updating the second cache line state of the requested data on the home node to a shared clean state.

16. The one or more non-transitory computer-readable storage media of claim 13, further comprising instructions, which when executed by the one or more processors, cause:

storing, in a requester scoreboard of the requester node, a scoreboard comprising a plurality of scoreboard entries, wherein each scoreboard entry of the plurality of scoreboard entries comprises a transaction identifier, sequence number, source node identifier, destination node identifier, and cache line address; and

storing, based on the sending of the first request for data, by the first hardware unit, a first scoreboard entry in the requester scoreboard for the first request for data, wherein the first scoreboard entry comprises a first transaction identifier, the first sequence number, a first source node identifier that identifies the requester node, a first destination node identifier that identifies the home node, and a first cache line address that identifies the data in the first request.

17. The one or more non-transitory computer-readable storage media of claim 16, further comprising instructions, which when executed by the one or more processors, cause:

storing, based on the sending of the second request for data, by the first hardware unit, a second scoreboard entry in the requester scoreboard for the first request for data, wherein the second scoreboard entry comprises the first transaction identifier, a second sequence number, the first source node identifier that identifies the requester node, the first destination node identifier that identifies the home node, and the first cache line address.

18. The one or more non-transitory computer-readable storage media of claim 17, further comprising instructions, which when executed by the one or more processors, cause:

in response to receiving the first data message containing the requested data from the home node, deleting the first scoreboard entry and the second scoreboard entry from the requester scoreboard.