Inter-chip interconnect protocol for a multi-chip system

• US 9,372,800 B2
• Filed: 03/07/2014
• Issued: 06/21/2016
• Est. Priority Date: 03/07/2014
• Status: Active Grant

• Current Assignee: Marvell Asia Pte Limited (Marvell Technology Group Limited)
• Original Assignee: Cavium, LLC (Marvell Technology Group Limited)
• Inventors: Akkawi, Isam, Kessler, Richard E., Asher, David H., Chin, Bryan W., Snyder, Wilson P. II
• Primary Examiner(s): Pham, Ly D
• Application Number:
  US14/201,513
• Publication Number:
  US 20150254183A1
• Time in Patent Office: 837 Days
• Field of Search: 711/141, 711/148
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0813 with a network or matrix configuration
• CPC Class Codes: G06F 12/0822 Copy directories local copy tags for implementing a bus snooping protocol G06F12/0831
• CPC Class Codes: G06F 12/0828 with concurrent directory accessing, i.e. handling multiple concurrent coherency transactions
• CPC Class Codes: G06F 12/0831 using a bus scheme, e.g. with bus monitoring or watching means
• CPC Class Codes: G06F 12/0833 in combination with broadcast means (e.g. for invalidation or updating)
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 13/1663 Access to shared memory
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/62 Details of cache specific to multiprocessor cache arrangements
• CPC Class Codes: G06F 2212/621 Coherency control relating to peripheral accessing, e.g. from DMA or I/O device
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A multi-chip system includes multiple chip devices configured to communicate to each other and share resources. According to at least one example embodiment, a method of providing memory coherence within the multi-chip system comprises maintaining, at a first chip device of the multi-chip system, state information indicative of one or more states of one or more copies, residing in one or more chip devices of the multi-chip system, of a data block. The data block is stored in a memory associated with one of the multiple chip devices. The first chip device receives a message associated with a copy of the one or more copies of the data block from a second chip device of the multiple chip devices, and, in response, executes a scheme of one or more actions determined based on the state information maintained at the first chip device and the message received.

Claims

1. A method of providing memory coherence between multiple chip devices of a multi-chip system, the method comprising:

maintaining, at a first chip device of the multi-chip system, state information indicative of one or more states of one or more copies of a data block, the data block stored in a memory associated with one of the multiple chip devices, the one or more copies of the data block residing in one or more chip devices of the multi-chip system;

receiving, by the first chip device, a message associated with a copy of the one or more copies of the data block from a second chip device of the multiple chip devices; and

in response to the message received, executing, by the first chip device, a scheme of one or more actions determined based on the state information maintained at the first chip device and the message received;

wherein the data block is stored in a memory attached to the first chip device and the message is indicative of a state, maintained at the second chip device, of a copy of the data block residing in the second chip device.

2. The method as recited in claim 1, wherein the one or more states of the one or more copies of the data block include at least one of:

an exclusive copy;

a shared copy;

a modified copy;

an owned copy; and

an invalid copy.

3. The method as recited in claim 1, further comprising determining the scheme of one or more actions, said determining includes:

upon receiving the message, detecting a discrepancy between the state, of the copy of the data block residing in the second chip device, indicated by the message received, and a state of the same copy of the data block in the state information maintained by the first chip device.

4. The method as recited in claim 3, wherein executing the scheme of one or more actions includes:

upon detecting the discrepancy, waiting for receipt of an other message, by the first chip device, from the second chip device.

5. The method as recited in claim 4, wherein executing the scheme of one or more actions further includes:

updating the state of the same copy of the data block, in the state information maintained by the first chip device, to a transient state until the other message is received from the second chip device; and

upon receiving the other message, updating the state of the same copy of the data block, in the state information maintained by the first chip device, to a state indicated by the other message received.

6. The method as recited in claim 4, wherein executing the scheme of one or more actions further includes at least one of:

upon receiving the other message, sending a response message to the second chip device;

upon receiving the other message, updating the state of the same copy of the data block in the state information maintained by the first chip device;

upon receiving the other message, sending a message to another chip device, other than the second chip device, of the multiple chip devices;

upon receiving the other message, updating the data block stored in the memory attached to the first chip device;

upon receiving the other message, updating a copy of the data block residing in the first chip device;

upon receiving the other message, causing at least one copy of the one or more copies of the data block to be invalidated.

7. The method as recited in claim 6, wherein updating the data block includes:

receiving an indication referring to a sub-block of multiple sub-blocks in the data block;

receiving a modified copy of the sub-block referred to by the indication; and

updating, in the data block, the sub-block referred to by the indication based on the modified copy of the sub-block received.

8. A method of providing memory coherence between multiple chip devices of a multi-chip system, the method comprising:

maintaining, at a first chip device of the multi-chip system, state information indicative of one or more states of one or more copies of a data block, the data block stored in a memory associated with one of the multiple chip devices, the one or more copies of the data block residing in one or more chip devices of the multi-chip system;

receiving, by the first chip device, a message associated with a copy of the one or more copies of the data block from a second chip device of the multiple chip devices; and

in response to the message received, executing, by the first chip device, a scheme of one or more actions determined based on the state information maintained at the first chip device and the message received, wherein the data block is stored in a memory attached to the second chip device and wherein the message is indicative of a state, maintained at the second chip device, of a copy of the data block residing in the first chip device.

9. The method as recited in claim 8, further comprising determining the scheme of one or more actions, said determining includes:

upon receiving the message, detecting a discrepancy between the state, of the copy of the data block residing in the first chip device, indicated by the message received, and a state of the same copy of the data block in the state information maintained by the first chip device.

10. The method as recited in claim 9, wherein executing the scheme of one or more actions includes:

upon detecting the discrepancy, sending a response message to the second chip device indicative of the state of the same copy of the data block in the state information maintained by the first chip device.

11. The method as recited in claim 10, wherein executing the scheme of one or more actions further includes:

storing information indicative of the message received until receipt of an other message from the second chip device.

12. The method as recited in claim 11, wherein executing the scheme of one or more actions further includes:

upon receiving the other message from the second chip device, providing the second chip device with a response to the other message received.

13. A multi-chip system comprising:

multiple chip devices;

a memory attached to one of the multiple chip devices, the memory storing a data block; and

a controller in the first chip device, the controller being configured to;

maintain state information indicative of one or more states of one or more copies of the data block, the one or more copies of the data block residing in one or more chip devices of the multiple chip devices;

receive a message associated with a copy of the one or more copies of the data block from a second chip device of the multiple chip devices; and

in response to the message received, execute a scheme of one or more actions determined based on the state information maintained at the first chip device and the message received

wherein the data block is stored in a memory attached to, or residing in, the first chip device and the message is indicative of a state, maintained at the second chip device, of a copy of the data block residing in the second chip device.

14. The multi-chip system as recited in claim 13, wherein the one or more states of the one or more copies of the data block include at least one of:

an exclusive copy;

a shared copy;

a modified copy;

an owned copy; and

an invalid copy.

15. The multi-chip system as recited in claim 13, wherein the controller is further configured to determine the scheme of one or more actions, in determining the scheme of one or more actions, the controller is further configured to:

upon receiving the message, detect a discrepancy between the state, of the copy of the data block residing in the second chip device, indicated by the message received, and a state of the same copy of the data block in the state information maintained by the first chip device.

16. The multi-chip system as recited in claim 15, wherein in executing the scheme of one or more actions, the controller is further configured to:

upon detecting the discrepancy, wait for receipt of an other message, by the first chip device, from the second chip device.

17. The multi-chip system as recited in claim 16, wherein in executing the scheme of one or more actions, the controller is further configured to:

update the state of the same copy of the data block, in the state information maintained by the first chip device, to a transient state until the other message is received from the second chip device; and

upon receiving the other message, update the state of the same copy of the data block, in the state information maintained by the first chip device, to a state indicated by the other message received.

18. The multi-chip system as recited in claim 16, wherein in executing the scheme of one or more actions, the controller is further configured to:

upon receiving the other message, send a response message to the second chip device;

upon receiving the other message, update the state of the same copy of the data block in the state information maintained by the first chip device;

upon receiving the other message, send a message to another chip device, other than the second chip device, of the multiple chip devices;

upon receiving the other message, update the data block stored in the memory attached to the first chip device;

upon receiving the other message, update a copy of the data block residing in the first chip device;

upon receiving the other message, cause at least one copy of the one or more copies of the data block to be invalidated.

19. The multi-chip system as recited in claim 18, wherein in updating the data block, the controller is configured to:

receive an indication referring to a sub-block of multiple sub-blocks in the data block;

receive a modified copy of the sub-block referred to by the indication; and

update, in the data block, the sub-block referred to by the indication based on the modified copy of the sub-block received.

20. A multi-chip system comprising:

multiple chip devices;

a memory attached to one of the multiple chip devices, the memory storing a data block; and

a controller in the first chip device, the controller being configured to;

maintain state information indicative of one or more states of one or more copies of the data block, the one or more copies of the data block residing in one or more chip devices of the multiple chip devices;

receive a message associated with a copy of the one or more copies of the data block from a second chip device of the multiple chip devices; and

in response to the message received, execute a scheme of one or more actions determined based on the state information maintained at the first chip device and the message received;

wherein the data block is stored in a memory attached to the second chip device and wherein the message is indicative of a state, maintained at the second chip device, of a copy of the data block residing in the first chip device.

21. The multi-chip system as recited in claim 20, wherein the controller is further configured to determine the scheme of one or more actions, in determining the scheme of one or more actions, the controller is further configured to:

upon receiving the message, detect a discrepancy between the state, of the copy of the data block residing in the first chip device, indicated by the message received, and a state of the same copy of the data block in the state information maintained by the first chip device.

22. The multi-chip system as recited in claim 21, wherein in executing the scheme of one or more actions, the controller is further configured to:

upon detecting the discrepancy, send a response message to the second chip device indicative of the state of the same copy of the data block in the state information maintained by the first chip device.

23. The multi-chip system as recited in claim 22, wherein in executing the scheme of one or more actions, the controller is further configured to:

store information indicative of the message received until receipt of an other message from the second chip device.

24. The multi-chip system as recited in claim 23, wherein in executing the scheme of one or more actions, the controller is further configured to:

upon receiving the other message from the second chip device, provide the second chip device with a response to the other message received.