Method and system for ordering I/O access in a multi-node environment

• US 10,592,459 B2
• Filed: 03/07/2014
• Issued: 03/17/2020
• Est. Priority Date: 03/07/2014
• Status: Active Grant

• Current Assignee: Marvell Asia Pte Limited (Marvell Technology Group Limited)
• Original Assignee: Cavium, LLC (Marvell Technology Group Limited)
• Inventors: Kessler, Richard E.
• Primary Examiner(s): Yimer, Getente A
• Application Number:
  US14/201,557
• Publication Number:
  US 20150254207A1
• Time in Patent Office: 2,202 Days
• Field of Search: 710 39
• CPC Class Codes: G06F 12/0822 Copy directories local copy tags for implementing a bus snooping protocol G06F12/0831
• CPC Class Codes: G06F 12/0833 in combination with broadcast means (e.g. for invalidation or updating)
• CPC Class Codes: G06F 13/10 Program control for peripheral devices G06F13/14 - G06F13/42 take precedence
• CPC Class Codes: G06F 13/20 for access to input/output bus
• CPC Class Codes: G06F 13/423 with synchronous protocol

Abstract

According to at least one example embodiment, a multi-chip system includes multiple chip devices configured to communicate to each other and share resources, such as I/O devices. According to at least one example embodiment, a method of synchronizing access to an input/output (I/O) device in the multi-chip system comprises initiating, by a first agent of the multi-chip system, a first operation for accessing the I/O device, the first operation is queued, prior to execution by the I/O device, in a queue. Once the first operation is queued, an indication of such queuing is provided. Upon detecting, by a second agent of the multi-chip system, the indication of queuing the first operation in the queue, initiating a second operation to access the I/O device, the second operation is queued subsequent to the first operation in the queue.

Claims

1. A method of synchronizing access to an input/output (I/O) device in a multi-chip system, the method comprising:

initiating, by a first agent of the multi-chip system, a first operation for accessing the I/O device, the first operation is queued, prior to transmission to the I/O device, in a queue;

providing an indication of queuing the first operation in the queue prior to transmitting the first operation to the I/O device, wherein the indication of queuing the first operation in the queue comprises modifying a value located in a memory component of the multi-node system;

monitoring, by a second agent of the multi-chip system, the value located in the memory component of the multi-node system;

upon detecting, by the second agent of the multi-chip system, the indication of queuing the first operation in the queue, initiating, by the second agent of the multi-chip system, a second operation to access the I/O device, the second operation is queued subsequent to the first operation in the queue;

transmitting the first operation to the I/O device; and

transmitting the second operation to the I/O device after transmitting the first operation.

2. The method as recited in claim 1 further comprising executing a synchronization operation by the first agent of the multi-chip system, the synchronization operation enforcing the indication to be provided only after the first operation is already queued in the queue.

3. The method as recited in claim 2, wherein executing a synchronization operation includes:

executing a first synchronization operation if the first agent and the second agent reside in the same chip device of the multi-chip system; and

executing a second synchronization operation if the first agent and the second agent reside in different chip devices of the multi-chip system.

4. The method as recited in claim 1, wherein the queue and the I/O device reside in a same chip device of the multi-chip system.

5. The method as recited in claim 4, wherein the I/O device and the first agent reside in the same chip device of the multi-chip system, and wherein initiating the first operation includes sending the first operation directly to the queue for ordering access to the I/O device.

6. The method as recited in claim 4, wherein the I/O device and the queue reside in a first chip device, the first agent resides in a second chip device of the multi-chip system, and wherein initiating the first operation includes:

sending the first operation to an other queue residing in the second chip device; and

forwarding the first operation from the other queue in the second chip device to the queue residing in the first chip device through an inter-chip interconnect interface.

7. The method as recited in claim 6 further comprising:

receiving, by the first agent, an acknowledgement of queuing the first operation from the first chip device, wherein providing the indication of queuing the first operation is in response to receiving the acknowledgement.

8. The method as recited in claim 4, wherein the second agent and the I/O device reside in the same chip device of the multi-chip system, and wherein initiating the second operation includes sending the second operation directly to the queue.

9. The method as recited in claim 4, wherein the I/O device and the queue reside in a first chip device, the first agent resides in a second chip device of the multi-chip system, and wherein initiating the second operation includes:

sending the second operation to an other queue in the second chip device; and

forwarding the second operation from the other queue in the second chip device to the queue residing in the first chip device through an inter-chip interconnect interface.

10. The method as recited in claim 1, wherein the I/O device resides on a first chip device of the multi-chip system, and both the first agent and the second agent reside on a second chip of the multi-chip system, the method further comprising providing an indication of synchronizing access to the I/O device locally within the second chip device.

11. The method as recited in claim 10, wherein the queue resides on the second chip device.

12. The method as recited in claim 1, wherein the first or second operation is a store operation for writing data to the I/O device.

13. The method as recited in claim 1, wherein the first agent or the second agent is a core processor of the multi-chip system.

14. A multi-chip system comprising:

multiple chip devices, at least one chip device of the multiple chip devices including an input/output (I/O) device, and each chip device of the multiple chip devices including;

an inter-chip interconnect interface configured to couple the chip device to one or more other chip devices of the multi-chip system,

multiple agents configured to initiate operations for accessing the I/O device, and

a queue configured to queue operation for accessing the I/O device generated by one or more agents of the multiple agents,

wherein a first agent of the multi-chip system is configured to, upon initiating a first operation for accessing the I/O device, provide an indication of queuing the first operation in the queue prior to transmission of the first operation to the I/O device wherein the indication of queuing the first operation in the queue comprises modifying a value located in a memory component of the multi-node system, the first operation is queued in the queue of the multi-chip system;

wherein a second agent of the multi-chip system is configured to monitor the value located in the memory component of the multi-node system, initiate a second operation to access the I/O device, in response to detecting the indication of queuing the first operation in the queue, the second operation is queued subsequent to the first operation in the queue; and

the queue is further configured to transmit the first I/O operation to the I/O device and transmit the second I/O operation to the I/O device after transmitting the first I/O operation.

15. The multi-chip system as recited in claim 14, wherein the first agent of the multi-chip system is further configured to execute a synchronization operation, the synchronization operation enforcing the indication to be provided only after the first operation is already queued in the queue.

16. The multi-chip system as recited in claim 15, wherein in executing a synchronization operation the first agent of the multi-chip system is configured to:

execute a first synchronization operation if the first agent and the second agent reside in the same chip device of the multi-chip system; and

execute a second synchronization operation if the first agent and the second agent reside in different chip devices of the multi-chip system.

17. The multi-chip system as recited in claim 14, wherein the queue and the I/O device reside in a same chip device of the multi-chip system.

18. The multi-chip system as recited in claim 17, wherein the I/O device and the first agent reside in the same chip device of the multi-chip system, and wherein in initiating the first operation the first agent is further configured to send the first operation directly to the queue for ordering access to the I/O device.

19. The multi-chip system as recited in claim 17, wherein the I/O device and the queue reside in a first chip device, the first agent resides in a second chip device of the multi-chip system, and wherein in initiating the first operation, the first agent is further configured to:

send the first operation to an other queue residing in the second chip device, the first operation is forwarded from the other queue in the second chip device to the queue residing in the first chip device through an inter-chip interconnect interface.

20. The multi-chip system as recited in claim 19, wherein the first agent is further configured to receive an acknowledgement of queuing the first operation from the first chip device, and wherein providing the indication of queuing the first operation is in response to receiving the acknowledgement.

21. The multi-chip system as recited in claim 17, wherein the second agent and the I/O device reside in the same chip device of the multi-chip system, and wherein in initiating the second operation, the second agent is further configured to send the second operation directly to the queue.

22. The multi-chip system as recited in claim 17, wherein the I/O device and the queue reside in a first chip device, the first agent resides in a second chip device of the multi-chip system, and wherein in initiating the second operation, the second agent is further configured to:

send the second operation to an other queue in the second chip device, the second operation is forwarded from the other queue in the second chip device to the queue residing in the first chip device through an inter-chip interconnect interface.

23. The multi-chip system as recited in claim 14, wherein the I/O device resides on a first chip device of the multi-chip system, and both the first agent and the second agent reside on a second chip of the multi-chip system, the second chip device is configured provide an indication of synchronizing access to the I/O device locally within the second chip device.

24. The multi-chip system as recited in claim 23, wherein the queue resides on the second chip device.

25. The multi-chip system as recited in claim 14, wherein the first or second operation is a store operation for writing data to the I/O device.

26. The multi-chip system as recited in claim 14, wherein the first agent or the second agent is a core processor of the multi-chip system.