System and method for port-to-port communications using direct memory access

• US 10,664,420 B2
• Filed: 04/19/2018
• Issued: 05/26/2020
• Est. Priority Date: 04/19/2018
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Inventors: Wu, Kenny, Karnowski, Mark, Smart, James, Shenoy, Ravi, Chhabra, Lalit, Gervasio, Jr., Gregorio, Le, Tuong, Nguyen, Vuong
• Primary Examiner(s): Shin, Christopher B
• Application Number:
  US15/957,264
• Publication Number:
  US 20190324926A1
• Time in Patent Office: 768 Days
• Field of Search: 710310
• CPC Class Codes: G06F 13/1663 Access to shared memory
• CPC Class Codes: G06F 13/1673 using buffers
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: G06F 2213/0026 PCI express

Abstract

A system, method, and adaptor that facilitate data transmission are described. One example of the disclosed system facilitates the chip-to-chip transport of header descriptors and payloads. The system may include a source chip, a destination chip, and a set of queues describing buffer memory locations for staging header descriptors and payloads to be transferred from the source chip to the destination chip, where the set of queues are directly accessible to the source chip and to the destination chip.

Claims

1. A system that facilitates chip-to-chip transport of header descriptors and payloads, comprising:

a source chip;

a destination chip;

a set of queues describing buffer memory locations for staging the header descriptors and the payloads to be transferred from the source chip to the destination chip, wherein the set of queues are directly accessible to the source chip and to the destination chip; and

a host device coupled to the source chip and the destination chip and that executes a setup routine to establish the set of queues in a non-host device memory prior to transfer of the header descriptors and the payloads from the source chip to the destination chip, wherein the transfer of the header descriptors and the payloads from the source chip to the destination chip flows through the set of queues in the non-host device memory and bypasses a central processing unit (CPU) of the host device after the set of queues have been established by the host device.

2. The system of claim 1, wherein the set of queues comprises a work queue (WQ), receive queue (RQ) pair, header buffer and payload buffer, wherein the transfer of the header descriptors and the payloads is initiated by a doorbell to the destination chip.

3. The system of claim 2, wherein a queue depth of the WQ and RQ pair is equal.

4. The system of claim 2, wherein the set of queues comprises the payload buffer and the header buffer, wherein an RQ payload contains payload buffer memory addresses where payloads are placed by the source chip, and wherein an RQ header contains header buffer memory addresses where a work queue entry (WQE) is created by the source chip with a header descriptor that provides information on a corresponding payload.

5. The system of claim 4, wherein the information provided in the header descriptor comprises at least one of frame header, optional header, and control attributes.

6. The system of claim 4, wherein the source chip maintains a phase state bit that tracks a state of a current valid phase, wherein the phase state bit is added to a last word in the WQE, and wherein the phase state bit is toggled when the WQ wraps to a predetermined index.

7. The system of claim 1, wherein the source chip rings a work queue (WQ) doorbell with a WQ identifier to the destination chip via a PCIe posted memory write transaction once data placements into the buffer memory locations are complete.

8. The system of claim 1, wherein the buffer memory locations are maintained at the destination chip.

9. The system of claim 1, wherein the buffer memory locations are accessed by the source chip through a network fabric.

10. The system of claim 9, wherein the network fabric comprises a PCIe network fabric, wherein the source chip writes the header descriptors and the payloads directly to the buffer memory locations without assistance from the host device, and wherein the destination chip retrieves the header descriptors and the payloads directly from the buffer memory locations.

11. The system of claim 1, wherein the buffer memory locations are co-located with one or both of the source chip and destination chip.

12. A method of enabling chip-to-chip transport of header descriptors and payloads, the method comprising:

establishing, by a host device prior to transfer of the header descriptors and the payloads from a source chip to a destination chip, a set of queues that describe buffer memory locations in a non-host device memory for staging the header descriptors and the payloads to be transferred from the source chip to the destination chip;

writing, by the source chip after the set of queues is established, the header descriptors and the payloads directly into the buffer memory locations described by the set of queues while bypassing a central processing unit (CPU) of the host device; and

reading, by the destination chip, header descriptors and the payloads directly from the buffer memory locations described by the set of queues while bypassing the CPU of the host device.

13. The method of claim 12, wherein the set of queues comprises a work queue (WQ), receive queue (RQ) pair, header buffer and payload buffer, wherein a queue depth of the WQ and RQ pair is equal, wherein the set of queues comprises the payload buffer and the header buffer, wherein an RQ payload contains payload buffer memory addresses where payloads are placed by the source chip, and wherein an RQ header contains header buffer memory addresses where a work queue entry (WQE) is created by the source chip with a header descriptor that provides information on a corresponding payload.

14. The method of claim 13, further comprising:

maintaining a phase state bit that tracks a state of a current valid phase; and

toggling the phase state bit when the WQ wraps to a predetermined index.

15. The method of claim 12, wherein the source chip rings a work queue (WQ) doorbell with a WQ identifier to the destination chip via a PCIe posted memory write transaction once data placements into the buffer memory locations are complete.

16. The method of claim 12, wherein the buffer memory locations are accessed by the source chip through a network fabric.

17. An adaptor, comprising:

a data source;

a data destination;

buffer memory; and

a set of queues describing buffer memory locations in the buffer memory, wherein the buffer memory provides a location for staging header descriptors and payloads to be transferred from the data source to the data destination, wherein the set of queues are setup by a host device prior to transfer of the header descriptors and the payloads from the data source to the data destination in order to enable direct transfer of the header descriptors and the payloads between the data source and the data destination while bypassing a central processing unit (CPU) of the host device, and wherein the buffer memory is a non-host device buffer memory.

18. The adaptor of claim 17, wherein the set of queues comprises a work queue (WQ), receive queue (RQ) pair, wherein a queue depth of the WQ and RQ pair is equal, wherein the set of queues comprises a payload buffer and a header buffer, wherein an RQ payload contains payload buffer memory addresses where the payloads are placed by the data source, and wherein an RQ header contains header buffer memory addresses where a work queue entry (WQE) is created by the data source with a header descriptor that provides information on a corresponding payload.

19. The adaptor of claim 17, wherein the data source rings a work queue (WQ) doorbell with a WQ identifier to the data destination, wherein the WQ doorbell includes a number of entries posted field or a source destination index field.

20. The adaptor of claim 17, wherein the data source comprises a first port of the adaptor and wherein the data destination comprises a second port of the adaptor.