Method and system for an on-chip completion cache for optimized completion building

• US 9,191,452 B2
• Filed: 12/20/2012
• Issued: 11/17/2015
• Est. Priority Date: 12/20/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Manula, Brian Edward, Sandven, Magne Vigulf, Bugge, Haakon Ording
• Primary Examiner(s): Maung, Zarni
• Application Number:
  US13/721,765
• Publication Number:
  US 20140181241A1
• Time in Patent Office: 1,062 Days
• Field of Search: 709/203, 709217-218, 709223-229, 709/250
• US Class Current: 1/1
• CPC Class Codes: H04L 49/358 Infiniband Switches
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A method for optimizing completion building is disclosed. The method involves receiving a work request by a host channel adapter (HCA), caching a portion of the work request in a completion cache in the HCA, wherein the cached portion of the work request includes information for building a completion for the work request, receiving, by the HCA, a response to the work request, querying the completion cache upon receiving the response to the work request to obtain the cached portion of the work request, and building the completion for the work request using the cached portion of the work request, wherein the completion informs a software application of at least a status of the work request as executed by the HCA.

Claims

1. A method for completion building, the method comprising:

receiving a work request by a host channel adapter (HCA);

caching a portion of the work request in a completion cache in the HCA, wherein the cached portion of the work request comprises information for building a completion for the work request;

receiving, by the HCA, a response to the work request;

querying the completion cache upon receiving the response to the work request to obtain the cached portion of the work request; and

building the completion for the work request using the cached portion of the work request, wherein the completion informs a software application of at least a status of the work request as executed by the HCA.

2. The method of claim 1, further comprising:

arming a completion cache timer corresponding to the cached portion of the work request, wherein the completion cache timer comprises a length of time that the cached portion of the work request remains cached; and

evicting a cache entry corresponding to the work request when the completion cache timer expires.

3. The method of claim 2, wherein the work request and the response to the work request are transmitted over an Infiniband®

network fabric.

4. The method of claim 3, wherein the length of time is equal to or greater than a round trip latency of the Infiniband®

network fabric.

5. The method of claim 2, wherein:

The completion cache timer comprises a credit associated with the cached portion of the work request, wherein the credit remains outstanding until the work request is processed and the response to the work request is received by the HCA, and

when a number of outstanding credits is higher than a predetermined threshold, an outgoing work request rate is throttled.

6. The method of claim 1, further comprising:

issuing the work request by the software application; and

committing the work request to a send queue in a host operatively connected to the HCA prior to transmission of the work request to the HCA.

7. The method of claim 6, wherein the completion cache comprises cache logic configured to:

determine that the send queue is stalled after the completion cache timer expires;

evict cached entries for the stalled send queue corresponding to the most recent send queue entries, wherein cached entries for the stalled send queue comprises the cached portion of the work request; and

prefetch the evicted cache entries for the stalled send queue when the stalled send queue begins receiving responses from the HCA corresponding to the oldest send queue entries.

8. The method of claim 1, further comprising:

executing the work request, by the HCA, to generate the response to the work request.

9. The method of claim 1, wherein the work request comprises one selected from the group consisting of a send, a RDMA write, RDMA read, and an atomic operation, and wherein the portion of the work request comprises metadata of the work request comprising at least one selected from a group consisting of:

an identifier for the work request, an OpCode of the work request, an identifier of a completion queue to which the completion for the work request is written, a location for placing read data for an RDMA read, and a length of a data transfer associated with the work request.

10. The method of claim 1, further comprising:

writing, by the HCA, the completion to a completion queue in the host, wherein the completion is used by the software application to determine the status of the work request.

11. The method of claim 1, wherein the response is an acknowledgement message acknowledging that the work request has been processed successfully, the method further comprising:

dividing, by the HCA, the work request into a plurality of packets, wherein the acknowledgement message is sent for each of the plurality of packets; and

building the completion using the cached portion of the work request only upon receiving all acknowledgement messages corresponding to the work request.

12. A host communication adapter (HCA) comprising:

transmitting processing logic configured to;

transmit a work request issued by a software application executing on a host operatively connected to the HCA; and

cache a portion of the work request in a completion cache, wherein the cached portion of the work request comprises information for building a completion for the work request;

responder processing logic configured to;

receive the work request from the transmitting processing logic; and

generate a response to the work request; and

a completion module configured to;

receive the response to the work request from the responder processing logic; and

build the completion for the work request, wherein the completion informs the software application of at least a status of the work request as executed by the HCA.

13. The host communication adapter of claim 12, wherein the completion cache comprises cache logic configured to:

arm a completion cache timer corresponding to the cached portion of the work request, wherein the completion cache timer comprises a length of time that the cached portion of the work request remains cached; and

evict a cache entry corresponding to the work request when the completion cache timer expires.

14. The host communication adapter of claim 13, wherein the length of time is equal to or greater than a round trip latency of the Infiniband®

network fabric.

15. The host communication adapter of claim 13, wherein

the completion cache timer comprises a credit associated with the cached portion of the work request, wherein the credit remains outstanding until the work request is processed and the response to the work request is received by the HCA, and

when a number of outstanding credits is higher than a predetermined threshold, an outgoing work request rate is throttled.

16. The host communication adapter of claim 12, wherein the host is configured to:

commit the work request to a send queue in the host prior to transmission of the work request to the HCA,

wherein the cached portion of the work request corresponds to an entry in the send queue of the committed work request.

17. The host communication adapter of claim 12, wherein the work request comprises one selected from the group consisting of a send, a RDMA write, RDMA read, and an atomic operation, and wherein the portion of the work request comprises metadata of the work request comprising at least one selected from a group consisting of:

an identifier for the work request, an OpCode of the work request, an identifier of a completion queue to which the completion for the work request is written, a location for placing read data for an RDMA read, and a length of a data transfer associated with the work request.

18. A system, comprising:

a host executing a software application, wherein the software application is configured to issue a work request; and

a host communication adapter (HCA) operatively connected to the host and configured to;

receive the work request;

cache a portion of the work request in a completion cache in the HCA, wherein the cached portion of the work request comprises information for building a completion for the work request;

process the work request and generate a response to the work request; and

build the completion for the work request using the cached portion of the work request, wherein the completion informs the software application of at least a status of the work request as executed by the HCA.

19. The system of claim 18, the HCA being further configured to:

arm a completion cache timer corresponding to the cached portion of the work request, wherein the completion cache timer comprises a length of time equal to or greater than a round trip latency of the Infiniband®

network fabric that the cached portion of the work request remains cached; and

evict a completion cache entry corresponding to the work request when the completion cache timer expires.

20. The system of claim 19, wherein:

The completion cache timer comprises a credit associated with the cached portion of the work request, wherein the credit remains outstanding until the work request is processed and the response to the work request is received by the HCA, and

when a number of outstanding credits is higher than a predetermined threshold, an outgoing work request rate is throttled.