Doorbell backpressure avoidance mechanism on a host channel adapter

• US 9,069,485 B2
• Filed: 12/20/2012
• Issued: 06/30/2015
• Est. Priority Date: 12/20/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Manula, Brian Edward, Bugge, Haakon Ording, Sum, Benny
• Primary Examiner(s): LEE, CHUN KUAN
• Application Number:
  US13/721,720
• Publication Number:
  US 20140181323A1
• Time in Patent Office: 922 Days
• Field of Search: 710/5
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: H04L 49/358 Infiniband Switches

Abstract

A method for processing commands includes receiving, for multiple commands, doorbells for writing to a send queue scheduler buffer on a host channel adapter (HCA). The send queue scheduler buffer is associated with a send queue scheduler. The method further includes detecting a potential deadlock of the send queue scheduler from processing a portion of the doorbells, writing a subset of the doorbells to a doorbell overflow buffer on a host, operatively connected to the HCA, based on detecting the potential deadlock, and discarding the subset by the send queue scheduler without processing the subset of the plurality of doorbells before discarding.

Claims

1. A method for processing commands in a host channel adapter (HCA), comprising:

receiving, for a plurality of commands, a plurality of doorbells for writing to a send queue scheduler buffer on the HCA, wherein the send queue scheduler buffer is associated with a send queue scheduler;

detecting a potential deadlock of the send queue scheduler from processing a portion of the plurality of doorbells;

writing a subset of the plurality of doorbells to a doorbell overflow buffer on a host, operatively connected to the HCA, based on detecting the potential deadlock;

discarding, from the HCA, the subset of the plurality of doorbells by the send queue scheduler, after writing the subset of the plurality of doorbells to the doorbell overflow buffer, without processing the subset of the plurality of doorbells before discarding; and

processing, from the doorbell overflow buffer on the host, the subset of the plurality of doorbells, after discarding the subset of the plurality of doorbells from the HCA, as resources of the HCA become available.

2. The method of claim 1, wherein the plurality of commands are written to a send queue on the host.

3. The method of claim 1, further comprising:

processing a remaining set of the plurality of doorbells after discarding the subset of the plurality of doorbells, wherein processing the remaining set comprises for each doorbell in the remaining set;

obtaining, from the doorbell, an identifier of a location in the send queue storing the command,

locking an available collect buffer on the HCA to obtain a locked collect buffer,

writing a command from the location to the locked collect buffer,

processing, from the locked collect buffer, the command to generate a packet,

issuing the packet to a port to send on a network,

writing a completion to host memory for the packet, and

releasing the locked collect buffer after writing the completion.

4. The method of claim 3, wherein the available collect buffer is dedicated to the send queue scheduler.

5. The method of claim 1, wherein detecting the potential deadlock comprises:

identifying a number of available collect buffers and a number of available entries in the send queue scheduler buffer; and

detecting the potential deadlock when the number of available collect buffers is less than a first threshold and the number of available entries is less than a second threshold.

6. The method of claim 1, wherein each doorbell of the plurality of doorbells comprises a queue pair identifier and a sequence number.

7. A host channel adapter (HCA) for processing commands, comprising:

a send queue scheduler buffer configured to store at least a portion of a plurality of doorbells corresponding to a plurality of commands; and

a send queue scheduler, operatively connected to the send queue scheduler buffer, and configured to;

receive, for the plurality of commands, the plurality of doorbells for writing to the send queue scheduler buffer;

detect a potential deadlock of the send queue scheduler from processing the portion of the plurality of doorbells;

write, based on detecting the potential deadlock, a subset of the plurality of doorbells to a doorbell overflow buffer on a host operatively connected to the HCA;

discard, from the HCA, the subset of the plurality of doorbells, after writing the subset of the plurality of doorbells to the doorbell overflow buffer, without processing the subset of the plurality of doorbells before discarding;

issue a read request to the host to obtain a doorbell of the subset of the plurality of doorbells, from the doorbell overflow buffer on the host, as resources of the HCA become available; and

process the doorbell, after discarding the subset of the plurality of doorbells from the HCA.

8. The HCA of claim 7, wherein the send queue scheduler is further configured to:

process a remaining set of the plurality of doorbells after discarding the subset of the plurality of doorbells, wherein processing the remaining set comprises for each doorbell in the remaining set;

obtaining, from the doorbell, an identifier of a location in the send queue having the command,

locking an available collect buffer on the HCA to obtain a locked collected buffer,

writing a command from the location to the locked collect buffer,

processing, from the locked collect buffer, the command to generate a packet,

issuing the packet to a port to send on a network,

writing a completion to host memory for the packet, and

releasing the locked collect buffer after writing the completion.

9. The HCA of claim 8, wherein the available collect buffer is dedicated to the send queue scheduler.

10. The HCA of claim 7, wherein detecting the potential deadlock comprises:

identifying a number of available collect buffers and an number of available entries in the send queue scheduler buffer; and

detecting the potential deadlock when the number of available collect buffers is less than a first threshold and the number of available entries is less than a second threshold.

11. The HCA of claim 7, wherein each doorbell of the plurality of doorbells comprises a queue pair identifier and a sequence number.

12. A system for processing commands, comprising:

a host comprising a doorbell overflow buffer; and

a host channel adapter (HCA) comprising;

a send queue scheduler buffer configured to store at least a portion of a plurality of doorbells corresponding to a plurality of commands; and

a send queue scheduler, connected to the send queue scheduler buffer, and configured to;

receive, for the plurality of commands, the plurality of doorbells for writing to the send queue scheduler buffer;

detect a potential deadlock of the send queue scheduler from processing the portion of the plurality of doorbells;

write a subset of the plurality of doorbells to the doorbell overflow buffer on the host based on detecting the potential deadlock;

discard the subset of the plurality of doorbells, after writing the subset of the plurality of doorbells to the doorbell overflow buffer, without processing the subset of the plurality of doorbells before discarding;

issue a read request to the host to obtain a doorbell of the subset of the plurality of doorbells, from the doorbell overflow buffer on the host, as resources of the HCA become available; and

process the doorbell, after discarding the subset of the plurality of doorbells from the HCA.

13. The system of claim 12, wherein the send queue scheduler is further configured to:

process a remaining set of the plurality of doorbells after discarding the subset of the plurality of doorbells, wherein processing the remaining set comprises for each doorbell in the remaining set;

obtaining, from the doorbell, an identifier of a location in the send queue having the command,

locking an available collect buffer on the HCA to obtain a locked collected buffer,

writing a command from the location to the locked collect buffer,

processing, from the locked collect buffer, the command to generate a packet,

issuing the packet to a port to send on a network,

writing a completion to host memory for the packet, and

releasing the locked collect buffer after writing the completion.

14. The system of claim 13, wherein the available collect buffer is dedicated to the send queue scheduler.

15. The system of claim 12, wherein detecting the potential deadlock comprises:

identifying a number of available collect buffers and an number of available entries in the send queue scheduler buffer; and

detecting the potential deadlock when the number of available collect buffers is less than a first threshold and the number of available entries is less than a second threshold.

16. The system of claim 12, wherein each doorbell of the plurality of doorbells comprises a queue pair identifier and a sequence number.

17. The system of claim 12, wherein the host further comprises a send queue configured to store the plurality of commands.