Method and system for infiniband host channel adaptor quality of service

• US 8,484,392 B2
• Filed: 05/31/2011
• Issued: 07/09/2013
• Est. Priority Date: 05/31/2011
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Manula, Brian Edward, Sandven, Magne Vigulf, Torudbakken, Ola, Bugge, Haakon Ording
• Primary Examiner(s): SHIN, CHRISTOPHER B
• Application Number:
  US13/149,449
• Publication Number:
  US 20120311597A1
• Time in Patent Office: 770 Days
• Field of Search: 710 52- 57, 710/22, 710/308, 710/310, 718/100, 709/213
• US Class Current: 710/56
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455

Abstract

A method for allocating resources of a host channel adapter includes the host channel adapter identifying an underlying function referenced in the first resource allocation request received from a virtual machine manager, determining that the first resource allocation request specifies a number of physical collect buffers (PCBs) allocated to the underlying function, allocating the number of PCBs to the underlying function, determining that the first resource allocation request specifies a number of virtual collect buffers (VCBs) allocated to the underlying function, and allocating the number of VCBs to the underlying function. The host channel adapter further receives command data for a command from the single virtual machine, determines that the underlying function has in use at least the number of PCBs when the command data is received, and drops the command data in the first command based on the underlying function having in use at least the number of PCBs.

Claims

1. A method for allocating resources of a host channel adapter, comprising:

receiving, by the host channel adapter from a virtual machine manager on a host, a first resource allocation request;

identifying, by the host channel adapter, an underlying function referenced in the first resource allocation request, wherein the underlying function corresponds to a single virtual machine located on the host;

determining, by the host channel adapter, that the first resource allocation request specifies a number of physical collect buffers (PCBs) allocated to the underlying function;

allocating, by the host channel adapter, the number of PCBs to the underlying function by storing the number in a control and status register (CSR);

determining, by the host channel adapter, that the first resource allocation request specifies a number of virtual collect buffers (VCBs) allocated to the underlying function;

allocating, by the host channel adapter, the number of VCBs to the underlying function by storing a base address register (BAR) in the CSR;

receiving, by the host channel adapter, first command data for a first command from the single virtual machine for sending packets on a network, wherein the first command data specifies an address of a VCB of the number of VCBs allocated to the single virtual machine;

determining, using the CSR, that the underlying function has in use at least the number of PCBs when the first command data is received; and

dropping the first command data in the first command based on the underlying function having in use at least the number of PCBs.

2. The method of claim 1, wherein, prior to dropping the first command data, the first command data is stored in a send queue located on the host, and wherein the method further comprises:

iterating, by the host channel adapter, through the send queue;

obtaining, by the host channel adapter, the first command data from the send queue after dropping the first command data;

storing, by the host channel adapter, the first command data in a dedicated PCB;

executing the first command in a pipeline in order to obtain packet data; and

sending, by the host channel adapter, the packet data on the network.

3. The method of claim 1, further comprising:

receiving, by the host channel adapter, second command data for a second command from the single virtual machine for sending packets on the network;

determining, using the CSR, that the underlying function has an available PCB when the second command data is received; and

storing the second command data in the available PCB.

4. The method of claim 3, further comprising:

receiving, by the host channel adapter from the virtual machine manager, a second resource allocation request;

identifying, by the host channel adapter, the underlying function referenced in the second resource allocation request;

determining, by the host channel adapter, that the second resource allocation request specifies a number of transport specific unit virtual lanes (TVLs) to allocate to the underlying function; and

updating a mapping table based on the number of TVLs to allocate number of TVLs to the underlying function.

5. The method of claim 4, further comprising:

referencing the available PCB in a virtual kick list corresponding to a transport specific unit virtual lane (TVL) of the number of TVLs when the entirety of the second command is received;

obtaining the second command from the virtual kick list;

executing the second command in a pipeline in order to obtain packet data using the second command data; and

sending, by the host channel adapter, the packet data on the network.

6. The method of claim 5, further comprising:

receiving, by the host channel adapter from the virtual machine manager, a third resource allocation request;

identifying, by the host channel adapter, the TVL referenced in the third resource allocation request;

determining, by the host channel adapter, that the third resource allocation request specifies a number of execution credits to allocate to the TVL; and

storing, by the host channel adapter, the number of execution credits,

wherein executing the second command comprises;

when an execution credit of the number of execution credits is available;

setting the execution credit;

obtaining the packet data using the second command data; and

releasing the execution credit after the packet data is obtained.

7. The method of claim 6, wherein the packet data is obtained directly from the command data.

8. The method of claim 6, wherein the packet data is obtained by performing direct memory access (DMA) from memory on the host using a location specified by the command data.

9. The method of claim 6, further comprising:

receiving, by the host channel adapter from the virtual machine manager, a fourth resource allocation request;

identifying, by the host channel adapter, the TVL referenced in the fourth resource allocation request;

determining, by the host channel adapter, that the fourth resource allocation request specifies a number of completion credits to allocate to the TVL; and

storing, by the host channel adapter, the number of completion credits,

wherein sending the second command comprises;

when a completion credit of the number of completion credits is available;

setting the completion credit;

sending the packet data on the network; and

releasing the completion credit after the packet data is sent.

10. A host channel adapter comprising:

a plurality of physical collect buffers (PCBs);

a first control and status register (CSR) corresponding to a first underlying function, wherein the first underlying function corresponds to a first virtual machine;

a second CSR corresponding to a second underlying function, wherein the second underlying function corresponding to a second virtual machine;

a service processor resource manager operatively connected to the first CSR and the second CSR and configured to;

receive, from a virtual machine manager on a host, a first resource allocation request;

identify the first underlying function referenced in the first resource allocation request;

determine that the first resource allocation request specifies a first number of PCBs allocated to the first underlying function;

allocate the first number of PCBs to the first underlying function by storing the first number in the first CSR;

determine that the first resource allocation request specifies a first number of virtual collect buffers (VCBs) allocated to the first underlying function;

allocate the first number of VCBs to the first underlying function by storing a base address register (BAR) in the CSR;

receive, from the virtual machine manager on a host, a second resource allocation request;

identify the second underlying function referenced in the second resource allocation request;

determine that the second resource allocation request specifies a second number of PCBs allocated to the second underlying function; and

allocate the second number of PCBs to the second underlying function by storing the second number in the second CSR; and

a PCB allocator configured to;

receive first command data for a first command from the first virtual machine for sending packets on an network, wherein the first command data specifies an address of a VCB of the first number of VCBs;

determine, using the CSR, that the underlying function has in use at least the number of PCBs when the first command data is received; and

drop the first command data in the first command based on the underlying function having in use at least the number of PCBs.

11. The host channel adapter of claim 10, further comprising:

a dedicated PCB; and

a send queue scheduler configured to transfer command data from a send queue located on a host to the dedicated PCB.

12. The host channel adapter of claim 10, further comprising:

a pipeline configured to execute commands.

13. The host channel adapter of claim 10, further comprising:

a virtual kick list for storing identifiers of PCBs comprising complete commands.

14. The host channel adapter of claim 10, further comprising:

a pipeline connected to the virtual kick list and configured to;

execute commands in order to obtain packet data; and

a port configured to;

send the packet data on the network.

15. The host channel adapter of claim 14, further comprising:

a mapping table comprising a mapping between underlying functions and transport specific unit virtual lanes (TVLs), wherein each underlying function is allocated a number of TVLs.

16. The host channel adapter of claim 15, further comprising:

a execution and completion credit CSR for each TVL, wherein the execution and completion credit CSR is configured to store a number of execution credits and a number of completion credits defining a number of commands processed for a corresponding TVL at a time.

17. A system comprising:

a host comprising;

a virtual machine manager configured to send a first resource allocation request and a second resource allocation request;

a first virtual machine; and

a second virtual machine; and

a host channel adapter operatively connected to the host and comprising;

a plurality of physical collect buffers (PCBs);

a first control and status register (CSR) corresponding to a first underlying function, wherein the first underlying function corresponds to the first virtual machine;

a second CSR corresponding to a second underlying function, wherein the second underlying function corresponding to the second virtual machine;

a service processor resource manager configured to;

receive the first resource allocation request;

identify the first underlying function referenced in the first resource allocation request;

determine that the first resource allocation request specifies a first number of PCBs allocated to the first underlying function;

allocate the first number of PCBs to the first underlying function by storing the first number in the first CSR;

determine that the first resource allocation request specifies a first number of virtual collect buffers (VCBs) allocated to the first underlying function;

allocate the first number of VCBs to the first underlying function by storing a base address register (BAR) in the CSR;

receive the second resource allocation request;

identify the second underlying function referenced in the second resource allocation request;

determine that the second resource allocation request specifies a second number of PCBs allocated to the second underlying function; and

allocate the second number of PCBs to the second underlying function by storing the second number in the second CSR; and

a PCB allocator configured to;

receive first command data for a first command from the first virtual machine for sending packets on a network, wherein the first command data specifies an address of a VCB of the first number of VCBs;

determine, using the CSR, that the underlying function has in use at least the number of PCBs when the first command data is received; and

drop the first command data in the first command based on the underlying function having in use at least the number of PCBs.

18. The system of claim 17, wherein the host further comprises:

a send queue configured to store the first command data; and

a queuing structure configured to store an address of the send queue after the first command data is dropped.

19. The system of claim 18, wherein the host channel adapter further comprises:

a send queue scheduler configured to;

iterate through the queuing structure;

identify, after the first command data is dropped, the send queue from an entry in the queuing structure while iterating through the queuing structure; and

initiate transmission of the first command data from the send queue to the host channel adapter.

20. The system of claim 19, wherein the plurality of PCBs comprises:

a dedicated PCB for the send queue scheduler.