Method and system for Infiniband host channel adapter multicast packet replication mechanism

• US 8,937,949 B2
• Filed: 12/20/2012
• Issued: 01/20/2015
• Est. Priority Date: 12/20/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Manula, Brian Edward, Tvete, Knut Hallvard, Schanke, Morten
• Primary Examiner(s): HO, DUC CHI
• Application Number:
  US13/721,704
• Publication Number:
  US 20140177629A1
• Time in Patent Office: 761 Days
• Field of Search: None
• US Class Current: 370/390
• CPC Class Codes: H04L 49/201 Multicast operation; Broadcast operation
• CPC Class Codes: H04L 49/358 Infiniband Switches

Abstract

A method for multicast replication by a host channel adapter (HCA) involving receiving a multicast packet, by a receive pipeline for processing packets of the HCA, storing, in a payload RAM within the HCA, a multicast packet payload corresponding to a data portion of the multicast packet, identifying, from a multicast header of the multicast packet, a plurality of destination underlying functions and a plurality of corresponding destination QPs to which the multicast packet is directed, wherein each destination underlying function of corresponds to a virtual machine located on a host, identifying, from the multicast header, information to be replicated for each multicast packet destination, injecting, by the HCA, a number of multicast packet descriptors corresponding to a number of the corresponding destination QPs into the receive pipeline of the HCA, and copying, from the payload RAM, the multicast packet payload to each of the corresponding destination QPs.

Claims

1. A method for multicast replication by a host channel adapter (HCA), comprising:

receiving, by the HCA, a multicast packet, wherein the HCA comprises responder processing logic for processing packets through a receive pipeline in the HCA;

storing, in a payload random access memory (RAM) within the HCA, a multicast packet payload corresponding to a data portion of the multicast packet;

identifying, from a multicast header of the multicast packet, a plurality of destination underlying functions and a plurality of corresponding destination QPs within a multicast group to which the multicast packet is directed, wherein each destination underlying function of the plurality of destination underlying functions corresponds to a virtual machine located on a host operatively connected to the HCA;

identifying, from the multicast header, information to be replicated for each multicast packet destination;

injecting, by the HCA, a number of multicast packet descriptors corresponding to a number of the corresponding destination QPs into the receive pipeline of the HCA, wherein each multicast packet descriptor comprises the information to be replicated, wherein the multicast packet descriptors are processed in the receive pipeline of the HCA; and

copying, from the payload RAM, the multicast packet payload to each of the corresponding destination QPs in host memory.

2. The method of claim 1, wherein identifying the plurality of destination underlying functions and the plurality of corresponding destination QPs further comprises:

extracting, from the multicast header, a multicast local destination identifier and a multicast global destination identifier; and

storing the multicast local destination identifier and the multicast global destination identifier in a multicast information data structure.

3. The method of claim 1, further comprising:

incrementing a payload reference counter when the multicast payload is stored in the payload RAM;

incrementing the payload reference counter for each copy of the multicast packet descriptor injected into the receive pipeline; and

decrementing the payload reference counter each time the multicast packet header is processed through the receive pipeline and when a dummy copy is processed through the receive pipeline,

wherein the payload of the multicast packet is flushed from the payload RAM when the payload reference counter is decremented to zero.

4. The method of claim 1, further comprising:

incrementing a multicast threshold counter for each MC LID/MC GID pair that is stored;

periodically comparing the multicast threshold counter value to a predetermined threshold; and

preventing new incoming multicast processing in the HCA when the multicast threshold counter value is greater than the predetermined threshold.

5. The method of claim 1, further comprising:

storing multicast packet header details in a data structure in the receive pipeline,

wherein the multicast packet header details are flushed when a command is transmitted indicating that a last copy of the multicast packet header has been injected into the receive pipeline.

6. The method of claim 5, further comprising:

storing, in a replication request data structure operatively connected to the receive pipeline, the information to be replicated for each multicast packet destination.

7. The method of claim 6, further comprising:

tracking a number of entries in the replication request data structure using a replication credit counter; and

decrementing the replication credit counter when the command is transmitted to the replication request data structure.

8. The method of claim 1, wherein the multicast packet is received via the Infiniband®

network, and wherein the HCA is an Infiniband®

HCA.

9. The method of claim 1, wherein each underlying function exposes a virtualized HCA associated with a corresponding virtual machine executing on the host, to the HCA.

10. The method of claim 1, further comprising:

injecting, by the HCA, a dummy copy of the multicast packet descriptor into the receive pipeline of the HCA, wherein when the dummy copy of the multicast packet descriptor completes processing in the receive pipeline, this indicates that all copies of the multicast packet have been replicated.

11. A host channel adapter (HCA) comprising:

a plurality of underlying functions, wherein each of the plurality of underlying functions corresponds to a virtual machine in a host operatively connected to the HCA;

a receive packet processing pipeline, the receive packet processing pipeline being configured to;

receive a multicast packet comprising a multicast header and a payload corresponding to a data portion of the multicast packet;

store, in a payload random access memory (RAM) within the HCA, the multicast packet payload;

a multicast inspection engine configured to;

identify, from the multicast header of the multicast packet, a subset of the plurality of destination underlying functions and a plurality of corresponding destination QPs within a multicast group to which the multicast packet is directed; and

identify, from the multicast header, information to be replicated for each multicast packet destination,

wherein the receive packet processing pipeline is further configured to;

inject a number of multicast packet descriptors corresponding to a number of the plurality of corresponding destination QPs identified by the multicast inspection engine into the receive packet processing pipeline, wherein each multicast packet descriptor comprises the information to be replicated; and

copy, from the payload RAM, the multicast packet payload to each of the corresponding destination QPs in host memory.

12. The HCA of claim 11, further comprising:

a multicast information data structure configured to store a multicast local destination identifier (MC LID) and a multicast global destination identifier (MC GID) from the multicast packet header; and

a replication request data structure configured to store information to be replicated for each multicast packet destination, wherein the information to be replicated is used in each multicast packet descriptor injected into the receive pipeline,

wherein the multicast information data structure and the replication request data structure are communication points between the receive packet processing pipeline and the multicast inspection engine.

13. The HCA of claim 12, further comprising:

a replication credit counter for tracking a number of entries in the replication request data structure,

wherein the replication credit counter is decremented when the command is transmitted to the replication request data structure.

14. The HCA of claim 12, wherein a multicast threshold counter is incremented each time the multicast information data structure stores a MC LID/MC GID pair.

15. The HCA of claim 14, wherein the multicast threshold counter is compared to a predetermined threshold and when the multicast threshold counter is greater than a predetermined threshold, new incoming multicast packets are not processed.

16. The HCA of claim 11, wherein the multicast packet is received via the Infiniband®

network, and wherein the HCA is an Infiniband®

HCA.

17. The HCA of claim 11, wherein each underlying function exposes a virtualized HCA associated with a corresponding virtual machine executing on the host, to the HCA.

18. The HCA of claim 11, further comprising:

a payload reference counter,

wherein;

the payload reference counter is incremented when the multicast payload is stored in the payload RAM,

the payload reference counter is incremented for each copy of the multicast packet descriptor injected into the receive pipeline,

the payload reference counter is decremented each time the multicast packet header is processed through the receive pipeline and when a dummy copy is processed through the receive pipeline, and

the payload of the multicast packet is flushed from the payload RAM when the payload reference counter is decremented to zero.

19. A system comprising:

a host comprising;

a first virtual machine comprising a first destination queue pair (QP); and

a second virtual machine comprising a second destination QP; and

a host channel adapter (HCA) operatively connected to the host and comprising;

a first underlying function corresponding to the first virtual machine;

a second underlying function corresponding to the second virtual machine;

a receive packet processing pipeline, the receive packet processing pipeline configured to;

receive a multicast packet comprising a multicast header and a payload corresponding to a data portion of the multicast packet;

store, in a payload random access memory (RAM) within the HCA, the multicast packet payload;

a multicast inspection engine configured to;

identify, from the multicast header of the multicast packet, the first and second underlying functions and corresponding first and second destination QPs within a multicast group to which the multicast packet is directed; and

identify, from the multicast header, information to be replicated for each multicast packet destination;

wherein the receive packet processing pipeline is further configured to;

inject a multicast packet descriptor for each of the corresponding first and second destination QPs identified by the multicast inspection engine into the receive packet processing pipeline, wherein each multicast packet descriptor comprises the information to be replicated; and

copy, from the payload RAM, the multicast packet payload to each of the first and second corresponding destination QPs in host memory.

20. The system of claim 19, wherein each of the first and second underlying functions exposes a virtualized HCA associated with the corresponding first and second virtual machines executing on the host, respectively, to the HCA.