Method and system for queue descriptor cache management for a host channel adapter

• US 9,256,555 B2
• Filed: 12/20/2012
• Issued: 02/09/2016
• Est. Priority Date: 12/20/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Manula, Brian Edward, Bugge, Haakon Ording, Gotwald, Frederick Luther
• Primary Examiner(s): Thammavong, Prasith
• Application Number:
  US13/721,699
• Publication Number:
  US 20140181409A1
• Time in Patent Office: 1,146 Days
• Field of Search: 711/113
• US Class Current: 1/1
• CPC Class Codes: G06F 12/123 with age lists, e.g. queue, most recently used [MRU] list or least recently used [LRU] list
• CPC Class Codes: G06F 13/12 using hardware independent of the central processor, e.g. channel or peripheral processor
• CPC Class Codes: G06F 13/1642 with request queuing
• CPC Class Codes: H04L 49/90 Buffering arrangements

Abstract

A method for managing a queue descriptor cache of a host channel adaptor (HCA) includes obtaining a queue descriptor from memory. The queue descriptor includes data describing a queue and the memory is located in a host system. The method further includes storing a copy of the queue descriptor in the queue descriptor cache of the HCA. The HCA accesses the copy of the queue descriptor to obtain the plurality of data, accesses the queue using the data, and updates the data to reflect the access to the queue. The method further includes calculating, using the data, a value corresponding to utilization of the queue, comparing the value against a threshold, fetching, if the value exceeds the threshold, a new copy of the queue descriptor from memory, and replacing the copy of the queue descriptor in the queue descriptor cache with the new copy obtained from the memory.

Claims

1. A method for managing a queue descriptor cache of a host channel adaptor (HCA), comprising:

obtaining a queue descriptor from memory, wherein the queue descriptor comprises a plurality of data describing a queue and wherein the memory is located in a host system;

storing a copy of the queue descriptor in the queue descriptor cache of the HCA;

accessing, by the HCA, the copy of the queue descriptor to obtain the plurality of data;

accessing, by the HCA and using the plurality of data, the queue;

updating, by the HCA, the plurality of data to reflect the access to the queue;

calculating, using the plurality of data, a value corresponding to utilization of the queue;

comparing the value against a threshold;

fetching, if the value exceeds the threshold, a new copy of the queue descriptor from memory; and

replacing the copy of the queue descriptor in the queue descriptor cache with the new copy obtained from the memory.

2. The method of claim 1,

wherein the queue comprises a plurality of entries,

wherein the plurality of data comprises the threshold, a software pointer, and a hardware pointer,

wherein the software pointer references a queue entry most recently accessed by the host system,

wherein the hardware pointer references a queue entry most recently accessed by the HCA, and

wherein the value corresponding to utilization of the queue represents a number of entries separating the software pointer and the hardware pointer.

3. The method of claim 2, wherein the software pointer in the queue descriptor in memory is updated after the host system accesses to the queue without updating the software pointer in the copy of the queue descriptor, and wherein the new copy of the queue descriptor comprises an updated software pointer.

4. The method of claim 3,

wherein the queue descriptor comprises a first portion and a second portion,

wherein the first portion comprises the hardware pointer,

wherein the second portion comprises the software pointer, and

wherein fetching the new copy of the queue descriptor comprises obtaining only the second portion of the queue descriptor from memory.

5. The method of claim 3, wherein the threshold represents an expected number of entries in the queue accessed by the HCA in an expected time required to fetch the new copy of the queue descriptor from memory.

6. The method of claim 5, further comprising:

identifying, by the host, an expected memory access latency of the HCA;

identifying, by the host, an expected queue access rate of the HCA;

multiplying, by the host, the expected memory access latency with the expected queue access rate to obtain the threshold; and

setting, by the host, the threshold within the queue descriptor.

7. The method of claim 2,

wherein the queue is one of a plurality of queues,

wherein the plurality of queues comprises:

a completion queue, configured to store completions;

a receive queue, configured to store references to data buffers in the memory;

a ring buffer, configured to store received data packets,

wherein the HCA is configured to read from the receive queue, and write to the completion queue and the ring buffer, and

wherein the host is configured to write to the receive queue, and read from the completion queue and the ring buffer.

8. The method of claim 5,

wherein the queue descriptor comprises a plurality of potential thresholds,

wherein each of the plurality of potential thresholds is associated with at least one selected from a group consisting of a different expected memory access latency and a different expected rate of queue consumption, and

wherein at least one of the plurality of potential thresholds is selected to be used as the threshold based on an observed memory access latency and an observed queue access rate.

9. A host channel adaptor (HCA), comprising:

descriptor fetch circuitry, comprising;

a queue descriptor cache, configured to store a copy of a queue descriptor comprising a plurality of data describing a queue,

wherein the queue descriptor is located in a memory of a host operatively connected to the HCA, and

wherein the queue is located in the memory and comprises a plurality of entries; and

cache control circuitry, configured to;

obtain the copy of the queue descriptor from the memory,

store the copy of the queue descriptor in the cache,

determine that the HCA has accessed the queue,

calculate a value representing a utilization of the queue,

compare the value against a threshold, and

fetch a new copy of the queue descriptor from memory when the value exceeds the threshold; and

a plurality of additional circuitry, configured to;

access the copy of the queue descriptor to obtain the plurality of data;

access, using the plurality of data, the queue; and

update the plurality of data to reflect the access to the queue.

10. The method of claim 9,

wherein the plurality of data comprises the threshold, a software pointer, and a hardware pointer,

wherein the software pointer references a queue entry most recently accessed by the host system,

wherein the hardware pointer references a queue entry most recently accessed by the HCA, and

wherein the value corresponding to the utilization of the queue represents a number of entries separating the software pointer and the hardware pointer.

11. The HCA of claim 10, wherein the threshold represents an expected number of entries in the queue accessed by the HCA in an expected time required to fetch the new copy of the queue descriptor from memory.

12. The HCA of claim 10,

wherein the queue descriptor comprises a first portion and a second portion,

wherein the first portion comprises the hardware pointer,

wherein the second portion comprises the software pointer, and

wherein the descriptor fetch circuitry is further configured to obtain only the new copy of the second portion of the queue descriptor from memory, and store the new copy in place of the second portion of the copy of the queue descriptor in the cache.

13. The HCA of claim 12,

wherein the queue descriptor comprises a plurality of potential thresholds,

wherein each of the plurality of potential thresholds is associated with at least one selected from a group consisting of a different expected memory access latency and a different expected rate of queue consumption, and

wherein one of the potential thresholds is selected to be used as the threshold based on an observed memory access latency and an observed queue access rate.

14. The HCA of claim 9, wherein the HCA is configured to send and receive packets over a channel-based switched fabric network.

15. A system comprising:

a host, comprising;

a processor, and

a memory configured to store;

a queue descriptor comprising a plurality of data describing a queue, comprising;

a hardware pointer configured to reference an entry in the queue most recently access by the HCA,

a software pointer configured to reference an entry in the queue most recently accessed by a host, and

a threshold; and

a queue comprising a plurality of entries; and

a host channel adapter (HCA) operatively connected to the host and comprising;

a cache configured to store a copy of the queue descriptor, and

cache control circuitry, configured to;

obtain the copy of the queue descriptor from the memory;

store the copy of the queue descriptor in the cache;

determine that the HCA has accessed the queue;

calculate a value representing the number of entries separating the software pointer and the hardware pointer;

compare the value against the threshold; and

fetch a new copy of the queue descriptor from the memory when the value exceeds the threshold; and

a plurality of additional circuitry, configured to;

access the copy of the queue descriptor to obtain the plurality of data;

access, using the plurality of data, the queue; and

update the plurality of data to reflect the access to the queue.

16. The system of claim 15, wherein the host is configured to update the software pointer of the queue descriptor located in memory after accessing the queue, and without the copy of the queue descriptor in the cache being changed.

17. The system of claim 15, wherein the host is configured to:

identify an expected memory access latency of the HCA;

identify an expected queue access rate of the HCA;

multiply the expected memory access latency with the expected queue access rate to obtain the threshold; and

set the threshold within the queue descriptor.

18. The system of claim 15,

wherein the queue is one of a plurality of queues stored in the memory,

wherein the plurality of queues comprises:

a completion queue, configured to store completions;

a receive queue, configured to store references to data buffers in the memory;

a ring buffer, configured to store received data packets,

wherein the HCA is configured to read from the receive queue, and write to the completion queue and the ring buffer, and

wherein the host is configured to write to the receive queue, and read from the completion queue and the ring buffer.

19. The system of claim 18, wherein the HCA comprises a plurality of queue descriptor caches comprising:

a completion queue descriptor cache stores a queue descriptor describing the completion queue;

a receive queue descriptor cache, configured to store a queue descriptor describing the receive queue; and

a ring buffer descriptor cache, configured to store a queue descriptor describing the ring buffer.

20. The system of claim 15, wherein the host system is configured to use the HCA to communicate over a channel-based switched fabric network.