Lockless multithreaded completion queue access

• US 10,031,786 B2
• Filed: 01/13/2016
• Issued: 07/24/2018
• Est. Priority Date: 01/13/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chen, Lior, Gavrilov, Constantine, Snast, Alexander
• Primary Examiner(s): Zhao, Bing
• Application Number:
  US14/994,987
• Publication Number:
  US 20170199772A1
• Time in Patent Office: 923 Days
• Field of Search: None
• CPC Class Codes: G06F 9/4812 by interrupt, e.g. masked
• CPC Class Codes: G06F 9/524 Deadlock detection or avoidance
• CPC Class Codes: G06F 9/526 Mutual exclusion algorithms

Abstract

Methods, computing systems and computer program products implement embodiments of the present invention that include identifying a first number of processors in a computer, and identifying a second number of interrupt request (IRQ) lines on a hardware acceleration device in the computer and coupled to the processors, the second number greater than or equal to the first number. Each of the IRQ lines is associated with one of the processors, and upon selecting a given IRQ line for an application thread, a given processor associated with the given IRQ line is identified. Execution of the application thread is initiated on the given processor, and using the given IRQ line, a completion queue is configured for the application thread. If the thread is executing on a different processor than the one managing the completion queue, then the management of the completion queue can be migrated to the processor executing the thread.

Claims

1. A method, comprising:

identifying a first number of processors in a computer;

identifying a second number of interrupt request (IRQ) lines on a hardware acceleration device in the computer and coupled to the processors, the second number greater than or equal to the first number;

associating, using a device driver by a first given one of the processors, each of the IRQ lines with one of the processors;

selecting, by the first given processor, a given IRQ line for an application thread;

identifying, by the first given processor, a second given one of the processors associated with the given IRQ line;

initiating, by the first given processor, execution of the application thread on the second given processor;

configuring through the device driver by the first given processor, using the given IRQ line, a completion queue for the application thread; and

prior to selecting the given IRQ line, identifying a plurality of event queues on the hardware acceleration device, and associating each of the IRQ lines with one of the event queues by the first given processor;

wherein a given one of the event queues is bound to the completion queue prior to the first given processor initiating execution of the application thread on the second given processor; and

wherein the second given processor is configured to lock the completion queue upon detecting that the given IRQ line is associated with a third given one of the processors by acquiring a spinlock for the completion queue, setting an processor owner flag to an invalid value, activating a read memory barrier, waiting for an in-use flag to clear, and releasing the spinlock.

2. The method according to claim 1, wherein selecting the given IRQ line for the application thread comprises selecting the given event queue associated with the IRQ line.

3. The method according to claim 1, and comprising performing by the second given processor:

conveying, by the application thread, an operation to the hardware acceleration device, receiving a notification indicating completion of the operation, locking the completion queue in response to the notification, retrieving completion information from the locked completion queue, and unlocking the locked completion queue.

4. The method according to claim 3, wherein locking the completion queue comprises setting an in-use flag for the second given processor, activating a write memory barrier for the second given processor, and disabling preemption for the second given processor.

5. The method according to claim 4, and comprising associating, by the second given processor, the given IRQ line with the second given processor, thereby associating the given event queue and the completion queue with the second given processor.

6. The method according to claim 1 wherein the hardware acceleration device is selected from a group consisting of a data networking device, a data compression device, a data encryption device and a non-volatile memory express device, and wherein the processor is selected from a group consisting of a physical processor and a virtual processor.

7. An apparatus, comprising:

a hardware acceleration device;

a memory; and

a first number of processors, wherein a first given one of the processors is configured;

to identify a second number of interrupt request (IRQ) lines on the hardware acceleration device, the second number greater than or equal to the first number,

to associate, using a device driver, each of the IRQ lines with one of the processors,

to select a given IRQ line for an application thread,

to identify a second given one of the processors associated with the given IRQ line,

to initiate execution of the application thread on the second given processor,

to configure through the device driver, in the memory and using the given IRQ line, a completion queue for the application thread, and

prior to selecting the given IRQ line, to identify a plurality of event queues on the hardware acceleration device, and to associate each of the IRQ lines with one of the event queues by the first given processor;

wherein a given one of the event queues is bound to the completion queue prior to the first given processor initiating execution of the application thread on the second given processor; and

wherein the second given processor is configured to lock the completion queue upon detecting that the given IRQ line is associated with a third given one of the processors by acquiring a spinlock for the completion queue, setting an processor owner flag to an invalid value, activating a read memory barrier, waiting for an in-use flag to clear, and releasing the spinlock.

8. The apparatus according to claim 7, wherein the hardware acceleration device comprises an adapter processor, and wherein the first given processor is configured to select the given IRQ line for the application thread by selecting the given event queue associated with the IRQ line.

9. The apparatus according to claim 7, wherein the second given processor executing the application thread is configured:

to convey an operation to the hardware acceleration device, to receive a notification indicating completion of the operation, to lock the completion queue in response to the notification, to retrieve completion information from the locked completion queue, and to unlock the locked completion queue.

10. The apparatus according to claim 9, wherein the second given processor is configured:

to lock the completion queue by setting an in-use flag for the second given processor, to activate a write memory barrier for the second given processor, and to disable preemption for the second given processor.

11. The apparatus according to claim 10, wherein the second given processor is configured to associate the given IRQ line with the second given processor, thereby associating the given event queue and the completion queue with the second given processor.

12. The apparatus according to claim 11 wherein the hardware acceleration device is selected from a group consisting of a data networking device, a data compression device, a data encryption device and a non-volatile memory express device, and wherein each of the first number of processors is selected from a group consisting of a physical processor and a virtual processor.

13. A computer program product, the computer program product comprising:

a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to identify a first number of processors in a computer;

computer readable program code configured to identify a second number of interrupt request (IRQ) lines on a hardware acceleration device in the computer and coupled to the processors, the second number greater than or equal to the first number;

computer readable program code configured to associate, using a device driver by a first given one of the processors, each of the IRQ lines with one of the processors;

computer readable program code configured to select, by the first given processor, a given IRQ line for an application thread;

computer readable program code configured to identify, by the first given processor, a second given one of the processors associated with the given IRQ line;

computer readable program code configured to initiate, by the first given processor, execution of the application thread on the second given processor;

computer readable program code arranged to configure through the device driver by the first given processor, using the given IRQ line, a completion queue for the application thread; and

computer readable program code configured, prior to selecting the given IRQ line, to identify a plurality of event queues on the hardware acceleration device, and to associate each of the IRQ lines with one of the event queues by the first given processor;

wherein a given one of the event queues is bound to the completion queue prior to the first given processor initiating execution of the application thread on the second given processor; and

wherein the second given processor is configured to lock the completion queue upon detecting that the given IRQ line is associated with a third given one of the processors by acquiring a spinlock for the completion queue, setting an processor owner flag to an invalid value, activating a read memory barrier, waiting for an in-use flag to clear, and releasing the spinlock.

14. The computer program product according to claim 13, wherein the computer readable program code is configured to select the given IRQ line for the application thread by selecting the given event queue associated with the IRQ line by the first given processor.

15. The computer program product according to claim 13, and comprising computer readable program code configured, by the second given processor:

to convey, by the application thread, an operation to the hardware acceleration device, to receive a notification indicating completion of the operation, to lock the completion queue in response to the notification, to retrieve completion information from the locked completion queue, and to unlock the locked completion queue.

16. The computer program product according to claim 15, wherein the computer readable program code is configured, by the second given processor:

to lock the completion queue by setting an in-use flag for the given processor, to activate a write memory barrier for the given processor, and to disable preemption for the second given processor.

17. The computer program product according to claim 16, and comprising computer readable program code configured to associate, by the second given processor, the given IRQ line with the second given processor, thereby associating the given event queue and the completion queue with the second given processor.