Ordered kernel queue for multipathing events

• US 8,904,063 B1
• Filed: 04/29/2013
• Issued: 12/02/2014
• Est. Priority Date: 12/30/2011
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Sandstrom, Harold M., Tao, Tao, Trivedi, Hitesh, Pellowski, Robert J.
• Primary Examiner(s): Vy, Hung T
• Application Number:
  US13/872,660
• Time in Patent Office: 582 Days
• Field of Search: 714/4.1, 713/164, 709/220, 709/223, 709/224, 707/752, 707/822, 711113-114, 710/36, 710/58, 710/38, 710/39, 710/40, 710/62
• US Class Current: 710/39
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

An improved technique for handling events in a multipathing driver employs an event queue and a queue manager that run in the kernel of a computing system. The queue manager receives events raised by the multipathing driver, as well as events raised by software constructs, such as application programs. Records of events are added to the event queue in the order the queue manager receives them. Event records may be consumed chronologically by external software. Preferably, the event queue is sufficiently large to store all events arising out of most predicted fault scenarios. Also, the queue manager is sophisticated and can perform certain diagnostic and analysis tasks without the aid of external software.

Claims

1. A method of managing events pertaining to input/output (I/O) activities in a computing system coupled to a storage array through multiple paths, the method comprising:

storing multiple first events within a multipathing driver running within a kernel of the computing system, the first events originating from within the multipathing driver;

storing multiple second events within the multipathing driver, the second events originating from a set of software constructs running outside the kernel of the computing system; and

accessing at least some of the first and second events from the multipathing driver by at least one of the set of software constructs running outside the kernel,

wherein the multipathing driver includes an event queue for storing the first events and the second events within the kernel.

2. The method as in claim 1, wherein the method further comprises the multipathing driver:

performing a diagnostic operation based on a set of event records stored in the event queue to produce a diagnostic result;

raising an event, the raised event including a message that summarizes the diagnostic result; and

storing a record of the raised event in the event queue, the record including the message that summarizes the diagnostic result pertaining to the set of event records.

3. The method as in claim 2, wherein storing the record of the raised event in the event queue includes storing the record of the raised event adjacent to the set of event records on which the diagnostic operation was performed.

4. The method as in claim 3, further comprising:

receiving, by a software interface between the set of software constructs and the multipathing driver, an instruction from one of the set of software constructs to wait for an event;

in response to the instruction, submitting a request from the software interface to the multipathing driver for a record of a next event to occur, the request including a timeout interval specifying an amount of time after which the multipathing driver will cease to wait for the next event;

after the next event occurs and is stored on the event queue within the timeout interval, returning the record of the event from the multipathing driver to the software interface; and

returning the record of the event from the software interface to the software construct that provided the instruction.

5. The method as in claim 4, further comprising:

the software interface receiving an instruction from one of the set of software constructs to run a query to obtain a set of records from the event queue, the query specifying a set of criteria;

submitting the query from the software interface to the multipathing driver;

executing the query by the multipathing driver, including applying the set of criteria to the records stored in the event queue;

to identify a set of matching records from the event queue that satisfy the set of criteria;

returning the set of matching records from the multipathing driver to the software interface; and

returning the set of matching records from the software interface to the software construct that provided the instruction to run the query.

6. The method as in claim 5, further comprising, prior to executing the query, applying additional query criteria particular to a client associated with the software construct, the additional query criteria stored in the multipathing driver in kernel memory.

7. The method as in claim 5, further comprising:

the software interface receiving an instruction from one of the set of software constructs to push an event onto the event queue of the multipathing driver, the event being defined by a set of event properties;

conveying the set of event properties from the software interface to the multipathing driver;

creating a new event record that includes the set of event properties; and

storing the new event record in the event queue in chronological order with the first events and the second events received by the queue manager.

8. The method as in claim 1, wherein the method further comprises:

receiving, by a software interface between the set of software constructs and the multipathing driver, an instruction from one of the set of software constructs to wait for an event;

in response to the instruction, submitting a request from the software interface to the multipathing driver for a record of a next event to occur;

after the next event occurs and is stored on the event queue, returning the record of the event from the multipathing driver to the software interface; and

returning the record of the event from the software interface to the software construct that provided the instruction.

9. The method as in claim 1, wherein the method further comprises:

receiving, by a software interface between the set of software constructs and the multipathing driver, an instruction from one of the set of software constructs to run a query to obtain a set of records from the event queue, the query specifying a set of criteria;

executing the query by the multipathing driver to identify a set of matching records from the event queue that satisfy the set of criteria;

returning the set of matching records to the software construct that provided the instruction.

10. The method as in claim 9, further comprising, prior to executing the query, applying additional query criteria particular to a client associated with the software construct, the additional query criteria stored in the multipathing driver in kernel memory.

11. The method as in claim 1, and wherein the method further comprises:

receiving, by a software interface between the set of software constructs and the multipathing driver, an instruction from one of the set of software constructs to push an event onto the event queue of the multipathing driver, wherein the event is defined by a set of event properties;

conveying the set of event properties from the software interface to the multipathing driver;

creating a new event record that includes the set of event properties; and

storing the new event record in the event queue in chronological order with the first events and the second events received by the queue manager.

12. The method as in claim 11, further comprising:

initiating, from one of the set of software constructs, a data storage activity that causes a set of events to be generated by the multipathing driver and stored in the event queue; and

pushing an event onto the event queue, the event including a description of the data storage activity that caused the set of events to be generated.

13. A method of managing events pertaining to input/output (I/O) activities in a computing system coupled to a storage array through multiple paths, the method comprising:

storing multiple first events within a multipathing driver running within a kernel of the computing system, the first events originating from within the multipathing driver;

storing multiple second events within the multipathing driver, the second events originating from a set of software constructs running outside the kernel of the computing system; and

accessing at least some of the first and second events from the multipathing driver by at least one of the set of software constructs running outside the kernel,

wherein one of the set of software constructs running on the computing system is associated with a client ID, and the method further comprises;

storing the client ID associated with the software construct and a filter criterion specific to the client ID in the multipathing driver within the kernel of the operating system;

subsequently receiving a request for event information from the software construct;

obtaining the client ID of the software construct;

obtaining the requested event information, including applying the filter criterion specific to the client ID to event records; and

providing a filtered list of event records to the software construct.

14. A host computing system, comprising:

a set of processors;

memory, coupled to the set of processors to form a specialized circuit, the memory including an operating system kernel, the operating system kernel including an input/output (I/O) driver,

wherein the specialized circuit is constructed and arranged to—

store multiple first events within a multipathing driver running within a kernel of the computing system, the first events originating from within the multipathing driver;

store multiple second events within the multipathing driver, the second events originating from a set of software constructs running outside the kernel of the computing system; and

access at least some of the first and second events from the multipathing driver by at least one of the set of software constructs running outside the kernel,

wherein the multipathing driver includes an event queue for storing the first events and the second events within the kernel.

15. The host computing system as in claim 14, wherein the specialized circuit is further constructed and arranged to:

perform a diagnostic operation based on a set of event records stored in the event queue to produce a diagnostic result;

raise an event, the raised event including a message that summarizes the diagnostic result; and

store a record of the raised event in the event queue, the record including the message that summarizes the diagnostic result pertaining to the set of event records.

16. The host computing device as in claim 14, wherein the specialized circuit is further constructed and arranged to:

receive, by a software interface between the set of software constructs and the multipathing driver, an instruction from one of the set of software constructs to wait for an event;

in response to the instruction, submit a request from the software interface to the multipathing driver for a record of a next event to occur; and

after the next event occurs and is stored on the event queue, return the record of the event to the software construct that provided the instruction.

17. The host computing device as in claim 14, wherein the specialized circuit is further constructed and arranged to:

receive, by a software interface between the set of software constructs and the multipathing driver, an instruction from one of the set of software constructs to run a query to obtain a set of records from the event queue, the query specifying a set of criteria;

execute the query by the multipathing driver to identify a set of matching records from the event queue that satisfy the set of criteria; and

return the set of matching records to the software construct that provided the instruction.

18. The host computing device as in claim 14, wherein the method further comprises, and wherein the specialized circuit is further constructed and arranged to:

receive, by a software interface between the set of software constructs and the multipathing driver, an instruction from one of the set of software constructs to push an event onto the event queue of the multipathing driver, wherein the event is defined by a set of event properties;

convey the set of event properties from the software interface to the multipathing driver;

create a new event record that includes the set of event properties; and

store the new event record in the event queue in chronological order with the first events and the second events received by the queue manager.

19. A non-transitory computer readable media including instructions which, when executed by a processor, cause the processor to perform a method of managing events pertaining to input/output (I/O) activities in a computing system coupled to a storage array through multiple paths, the method comprising:

storing multiple first events within a multipathing driver running within a kernel of the computing system, the first events originating from within the multipathing driver;

storing multiple second events within the multipathing driver, the second events originating from a set of software constructs running outside the kernel of the computing system;

accessing at least some of the first and second events from the multipathing driver by at least one of the set of software constructs running outside the kernel;

initiating, from one of the set of software constructs, a data storage activity that causes a set of events to be generated by the multipathing driver and stored in an event queue; and

pushing an event onto the event queue, the event including a description of the data storage activity that caused the set of events to be generated.