Utilizing Input/Output Paths For Failure Detection And Analysis

• US 20110276831A1
• Filed: 05/05/2011
• Published: 11/10/2011
• Est. Priority Date: 05/05/2010
• Status: Abandoned Application

• Current Assignee: Kaminario Limited
• Original Assignee: Kaminario Limited
• Inventors: Koren, Benny, Doron, Tal, Perelstein, Itzhak, Atzmony, Yedidia
• Application Number:
  US13/101,414
• Publication Number:
  US 20110276831A1
• Time in Patent Office: Days
• US Class Current: 714/37
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36

Abstract

Systems and methods for failure monitoring in a storage system. In some cases, a failed entity is detected based on an analysis of at least the indications obtained in return for input/output commands sent to multiple entities in the storage system. In some of these cases, it is also determined whether the failure is enduring or transient.

Claims

1. A method of failure monitoring in a storage system, comprising:

sending input/output commands to a plurality of entities in a storage system, and obtaining indications of results in return; and

if there is at least one indication of failure result detected, then analyzing at least said obtained indications of results in order to determine a reason for said detected indication of failure result.

2. The method of claim 1, wherein said input/output commands correspond to each predetermined path in said storage system.

3. The method of claim 1, wherein said input/output commands correspond to each predetermined path in said storage system for which indications of results are currently desired.

4. The method of claim 1, further comprising:

obtaining at least one indication of failure result which is not in return for said sent input/output commands but is related to at least one other input/output command, wherein said analyzed obtained indications includes said at least one obtained indication of failure result related to said at least one other input/output command.

5. The method of claim 4, wherein said sending occurs after obtaining an indication of failure result relating to another input/output command.

6. The method of claim 1, wherein said sending occurs independently of obtaining of any indication of failure result.

7. The method of claim 1, wherein an indication of a success result or alternatively of a failure result is obtained when no response is received.

8. The method of claim 1, wherein said detected indication of failure result was obtained not in return for said sent input/output commands but is related to another input/output command.

9. The method of claim 1, wherein said detected indication of failure result was obtained in return for a sent input/output command.

10. The method of claim 1, wherein said analyzing includes:

analyzing indications of results corresponding to a predefined number of rounds.

11. The method of claim 1, wherein said analyzing includes:

analyzing indications of results corresponding to a period of time which is at least equal to a grace period in which a failure would not yet be defined as enduring.

12. The method of claim 1, wherein input/output commands are sent at a higher rate than a reciprocal of shortest timeout of said system.

13. The method of claim 1, wherein said obtaining in return includes:

obtaining in return an indication of a result relating to handling of a previous input/output command of different origin instead of an indication of a result for handling one of said sent input/output commands.

14. The method of claim 1, further comprising taking action based on said reason.

15. The method of claim 1, wherein said multiple entities are on a plurality of levels in said storage system.

16. The method of claim 1, wherein said analyzing includes:

for each obtained indication of failure, adding or retaining failure notation for reporting entity and for any other entity on a predetermined path for said corresponding command;

for each obtained indication of success which is not a repeated indication, adding explicit success notation for reporting entity and for any other entity on a predetermined path for said corresponding command or implicitly noting success by omission;

determining if there are any entities noted only for failure and not for success;

if there is only one entity noted only for failure and not for success, then identifying failure of said entity as reason for said indication of failure; and

if there is more than one entity noted only for failure and not for success, then deciding that reason for said indication of failure is failure of more than one entity.

17. The method of claim 16, wherein said analyzing further includes:

determining whether or not failure of said entity or entities is transient or enduring.

18. The method of claim 16, wherein said analyzing further includes:

if there is no entity noted only for failure and not for success, then determining that said reason is inconclusive.

19. A storage system comprising:

a failure monitoring controller including;

a command generator for generating and sending input/output commands to entities in said storage system;

an indication obtainer for obtaining indications of results in return for said sent input/output commands,

a failure result indication detector for detecting if there is at least one indication of failure result;

and an analyzer for analyzing indications of results, if at least one indication of failure result has been detected, in order to determine a reason for said detected indication of failure result.

20. The system of claim 19, further comprising:

at least two entities.

21. The system of claim 20, wherein said system includes at least one predetermined path for transferring input/output commands among said at least two entities.

22. The system of claim 20, wherein an entity which receives an input/output command from said failure monitoring controller is configured to provide an indication of result to said failure monitoring controller explicitly, or implicitly by non-response.

23. The system of claim 22, wherein said entity is configured to handle said input/output command and provide an indication of result of said handling explicitly, or implicitly by non-response.

24. The system of claim 23, wherein said entity is also configured to instead provide an indication of result of a previously received and handled similar input/output command not originating from said failure monitoring controller explicitly, or implicitly by non-response.

25. The system of claim 19, wherein said controller further includes:

a timer for providing timing to said controller.

26. The system of claim 19, wherein said controller further includes:

a memory for storing indications of results.

27. The system of claim 19, wherein said controller further includes:

a failure follow up module for performing an action based on said reason.

28. The system of claim 20, wherein said failure monitoring controller is physically divided into two or more units.

29. The system of claim 28, wherein said physically divided failure monitoring controller includes a plurality of at least one selected from a group comprising:

command generators, indication obtainers, failure result indication detectors, and analyzers.

30. An entity in a storage system comprising:

a command receiver for receiving input/output commands originating from a host, and for receiving input/output commands originating from a failure monitoring controller;

an origin detector for detecting that a received command originates from a failure monitoring controller and is a candidate for comparing;

a command comparer for comparing said detected command with at least one previously handled command originating from a host to determine if similar or different;

a command handler for handling said detected command if different, or for not handling said detected command if similar; and

an indication returner for explicitly or implicitly returning an indication of result of said similar previously handled command originating from said host if similar, or for explicitly or implicitly returning an indication of result of said handled detected command if different.

31. The entity of claim 30, wherein said origin detector detects that said received command is a candidate, only if said command is received directly from said failure monitoring controller.

32. The entity of claim 30, wherein said origin detector detects that said received command is a candidate inherently because said command originates from said failure monitoring controller.

33. The entity of claim 30, further comprising:

memory for storing host-originating commands.

34. A method of handling input/output commands in a storage system comprising:

receiving an input/output command;

detecting that said input output command originates from a failure monitoring controller and is a candidate for comparing;

comparing said detected command with at least one previously handled command originating from a host to determine if similar or different;

handling said detected command if different, or not handling said detected command if similar; and

explicitly or implicitly returning an indication of result of said similar previously handled command originating from said host if similar, or explicitly or implicitly returning an indication of result of said handled detected command if different.