Systems and methods for managing stalled storage devices

• US 7,984,324 B2
• Filed: 03/27/2008
• Issued: 07/19/2011
• Est. Priority Date: 03/27/2008
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Eng, Brian, Passey, Aaron J., Akidau, Tyler A., Daud, Asif, Maykov, Ilya
• Primary Examiner(s): Baderman; Scott T
• Assistant Examiner(s): BUTLER, SARAI E
• Application Number:
  US12/057,298
• Publication Number:
  US 20090248975A1
• Time in Patent Office: 1,209 Days
• Field of Search: 714/5, 714/47, 714/48, 714/54, 714 42- 44, 714/56
• US Class Current: N/A
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/30 Monitoring

Abstract

Embodiments relate to systems and methods for managing stalled storage devices of a storage system. In one embodiment, a method for managing access to storage devices includes determining that a first storage device, which stores a first resource, is stalled and transitioning the first storage device to a stalled state. The method also includes receiving an access request for at least a portion of the first resource while the first storage device is in the stalled state and attempting to provide access to a representation of the portion of the first resource from at least a second storage device that is not in a stalled state. In another embodiment, a method of managing access requests by a thread for a resource stored on a storage device includes initializing a thread access level for an access request by a thread for the resource. The method also includes determining whether the storage device, which has a device access level, is accessible based at least in part on the thread access level and the device access level and selecting a thread operation based at least in part on the determination of whether the storage device is accessible. The thread operation may be selected from attempting the thread access request if the device is accessible and determining whether to restart the thread access request if the device is not accessible.

Claims

1. A method for managing access to storage devices, the method comprising:

determining that a first storage device is available but stalled based on at least a response time for one or more access requests to the first storage device, wherein a first resource is stored on the first storage device;

transitioning the first storage device to a stalled state;

receiving an access request for at least a portion of the first resource while the first storage device is in the stalled state; and

attempting to provide access to a representation of the portion of the first resource from at least a second storage device that is not in a stalled state.

2. The method of claim 1, wherein determining that a first storage device is stalled comprises evaluating whether a response to an access request to the first storage device is slow.

3. The method of claim 2, wherein evaluating whether a response to an access request to the first storage device is slow comprises determining whether the response takes-longer than a first time period.

4. The method of claim 2, wherein evaluating whether a response to an access request to the first storage device is slow comprises determining whether a frequency of slow responses is above a frequency threshold, wherein a slow response comprises a response taking longer than a second time period to complete.

5. The method of claim 4, wherein the frequency is determined based at least in part on a number of slow requests and a number of non-slow requests.

6. The method of claim 2, wherein evaluating whether a response to an access request to the first storage device is slow comprises determining, for a third time period during which the first storage device is continuously busy, whether a number of slow responses is above a threshold number.

7. The method of claim 2, wherein evaluating whether a response to an access request to the first storage device is slow is based at least in part on one or more device parameters specific to a system comprising the first storage device and at least the second storage device.

8. The method of claim 7, wherein the one or more device parameters comprise at least one of a time resolution of a timing clock, a retry time for access requests to the first storage device, a seek time for the first storage device, an input/output rate of the first storage device, a kernel delay, a system load, a load of the first storage device, and a cache flush time.

9. The method of claim 7, further comprising monitoring at least one of the one or more device parameters and using at least one monitored device parameter for evaluating whether a response to an access request to the first storage device is slow.

10. The method of claim 1, wherein determining that a first storage device is stalled comprises determining whether an access request to the first storage device returns an error message.

11. The method of claim 10, wherein the error message comprises an error correction code message.

12. The method of claim 1, wherein transitioning the first storage device to a stalled state comprises:

aborting pending access requests to the first storage device;

communicating a message to at least the second storage device that the first storage device is or will be in the stalled state; and

restarting the aborted pending access requests.

13. The method of claim 1, wherein the representation of the portion of the first resource comprises a copy of the portion or parity information associated with the portion.

14. The method of claim 1, wherein attempting to provide access to a representation of the portion of the first resource comprises successfully providing access to the representation from at least the second storage device.

15. The method of claim 1, wherein attempting to provide access to a representation of the portion of the first resource comprises providing access to some or all of the portion from the first storage device if the second storage device becomes unavailable or if the representation obtainable from at least the second storage device is incomplete.

16. The method of claim 1, further comprising directing write requests to at least one storage device that is not in a stalled state rather than to the first storage device when in the stalled state.

17. The method of claim 1, further comprising:

determining that the first storage device is no longer stalled;

selecting a non-stalled state; and

transitioning the first storage device to the non-stalled state.

18. The method of claim 17, wherein determining that the first storage device is no longer stalled comprises performing one or more diagnostic tests to evaluate responses of the first storage device to access requests, and wherein selecting the non-stalled state depends at least in part on an outcome of the one or more diagnostic tests.

19. The method of claim 17, wherein the non-stalled state comprises a normal state, and the method further comprises directing future access requests for resources on the first storage device to the first storage device.

20. The method of claim 17, wherein the non-stalled state comprises a down state, and the method further comprises directing future access requests for resources on the first storage device to one or more storage devices that are not in a down state.

21. The method of claim 17, wherein the non-stalled state comprises a soft-failed state, and the method further comprises moving a representation of resources on the first storage device to one or more operational storage devices.

22. The method of claim 17, wherein transitioning the first storage device to a non-stalled state comprises communicating a message to at least the second storage device that the first storage device is or will be in the non-stalled state.

23. The method of claim 1, wherein the first storage device and at least the second storage device comprise at least a portion of a file system.

24. The method of claim 23, wherein the file system comprises a distributed file system.

25. The method of claim 24, wherein at least one of the first storage device and the second storage device comprises a node of the distributed file system.

26. The method of claim 1, wherein at least one of the first storage device and the second storage device comprises a magnetic storage medium or an optical storage medium.

27. The method of claim 26, wherein the magnetic storage medium comprises at least a portion of a hard disk drive.

28. A computer-readable medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform a method for managing access to storage devices, the method comprising:

determining that .a first storage device is available but stalled based on at least a response time for one or more access requests to the first storage device, wherein a first resource is stored on the first storage device;

transitioning the first storage device to a stalled state;

receiving an access request for at least a portion of the first resource while the first storage device is in the stalled state; and

attempting to provide access to a representation of the portion of the first resource from at least a second storage device that is not in a stalled state.

29. A system for managing storage on storage devices, the system comprising:

a first storage device configured to store a first resource;

a second storage device configured to store a representation of at least a portion of the first resource; and

a software module configured to;

determine that the first storage device is available but stalled based on at least a response time for one or more access requests to the first storage device;

transition the first storage device to a stalled state;

receive an access request for the at least a portion of the first resource while the first storage device is in the stalled state; and

attempt to provide access to the representation of the portion of the first resource from at least the second storage device.

30. The system of claim 29, wherein to determine that the first storage device is stalled, the software module is further configured to evaluate whether a response to an access request to the first storage device is slow.

31. The system of claim 30, wherein to evaluate whether the response to the access request to the first storage device is slow, the software module is further configured to determine whether the response takes longer than a first time period.

32. The system of claim 30, wherein to evaluate whether the response to the access request to the first storage device is slow, the software module is further configured to determine whether a frequency of slow responses is above a frequency threshold, wherein a slow response comprises a response that takes longer than a second time period to complete.

33. The system of claim 32, wherein the frequency is determined based at least in part on a number of slow requests and a number of non-slow requests.

34. The system of claim 30, wherein to evaluate whether the response to the access request to the first storage device is slow, the software module is further configured to determine, for a third time period during which the first storage device is continuously busy, whether a number of slow responses is above a threshold number.

35. The system of claim 30, wherein the software module is configured to evaluate whether the response to the access request to the first storage device is slow based at least in part on one or more device parameters specific to the system.

36. The system of claim 35, wherein the one or more device parameters comprise at least one of a time resolution of a timing clock, a retry time for access requests to the first storage device, a seek time for the first storage device, an input/output rate of the first storage device, a kernel delay, a system load, a load of the first storage device, and a cache flush time.

37. The system of claim 35, wherein the software module is further configured to monitor at least one of the one or more device parameters and to evaluate whether the response to the access request to the first storage device is slow based at least in part on one or more monitored device parameters.

38. The system of claim 29, wherein to determine that the first storage device is stalled, the software module is further configured to determine whether an access request to the first storage device returns an error message.

39. The system of claim 38, wherein the error message comprises an error correction code message.

40. The system of claim 29, wherein to transition the first storage device to the stalled state, the software module is further configured to:

abort pending access requests to the first storage device;

communicate a message to at least the second storage device that the first storage device is or will be in the stalled state; and

restart the aborted pending access requests.

41. The system of claim 29, wherein the representation of the portion of the first resource comprises a copy of the portion or parity information associated with the portion.

42. The system of claim 29, wherein to attempt to provide access to the representation of the portion of the first resource, the software module is further configured to provide access to the representation from at least the second storage device.

43. The system of claim 29, wherein to attempt to provide access to the representation of the portion of the first resource, the software module is further configured to provide access to some or all of the portion from the first storage device if the second storage device becomes unavailable or if the representation obtainable from at least the second storage device is incomplete.

44. The system of claim 29, wherein the software module is further configured to direct write requests to at least one storage device that is not in a stalled state rather than to the first storage device when in the stalled state.

45. The system of claim 29, wherein the software module is further configured to:

determine that the first storage device is no longer stalled;

select a non-stalled state; and

transition the first storage device to the non-stalled state.

46. The system of claim 45, wherein the software module is configured to perform one or more diagnostic tests to evaluate responses of the first storage device to access requests, and to select the non-stalled state based at least in part on an outcome of the one or more diagnostic tests.

47. The system of claim 45, wherein the non-stalled state comprises a normal state, and the software module is configured to direct future access requests for resources on the first storage device to the first storage device.

48. The system of claim 45, wherein the non-stalled state comprises a down state, and the software module is configured to direct future access requests for resources on the first storage device to one or more storage devices that are not in a down state.

49. The system of claim 45, wherein the non-stalled state comprises a soft-failed state, and the software module is configured to move a representation of resources on the first storage device to one or more operational storage devices.

50. The system of claim 45, wherein to transition the first storage device to a non-stalled state, the software module is further configured to communicate a message to at least the second storage device that the first storage device is or will be in the non-stalled state.

51. The system of claim 29, wherein the first storage device and at least the second storage device comprise at least a portion of a file system.

52. The system of claim 51, wherein the file system comprises a distributed file system.

53. The system of claim 52, wherein at least one of the first storage device and the second storage device comprises a node of the distributed file system.

54. The system of claim 29, wherein at least one of the first storage device and the second storage device comprises a magnetic storage medium or an optical storage medium.

55. The system of claim 54, wherein the magnetic storage medium comprises at least a portion of a hard disk drive.

56. A method for managing access to storage devices, the method comprising:

determining that a first storage device is stalled, wherein a first resource is stored on the first storage device;

if the stall is related to an access request, transitioning the first storage device to a stalled state;

if the stall is not related to an access request, transitioning the first storage device to a soft-failed state, wherein, while the first storage device is in the soft-failed state, the first storage device is read-only;

receiving an access request for at least a portion of the first resource while the first storage device is in the soft-failed state or the stalled state;

if the first storage device is in the stalled state, attempting to provide access to a representation of the portion of the first resource from at least a second storage device that is not in a stalled state; and

if the first storage device is in the soft-failed state, providing read-only access to the portion of the first resource from the first storage device.